Packagemanager-LFS-PackageUser-System git · main
Package-user based package manager for LFS/BLFS
Shell 63.7% Markdown 35.8%git clone https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System.gitwget https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System/archive/Packagemanager-LFS-PackageUser-System.tar.gzBug fixes And stack updates
BUILD.md | 141 +- HANDOFF.md | 2969 +-------------------------------- Makefile | 29 +- README.md | 43 +- blfs | 282 +--- kernel.conf | 40 +- lfs | 45 +- lfs-helper | 1618 +----------------- lfs-kernel | 869 ++-------- lfs-phases | 93 +- packagemanager | 3278 +++--------------------------------- stacks/kernel-sway.sh | 23 - stacks/machine.conf | 245 +-- stacks/network.stack | 51 +- stacks/services.stack | 111 +- stacks/shell-env.sh | 155 +- stacks/sway-session.sh | 16 +- stacks/sway.stack | 411 ++--- test_lfs_crosschain.sh | 4330 +----------------------------------------------- 19 files changed, 601 insertions(+), 14148 deletions(-) diff --git a/BUILD.md b/BUILD.md index 18b09b2..0f8dcbf 100644 --- a/BUILD.md +++ b/BUILD.md @@ -80,7 +80,7 @@ Worth setting before `run`: | `locale`, `charmap`, `paper_size`, `timezone` | `en_US.UTF-8`, `UTF-8`, `A4`, – | baked into chapter 9 | | `hostname`, `network` | `lfs`, `dhcp` | `static` writes interface files; `dhcp` expects dhcpcd from BLFS | | `main_user` | – | your login. `init-accounts` creates it (group `users`) and asks for root's and its password. Blank = root only, no password | -| `makeflags` | `-j$(nproc)` — the LFS build's job count | | +| `makeflags` | `-j$(nproc)` | | | `remove_tools` | no | delete `/tools` at the end; it is your way back if chapter 8 needs redoing | | `bootloader`, `esp` | `none`, – | `refind` + `/boot/efi` generates a rEFInd step that only *adds* an entry | | `collector_import_file` | – | groups export from a previous build; skips every sharing question | @@ -169,7 +169,7 @@ offline. The rest needs the network `resolv` just gave you. |---|---| | `/etc/pkgusr/stacks/machine.conf` | this machine's build options (`[mesa] gallium-drivers = radeonsi`, `[iptables] configure_args = --enable-nftables`, `[dbus] systemd = disabled`). Merged into generated scripts when they are born, so set it **before** the stack runs | | `/etc/pkgusr/stacks/package-fixes.conf` | workarounds the tools ship. Replaced on every `lfs run`; put yours in `machine.conf`, which wins | -| `/etc/pkgusr/stacks/*.stack` | which packages, in order; `avoid` lines; pinned git refs; `expect=` feature probes | +| `/etc/pkgusr/stacks/*.stack` | which packages, in order; `avoid` lines; pinned git refs | | `/etc/pkgusr/desktop-user` | the login that owns the desktop. Set once with `LFS_DESKTOP_USER=<name>`; `sway-session` and `mytools` both read the file | Files under `/etc/pkgusr/` are yours once edited. `make install` updates a @@ -178,34 +178,6 @@ copy you never touched, otherwise it lands as `.new` ### The stacks, in order -**Updating part of a system.** A stack run builds what is MISSING; it never -updates an installed package. To move packages to what the book now carries: - - packagemanager update # everything it has moved past - packagemanager update --stack media # just that chain - packagemanager update --stack media --select --run # pick from them - packagemanager update webkitgtk --run # one, deliberately - -`update <pkg>` also reports which of that package's requirements are older -here than the version its page was written against -- not a refusal, just -the numbers. - -`media` is the codec / gstreamer / WebRTC / webkitgtk chain. sway.stack -pulls it in with `include media`, so a full desktop build is still one -command -- and when a codec or webkitgtk needs redoing you rebuild just that -part: - - packagemanager stack media --run - packagemanager probe - -`vision` is separate and optional: OpenCV 5, hours of compiling, nobody's -dependency. `packagemanager stack vision --run` when you want it. - -`diag` is optional too: glxinfo, eglinfo, vulkaninfo, vainfo, nvtop. -GPU vendor (for nvtop) is set at the top of `diag.stack`. -`packagemanager stack diag --run`. - - ```sh usermod -aG audio,video,input <name> # the main_user from the interview export LFS_DESKTOP_USER=<name> @@ -214,59 +186,13 @@ packagemanager stack sway --check-refs # every git ref still exists? packagemanager stack sway --run # desktop: wlroots, sway, foot, webkit, pipewire, mesa packagemanager stack network --check-refs packagemanager stack network --run # wpa_supplicant, iptables-nft, net-tools -packagemanager stack services --run # bootscripts, fcron, acpid, mytools, shell-env, app-users +packagemanager stack services --run # bootscripts, fcron, acpid, mytools, shell-env ``` `sway` includes `check kernel-sway`, which reads the running kernel's config and fails naming the missing option. In the chroot that is the host kernel; section 6 builds the real one. -**How many jobs.** Four build systems, and only one of them reads -`MAKEFLAGS`: - -| what | reads | set it in | -|---|---|---| -| make | `MAKEFLAGS` | `/etc/pkgusr/build.env`: `MAKEFLAGS=-j13` | -| cmake | `CMAKE_BUILD_PARALLEL_LEVEL` | derived from the above, or `--jobs` | -| cargo | `CARGO_BUILD_JOBS` | same | -| ninja | nothing at all | `machine.conf` `[all] jobs = 13` | -| rust (`./x.py`) | `CARGO_BUILD_JOBS` **and** `--jobs N` | same `[all] jobs` — it patches `x.py` and `cargo` lines too | - -A `jobs` value is patched into the script **when the script is generated**, -so a package built before you set it needs `--regenerate` to pick it up. Any -line that already names `-j` or `--jobs` is left alone. - -`lfs-helper build <pkg> --jobs 13` overrides all of them for one build. A -named `machine.conf` section beats `[all]`, which is how webkitgtk keeps its -lower count for RAM. - -**Order matters for autodetected features.** A package built inside another -package's dependency plan is built in the *book's* order, not the stack's — -so an optional library like `alsa-lib` must be listed before anything that -autodetects it, or ffmpeg and SDL3 come out with no sound and no error. An -entry can prove its own features afterwards: - - book ffmpeg expect="ffmpeg -hide_banner -devices | grep -qw alsa" - -A failing probe leaves the entry unrecorded, so the next run rebuilds it. -Probe the artifact that proves the feature: for a GStreamer plugin that is a -registered element (`gst-inspect-1.0 webrtcbin`), not a file — the library is -built either way. - -Two reports cover the chains where this bites: - - packagemanager stack network --only net-wifi --run # wifi - packagemanager stack sway --only webrtc --run # camera, mic, WebRTC - -**After a stack finishes, do the second pass.** Optional features are decided -when a package configures, so anything built before a dependency it can use -was built without it — and some of those cannot be reordered at all (cairo -requires glib2, so it cannot precede it): - - packagemanager probe # do the features exist? - packagemanager late-deps # orderable / circular / cosmetic - packagemanager late-deps --rebuild --run # in dependency order - Useful while a stack runs: ```sh @@ -289,9 +215,7 @@ generated, so it can be edited before the first build. | `ESP`, `KERNEL_NAME` | *which* kernel this is. `vmlinuz_new` lands beside the working one; `vmlinuz` replaces it. Previous file is kept as `_old` | | `JOBS` | leave a core free | | `CONFIG_FROM`, `CONFIG_REFRESH` | `running` + `olddefconfig` for unattended; `oldconfig`/`menuconfig` to decide yourself | -| `CONFIG_SET` | options this machine always wants (`LEGACY_TIOCSTI=n`), applied after the refresh and read back | -| `FIRMWARE_FROM`, `FIRMWARE_ONLY` | `git` and a short list. A directory name covers a directory; a *driver* asks the loader for a flat path, so Intel wireless needs `iwlwifi-*.ucode` as well as `intel`. Empty means 10 GB | -| `REGDB` | yes. cfg80211's regulatory database is wireless-regdb, not linux-firmware; do **not** name it in `FIRMWARE_ONLY` | +| `FIRMWARE_FROM`, `FIRMWARE_ONLY` | `git` and a short list (`amdgpu amd-ucode amd intel`). Empty means 10 GB | | `HEADERS`, `V4L2LOOPBACK`, `BOOTLOGO` | as you like | ```sh @@ -303,15 +227,6 @@ The kernel is a package: `packagemanager remove linux` takes it away. The tree of the last kernel known to boot is never deleted, and nothing is recorded as known-good until it has actually booted. -Firmware is its own package and its own command, so a missing blob does not -cost a kernel build: - -```sh -lfs-kernel firmware # install what FIRMWARE_ONLY names -lfs-kernel firmware --missing # what the running kernel asked for and - # did not get, and where it actually lives -``` - --- ## 7. Bootloader @@ -339,49 +254,12 @@ dmesg | grep -i firmware # the definitive FIRMWARE_ONLY list packagemanager verify ``` -If something that needs root privileges fails (`su -` reporting `setgid: -Operation not permitted`, `passwd` refusing to write), the setuid bit was -refused at install time — a package user may not set one, and a bit it *can* -set makes the program run as the package user rather than as root: - -```sh -lfs-helper setid # what is outstanding, and what is on disk -lfs-helper setid --run # chown root, set the bit, read it back -``` - -If a device is missing, ask what firmware it wants. This works in the chroot -too: the list comes from the tree's own modules and this machine's hardware, -not only from the running boot, and each line says which source named it. - -```sh -lfs-kernel firmware --missing # boot / modules / config, per line -lfs-kernel firmware --missing --fix -``` - -`boot` lines are things a driver actually asked for and did not get — the -reliable list. For each of those it also says whether the same file is -already on disk under another path (a `FIRMWARE_ONLY` entry keeps the -repository's path; the loader wants its own) or whether an older firmware -API of it is installed, which means the checkout needs updating rather than -a different path. `modules` lines are candidates: a module declares firmware for -every chip it supports, and yours loads only a few, so they are summarised -per module rather than reported as failures. `--fix --all` installs them -anyway. - Things a first boot commonly still wants: -- **wifi:** `/etc/sysconfig/ifconfig.<iface>` (the real interface name — - `wlp3s0`, not `wifi0`) with `SERVICE=wpa`, and one file per SSID in - `/etc/sysconfig/wpa_supplicant/`, chmod 600. The network stack installs - `dhcpcd`; `packagemanager stack network --only net-wifi --run` reports - whatever is still missing, including which hand-off variable the installed - `wpa` service reads. +- **wifi:** `/etc/sysconfig/ifconfig.wifi0` with `SERVICE=wpa`, and one + file per SSID in `/etc/sysconfig/wpa_supplicant/`. - **certificates:** `make-ca -g` weekly, as a cron entry in `/etc/cron.weekly/` (the systemd book used a timer). -- **power:** `halt`, `poweroff`, `reboot` and suspend belong to the - `<collector_prefix>_power` group; `cfg power` in the services stack sets - that up, and members must log in again before it counts. Check it with - `/etc/rc.d/init.d/power-perms status`. - **desktop:** log in on the active tty as the desktop user, run `sway`. The seat comes from `/usr/bin/seatd-<user>`, a setuid wrapper `mytools` built for that one name. If it fails, the launcher checks the wrapper, @@ -416,12 +294,5 @@ start again from section 2 with `collector_import_file` set. | a chroot step fails | `/usr/src/lfs-pkgusr/logs/<step>.log`; `lfs-helper check-toolchain` | | a stack entry fails | `packagemanager errors`; `--only <entry> --run` to retry one | | files owned by nobody | `lfs-helper fix-orphans --run`, `packagemanager files_with_broken_id` | -| a feature is missing from a package that built fine | `packagemanager late-deps` — it was built before that dependency existed; `--rebuild --run` fixes it | -| a package cannot install into its own directory | check who owns the parent; `packagemanager reclaim <pkg> --run` | | ownership drifted | `packagemanager verify --fix`; `lfs-helper fix-ownership --run` | -| `su`/`passwd`/`mount` say "Operation not permitted" | `lfs-helper setid` — a refused setuid bit | -| a device has no driver | `lfs-kernel firmware --missing` | | tools and scripts disagree | `lfs build-system gen-chroot-scripts --run --overwrite` (host) | - -The regression suite runs in pieces: `bash test_lfs_crosschain.sh --list`, -then `--from N --to M` or `--only <text>`. diff --git a/HANDOFF.md b/HANDOFF.md index de7d817..beafb07 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -7,25 +7,23 @@ account per package, so every file on the system has an owner, and Written for a **SysV** system (no systemd) built from the **systemd** BLFS book, because BLFS stopped publishing a SysV edition after 12.4. - all six tools at 1.14.299 (lfs-backup has its own: 1.3.0) - lfs 8b4b96e - lfs-helper 79163cd - packagemanager 9eb593a - blfs a4b278f - lfs-phases 8077458 - lfs-kernel 219215c (no version line of its own) - lfs-backup c73acaa (1.3.0; replaces lfs-config) + all five tools at 1.14.146 + lfs c0858da + lfs-helper bf6756c + packagemanager 367a690 + blfs 877e4cb + lfs-phases ba72201 + lfs-kernel 1aa3396 (no version line of its own) lfs-sysvbook 62ebe64 (1.0.0; untouched since it was written) **`BUILD.md` is the one document for building a whole system**, host to desktop, with what to configure before each step. README.md points at it. Keep it true: when a command or a config key changes, it changes there too. -`test_lfs_crosschain.sh` is the regression suite. It runs in pieces now: -`--list` numbers the blocks, `--from N --to M` or `--only N,M|<text>` runs -some, and a selection re-executes a generated subset (preamble + blocks + -summary) whose path it prints. rc.site now SKIPs when the book cannot have -it, so a clean run reads clean. +`test_lfs_crosschain.sh` is the regression suite. It no longer finishes inside +one command slot in the authoring sandbox; run it whole on a real machine, or +one block at a time (each starts `# ---- <title>`). One check fails by +environment only: **rc.site** needs a SysV LFS book on disk. The full per-release history — 144 entries, every bug and what it taught — is `HANDOFF-history.md`. It is worth grepping when something looks familiar; it @@ -48,34 +46,13 @@ acpid, fcron, blfs-bootscripts) and bash-completion. ESP, modules with a depmod index, headers, v4l2loopback, and 15 embedded firmware blobs. `lfs-kernel verify` reports 32 checks green. -**7.2.4 has booted** (`Linux version 7.2.4 ... #6 SMP`, built by `p_linux`). -Whether `kernel.last-good` was updated is unchecked; `lfs-kernel verify` -answers it. +**The machine is still running 7.2.0.** 7.2.4 is on the ESP as +`vmlinuz_new`, beside the working `vmlinuz`, and has never been booted. +Nothing is recorded as known-good yet, which is correct and deliberate. **Nothing in the desktop has ever run.** The last sway attempt failed in the chroot for want of `/run/udev`, which is the expected result there. -**Two things the boot exposed, both now fixed in the tools:** - -- **`su -` is fixed** (`lfs-helper setid --run` did it). -- **No wifi.** The AX200 is detected and has no firmware: - `Direct firmware load for iwlwifi-cc-a0-77.ucode failed`, plus - `failed to load regulatory.db`. `intel` in FIRMWARE_ONLY installs - `intel/`, but the driver asks the loader for a FLAT path, so it never - matched. Fix on the machine: `lfs-kernel firmware --missing --fix`, then - `modprobe -r iwlwifi; modprobe iwlwifi`. The shipped kernel.conf now lists - `iwlwifi-*.ucode regulatory.db*` (their edited copy will arrive as `.new`). - It works in the chroot now: the list is derived from the tree's own - modules and this machine's hardware, not only from dmesg. `regulatory.db` - is genuinely not in linux-firmware -- it is wireless-regdb, and the - command says so. -- **`su -` says `setgid: Operation not permitted`.** shadow installs su with - `install -m 4755`; the install wrapper downgraded it to 755 and, unlike the - chmod wrapper beside it, recorded nothing. Fix on the machine: - `lfs-helper setid --run`. This is very likely the same cause as the open - "root cannot log in" item — check `ls -l /usr/bin/su /usr/bin/passwd` - before assuming anything about /etc/shadow. - --- ## What to do next, in order @@ -91,19 +68,12 @@ chroot for want of `/run/udev`, which is the expected result there. group. If it fails, `stacks/sway-session.sh`'s launcher checks the three things that were each a separate bug: the wrapper, `/run/udev`, and whether this tty is the active console. -4. **Bring up wifi.** The interface is `wlp3s0` (predictable name, not - `wifi0`), so the file has to be `/etc/sysconfig/ifconfig.wlp3s0`. What - is still missing: - - packagemanager stack network --run # dhcpcd, which was NEVER in it - packagemanager stack network --only net-wifi --run # what else is missing - - then `/etc/sysconfig/ifconfig.wlp3s0` (`ONBOOT=yes`, `IFACE=wlp3s0`, - `SERVICE=wpa`, and the hand-off variable that `/lib/services/wpa` - actually reads — `net-wifi` prints it) and a per-SSID file under - `/etc/sysconfig/wpa_supplicant/`, chmod 600. Their `wifi` script calls +4. **Bring up wifi.** `wifi` is in `/bin`, `wpa_supplicant`/`wpa_cli` in + `/usr/sbin`. Still missing: `/etc/sysconfig/ifconfig.wifi0` (with + `SERVICE=wpa`) and the per-SSID files under + `/etc/sysconfig/wpa_supplicant/`. Note their `wifi` script calls `ifup $IFNAME wifi0` with `IFNAME` unset while `INTERFACE` is set and - unused — one of the two should go, and `wifi0` is the wrong name now. + unused — one of the two should go. 5. **make-ca's weekly job.** The systemd book enables a timer, which the shim skips. fcron exists now, so add the 12.4 book's cron entry by hand: @@ -129,9 +99,6 @@ and whether `/bin/bash` is in `/etc/shells`. blfs turns book pages into install scripts lfs-phases the phase runner every generated script sources lfs-kernel build/install/verify a kernel and its firmware - lfs-backup back up backup.conf's files; restore creates what is - missing and writes .new beside everything else - (replaced lfs-config in 1.14.291) lfs-sysvbook graft the 12.4 SysV flavour onto a systemd-only LFS book -> 13.0-sysv (LFS only; BLFS has no counterpart yet, see deferred). Its own notes: @@ -165,15 +132,6 @@ Makefile's `TOOLS` line, so a new tool needs no second edit. These are the ones that cost releases. They are worth applying to new code without waiting for the failure that proves them again. -**A feature is not proved by a file existing.** ffmpeg and SDL3 installed, -passed `have=`, and had no ALSA, because the library was built after them. -An autodetected feature needs its dependency placed by entry order and a -probe (`expect=`) that reads the built artifact. - -**A repair must not be able to fail the thing it is repairing.** It runs -after something already went wrong, so every input it reads may be missing -and every variable it names must be one the caller actually sets. - **Confirm your own effects.** A zero exit is not proof: a phase whose log says "make: *** Error 1" failed whatever it returned; a write must be read back (`usermod -d` returned 0 and changed nothing, three times); a step that @@ -229,2885 +187,7 @@ booted, because an empty file looked like something to fill in. --- -## 1.14.239 -- shadow stops at configure: --disable-logind - - checking for sd_session_get_remote_host in -lsystemd... no - configure: error: libsystemd not found. If logind integration is not - required, disable it using the --disable-logind option. - -shadow 4.20 auto-detects logind, and the book's page -- written for a -systemd host where libsystemd is simply present -- passes no switch either -way. On this machine it stops. configure names the flag itself, so: - - [shadow] - configure_args = --disable-logind - -Verified: the flag lands on the book's own `./configure` line. - -This IS the case `book =` was built for -- a page whose COMMANDS do not -suit this machine -- but the 12.4 page would also mean shadow 4.18.0, and -one switch is cheaper than four versions back. Recorded next to the setting. - -**shadow was interrupted mid-build**, so on the machine: - - packagemanager update shadow --reinstall --regenerate --run --yes - -`--regenerate` matters: machine.conf is applied when the script is written, -and the script on disk was written before that section existed. - -## 1.14.240 -- `book` is for the tool, and it reached the build line - - meson setup -Dbook=/usr/share/blfs/books/BLFS-BOOK-12_4-nochunks.html .. - ERROR: Unknown option: "book". - -`[linux-pam] book = ...` says WHICH BOOK to read that package from. It is -addressed to the tool, and every other key in a machine.conf section is a -BUILD option, so it went straight onto the meson line. `jobs` and -`configure_args` were each popped out by hand; `book` was new and nobody -popped it. - -A reserved SET now, so the next such key cannot leak the same way. -`_warn_unknown_meson_opts` could not have saved this: at script-generation -time the source is not unpacked, so there is no meson.options to check -against. - -Asserted with a fixture carrying BOTH kinds in one section -- `book` must -not reach the build line, `jobs` must still become `ninja -j4`. - -Two things visible in that script worth noting: - - * the override WORKS -- `required=('Shadow-4.18.0')`, - `name_version='Linux-PAM-1.7.1'`, `pkgusr_book_mirror=.../12.4`: the - whole page came from the older book, which is the design. - * 1.14.238 said linux-pam does not need the override (13.1's page needs - nothing from systemd to build, and `install` drops avoided packages by - itself). With this fixed, either choice now works; the newer page - still means 1.7.2 instead of 1.7.1. - -## 1.14.241 -- zathura and imv, and a package that was built and avoided - -Neither is in the BLFS book, so they are `git` entries in **sway.stack** -(with the other applications, not media.stack -- media is the codec chain, -and a PDF viewer is not a codec). - -CHECKED with git ls-remote, all four at once: - - * **girara, zathura, zathura-pdf-poppler share the tag 2026.07.18** -- - they are one project released together, and mixing tags is the usual way - to end up with a zathura that cannot open a PDF. The suite asserts one - tag across the three. - * **imv v4.3.0**; its repo answers on both exec64/imv and eXeC64/imv. - -**zathura is three packages.** girara is its widget library, the core opens -no formats at all, and the backend is a separate plugin -- a zathura with no -plugin starts, shows a grey window and says nothing. So the probe looks for -`/usr/lib/zathura/libpdf-poppler.so`, not for the binary. - -**poppler left avoid.stack.** It was there as libcupsfilters' child; -zathura's backend wants it, so it is a `book poppler` entry now. mupdf -stays avoided -- it is the other backend and one is enough. - -**And the check for that class found a real contradiction:** `avoid openssh` -was in avoid.stack while network.stack BUILT openssh with the install-sshd -bootscript. Two files disagreeing about one package, and whichever a plan -saw first won. Removed, and every stack is now checked for it. - -One test bug fixed on the way: the accounts-before-openssh assertion -compared an AVOID entry (index 50) against a build entry (73) and called -the order wrong. It filters to build entries now -- the cost of every -stack including the exclusions is that an index means nothing unless you -say which kind you mean. - -## 1.14.242 -- cmake will not overwrite a symlink, and journald came back - -**tiff.** Round 1 of the repair worked -- `/usr/lib/pkgconfig/libtiff-4.pc -belongs to p_libtiff -- handing it to p_tiff` -- and the retry then died on - - file INSTALL cannot duplicate symlink ... at /usr/lib/libtiff.so.6 - because: File exists - -Ownership was never the problem: cmake refuses to REPLACE a symlink at all, -so handing it over changes nothing. It has to go. The repair removes a -symlink owned by ANOTHER PACKAGE USER (never root's, and it says whose it -was) and lets the install create its own. - -The extractor also had to grow: in the symlink message the paths are NOT -quoted, unlike `cannot copy file "A" to "B"`, which is why it saw nothing -and the round did nothing. - -**p_libtiff AND p_tiff are the same upstream package** -- 12.4 titles the -page libtiff, 13.1 titles it tiff. Two accounts, two copies of one -library, and whichever installed last owns every file. Reported here, NOT -fixed: `packagemanager which-package /usr/lib/libtiff.so.6` says who holds -it today, and one of the two accounts should be retired deliberately. - -**webkitgtk: `fatal error: systemd/sd-journal.h`** -- my regression. The -machine's own conf had `ENABLE_JOURNALD_LOG = OFF`; the [webkitgtk] section -I shipped at 1.14.203 did not, so taking the shipped version dropped it and -webkit autodetected journald. Nine thousand files in. Restored, with -ENABLE_BUBBLEWRAP_SANDBOX = OFF beside it (also in the machine's conf and -also missing from mine). - -When a shipped section replaces a hand-made one, every switch that is -missing from it is a switch silently turned back to its default. `diff -machine.conf machine.conf.new` before taking one. - -## 1.14.243 -- poppler's soft-mandatory deps, and bubblewrap confirmed - -**ENABLE_BUBBLEWRAP_SANDBOX = OFF is in the shipped [webkitgtk] section** -(1.14.242, beside ENABLE_JOURNALD_LOG). Both were restored in the same -edit; `grep -A9 '^\[webkitgtk\]' machine.conf` shows the pair. - -**poppler stops at configure for nss, and would then stop twice more.** -cmake calls these "soft mandatory" -- absent, they fail the configure and -name the switch themselves: - - Could not find the 3.98 version of NSS3. If you're not interested in - the features it provides set the cmake ENABLE_NSS3 option to OFF - -The point nobody had noticed: **nss, boost and gpgmepp are in avoid.stack -AS POPPLER'S CHILDREN** -- they were avoided because poppler was. Promoting -poppler out of that list (1.14.241, for zathura's backend) brought all three -back into question, and each would have failed in turn, one long configure -apart. They serve PDF signature verification and the splash backend; -zathura's poppler-glib backend needs none of it: - - [poppler] - ENABLE_NSS3 = OFF # OBSERVED -- the failure above - ENABLE_GPGME = OFF # ANTICIPATED, same avoid list, same mechanism - ENABLE_BOOST = OFF # ANTICIPATED - -The comment marks which was seen and which was inferred, and the three -children STAY avoided -- turning the features off is the whole point. -Verified the switches reach the book's `cmake -D CMAKE_BUILD_TYPE=...` line -(it sits after `&&`, which 1.14.222 taught the merge to handle). - -**The general rule worth carrying:** a package promoted out of avoid.stack -brings its avoided children back into question. The suite now checks this -particular case; the pattern will recur the next time something is -promoted. - -## 1.14.244 -- two invented values in one entry - -Both mine, both from guessing instead of reading, and both caught by -machinery that was working correctly: - - * **`-Dsqlite=enabled`** stopped meson dead: `Unknown option: "sqlite"`. - zathura has no such option. Its meson.build takes sqlite - unconditionally -- `dependency('sqlite3', version: '>=3.35.0')` -- and - its only options are synctex, seccomp, landlock, manpages, tests-x11, - tests-wayland and convert-icon. Dropped; sqlite3 is already installed - (p_sqlite). - * **`have=/usr/lib/pkgconfig/girara-gtk3.pc`** made the stack refuse to - record a girara that had installed perfectly: - girara finished, but /usr/lib/pkgconfig/girara-gtk3.pc is still - missing -- NOT recording it as done - girara's meson.build says `pkg.generate(name: 'girara')`, so the file - is **girara.pc**. That message is the have= check doing its job on a - wrong have=, which is the better failure of the two. - -The pattern, three sessions running now (librice's four wrong guesses, the -zathura option, the girara filename): an entry's option names and installed -filenames must come from the source, and the source is one `git clone ---depth 1` away. Asserted both, by the shape that was wrong. - -(The run also reported `misplaced home: p_zathura is recorded at -/usr/src/p_zathura` and moved it to /usr/src/pkgusr/p_zathura by itself -- -that check earning its keep.) - -## 1.14.245 -- zathura also wants libxxhash - - Run-time dependency libxxhash found: NO (tried pkgconfig and cmake) - meson.build:57:9: ERROR: Dependency "libxxhash" not found - -The BLFS book mentions xxhash eight times and gives it **no page**, so the -entry is ours. Checked in the checkout rather than guessed, for once -without a correction afterwards: - - * newest tag **v0.8.3**; - * it ships `libxxhash.pc.in`, so the pkg-config module really is - `libxxhash` -- the name meson asks for -- and `make install` runs - `install_libxxhash.pc` among its targets; - * a plain Makefile, no meson or cmake: `PREFIX` and `LIBDIR` are what it - takes. - -Placed at 143, between girara (142) and zathura (144); the suite fails if -either dependency lands after zathura. - -Everything else in that run was right: girara `installed` with the -corrected have=, poppler `done` with the three ENABLE_*=OFF switches, and -poppler and openssh both gone from the "never built by this stack" line. - -## 1.14.246 -- a book entry was recorded done without being checked - - pkg-config --list-all | grep -i poppler -> nothing - ...while the stack said: book poppler done (stack cache) - -poppler was never installed, the stack believed it was, and every later run -skipped it -- so zathura-pdf-poppler kept failing on -`Dependency "poppler-glib" not found` for a dependency the plan was certain -it had. - -The "is it actually there afterwards?" guard existed and covered **git and -tar only**. A `book` entry went straight to `_stack_mark_done`. Now every -kind answers the same question: git/tar by have=/prog=, book by the package -state that `verify` reads -- and an empty manifest is not an install. - -Also asked for and added: - - [poppler] - ENABLE_GLIB = ON - -cmake autodetects the glib frontend, and that frontend is the entire reason -poppler is in this stack. The thing a chain exists for should be asked -for, not inferred -- the gst-plugins-ugly `gpl` rule again. - -### Why the dependency was not "pulled in" - -It cannot be. girara, xxhash, zathura and the backend are `git` entries, -and dependency resolution only works for `book` entries -- required= and -recommended= come from the book page. A git entry has no page, so nothing -can know zathura needs girara. That ordering is by hand in the stack file, -which is why the suite asserts it. - -## 1.14.247 -- the fourth soft-mandatory switch, and how to stop guessing - - Could not find the 6.4 version of Qt6Core. If you're not interested in - the features it provides set the cmake ENABLE_QT6 option to OFF - -`ENABLE_QT6 = OFF`. qt6 is in avoid.stack and poppler's page passes only -`ENABLE_QT5=OFF`, because a BLFS system is expected to have qt6. - -That is **four** of these found one configure at a time -- NSS3, then GPGME -and BOOST (guessed from the avoid list), now QT6 -- each a full cmake run -apart. The list was never a guessing game: poppler's own CMakeLists names -every one, and on that machine the file is right there: - - grep -n find_soft_mandatory_package \ - /usr/src/pkgusr/p_poppler/src/poppler-*/CMakeLists.txt - -That is in machine.conf beside the switches now, to be read once when -poppler moves version rather than rediscovered a configure at a time. The -suite checks the four switches and that the grep is written down. - -The general shape, for the next package like this: when a build names a -switch in its own error message, that switch has SIBLINGS, and the -mechanism that produced the message (here `find_soft_mandatory_package`) is -greppable in the source that is already unpacked. - -## 1.14.248 -- `update` is a compound phase too - -Asked whether there is a general problem with rebuilding the whole package. -There was: - - !! phase 'install' FAILED <- the inner phase that failed - /usr/share/gir-1.0 -> group nimgnu_gir (owner p_glib2, + p_poppler) - # retrying poppler (phase update), round 1 ... - unpacked into /usr/src/pkgusr/p_poppler/src/poppler-26.08.0 - -- The C compiler identification is GNU 15.2.0 <- from scratch - -145 files were already installed and the grant had fixed the one thing that -was wrong. The retry re-unpacked the tarball and re-ran cmake anyway. - -`_retry_phase` mapped the inner failure for `all` and for nothing else, so -a failure inside `update` retried `update` -- and `update` re-unpacks. Both -compound phases use the inner phase now; a single-phase run keeps its own -(1.14.209). - -That is the third time this exact shape has cost a rebuild: 1.14.70 (foot, -seven rounds), 1.14.209 (the second round re-cloning), and now the compound -phase that was not in the list. The rule: **whenever a retry picks a -phase, it picks the one that FAILED, and the log says which that was.** - -The grant itself worked exactly as intended, and is worth noting because it -is the good half of that output: `/usr/share/gir-1.0 is owned by 'p_glib2' --- asking rather than opening it to every package`, then a collector group -for the two of them. - -## OPEN BUG -- a skipped step leaves its privileged half live (poppler) - - make: *** No rule to make target 'install'. Stop. - !! phase 'install' FAILED (exit 2) ...after 218 files installed fine - -**Diagnosed, NOT fixed.** Generated from the real book, poppler's install -phase comes out as: - - ninja install - install -v -m755 -d /usr/share/doc/poppler-26.08.0 && - cp -vr ../glib/reference/html /usr/share/doc/poppler-26.08.0 - ## SKIPPED -- the book says: If you downloaded the additional encoding data package... - # tar -xf ../../poppler-data-0.4.12.tar.gz && - # cd poppler-data-0.4.12 - make prefix=/usr install <- LIVE - -The book writes an optional step as a PAIR: an unprivileged block that -unpacks and `cd`s into a sub-package, then a `<pre class="root">` block that -installs it. The guard comments out the first half and leaves the second -running -- so `make` runs in poppler's own ninja build directory, which has -no Makefile. The tell is the `cd`: whatever follows a skipped block that -ends by changing directory runs in that directory or not at all. - -I wrote that rule into both emitters (`extract_commands` and -`extract_commands_phased`) and **neither fired** -- the regenerated script -was byte-identical. **1.14.262 found why: there is no third path.** The -index cache was keyed on the book file and a hand-bumped CACHE_VERSION, so -the script was rendered from data the OLD code had extracted; the patches -never ran. The cache key now includes blfs's build id. The reverted rule -is probably right as written -- re-apply it and regenerate. Both patches are REVERTED rather than shipped: code that -fails the one case it was written for is worse than an admitted bug, and -the next person needs the diagnosis, not my dead code. - -Still reproducible from the rendered book (poppler-26.09.0, 1.14.262). -Where to start: re-apply the `cd` rule, then regenerate poppler -- - - blfs --book-file <the 13.1 book> script poppler -o /tmp/x - --- and find which emitter actually appends `make prefix=/usr install`. - -**Meanwhile, on the machine:** poppler IS installed (218 files, and -`ninja install` plus the doc copy both ran). Delete that one line from the -script and finish: - - sed -i '/^make prefix=\/usr install$/d' /usr/src/pkgusr/p_poppler/install_Poppler-26.08.0 - lfs-helper build poppler --phase install --force - pkg-config --modversion poppler-glib - -Do NOT `--regenerate` poppler until this is fixed: it will put the line -back. - -## 1.14.250 -- `home` was not in scope, so the marker was never written - - # tracked 218 file(s) for 'poppler' - /usr/bin/lfs-helper: line 7345: home: unbound variable - # poppler: done (all phases complete) - -poppler installed. The line between those two is mine: the -`.pkgusr-installed` marker from 1.14.236 reads `$home`, which is a local of -the INSTALL function and not of the one that prints "done (all phases -complete)". Under `set -u` that aborted the block -- so the marker, the -entire point of 1.14.236, was never written for any package. - -It resolves the home the way every other caller here does, -`pkgusr_home_for "$name"`, and only writes when install_last is really -there. Asserted three ways: the old `$home` read must be gone, the -resolver must be used, and the marker must still sit within a few lines of -the success line. - -Worth naming the shape: 1.14.236 was tested by calling the PREDICATE -(`gather_user().version` with and without a marker) and never by running -the code that writes it. A test that exercises the reader and not the -writer proves half a feature. - -## 1.14.251 -- application homes in /usr/src, and imv's missing DSO - - ls /usr/src/ - cfg lfs-pkgusr linux p_mytools pkgusr u_browser-big u_browser-mini u_claude u_test - -**An application user is not a package user.** `add_package_user` creates -the account -- it handles the uid range and the matching group -- and it -puts the home where PACKAGE homes go. A package user's home is a build -tree; an application user's home holds a browser profile, a .config, a -Downloads directory. It belongs in /home, which is also where the shared -links and the XDG skeleton expect it. - -`_relocate_app_home` moves it with `usermod -m` (contents included) and -chmods 0750 -- but ONLY from the package tree, and only when the target -does not exist. A home someone has already placed elsewhere is left alone -and said so. - -For the four accounts already made: - - usermod -d /home/u_claude -m u_claude && chmod 0750 /home/u_claude - usermod -d /home/u_browser-mini -m u_browser-mini && chmod 0750 /home/u_browser-mini - usermod -d /home/u_browser-big -m u_browser-big && chmod 0750 /home/u_browser-big - usermod -d /home/u_test -m u_test && chmod 0750 /home/u_test - -**imv: `undefined reference to symbol 'ubrk_close_78' ... DSO missing from -command line`.** icu splits its API across libicuuc (common), libicui18n -and libicuio; imv's meson names libicuio, calls into libicuuc, and links -with -Wl,--no-undefined -- so the indirect dependency is an error, not a -warning. `-D c_link_args=-licuuc` names it. Nothing is patched. - -### (FIXED in 1.14.252 -- see above) shadow's install strips su's setuid bit - -Named, not fixed, and it needs its own session: the LFS Shadow page sets -4711 on /usr/bin/su and the BLFS page does not, so a BLFS-driven reinstall -of shadow leaves su non-setuid -- and the same applies to passwd, chage, -gpasswd and newgrp. `lfs-helper setid` records bits a PACKAGE USER was -refused; these are bits the SCRIPT never asks for, so there is nothing to -record. What it wants is a declared list of required setuid/setgid files -(the accounts.conf shape), enforced after any install that touches them. -Until then, after any shadow install: - - ls -l /usr/bin/su /usr/bin/passwd /usr/bin/chage /usr/bin/gpasswd /usr/bin/newgrp - chmod 4711 /usr/bin/su - -## 1.14.252 -- pam then shadow, and the modes are FIXED not reported - -Asked for both, and both were right. - -**`cfg shadow-pam`, immediately after `book linux-pam` in services.stack.** -Installing PAM changes nothing on its own: until shadow is rebuilt against -it nothing reads /etc/pam.d, so `su - u_claude` kept asking for a password -however correct su_u_user was. The book's Linux-PAM page ends with "You -should now reinstall the Shadow package"; that is now a step rather than a -sentence. It checks its own effect with `ldd /usr/bin/su | grep libpam`, -and with no PAM installed it says so and does nothing. - -**And the setuid bits are applied, not warned about.** The rebuild strips -them -- the BLFS Shadow page never sets them, the LFS one does -- so the -very rebuild that makes PAM usable broke su. Declared in -**stacks/setid.conf**: - - /usr/bin/su | 4711 | root | - - /usr/bin/passwd | 4711 | root | - - /usr/bin/chage | 4711 | root | - - /usr/bin/gpasswd | 4711 | root | - - /usr/bin/newgrp | 4711 | root | - - /usr/sbin/unix_chkpwd | 4755 | root | - - -`-` in the group column KEEPS what is there, which is what was asked for: -the group stays p_shadow (ownership follows the installer in this system; -only the USER must be root for a setuid bit to mean root). Result on a -test file: `4711 root:p_shadow`. - -Two details that decide whether this works at all: - - * **chown CLEARS the setuid bit**, so chmod runs last. The suite checks - the line order, not just that both calls exist. - * it reads the mode BACK afterwards -- a setuid bit that did not take is - the entire failure this step exists to end. - -`lfs-helper setid` is still the right tool for bits a build ASKED for and -was refused; these are bits no script ever asks for, which is why they need -declaring. - -## 1.14.253 -- `pam_start: error 26` -- linked, with nothing to read - - su: pam_start: error 26 - -PAM_ABORT: there is no /etc/pam.d/su. So the rebuild in 1.14.252 worked -- -su links libpam now -- and left it worse than before, because a PAM-linked -su with no service file cannot fall back to anything. - -`update` runs install. It does NOT run configure, and those files are -written by the CONFIGURE phase: linux-pam's writes /etc/pam.d/system-auth, -system-account, system-session, system-password; shadow's writes -/etc/pam.d/su, login, passwd, chage and the rest. Both were skipped by -every `update --reinstall` of either package. - -`cfg shadow-pam` now checks for /etc/pam.d/su and /etc/pam.d/system-auth, -runs `packagemanager script configure linux-pam` and `... shadow` when they -are missing, and reads back that both files exist -- naming the error they -produce if they do not. - -The general shape, worth keeping: **install and configure are different -phases, and a package whose configure writes system files is not finished -by an install.** That is the third face of the same coin as -"/etc/pam.d/su_u_user is inert until shadow links pam" and "a rule on disk -is not a rule in force". - -## 1.14.254 -- config BEFORE the rebuild (the window where nothing builds) - -1.14.252/253 rebuilt shadow and THEN wrote /etc/pam.d. Between those two -there is a state where su links libpam and has no service file -- and every -build in this system runs as its package user through `su - p_<pkg>`, so -that window is one where **nothing can be built**, including the configure -phases that would write the missing file: - - # packagemanager script configure linux-pam - su: pam_start: error 26 - -Root's own su survives it because pam_rootok is sufficient -- which is why -`su - p_bash` still worked and reported the pam failure as `(Ignored)`. -That is the only reason this was recoverable. - -So the step is reordered: **write the config, then rebuild.** A run that -stops midway now leaves pam config and a non-pam su (harmless) rather than -a pam su and no config (nothing builds). Asserted by LINE ORDER, not by -the presence of both calls. - -One more bug in the same file: step 1 called `_pam_installed` forty lines -above where it was defined. Under `set -u` that is a command that does not -exist yet, so the config would silently never have been written. Both -helpers are defined before either step, and the suite checks that too. - -### If it ever happens again - -Root can write those files directly -- no su involved. The contents are on -the book's Linux-PAM and Shadow pages; /etc/pam.d/su wants the `pam_exec.so -quiet /etc/pam.d/su_u_user` line FIRST, then pam_rootok, then pam_unix. - -## 1.14.255 -- the noise on every build: package users have no shadow entry - - su: Authentication service cannot retrieve authentication info - (Ignored) - -Not cosmetic, because **every build in this system is a `su - p_<pkg>`**: -that line will now appear in every package build and every log. pam_unix -reads /etc/shadow; `add_package_user` writes /etc/passwd and leaves no -shadow entry, so pam_unix cannot find the account. It is "(Ignored)" only -because pam_rootok already succeeded for root -- for a non-root caller it is -a real failure. - -`cfg shadow-pam` step 4 lists the accounts with no /etc/shadow entry, backs -up /etc/shadow, runs **pwconv** (shadow's own tool: it adds a locked entry -for every passwd entry that lacks one), and reads back whether they -appeared -- naming any that did not. - -**UNVERIFIED:** detection, backup and read-back were exercised against a -fixture; `pwconv` itself was not, because it edits the real /etc/shadow and -there is no safe way to run that here. The script says so where it runs it. -If pwconv turns out not to cover these accounts, the read-back names them -and the backup is beside the file. - -### State after this session - -pam config in place, `/usr/bin/su` links libpam, 6 setid files checked and -0 corrected (the bits were already right). What remains is one test: - - su - u_browser-mini -c id - -and, if that works, the home moves (`usermod -d /home/u_<n> -m u_<n>`) plus -`packagemanager stack services --only app-users --run` to recreate the -~/Shared links and the SHARED_* fragment against the new homes. - -## 1.14.256 -- the shared directory was built under /usr/src too - - $ mkdir -p /usr/src/u_claude/Shared - /home/n76310/Shared/Claude -> /usr/src/u_claude/Shared - -The homes had been moved to /home with usermod, and the next -`cfg app-users` run rebuilt Shared in the package tree anyway. Cause: -`appuser_home()` returned **BASE_DIR** -- so an application user's home, -its shared directory and its XDG profile link all defaulted to /usr/src. -1.14.251 moved the ACCOUNT's home and left three other places reading the -old default. - -Two changes: - - * `appuser_home()` defaults to **/home** (still configurable as - `appuser_home`). - * `_app_home(u)` asks **passwd first**, then falls back -- so an account - whose home has already been moved keeps it, which is exactly the case - that failed. The shared directory and the .bash_profile link both use - it. - -### STILL OPEN from the same output - - * **step 4 (pwconv) did not run.** `shadow-pam` printed neither "every - account has an /etc/shadow entry" nor the missing list, so the block was - not reached -- and `su - p_bash` still says "cannot retrieve - authentication info". The installed script is the thing to check: - grep -n pwconv /etc/pkgusr/stacks/shadow-pam.sh - If that is empty, `make install` kept an older copy (it keeps a file you - have edited, and says so). If it is present, the guard above it - (`[ -r /etc/passwd ] && [ -r /etc/shadow ]`) is what to instrument. - * **/etc/pkgusr/skel-u_xdg/.bash_profile does not exist** on that - machine, so the profile link fails with ENOENT for every account. The - file is in the zip under skel-u_xdg/; `make install` reported "kept your - existing .bash_profile" for it, which cannot be true if the file is - missing -- that message is worth distrusting. - -### Corrective commands for the machine - - for u in u_claude u_browser-mini u_browser-big; do - mkdir -p /home/$u/Shared && chown $u:$u /home/$u/Shared && chmod 2770 /home/$u/Shared - cp -a /usr/src/$u/Shared/. /home/$u/Shared/ 2>/dev/null || true - rm -f "/home/n76310/Shared/$(echo ${u#u_} | sed 's/^./\U&/')" - done - packagemanager stack services --only app-users --run # relinks to /home - -## 1.14.257 -- there is no /etc/shadow on this machine - - # ls -l /etc/shadow - ls: cannot access '/etc/shadow': No such file or directory - -That is the whole explanation for `su: Authentication service cannot -retrieve authentication info` on every package build: pam_unix reads -/etc/shadow and there is none, so it can look up NO account. The passwords -live in /etc/passwd, the traditional way, which is what LFS leaves when -Shadow is built without PAM. - -And my guard was `[ -r /etc/passwd ] && [ -r /etc/shadow ]` -- so the step -skipped in silence on precisely the system that needs it. That is why the -earlier run printed nothing either way. - -`pwconv` CREATES /etc/shadow from /etc/passwd, moving each hash out and -leaving `x` behind. That changes how the machine authenticates, so the new -branch: - - * backs up **/etc/passwd** (the file that changes), not /etc/shadow; - * requires `^root:` in BOTH files afterwards before calling it done; - * chmods /etc/shadow 600 and reports the entry count; - * on any failure restores passwd, removes the half-made shadow, and says - nothing changed; - * prints the undo by hand: `cp -p <backup> /etc/passwd && rm -f /etc/shadow`. - -**CONFIRMED ON THE MACHINE.** The rollback path was exercised here -(shadow absent, passwd restored); the SUCCESS path was then run for real: -pwconv created /etc/shadow, root survived, and - - # su - p_bash -c id - uid=10006(p_bash) gid=10006(p_bash) groups=10006(p_bash),9999(install) - --- silent, no "cannot retrieve authentication info". That line is gone -from every future build log. The remaining `cannot set terminal process -group` / `no job control` pair on an interactive `su -` is unrelated to -pam -- and CONFIRMED on the machine: shadow 4.20's su calls setsid() for -the target shell when the CALLER IS ROOT and the kernel still allows legacy -TIOCSTI (`/proc/sys/dev/tty/legacy_tiocsti` not 0), so the user cannot push -keystrokes into root's terminal. The shell gets its own session (SID = its -PID, TTY `?`), and bash has no job control. Fix is the kernel's: -`dev.tty.legacy_tiocsti = 0` (sysctl) or CONFIG_LEGACY_TIOCSTI=n -- the -latter is in kernel.conf's CONFIG_SET since 1.14.277. - -It is also worth asking whether this is wanted at all: the system works -today, and the only symptom is a noisy line on every build. Converting to -shadow passwords is the conventional answer and PAM expects it -- but -"leave it alone" is a legitimate choice, and then the line is cosmetic. - -### And the Shared layout is right now - - Claude -> /home/u_claude/Shared - Browser-mini -> /home/u_browser-mini/Shared - Browser-big -> /home/u_browser-big/Shared - -with `.bash_profile already points at the XDG profile` for all three. One -thing still open there: `/tmp/xdg-n76310 is group 'n76310', which is not a -shared group` -- the session-group half of --shared, which needs the sway -session up with a proper runtime group. - -## 1.14.258 -- apply the setid bits, and four that were missing - -Two things from shadow's install output. - -**It printed nine chmod lines and walked away.** -`report_setid_requests` runs AS ROOT, immediately after the install -- the -one moment when the thing that is missing can simply be done. It now calls -`cmd_setid --run` (which applies each recorded request and reads the mode -back) and reports what it DID. Non-root callers still get the old advice, -because then it really is advice. Same lesson as the PAM file's mode and -/etc/pam.d/su: warning about a one-line fix you are already privileged to -make is not help. - -**And setid.conf declared five of shadow's nine.** chfn, chsh, newgidmap -and newuidmap were missing -- read off that request list, not guessed. Ten -lines now. - -### On the other question: do shadow and pam each get the other's features? - -**Shadow has to be built TWICE, and that is unavoidable.** Their pages -require each other: Linux-PAM's page lists Shadow as required, and shadow -only gains PAM support if libpam exists when it configures. So the order is - - 1. shadow (no PAM -- this is what the LFS build leaves) - 2. linux-pam (needs shadow) - 3. shadow again (now with PAM) <- `cfg shadow-pam` does this - -PAM itself does NOT need a second pass: nothing in its configure depends on -a feature shadow gains later. So one rebuild, of shadow, in that order -- -which is exactly what services.stack does, with `cfg shadow-pam` -immediately after `book linux-pam`. The step checks `ldd /usr/bin/su | -grep libpam` rather than assuming, so a shadow that somehow already has it -is not rebuilt. - -### Asked for and NOT built: a general post-install root file - -The idea is right and the mechanism half exists (`.pkgusr-setid.requests` -is exactly this, for one kind of request). A general -`$home/.pkgusr-root.post` run by root after a successful install would cover -setid bits, udev reloads, ldconfig, bootscript symlinks -- everything that -currently needs a named repair. - -It is NOT implemented here because of one hard requirement: that file must -be **root-owned and unwritable by the package user**, and the runner must -verify that before executing a line of it. A file inside the package -user's own home, written by root, that root later executes -- if the -package user can touch it, every package on this system has a root -escalation. The verification (ownership, mode, no symlink, parent -directory not writable) is the feature, and it wants writing deliberately -rather than at the end of a long session. - -## 1.14.259 -- root:p_shadow, and a backtick that ran a command - -Both mine, both visible in that output. - -**`root:root` threw away the package group.** Only the USER has to be root -for a setuid bit to mean root; the group is how this system records which -package owns a file, which is the whole point of package users. -setid.conf already used `-` for exactly this -- the REQUEST path -(`cmd_setid --run`, which handles the bits a build asked for and was -refused) forced root. It keeps the group the file already has now: - - /usr/bin/su is now 4755 root:p_shadow - -**And the report printed twice, mid-sentence:** - - applied. Setuid/setgid bits refused during the build ... - ... every setid bit this system needs is in place lists anything still outstanding. - -My success line said `` `lfs-helper setid` `` inside double quotes, so the -shell RAN it and pasted a whole second report into the middle of the -sentence. Single quotes. A backtick in a double-quoted string is a -command, not punctuation -- and this is the fourth time this session that -a quoting mistake produced output that looked like a tool bug. - -Test-harness note in the same spirit: the first version of the group check -created its fixture as root, so root:root came out either way and the -assertion compared nothing. It chgrps to a real non-root group first now. - -## 1.14.260 -- two tables disagreeing, seventeen changes to the wrong answer - -1.14.259 made the REQUEST path keep the group. The output then read: - - /usr/bin/su is now 4755 root:p_shadow <- request table - ... - /usr/bin/su is now 4755 root:root <- "upstream" table, one line later - Applied 17 change(s). - -`cmd_setid` has TWO tables: the bits a build asked for and was refused, and -`_setid_known` -- what upstream installs, for packages whose request was -never recorded. The second hard-coded `root` in its group column, so it -ran immediately after the first and undid it. Nine files changed twice. - -`-` in that column now means KEEP WHAT IS THERE, and the shadow rows use -it. A file already at `4755 root:p_shadow` is recognised as correct and -skipped, so the two tables agree instead of fighting. - -**A setGID entry keeps its NAMED group**, because there the group IS the -purpose: dbus-daemon-launch-helper must be group messagebus, fcrontab group -fcron. That row is unchanged and asserted unchanged. - -The general rule, now written in the table's own header: **for setuid, only -the owner matters and the group belongs to whoever installed the file; for -setgid, the group is the feature.** - -## 1.14.261 -- the setid pass locked linux-pam out of its own file - - rm: cannot remove '/etc/pam.d/other': Permission denied - /etc/pam.d -> already group nimgnu_bash-completion, and 'p_linux-pam' is a member - # retrying linux-pam (phase install only -- the build is done), round 1 ... - PermissionError: [Errno 1] Operation not permitted: '/usr/sbin/unix_chkpwd' - -Round 1 was right and worked. The second failure is one I created two -releases ago: **/usr/sbin is sticky**, so only a file's owner may unlink it -there. linux-pam installs /usr/sbin/unix_chkpwd 4755, the post-install -setid pass chowns it to root -- which is exactly what a setuid bit needs -- -and the next reinstall can no longer replace its own file. EPERM, not -EACCES, which is why no permission repair recognised it. - -Three parts to the fix, all in `_take_cmake_dest`: - - * **meson's wording**: `Operation not permitted: '<path>'` extracts the - path. cmake says `cannot copy file "A" to "B"`; python says this. - * **the GATE learned it too** -- it only accepted cmake's phrasing, so the - function returned before looking. The same two-places-one-fact shape as - 1.14.222. - * **root's files are still off limits, with one exception**: a path the - package's OWN manifest claims. That is the file it installed and we - then gave to root; handing it back for the install is safe, and the - setid pass re-applies the bit afterwards -- so the loop closes instead - of alternating. A root-owned file no manifest claims is left alone, - which the suite checks with two files and one manifest. - -The lesson is about the shape, not the file: **a repair that changes -ownership creates work for the next install of the same package.** The -setid pass and this take-back are two halves of one cycle, and either alone -is a bug. - -## 1.14.299 -- a package and the thing that starts it come from different tarballs - -"dhcpcd does not start at boot; did we forget its init script?" There is no -init script to forget: BLFS ships no /etc/rc.d/init.d/dhcpcd at all. The only -target is install-service-dhcpcd (/lib/services/dhcpcd), which services.stack -already installs, and that script is started by nothing on its own -- the LFS -`network` script reads /etc/sysconfig/ifconfig.<iface> at boot and hands each -interface to the service ITS file names. With no such file the machine comes -up with dhcpcd installed, its service script in place, and no address. -`lfs config network dhcp` writes no interface files on purpose, and nothing -else wrote one either. - - * **stacks/bootscripts.conf** -- the services.stack install= list as a - MAPPING: `package | proof | targets | creates`. proof and creates take - several comma-separated alternatives (a daemon that moved between /sbin - and /usr/sbin is still installed; /lib/services and /usr/lib/services are - the same service). - * **cfg bootscripts** installs the targets whose package IS here and whose - script is NOT, skipping packages that are absent (an init script for - something uninstalled is a symlink in rc?.d that fails at boot), and then - **checks the disk rather than make's exit status** -- a target that - prints nothing and installs nothing is the failure that matters. make - runs as the tree's owner, so the files keep saying which package user - installed them, with root as a fallback that says so. - * The same step writes /etc/sysconfig/ifconfig.<iface> for a WIRED - interface that has none: ONBOOT, IFACE, SERVICE=dhcpcd, DHCP_START/STOP. - That file holds no secret, which is exactly why it can be written here -- - a wireless one needs an SSID and a key, so those are still left to - `check net-wifi`, which prints the file to paste. - * The entry carries `expect=`, re-checked when cached, so a package - installed later is noticed on the next run instead of on the next boot. - -Tested: a fake bootscripts tree with three targets, one package installed, -one absent and one target that lies (prints success, installs nothing) -- the -absent package skipped, the two real ones installed, the liar caught by the -after-check and the step exiting 1; a second run changing nothing; the -interface file written for eth0, wlan0 left to net-wifi, and an existing file -naming another service left alone. NOT tested: a real blfs-bootscripts -Makefile, and a boot. - -## 1.14.298 -- we wrote the file that unsets the locale - -The i18n.sh on the machine was not BLFS's. It was OURS -- -`lfs-helper install_shell_startup_files`, which wrote: - - # Locale. Set LANG here (e.g. en_US.UTF-8); an unset LANG means the C - # locale, which is a choice, not an accident. - if [ -z "${LANG:-}" ] && command -v locale >/dev/null 2>&1; then - : # export LANG=en_US.UTF-8 - fi - -The export is commented out, and the comment argues that this is deliberate. -It is not a choice anyone makes: nothing else on the system sets LANG either, -so every machine built with these tools sits in the C locale until a terminal -complains, and the only repair is to hand-edit a generated file. The three -previous releases chased this through /etc/profile and BLFS's own placeholder -without ever looking at who wrote the file. - - * lfs-helper writes an i18n.sh that reads /etc/locale.conf -- the file LFS - uses and the one `cfg shell-env` now writes from `lfs config locale`. It - carries `# written by lfs-helper`, and with no locale.conf it does - nothing, exactly as before. `_write_if_absent` still keeps an existing - file, so this only helps new systems. - * shell-env.sh therefore recognises all THREE writers of that file as ours - to replace when it produces no LANG: its own marker, `# written by - lfs-helper`, the book's `Begin /etc/profile.d/i18n.sh`, and -- for - machines already built -- the commented-out line `: # export LANG=` - itself. A file nobody here wrote is still left alone and reported. - * The replacement keeps `LESSCHARSET=utf-8`, which lfs-helper's version set - and less(1) reads instead of the locale. - -Tested: shell-env against a byte-for-byte copy of the machine's i18n.sh -(recognised, backed up, replaced, LESSCHARSET kept, LANG then set from the -locale.conf it had just written), and lfs-helper's new fragment sourced in an -empty environment with a locale.conf present (LANG set, rc 0) and absent -(nothing, rc 0). - -## 1.14.297 -- i18n.sh was already there, and /etc/profile.d belongs to nobody - -1.14.296 would not have fixed the machine it was written for. `ls -/etc/profile.d` showed i18n.sh already present (root, Sep 4, from the BLFS -page) -- so "exists and is not ours, left alone" would have been the whole of -it, and $LANG would still have been empty. The file was never the problem: -it does `. /etc/locale.conf`, and /etc/locale.conf did not exist. - - * **i18n.sh is judged by what it produces, not by who wrote it.** It is - sourced in an empty environment (TERM=xterm, because both the LFS and - BLFS versions hard-code C.UTF-8 when TERM=linux) and the LANG that comes - out is compared with the one this machine should have. Right -> left - alone, whoever wrote it. Wrong, and it carries the book's header -> - backed up and replaced. Wrong and hand-written -> left alone and - reported. locale.conf is written FIRST, so the book's own file usually - starts working and is never touched. - * **Root writes it, root owns it.** /etc/profile.d/pkgusr-system.sh was - owned by p_gstreamer10: `env --system` opens the existing file for - writing, which keeps its owner, and it was first created inside a build - window. A package owning a file no package installed is wrong on a - package-user system, and every login shell sources that directory -- - whoever may write there runs code as everyone who logs in. `env - --system`, shell-env.sh (dircolors, bashrc, locale.conf, i18n.sh) and - app-users.sh (shared-dirs.sh, xdg-runtime-share.sh, /usr/bin/xdg-runtime-share) - now re-assert root:root on every run, as the PAM file already did. - * **/etc/profile.d joins SHARED_DIR_PATTERNS.** rust, bash-completion, - gettext and the book's own config pages each drop ONE file in it, so the - first package to create it owned it -- p_libnice here -- and every later - package had to be granted it by hand. - * Permissions there are right as they are: /etc/profile SOURCES these - files (`. $script`), it does not execute them, so 644 is correct and +x - would mean nothing. Contrast /etc/pam.d/su_u_user, which pam_exec really - does run and which app-users forces to 755. - -Tested: the locale block against a stock BLFS i18n.sh with no locale.conf -(writes locale.conf, leaves the script alone -- the machine's own case), an -i18n.sh that yields nothing with locale.conf present (backed up, replaced), -and a hand-written one (left alone, reported); is_shared_dir on -/etc/profile.d with and without a trailing slash. NOT tested: the chowns on -a system where the files are really owned by a package user. - -## 1.14.296 -- nobody set $LANG - - warning: foot: 'C' is not a UTF-8 locale, falling back to 'C.UTF-8' - -Nothing was built wrong. LFS's own /etc/profile reads /etc/locale.conf, but -BLFS's "The Bash Shell Startup Files" page REPLACES /etc/profile with one that -reads /etc/profile.d/*.sh instead -- and i18n.sh, the piece of that page which -exports LANG, is a separate block, easy to walk past. Take that page (this -machine did: its /etc/bashrc is the BLFS one) and the locale quietly falls back -to C. `lfs config locale` was set, and nothing after the chroot ever read it. - - * shell-env.sh -- the step that exists because that page is config and no - package stack runs it -- now also does the locale, and writes BOTH files, - so it works whichever /etc/profile is installed: /etc/locale.conf - (`LANG=`) and /etc/profile.d/i18n.sh (sources it, exports LANG, and - LC_ALL when set). - * The locale is taken, in order, from $LFS_LOCALE, /etc/locale.conf, - `locale` in /usr/share/lfs/config.json (what `lfs config locale` wrote), - then en_US.UTF-8, then C.UTF-8 -- and only if `locale -a` really has it. - **A name is not a locale until glibc has built it**: locale.conf may name - one nobody generated, which looks exactly like setting nothing. That is - reported with the localedef line to fix it, as glibc's package user -- - the locale archive is glibc's, and root would leave root's files in it. - * An existing /etc/locale.conf is never rewritten; a hand-written i18n.sh - is left alone. A non-UTF-8 choice is reported, not overridden. - * services.stack gives `cfg shell-env` an `expect=` -- /etc/bashrc, - /etc/locale.conf and i18n.sh. Without it a machine that ran this step - before the locale was part of it stays `done (stack cache)` for ever and - the terminal keeps complaining with nothing to say why. - -Tested: the locale block with a de_DE.UTF-8 asked for and not installed (it -says so, names the localedef line, falls back), run twice (second run changes -nothing), the generated i18n.sh sourced in a shell with LANG unset (sets it, -returns 0), and a hand-written i18n.sh left alone; the expect= expression run -as the stack runs it. NOT tested: a real login, and a machine that has -de_DE.UTF-8. - -## 1.14.295 -- a menu entry has no terminal, and su wants one - -Three from the machine after 1.14.294: the .desktop entries did nothing, the -/etc/bashrc hook refused to apply, and /root filled with tarballs of homes -that held nothing. - - * **The launcher now allocates a pty when there is no tty.** Run from a - terminal, `~/bin/reddit` worked; clicked in the menu it did nothing -- - `su` fails without a terminal and its message goes to a stderr nobody - reads. The old hand-made entry worked around it with - `Exec=script -c 'reddit' /dev/null`, which is the same fix in the wrong - place: every entry had to remember it. The launcher now branches on - `[ -t 0 ] && [ -t 1 ]` -- `exec su` as before with a terminal, otherwise - `script -qec "$(printf '%q ' su ...)" /dev/null`, which gives su a pty, - sends the typescript to /dev/null and returns the program's own exit - status. A non-zero status with no terminal to print to is passed to - `notify-send` when it exists. - * **Menu entries get `Categories=`** -- Network;WebBrowser; for the - browsers, AudioVideo;Video;Player; for mpv, Utility; otherwise. Without - one an entry is "Other" at best and invisible at worst. - * **/etc/bashrc is recognised by `# Begin /etc/bashrc`**, its first line in - both the BLFS file and the one shell-env.sh writes. 1.14.294 looked only - for shell-env.sh's own line, so a machine carrying the BLFS file got the - line printed and the variables stayed unreachable from a terminal. One - line is appended, after a backup, starting with a newline -- the file on - this machine ends `# End /etc/bashrcCK=$(tput setaf 0)`, which is what an - append without one leaves behind. - * **No archive for a home that was never used.** The tarball exists for a - browser profile -- logins, cookies, history, downloads, which - `userdel -r` would take with it. A home holding only the .bash_profile - symlink these tools made and an empty Shared has none of that, so - `_home_is_untouched` skips it: a purge of fourteen fresh accounts now - leaves nothing in /root. When archives ARE kept, purge ends by saying - how many, how to look inside one and how to drop them. - -Tested: the launcher with and without a terminal, su replaced by a printer -that reports whether it got a tty (no tty -> wrapped in script, su sees a -tty; tty -> exec su direct), arguments and the default page surviving the -extra quoting, and a failing su's exit status coming back out; the archive -skip against an untouched home and one with a .mozilla in it, dry and real; -the bashrc hook against a BLFS-style file, twice (appends once, backs up -once). NOT tested: a real su from a real menu. - -## 1.14.294 -- the variables were right and invisible; simpleicons dropped OpenAI - -From the machine after 1.14.293 ran clean: `~/Shared` held the ten new links, -`shared-dirs.sh` was correct, and `$SHARED_<TAB>` still offered only -BROWSER_BIG, BROWSER_MINI, CLAUDE -- the three from the login BEFORE the -change. - - * **/etc/profile.d is read by LOGIN shells.** A terminal opened inside - sway is not one: it reads ~/.bashrc and shows whatever the session - inherited at login. The step now also sources the fragment from - /etc/bashrc -- the file every interactive shell reads, and the one - shell-env.sh writes, recognised by its `Based on the BLFS "Bash Shell - Startup Files" page` line. Anything else is left alone with the line - printed. `APP_USERS_BASHRC` overrides the path for testing. - * **The fragment now takes variables away too.** An account removed after - login left `SHARED_X` in the environment pointing at nothing, inherited - by every shell the session started -- so a correct removal looked like it - had failed. It drops every `SHARED_*` whose directory is gone (`env` + - sed + `eval`, POSIX, tested under dash) before exporting what is there, - and is safe to source again at any time: `. /etc/profile.d/shared-dirs.sh` - refreshes a running shell. - * **simpleicons.org has dropped `openai` and `grok`** (confirmed: both 404 - in the upstream repo, `claude` still there). That is why chatgpt, nrwgpt - and grok got no icon. Those three lines now carry Wikimedia URLs in the - stable `Special:FilePath/<File.svg>` form. - * **`#RRGGBB` at the end of the icon field** paints a one-colour SVG after - it is fetched (`_recolor_svg`): fills in attributes and in `style=`, - `fill="none"` left alone, a root `fill` added when there is none. It - replaces simpleicons' `/slug/colour` service, which cannot help for a - brand it no longer carries -- nrwgpt is the ChatGPT mark in sky blue. - The `.from` stamp records URL+colour, so changing only the colour - refetches. - * A launcher that is not ours is still left alone, but now says - `rm <path> and run this again` -- launchers from an older version are not - recognised either, so "left alone" had quietly kept the OLD behaviour (no - default page, no --app-id) on chatgpt, nrwgpt, grok, mercury, deepseek - and youtube. - -Tested: the fragment under bash and dash with a dead variable, a moved one -and a dangling link (dead dropped, moved corrected, dangling skipped); the -/etc/bashrc hook against a shell-env.sh file and a foreign one; `_recolor_svg` -on a real simple-icons file and on one with attribute, `style=` and `none` -fills; the launcher suite re-run unchanged. NOT tested: the Wikimedia -downloads (this machine cannot reach them). - -## 1.14.293 -- undoing the application accounts, and a default page per launcher - -Three things the last release left half-done: removal stopped at the account, -a search launcher with nothing typed landed on an empty search page, and the -conf did not say which window is which. - - * `packagemanager user purge [--run] [--yes] [--keep-home] [--no-archive] - [--accounts a,b] [--conf F]`: the whole undo. Every account in - app-users.conf, plus **strays** -- accounts with the user prefix whose - home is under `appuser_home` but whose line is gone (exactly - u_browser-mini/u_browser-big after the layout change) -- each through - `user delete`, so each home is archived to /root first. Then what no - per-account command could remove because the next account still needed - it: dead ~/Shared links, the hub and the icon dir when empty, - /etc/profile.d/shared-dirs.sh, /etc/profile.d/xdg-runtime-share.sh, - /usr/bin/xdg-runtime-share (each only if it carries the app-users - marker), /etc/pam.d/su_u_user, the `pam_exec` line in /etc/pam.d/su - (backup first), appuser_home when empty, and the session group last. - Kept on purpose: the /root archives and app-users.conf. - * `user delete` now removes **every dangling link** in ~/Shared, not only - the ones pointing into this home. A home moved after the link was made - (usermod -d, or the move to /home/shared_user) left the link behind, and - shared-dirs.sh enumerates that directory -- which is why - `$SHARED_BROWSER_MINI` outlived the account. Both commands end by - naming the SHARED_* variables an already-open shell still has, and the - `unset` for them: that file is read at LOGIN. - * `cfg app-users` takes a shared directory away again: a line changed to - `dir=no` has its ~/Shared/<Name> link removed when the target is empty - or dangling, and is reported, not touched, when it still holds files. - * Launcher templates take a default: `...?q={https://www.reddit.com/}`. - With something typed it is the search; with nothing typed the word - becomes what is inside the braces, instead of the URL cut at its `?` - (`/search/` -- a search page for a search nobody made). Plain `{}` is - unchanged. The fallback cannot contain `|`: that is app-users.conf's - field separator. `_write_launcher` now fills a plain template with - .replace() rather than %-formatting bash full of `%02X` and `${w%%...}`. - * app-users.conf: `--app-id <name>` spelled out on every browser line - except the four generic ones (browser-mini, browser-big, google, - duckduckgo), which keep the automatic one. Shared directory (and - `--download-dir ~/Shared`) on the accounts that want it; **not** on - dokus, reddit and youtube -- what they download stays in the account. - Default pages for google, duckduckgo, github, codeberg, reddit, youtube. - Icons: ARD and arte from Wikimedia, nrwgpt as the OpenAI mark in sky - blue (`https://cdn.simpleicons.org/openai/38BDF8`). - -Tested: launchers generated and run with su replaced by a printer (default -page with no arguments, search with `&` and an umlaut, `{}` passthrough, `~/` -expansion); the conf through the step with a stubbed packagemanager (every -line parses, the four accounts aggregate as expected); the dir=no cleanup -against an empty, a dangling and a non-empty link; purge dry and real against -a built fake state -- two accounts (one a stray), dead links, the marker -files, the su line, the group -- verified afterwards by hand. NOT tested: a -real su, and the icon downloads (this machine cannot reach the CDN). - -## 1.14.292 -- application accounts: several launchers, {} arguments, icons, delete - -From the user's app-users.conf (already written as `user | name | app ...`, -which the step could not read -- hence `~/bin/browser-mini` still running -u_browser, an account never created): - - * app-users.conf: `account | name | app | share | dir | desktop | launcher | - groups | icon`, one line per LAUNCHER. Accounts are aggregated: the - strongest share, any dir, all groups; created/set up once. Then - `packagemanager user launcher <account> <name> --app ... [--icon]` per - line. The old 7-column format (header `name`) still reads. - * Launcher (packagemanager `_write_launcher`): the argv is built in the - script -- a word `{}` = the arguments; `{}` inside a word = URL-encoded - search term, or with no arguments the word cut before its `?`; no `{}` = - nothing passed (claude's "a second address:" came from an extra '' - argument). `~/` in the app = the ACCOUNT's home. browser-mini/big get - `--app-id <name>` when missing. `exec su - u -c 'exec "$0" "$@"' -- argv`. - * Menu entry: named after the launcher; Exec %U only with {}; Icon from - `_fetch_icon`: URL, simpleicons slug, empty -> the name, - -> none; fetched - once into ~/.local/share/icons/lfs-apps (.from records the URL). - * `user delete <account> [--run] [--keep-home] [--no-archive]`: application - accounts only (prefix, home under appuser_home or the package tree), - refused while it runs anything; home archived to /root/<u>-home-<date> - .tar.gz (0600) first; removes its launchers/menu entries/icons (only ones - that run as it), ~/Shared links into its home, its su_u_user line, - account, group, home. `user remove-launcher <name> [--run]` for one - launcher. Both remind that the conf line must go too. - * The shipped app-users.conf is the user's list, corrected: google/ddg/ - youtube/reddit/github/codeberg search URLs, chatgpt/gemini pointed at - their own sites (both were claude.ai), codeberg .org, download dirs - ~/Shared (were /usr/src/u_*), media accounts full, broken ard/zdf/mpv - lines completed, icon slugs where the name differs (openai, - googlegemini). - -Tested: generated launchers run with su replaced by a printer (search term -with umlaut and &, no args, fixed URL with an extra arg, ~/ expansion), the -conf parser against the shipped file with a stub packagemanager, delete dry -and real (archive, launchers, link, pam regex), remove-launcher leaving a -hand-made file. NOT tested: real su, icon downloads. - -On the machine: u_browser-mini and u_browser-big stay until deleted. - -## lfs-backup 1.3.0 -- merge - -`merge <archive> [--plan f] [--only/--skip]`: the archive's existing .new -files one at a time, `diff -u` (MERGE_LINES, default 60), then k (keep live, -remove .new), n (take the backup's CONTENT; live owner/mode stay; .new -removed), e (MERGE_TOOL or vimdiff live .new, else $EDITOR .new; asks again), -s, q. passwd/group/shadow/gshadow refuse n. Answers from /dev/tty, or stdin -without one. Tested with piped answers (k, n twice with a 600 live file, -refusal for passwd); suite block extended. - -## lfs-backup 1.2.2 -- plan lines keyed by kind; absolute targets - -The user's first plan: `~/.bashrc` etc. exist in [user] and [root]; the plan -named them identically and restore took the FIRST matching line, so root's -file followed the user's line. And `~/` targets read as /root to someone -checking as root (953 "same" ALE files were the main user's, correctly). -Section column is now `kind:section`, the lookup key is kind:name, and the -target column is the resolved path (~/ still accepted). Older plans are -refused with "write a new one". Tested: user line skipped, root line kept. - -## lfs-backup 1.2.1 -- secrets restored 0600-or-less - -Second real backup: excludes worked (1043 files, 9.3 MB; no .git, test, -*~, .netrwhist). The host modes are unchanged (gshadow 644, Wi-Fi keys -644/664), so restore now drops group/other bits for shadow, gshadow, -wpa_supplicant/*, ssh host keys, ~/.ssh/id_*, authorized_keys, and says so. -Tested with a 664 key and a 644 gshadow -> 600 each. - -## lfs-backup 1.2.0 -- restore plan, excludes, ownership by kind - - * `plan <archive> [file]` writes `action | state | section | name | target` - per file (identical -> skip); `restore/clean --plan file` follow it: skip - lines are left alone, the target column may be ~/... (home of that kind) - or absolute. Unknown actions / short lines stop with the line numbers. - * `!pattern` lines exclude matching names everywhere (bash [[ == ]], so * - crosses /). Shipped: plugin .git and test trees, *~, .netrwhist. - * Owner on restore by KIND: [root] root:root always; [user] the main user - (group kept only if it is another existing group); [system] recorded - names that exist, never the main user, else root. - -From the user's first real backup (manifest reviewed): root's .bashrc and -.bash_profile were links into ~/LFS/lfs-config and recorded as n76310's -- -by name that restores a root dotfile the main user can edit. Also on the -HOST, not the tool: /etc/passwd, /etc/group, /etc/hosts owned by uid 9999 -mode 664/774; /etc/gshadow 644 (world-readable hashes); most -wpa_supplicant PSK files 644/664; /etc/vimrc owned by n76310 664. -/usr/share/vim/vimfiles held only cmake's files -- dropped from the config. - -Tested: plan with a skip and a moved target, root file ownership, clean by -plan, a malformed plan line; suite block extended. - -## lfs-backup 1.1.0 -- named sections, --only/--skip, clean - - * `[user:vim]`, `[system:accounts]` ...: a section's NAME (default: its - kind) is recorded per file in the manifest (7th column; older archives - fall back to the kind). `--only a,b` / `--skip a,b` select by name or - kind for create, restore and clean. `sections [archive]` lists them. - * `clean <archive> [--run]`: removes <target>.new for the archive's files - only when its content is still what restore wrote; an edited .new is - kept and listed; .new files from anything else are never touched. - * The shipped backup.conf is the user's list, sectioned, plus vim - (~/.vimrc, ~/.vim for user and root; /etc/vimrc, /usr/share/vim/vimfiles) - and /etc/pkgusr/backup.conf itself. /etc/rc.d is NOT taken whole: its - init.d scripts and rc?.d links come from the bootscript packages. - -Tested in the sandbox (sections from config and archive, --skip, --only, -clean with one edited .new); suite block extended. - -## 1.14.291 -- lfs-backup replaces lfs-config - -lfs-config was a git-repo manager (init/pull/push/adopt/mirror/undo). The -user wanted a backup: one list of files, carried to a new system, put back -without replacing anything. - - * /etc/pkgusr/backup.conf: [user] (~ = PKGUSR_SYS_MAIN_USER's home, read - from the environment or /etc/profile.d/pkgusr-system.sh), [root], - [system]; globs; a directory takes its files. Ships the user's list plus - suggestions (passwd/group/gshadow with shadow, hostname, hosts, resolv, - inittab, clock, console, profile, inputrc, sysctl, modprobe.d, sshd + - host keys, app-users.conf, packagemanager.conf, ~/.gitconfig, ~/.ssh/config). - * `create [--run] [archive]` (root): tar.gz with a manifest of mode + owner - and group NAMES (ids resolved through the tree's own passwd/group); a - symlink is stored as its target file. Archive 0600 (secrets). - * `restore <archive> [--run]` (root): missing -> created with mode/owner - (old main user's name -> this system's main user; unknown names -> root, - reported); identical -> left, mode differences reported; different -> - <file>.new; passwd/group/shadow/gshadow ALWAYS .new. Missing parent dirs - under a home belong to that user. The .new list is printed at the end; - an identical existing .new is not rewritten. - * LFS_BACKUP_ROOT restores into (or backs up from) a mounted tree. - * Makefile installs lfs-backup + backup.conf (yours kept, .new beside), - removes a retired /usr/bin/lfs-config. - -Tested end to end in the sandbox (list, create, show, restore dry and real, -re-run); suite block replaced with a functional one. - -## 1.14.290 -- machine.conf [gstreamer10] gst_debug = true - -The user wants GST_DEBUG logging. The book configures the core with -`-D gst_debug=false`; the machine option REPLACES that switch (tested: -apply_machine_opts turns the script's `-D gst_debug=false` into -`-Dgst_debug=true`). `gstreamer:gst_debug` is the subproject form -- not -applicable to BLFS's separate core package, not added. Plugin packages must -be rebuilt after the core: gstconfig.h's GST_DISABLE_GST_DEBUG is compiled -into each of them. No tool changed; the version moves with the stack file. - -## 1.14.289 -- launchers: a stale account, and an empty argument - - ~ > browser-mini - No passwd entry for user 'u_browser' - -~/bin/browser-mini was written when the account was still u_browser; every -later setup said "already exists -- left alone". Launchers and menu entries -that carry this tool's header ("# Runs ... as the application user ..." / -"Comment=Runs as the application user ...") are now rewritten when they -differ from what would be written; files without it are left alone. - -The CURRENT writer was also wrong: `su - u -c "app $(printf '%q ' "$@")"`, -and printf '%q ' with no arguments prints '' -- every launch without a URL -passed one empty argument. Now `exec su - u -c 'app "$@"' -- <name> "$@"`: -shadow su passes what follows the user name to the shell (checked in -su.c 4.20.2), so bash gets $0=<name> and exactly the typed arguments (n=0 -with none). Tested the writer (stale rewritten, current kept, hand-made -kept) and the argument count. - -Pre-existing in the upload: "a launcher could point at a program that does -not exist" (its sed range no longer matches the call). - -## 1.14.288 -- `script install mesa` ran the OLD version's script - - $ build p_mesa --script /usr/src/pkgusr/p_mesa/install_Mesa-25.3.5 --phase install --force - (Mesa 26.1.7 is what is installed) - -`_versioned_script_in` returned the alphabetically first install_<name>-<ver> -in the home, and every version's script stays there. It now takes the one -install_last names, else the newest by mtime. Tested both paths; suite runs -it on a fake home. - -Also: `# user 'p_<pkg>' already exists` is no longer printed on every build -(the build calls add-user with _QUIET_EXISTING=1); `lfs-helper add-user` -still says it. - -## 1.14.287 -- three undefined names (pyflakes) - - could not work out what WebKitGTK-2.52.5 depends on -- NameError: name 'ignore' is not defined - - * cmd_install's one-package missing-dependency check used `ignore`, a - local of build_install_plan. Every non-recursive install printed the - NameError instead of its missing dependencies. It now builds the same - set from args.ignore. - * cmd_script_status: `versions.get(...)` in a block that cannot currently - run (its states are script states); now `assess(u, None, run_cmd=False)`. - * lfs `_accounts_root`: `read_config()` does not exist; the NameError was - swallowed, so a configured pkgusr_subdir/cfguser_subdir was ignored. - Now load_config(). - -pyflakes finds nothing else but STACK_DIR (globals()-guarded on purpose). -Suite: runs pyflakes on packagemanager, lfs, blfs when it is installed. - -## 1.14.286 -- the session group: <prefix>_xdg-runtime - - /tmp/xdg-n76310 is group 'n76310', which is not a shared group -- u_x was not added - Set up a runtime group for the session first, then re-run this. - -Nothing ever set one up. The user's decision: one named group, -nimgnu_xdg-runtime, on the main user's XDG runtime directory; every shared -u_* account and the main user in it. - - * packagemanager `xdg_runtime_group()` = collector prefix + "xdg-runtime"; - `_join_xdg_runtime_group` creates it (collector range), adds main + u, - and chgrp/chmod 0750 the runtime directory if it exists now. The old - "read the group off the directory" rule and its tests are replaced. - * The directory is recreated each boot, so the SESSION applies the group: - app-users writes /usr/bin/xdg-runtime-share (dir 0750 + g+rw on - wayland-N and pipewire-0 sockets, polling ~30 s as they appear) and - /etc/profile.d/xdg-runtime-share.sh (dir only, at login). Not the - session bus: dbus refuses other uids. The user's sway config is not - edited; the step prints the `exec xdg-runtime-share` line to add. - * app-users expect= requires the helper, so the cached step re-runs. - -Tested: the generated helper, run as a non-root member, grouped the dir -(0750) and the wayland socket (g+rw), left the .lock alone. Not tested in a -real session: whether pipewire lets another uid in beyond the socket mode. - -## 1.14.285 -- "done" is not "current", and media's entries were done by sway - - media 1/19 done (with 189 of 215 packages installed) - ... ~70 lines of "(book is newer)" in `stack media --status --full` - -By design a stack run builds what is MISSING and never updates what is -installed; `update --stack <name>` does that. Two displays hid it: - * done counts read only the stack's OWN cache. sway includes media, so - media's entries were recorded under sway. The list and --status now - count an entry done if ANY stack's cache has it. - * --status printed the per-package "(book is newer)" only with --full. It - now prints `behind : N installed older than the book (...)` and the - `update --stack` command. The list shows "N entries behind the book" - (top-level entries only -- the dependency tree is --status's job). -Tested: the list with stubbed book/user data. Status blocks' failures are -pre-existing in the uploaded zip. - -Noticed in that status (unverified): spirv-llvm-translator and libclc show -installed at 22.x, but both last ended in failures under the record bug -- -check `llvm-spirv --version`. libical 3.0.20 -> 4.0.5 is a major (soname) -bump: update it with -R. - -## 1.14.284 -- `packagemanager stack` lists the stacks - -`name` is optional now. Without it: every *.stack in /etc/pkgusr/stacks -(and stacks/ beside the tool if different), its description (the header's -text after " -- "), buildable entries done/total from the stack cache (the -included avoid list is not counted), and a `.new waiting` note. avoid.stack -shows as "never built". Suite: the sway line is listed. README shows it. - -## 1.14.283 -- app-users' probe judged accounts it does not manage - -The 1.14.282 probe failed after a correct run: it checked EVERY u_* account, -and u_test (made by hand, not in app-users.conf) is still /home/u_test. A -permanently failing probe also re-runs the cached step on every stack run. -It now checks only the names in app-users.conf. Tested with a stub getent: -a managed account in /home fails, an unmanaged one does not. - -The failure text for a cfg entry said "MISSING A FEATURE ... optional -dependency" and suggested installing one. A cfg entry now gets: the step ran, -what it keeps true is still false, read its output, run it again. - -## 1.14.282 -- application homes in one subdirectory, as configured - - # ls /home - n76310 tester u_browser-big u_browser-mini u_claude u_test - -The machine's layout is appuser_home=/home/shared_user (history, "the user's -system"). `_relocate_app_home` ignored it and moved every account to a -hard-coded /home/<user>; `_setup_application_user` chmod-ed -appuser_home()/<user> instead of the real home; `lfs` defaulted -appuser_home to /usr/src. - - * APPUSER_HOME_DEFAULT = /home/shared_user (packagemanager), the same in - lfs's layout defaults; `appuser_home` in the config still overrides. - * Relocation targets appuser_home()/<user>, from the package tree OR the - old /home/<user>; creates the subdirectory root:root 755; re-points the - main user's ~/Shared/<Name> links into the new home. - * `user setup` relocates too, so `cfg app-users` moves existing accounts; - its expect= fails while any u_* home sits directly in /home, so the - cached entry re-runs. - -Tested in the sandbox: home moved with contents, 0750, parent created, link -re-pointed. Tests pinning the old /home and /usr/src defaults updated. - -## 1.14.281 -- no version record was written on a normal install - - .pkgusr-installed: LLVM-21.1.8 install_last: LLVM-22.1.8 llvm-config: 22.1.8 - -The "why is LLVM in the plan again" answer. cmd_build's success has two -branches: `--phase all|update` (every packagemanager install) and a -single-phase run finishing the last phase. .pkgusr-installed was written -ONLY in the second. VERSION, written before both, parsed only -name_version="..." -- the generator writes '...' -- so it was never written -for BLFS scripts either. Readers fell back to install_last, staged before -the build (the webkitgtk/LLVM "failed build advanced the version" bug). -1.14.273's seeding then pinned the OLD version with nothing to update it. - -`_record_installed` writes both records, parsing either quote style, and -both success branches call it. Suite: both calls, and a functional run of -the function; fails on 1.14.280. - -On the machine: markers seeded since 1.14.273 are stale for every package -that installed successfully since -- listed by a one-off in chat, set by hand -(LLVM: 22.1.8). - -The manifest cleanup found the pollution in 11 manifests and 4 pkg.lst -files, not only LLVM's (gst10-plugins-bad, libclc, libnftnl, librice, -libsndfile, mesa, opencv, pkgusr, shadow, webkitgtk). - -## 1.14.280 -- the earlier-stages claim took root's files from LLVM's manifest - - # claiming files from earlier stages of p_llvm ... - # gave 'p_llvm' 13 path(s) from its earlier build stages - -1.14.278 stopped the lease lending LLVM's polluted root entries, but the -claim step (before every phase) CHOWNS root-owned manifest entries to the -package for good: su_u_user and the tools became p_llvm's. The claim exists -for chapter 7's root-built leftovers under the same step name. - -Now: once the package has installed as itself (.pkgusr-installed or VERSION -in its home), only its -tmp/-pass1/-pass2/libstdcpp stage manifests are -claimed from, never its own. Claim blocks 87/197/210/214 pass. - -On the machine: chown the 13 back to root and run the 1.14.278 manifest -cleanup. Why LLVM is in the plan as "update" again is NOT known yet -- -its record should say 22.1.8 after the run that reached libclc; asked for -.pkgusr-installed and install_last. The install phase re-runs cmake+ninja -(the compile is in install_pkg) but the build tree is kept, so ninja should -have little to do. - -Pre-existing (upload): block "a failed build makes no install claim" fails -on libdisplay-info and poppler after sway.stack's git marker. - -## 1.14.279 -- firmware: patterns at any depth, flat names from WHENCE - - !! FIRMWARE_ONLY asks for 'iwlwifi-*.ucode', which is not in .../linux-firmware - intel/iwlwifi/iwlwifi-100-5.ucode ... - !! this config builds firmware INTO the kernel, so the build cannot proceed - -linux-firmware moved the iwlwifi blobs into intel/iwlwifi/ and keeps the flat -names drivers ask for as `Link:` lines in WHENCE (its copy-firmware.sh makes -the links). The sparse checkout already fetched the files at their new depth -(no-cone patterns match anywhere); the install looked at the top level only. - - * A pattern entry now matches files at any depth (and counts WHENCE link - names); files keep their paths (cp --parents). - * After a selective install, every WHENCE `Link:` whose target was - installed is created, never over a real file. `intel` alone now yields - the flat iwlwifi names too. - * No-cone sparse checkouts include /WHENCE. - -Tested on a fake checkout: both blobs copied under intel/iwlwifi/, three -links made, a link to a missing target skipped. WHENCE's exact format is -from memory (gitlab unreachable). - -**Also fixed:** 1.14.270 broke two existing suite checks -- the mesa probe -must use `test -e a -o -e b`, and the llvmpipe comment must sit directly -above `book mesa`. libglvnd now comes before that comment. - -**Pre-existing suite failures** (present in the uploaded zip, not touched): -alsa option for gst-plugins, mesa software gallium/vulkan drivers in -machine.conf, webkitgtk USE_LIBRICE / USE_WOFF2 / woff2 probe, openh264 -out-of-book markings. - -## 1.14.278 -- LLVM's manifest swallowed root's files, and the lease lent them - - # leased /etc/pam.d/su_u_user (755 root:root) to p_llvm for this phase - # leased /usr/bin/lfs (755 root:root) to p_llvm for this phase ... 14 files - -LLVM's compile is generated into install_pkg (OPEN, 1.14.273), so its -install window lasted hours. Root ran `make install` of these tools and -`cfg app-users` inside it; the touched-file scan is by time window and keeps -root-owned files, so they entered LLVM's manifest. The lease then treated -each as LLVM's own root file and chowned su_u_user -- run by root on every -su -- to p_llvm, every install round. - - * **Scan:** when a phase ran as the package user (not as_root, not - chown-after), a root-owned file it "touched" is not recorded -- it cannot - have written one; a leased file it rewrote is its own after settle. - * **Lease:** a root-owned file is lent only if it is setid (bit or - setid.conf) or the package's configure commands write it. A manifest - claim alone is no longer enough. Suite block 111 case 7; fails on - 1.14.277. - * Existing polluted manifests: one-off cleanup in chat (tool files, - su_u_user, S99power-perms, completions). - -**libclc: "Producer LLVM22.1.8, Reader LLVM 21.1.8"** -- llvm-spirv is still -the 21 build. spirv-llvm-translator failed at 22.1.5 before the 1.14.273 -marker fix, so its record says 22.1.5 and the plan skipped it (same poisoning -as LLVM's). Fix on the machine: rebuild it explicitly, then libclc, mesa. -After an LLVM major bump, `packagemanager update llvm -R` rebuilds dependents. - -## 1.14.277 -- kernel.conf CONFIG_SET - -lfs-kernel had no way to say "this machine always wants option X": the config -came from the running kernel, a refresh answered new symbols, and anything -else meant menuconfig by hand every time. `CONFIG_SET="SYM=val ..."` (y, m, -n, number, string) is applied after the refresh with scripts/config, resolved -with olddefconfig, and READ BACK -- a value the kernel reverts (missing -dependency, unknown symbol) is named and stops before compiling. - -Shipped value: `LEGACY_TIOCSTI=n` (shadow 4.20 su detaches root's `su -` -shells while TIOCSTI is allowed). An edited kernel.conf is kept by -`make install` (.new beside it), so an existing machine adds the line by hand. - -Tested: the apply/read-back block on a fake tree (disable taken; an unknown -symbol reported, exit 1). Kernel blocks 155 and 401 pass. Not built. - -## 1.14.276 -- lfs-helper regroup-dir - - /etc/pam.d p_bash-completion:nimgnu_bash-completion drwxrwxr-x - -A directory shared through the wrong group. `lfs-helper regroup-dir <dir>`: - * installers = packages whose manifest or pkg.lst has a direct child of it, - package/config accounts owning an entry in it, and its owner; - * asks the new name (1 dir-named, 2 owner-named, typed, q) -- pam.d gives - nimgnu_pam-d; - * members of the old group keep it only if they own, or install into, - another directory of that group -- found on EVERY real filesystem - (a missed mount would have dropped memberships); - * prints the plan, applies only on y; updates collector-map and granted. - * Leaves the old group in place and says when it is on no directory. - -Tested in the sandbox: installers found, two members dropped, the owner and a -member of another directory kept, q cancels, new group at 90000. - -(The 1.14.275 open item on /etc/pam.d stays: it should be root's. This -command at least narrows who can write it.) - -## 1.14.275 -- su to application users: four faults, one symptom - - ~ > su - u_browser-mini - su: Authentication failure - su: Authentication token manipulation error - - 1. **/etc/pam.d/su had no pam_exec line.** Shadow's configure rewrote su - (1.14.263); `cfg app-users` was `done (stack cache)` and never re-ran. - **A cached cfg entry with expect= is now re-checked on every run** and - re-run when the check fails. shadow-pam and app-users got expect= - probes (su setuid root + pam.d files; su calls su_u_user, root owns it, - caller check present). Probe tested against good and bad files. - 2. **SECURITY: su_u_user compared the TARGET with a list containing it** - (`[[ "$USERS" =~ "$PAM_USER" ]]`, empty PAM_RUSER filled with PAM_USER) - -- always true: any account could su to every listed account without a - password once the pam line was there. Now `[[ -n $PAM_RUSER ]] || exit - 1` and a whole-word check on PAM_RUSER. Existing files are repaired in - place with a backup. Tested: main user passes, another u_ and an empty - caller are refused. - 3. **SECURITY: su_u_user was owned by p_librice** (attributed to librice's - build window) -- a package user could replace a script root runs on - every su. app-users forces root:root 755 on every run. - 4. **"token manipulation error"** = the account step demanding a new - password: /etc/shadow last-change 0. app-users locks an EMPTY password - field (Enter would have logged in) and sets last-change when it is 0. - Tested on a sandbox account. - -**Open:** /etc/pam.d itself is p_bash-completion:nimgnu_bash-completion 775 --- every member (bash-completion, gstreamer10, linux-pam, shadow) can replace -any PAM file. It should be root-owned; collector grants on /etc/pam.d need a -rule. - -## 1.14.274 -- the slow scan, and CMake's wrapped permission error - -**"scanning the tree ... takes a moment" took minutes** after a two-second -install. Every phase end walked the whole tree five times (snapshot, -snapshot_dirs twice, two `find -newer`), and every walk used -`-not -path "$SRCROOT/*"` -- which filters AFTER descending, so each one -went through /usr/src: every package's unpacked sources and build trees. -Now `-prune` (same output: compared old vs new on a synthetic tree, both -snapshot lists and both touched lists identical), snapshot_dirs once, and one -walk for touched files+dirs. Scan-related suite blocks pass; block 240's -string check accepts the pruned form. - -**nvtop stopped after one round** on a plain permission error. CMake wraps -it: `...metainfo.xml": Permission` / `denied.` -- no line matched, round 2 -found nothing and the loop ended. `unwritable_dirs_from_log` joins each -`file INSTALL` block; the repair gate also accepts "file INSTALL cannot". -Tested on nvtop's log text. - -The retry loop itself already runs while each round fixes something (cap -20, stops when the same directories repeat) -- it was the parser. - -## 1.14.273 -- a failed LLVM update was taken as installed - - CMake Error ... Could not find a configuration file for package "LLVM" - compatible with requested version "22.1.0" ... version: 21.1.8 - -The mesa plan was 9 entries (LLVM failed on its patch, 1.14.272); the -re-run's was 6, without LLVM. `User.version` reads `.pkgusr-installed` -(written after success) and, for a package built before that marker -existed, falls back to `install_last` -- which lfs-helper replaces BEFORE -the build. So LLVM's failed 22.1.8 attempt made it "current". - -The staging step now seeds `.pkgusr-installed` from the OLD install_last -before replacing it (only when the marker is absent). A failure then leaves -the old version recorded. Tested: the version extraction. Suite: grep. - -On the machine the record is already wrong -- one-time fix: - echo LLVM-21.1.8 > /usr/src/pkgusr/p_llvm/.pkgusr-installed - -### OPEN -- LLVM's compile is generated into install_pkg - -The page creates /etc/clang/*.cfg as root BEFORE cmake (on purpose: SSP for -compiler-rt), and the phase split starts install at the first root block, -so cmake + ninja + `ninja install` all land in install_pkg. It works, but -"re-run only the install step (no recompile)" is false for LLVM, and a -repair round in install recompiles. Needs a rule for a root block the page -says must precede the build. - -## 1.14.272 -- extra downloads were saved under the tarball's name - - HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable - The file is already fully retrieved; nothing to do. - patch: **** Can't open patch file ../llvm-22.1.8-upstream_fix-1.patch - -`_pp_unpack` called `_pp_fetch "$li"` for each additional_links entry with no -output name, and `_pp_fetch` defaults the name to `$pkg` (the openh264 -rename). So every patch was `wget -c -O <the tarball>`: 416 when the tarball -is bigger (patch silently missing), and appended onto it when smaller. Now -`_pp_fetch "$li" "$f"`, and a missing file after the fetch is an error. - -Every BLFS page with a patch or extra tarball was affected since that rename. -Tested with a stub wget. Suite: grep in "one list, two entry points". - -The run was mesa's dependency plan updating LLVM to the 13.1 book's 22.1.8 --- hours; there is no skip-dependencies switch. - -## 1.14.271 -- libglvnd taking Mesa's headers was called a bundled copy - - /usr/include/GLES/egl.h belongs to p_mesa -- handing it to p_libglvnd - ! libglvnd has now taken three files, one per round, from 'p_mesa'. - ! It is building a BUNDLED COPY of that package ... - -It was not. Mesa with glvnd=disabled owned GL/GLES/EGL/KHR headers and -libGL; libglvnd legitimately replaces them. meson stops at the first file -it may not chmod, so the log named one per round, and the 1.14.116 streak -rule (3 single handovers from one owner) fired. - -`meson_takeover_from_log` now, after a handover, reads -`meson-info/intro-installed.json` (found by walking up from the source path, -no further than the package user's home) and hands over EVERY planned -destination still owned by that same package -- one round. Not done for a -subprojects/ source, and only a subprojects/ source now counts toward the -bundled-copy streak, so libdisplay-info's case still stops. - -Tested in the sandbox with two users: 1 logged + 3 planned files moved, a -file outside the plan untouched; a subproject source still sets the streak. -Suite: grep checks; "the directory was never the problem" and "one file per -round" blocks still pass. - -**Open:** directories (GL/, EGL/, KHR/) owned by p_mesa still get one -collector-group question each. - -## 1.14.270 -- Mesa with glvnd; libglvnd in sway.stack - -The user asked for it. `[mesa] glvnd = enabled` in machine.conf, and a -`git libglvnd` entry directly before `book mesa` in sway.stack. - - * **libglvnd is NOT a BLFS page** (the book only links GLFS). An earlier - chat reply said it was -- wrong. URL is the canonical freedesktop - gitlab; tag v1.7.0 and option names CHECKED on NVIDIA's GitHub copy. - Default options. - * **One go, or GL breaks**: between libglvnd's install and Mesa's - rebuild, libGL.so.1 is glvnd's with no vendor. - packagemanager stack sway --only libglvnd,mesa --run - Existing binaries keep working (same sonames). - * Mesa's probe now also requires 50_mesa.json and libGLX_mesa.so.0, and - its `a; b` form (first test ignored) became one expression. - * **`--clean` never deletes a file another package owns.** Mesa's old - manifest lists libGL.so.1, which libglvnd installed before the Mesa - rebuild -- older than the install start, not rewritten: `_orphan_sweep` - would have deleted the GL library. It now skips files not owned by the - package user. Tested with a stub remover. - -Suite: glvnd-before-mesa order and the sweep guard, in "one list, two entry -points". Not built. - -## 1.14.269 -- mesa-demos: libdecor's demo wants opengl.pc - - subprojects/libdecor-0/demo/meson.build:8:9: ERROR: Dependency "opengl" not found - -Not a missing stack entry. opengl.pc is libglvnd's, and machine.conf has -`glvnd = disabled` for Mesa, so Mesa ships gl.pc/egl.pc/glesv2.pc and no -opengl.pc -- by design. Only libdecor's DEMO (a meson subproject pulled in -by mesa-demos' wayland demos) asks for it. diag.stack passes -`-Dlibdecor-0:demo=false`, and wipes `build/` before `meson setup` in both -meson entries, since a failed setup leaves it half-configured. - -**The failure box misleads here**: "packagemanager install opengl" -- there -is no such package, and installing libglvnd without rebuilding Mesa against -it would not help. Not changed; worth a pc-module note later. (1.14.270 -switched the machine to glvnd.) - -**machine.conf `[mesa] gbm = enabled`.** Mesa builds gbm by default on -Linux, and sway works, so it is there. Named so a Mesa that cannot build -it fails at configure rather than finishing and breaking wlroots. No -rebuild needed. - -`libdecor-0:demo` is from memory (gitlab unreachable) -- if meson says -"Unknown option", the name is in -`subprojects/libdecor-0/meson_options.txt` in the clone. - -## 1.14.268 -- diag.stack: GPU diagnostic tools - -New optional stack, none of it in the book: mesa-demos (glxinfo, eglinfo), -vulkan-tools (vulkaninfo), libva-utils (vainfo), nvtop. - - * **vulkan-tools and libva-utils pick their tag at build time** from the - installed version (`pkg-config --modversion vulkan` / `libva`). Both - must match what is installed, and a fixed pin breaks on the next book. - Tag patterns checked with git ls-remote. - * **nvtop's vendor is the only choice in the file**, in a marked block at - the top. Every backend defaults ON on x86, so all are named; ON is - AMD only (this machine). - * **mesa-demos is NOT CHECKED**: gitlab.freedesktop.org was unreachable. - Pinned mesa-demos-9.0.0; option names from memory -- first thing to - look at if it fails. - -Tested: parsed, the four generated scripts pass `bash -n`, libva-utils tag -selection on a real clone. The conventions and version suite blocks pass. -Not built. - -## 1.14.267 -- id ranges, and cfg power's three false failures - -**Ranges, one per kind of id.** - - package users + private groups 10000-20000 - collector groups (no account) 90000-98999 - application users (u_*) 30000-39999 - -Found on the machine: `nimgnu_python-modules`, `-xorg7-input-driver`, -`-perl-modules` were group AND login account at 10096/10147/10162, home -`/usr/src/pkgusr/p_nimgnu_...` -- `sysgroup create` ran the package-user -helper. It now calls `_create_collector_group`, group only. `u_*` were -at 10324-10349 among the package users; `user create` now uses -`APP_UID_MIN/MAX`. `nimgnu_power` was 997 (power.sh: system groupadd); -it now asks `packagemanager sysgroup create`. - -The machine's existing ids are repaired by a one-off script given in chat -(dry run by default): removes the collector accounts, renumbers collector -groups into 90000+, u_* into 30000+, re-owns files on every real and tmpfs -mount, and puts each file's mode back -- chown/chgrp clear setuid, which -is how 1.14.265 lost su. Tested in the sandbox on fake accounts: setuid -bit, group members and home ownership kept. - -**power.sh, three false `!!`:** - * poweroff/reboot are symlinks to halt; `stat` read the link (777). It - now resolves each path and handles every target once. - * it linked only into rcsysinit.d; LFS's sysinit runlevel is rcS.d. - * the two udev lines that never matched (KERNEL=="state", =="bootlog") - are now removed, not explained -- the user asked. No bootlog account - or group is needed. - -Suite: greps in "one list, two entry points"; the 1.14.266 files fail all -five. Tested: power.sh's udev cleanup on sample rules. - -## 1.14.266 -- cfg power could not find the collector prefix - - !! no collector_prefix in /etc/pkgusr/packagemanager.conf - -power.sh parsed packagemanager.conf with the lfs config's JSON pattern -(`"collector_prefix": "..."`); that file is `key=value`. It could never -match, and on a booted system with no lfs config it stopped the stack. Two -parsers, one fact, again. - -packagemanager now exports the RESOLVED `LFS_COLLECTOR_PREFIX` and -`LFS_MAIN_USER` to every cfg script: environment, lfs config, -packagemanager.conf, then the collector groups on the system -- never a -built-in default. power.sh reads the variable, and falls back to the -conf's real `collector_prefix=` line. - -Suite: grep checks in "one list, two entry points"; the old power.sh fails -them. Tested: power.sh's group step with and without the variable. - -**The stack cache hides the repair.** `stack services --run` printed -`cfg shadow-pam done (stack cache)` and did not run it, so the stripped -setid bits stayed stripped. `--only shadow-pam --run` runs it. Whether a -cfg entry should ever be cached is open. - -## 1.14.265 -- the earlier-stages claim stripped every setuid bit - - ~ > su - - su: Authentication service cannot retrieve authentication info - -On the booted machine su had no u+s, nor had the rest. The cause is in -the configure output of 1.14.263's run: - - # gave 'p_shadow' 16 path(s) from its earlier build stages - -`_claim_earlier_stages` runs before EVERY phase and chowns root-owned -manifest entries to the package user -- its own manifest included, not only --tmp/-pass stages. chown clears setuid, so su, passwd, chage ... went -4711 root -> 755 p_shadow. The lease then saw files the installer already -owned and had nothing to restore. The claim fired before the lease could. - -`_claim_candidates` now skips any file with a setuid or setgid bit; those -are the lease's (copy aside, restore on failure, setid.conf on success). -Suite block 111 case 6; checked to FAIL against 1.14.264. - -**On the machine:** the bits are already gone, and shadow-pam's step 3 -applies setid.conf -- that is the stack's own repair. It needs root, and -su is what broke: log in as root on a tty, or run it from the chroot. - -**`update --clean` does not touch this.** It is opt-in, removes only -old-manifest files the new install did not rewrite, and never under /etc, -/var, /boot, /home, /root, /srv. - -## 1.14.264 -- a package reclaims its own files - -The rule, from the user: **a package owns what it installs, whoever holds -the file now.** The lease already settled a rewritten file to its -installer, but on INSTALL it only leased the manifest's ROOT-owned entries. -A manifest file another package had taken was not leased, so a bash -redirect or `cp` over it failed with a wording no repair parses -- the same -shape as shadow's /etc/pam.d/su. - -`_lease_candidates` now takes every manifest entry not owned by the -installer. The guards stay in `lease_files`: package accounts only, root -only when the manifest claims it, no symlinked directory. On success a -rewritten file stays with the installer; an unchanged one goes back. - -Suite block 111, case 5; checked to FAIL against the 1.14.263 lfs-helper. -Not run on the machine. - -## 1.14.263 -- the lease: hand files over before a phase, not after it fails - - /usr/src/pkgusr/p_shadow/install_Shadow-4.20.2: line 153: /etc/pam.d/su: Permission denied - -/etc/pam.d/su was p_linux-pam's, 644. The repair loop recognised nothing -in a bash redirect's wording, granted /etc/pam.d (never the problem) and -stopped. Every tool words that refusal differently, and each needed its own -parser -- and the setid pass (root takes the file) and the 1.14.261 -take-back (a repair hands it back) were two halves of one cycle. - -**The lease**, in `run_phase_as` -- the one runner -- around every -install/configure/all/update phase, as root: - - * **before:** each file the phase will write that another account owns - is COPIED to `$STATE/leases/<pkg>/` with owner and mode, and chowned - to the installer. Copied, never moved: the phase starts with - `su - p_<pkg>`, and su, its pam file and unix_chkpwd must exist. - * **failure** (including exit 0 with a failure in the log -- it settles - AFTER that check): every copy back, owner and mode, read back. - * **success:** a setid.conf path gets its policy; an unchanged file goes - back to its owner; a rewritten one belongs to the installer. - * a ledger left by a killed run is restored before the next lease. - -**Which files:** the manifest's ROOT-owned entries (install), and what -`configure_pkg` writes (configure) -- `lfs-helper script-targets`: redirects, -tee, install/cp/mv/ln destinations, sed -i, for-loops expanded, heredoc -bodies and comments skipped, /tmp /dev /proc /sys /run ignored. - -**Security:** a root-owned file is leased only when this package's manifest -claims it (the script is the package user's to edit; one line could name -/etc/shadow). Non-package accounts' files never. No path through a -symlinked directory. The first install over another package's file cannot -be predicted, so the repair loop remains the fallback -- minus its root -take-back, which is gone. - -**One setid table.** `_setid_known` had its own list saying 4755 while -setid.conf said 4711, so `cfg shadow-pam` and `setid --run` corrected each -other. It reads setid.conf now (the dbus helper line moved there), and a -requested bit on a path setid.conf names is left to setid.conf. - -**The stale script on the machine.** `packagemanager stack services` run -from ~/lfs read ~/lfs/services.stack, and `cfg` entries run -`<that dir>/<name>.sh` -- an old shadow-pam.sh. packagemanager now warns -when a stack or cfg script found there differs from /etc/pkgusr/stacks. - -shadow-pam asks `lfs-helper script-targets` for its pam.d set (one parser). -Stack stamps: all at 1.14.263 -- 1.14.262 bumped the tools and not the -stacks, which the suite reports. - -**Tested:** `run-script` end to end with real su, two throwaway package -accounts and sticky directories (failed configure restored; install over a -root 4711 file settled to 4711 root:<pkg>; successful configure owned by -the installer; interrupted lease restored). Suite block 111 does the same -without su; it FAILED until the fixture left /tmp (script-targets ignores -/tmp) and `_setid_conf` survived `set -u`; checked to FAIL against the -1.14.262 lfs-helper. Whole suite, all 429 blocks, run against both trees: -identical FAIL lines except the stack-version ones 1.14.263 fixes -- the -1.14.262 zip shipped stacks still stamped 1.14.261. - -**Not tested:** a real package build on the machine. - -### Suite: pre-existing, not from this release - - * Block "one quote character, three symptoms" deletes the suite's - counter directory; every PASS/FAIL total after it reads empty. Same - on the 1.14.262 tree. - * `--only` of a block that uses `_hlv` without block 114 ("the version - is what FINISHED") died on `_hlv: unbound variable`. Fixed for block - 121 ("one list, two entry points"), which now sets it itself; other - blocks may share the shape (grep `_hlv` before relying on --only). - -## 1.14.262 -- shadow's pam.d files, and the cache that hid three fixes - - passwd: pam_start() failed, error 26 - login: PAM Failure, aborting - -**The cache.** The first try at this fix did not fire either, which is how -the real bug turned up: blfs's index cache was keyed on the book's size, -mtime and a hand-bumped CACHE_VERSION -- not on the code. An edited -extractor rendered from the index the OLD code built. That is the whole -story of 1.14.212 and both poppler patches "not firing". `_book_stamp` now -includes `_build_id()`, so any edit to blfs re-indexes once (~20s). - -**The walk.** Shadow's commands are in `div.configuration > div.sect3 > -div.sect4 > pre.root`, and `extract_commands` dropped every div that was -not a note. It now descends into `div.sect4`/`div.sect5` with the same -rules (guards included) and keeps their titles as `##` comments. The top -level is unchanged byte for byte. - -**Proved against a rendered book, not a sample.** The BLFS source was -cloned and rendered (`make nochunks REV=systemd`). Before and after, for -all 146 pages with a configuration section: 141 scripts byte-identical, no -build or install phase changed anywhere, and exactly five configure phases -gained blocks -- the five pages the XML says have commands in sub-sections: -Shadow, MIT-Kerberos, NFS-Utils, SANE, Samba. - - * Shadow: login.defs, login, passwd, su, chpasswd+newusers, chage, the - `for PROGRAM` loop, access and limits renames. 16 pam.d files. - * NFS-Utils, SANE: the new blocks are all guarded (SKIPPED). - * Samba: `systemctl enable` lines (the shim's job); smb.conf examples - have no command markup and stay out. - * **MIT-Kerberos: see OPEN below.** - -**`cfg shadow-pam` checks the SET.** It read two files and said "in -place" while fourteen were missing. Now `_book_pam_set` reads each -package's `configure_pkg` (every `/etc/pam.d/<n>` on a live line, plus the -words of a `for VAR in ...` loop writing `/etc/pam.d/${VAR}`) -- linux-pam 5, -shadow 16. The script comes from `packagemanager script fetch`; if it names -no pam file (the stale pre-1.14.262 one), it is fetched again with -`--regenerate`, so a correct script is never regenerated just for a check. - -**The su trap.** Shadow's su block has no pam_exec line. app-users greps -for `su_u_user` and re-adds it, and runs after shadow-pam in services.stack; -the suite now asserts that order. shadow-pam says so when it happens. - -Suite: block 110 renders a shadow-shaped page through the generator (with a -fresh BLFS_STORE), and runs `_book_pam_set` on the output. Checked that it -FAILS five ways against 1.14.261's blfs. - -### OPEN -- MIT-Kerberos's configure phase is now a transcript - -It gained a KDC walkthrough: `kdb5_util create -r __EXAMPLE_ORG__ -s`, -`kadmin.local` followed by lines like `kadmin.local: addprinc ...` (typed -INTO the program, not shell), `kinit`. stdin is closed, so nothing hangs; -the phase fails loudly instead of silently doing nothing, as before. Not in -any stack. The fix is a "prose is not code" rule: a line `<prog>: ...` -after a bare `<prog>` line is a transcript -- comment the block. - -### Small: shadow's login.defs block is not idempotent - -A second configure run copies the ALREADY-edited login.defs over -`login.defs.orig`. Harmless for login; the pristine copy is lost. - -## GAP IN THIS DOCUMENT -- 1.14.180 to 1.14.238 - -The per-release notes for **1.14.180 through 1.14.238** are NOT in this -file. They were written each release and are not here now; the section -below jumps from 1.14.179 to 1.14.178. I do not know which edit dropped -them, and I am not going to reconstruct fifty-eight sessions from memory -- -invented notes would be worse than an admitted gap. - -Where the real record is: - - * every release's zip contains the HANDOFF.md of that release. The - newest zip that still has a given section is the place to read it. - * the CODE carries its own reasons: every fix in that range went in with - a comment naming the failure it came from, and - `test_lfs_crosschain.sh` carries an assertion per fix with the release - number in its comment. `grep -n '1\.14\.2' test_lfs_crosschain.sh` - lists them in order and is a usable index of what changed and why. - -What that range covered, from the assertions, in one line each: the -swallowed-failure dialects; the stale-destination and cmake-destination -repairs; job counts reaching make/cmake/cargo/ninja; the `[all]` section; -media.stack and `include`; avoid.stack and every stack including it; -app-users.conf with create-or-setup and the PAM script; `env --system`; -librice; the version marker; and the per-package `book =` override. - -## This session (1.14.179) -- lfsconf is now lfs-config, and it installs - -It was `lfsconf`, living inside the repo it managed: the repo carried a copy -of the tool, the tool had no version anyone could compare with the others, -and `make install` did not know it existed. It is `lfs-config` now, on the -Makefile's TOOLS line -- so install, uninstall, the stamp file and the chroot -refresh all pick it up with no second edit. - -**The point of it is the other machine**, and that half was missing: - -- **The repo records whose config it is** (`repo-user`). A directory's owner - answers that only on the machine that made it; cloned elsewhere it is - whoever ran git, while every `~` path and every owner field still means the - original user. -- **`lfs-config install [--as USER]`** maps the recorded user to this one - (`n76310:n76310` -> `chris:chris`), expands `~` to the target's home, - refuses a user that does not exist here, and **keeps what it replaces** as - `<path>.orig-<timestamp>`. -- It does NOT rewrite file contents. It lists the tracked files that still - mention the old name, because a username inside a config file may be a - path, a group, an SSID comment or a mistake, and only the reader can tell - which. The suite fails if a `sed -i` ever appears there. -- **`lfs-config init`** makes the repo. Creating one used to be a side - effect of every command -- `--help` in a directory with no repo wrote a - manifest into ~/LFS/lfs-config and printed two errors before the help. - -Tested end to end: collected as n76310, installed as chris, owner mapped, -original kept, the one file mentioning n76310 named. - -## Previous session (1.14.178) -- the gate decides, the loop acts - - # /usr/share/doc/libtiff-4.7.1/tiff is p_libtiff's own output ... - # removing it so this install can replace it - ! libtiff FAILED ... gate: recognised, rounds run: 0 - -The repair did exactly the right thing and the package failed anyway. I -called it from the GATE -- the code that decides whether a retry is worth -attempting -- so the directory was removed before the retry loop started, -and the loop counts only what IT fixed. Nothing re-ran the phase, and the -obstacle was already gone. - -The gate now only greps for the message; `_clear_stale_dest` runs inside the -loop beside the other repairs, and setting `_fixed_something=1` is what -causes the retry. A second round finds the path gone and returns 1, so the -loop breaks instead of spinning. - -Two things this cost, both now asserted: - -- a repair called from the gate is a bug in itself: block 375 fails if - `_clear_stale_dest` appears with `_perm_ish=1`. -- **`_slice_fn` stops at the first line starting with `}`**, and cmd_build is - 1700 lines with several of those -- so two assertions were matching against - an empty slice and passing. They grep the file now. A test that cannot fail - is worse than no test. - -Resume: `packagemanager late-deps --rebuild --run` -- 45 left, and libtiff -should clear itself and carry on this time. - -## Previous session (1.14.177) -- the repair failed the build it was repairing - - /usr/bin/lfs-helper: line 5914: user: unbound variable - update failed for git (exit 1) - -git's install had just finished -- every hardlink, every alias, 345 files -tracked. Then yesterday's repair hook, `_clear_stale_dest "$log" "$user"`, -read a variable that does not exist: `cmd_build` calls the package user -`owner`, and `set -u` turned a successful install into a failed one with a -line number and no explanation. - -Two fixes, and the second is the rule: - -- `${owner:-}`, which is what that function calls it. -- `_clear_stale_dest` returns 1 on an empty user instead of running. **A - repair must not be able to fail the thing it is repairing**: it runs after - something already went wrong, so anything it does must be safe when its - inputs are not what it expected. - -Block 375 asserts both, and that `cmd_build` never reads `$user` at all -- -checked by putting the bug back and watching two assertions fire. - -Resume: `packagemanager late-deps --rebuild --run`. git is installed -(the install completed); it will be re-checked and skipped or redone -quickly. - -## Previous session (1.14.176) -- a book command that works once - -The rebuild pass stopped on libtiff, 38 packages from the end: - - mv: cannot overwrite '/usr/share/doc/libtiff-4.7.1/tiff': Directory not empty - -Nothing was wrong with the system. libtiff's page moves its html docs into a -versioned directory; the first install creates that directory, and the -second cannot move onto it. The destination was **the package's own output -from last time**, and a reinstall means replacing it -- but the repair gate -did not recognise the message ("gate: did not recognise this failure, rounds -run: 0"), so a 38-package pass stopped on a directory the package itself had -put there. - -`_clear_stale_dest` reads `cannot overwrite '<path>'` out of the log and -removes that exact path -- **only when the package being installed owns -it**. A directory owned by anyone else is a conflict, not stale output, and -deleting it would be the worst repair available: that case warns and leaves -it. Proved on real text in block 375, all three outcomes (own output -cleared, other owner refused, unrelated failure untouched). - -Resume with: - - packagemanager late-deps --rebuild --run - -libtiff will be retried from its install phase; what already succeeded is -recorded and does not repeat. - -## Previous session (1.14.175) -- the help was two lists, both stale - -No, it was not up to date. - -- The grouped `packagemanager --help` -- the part a reader actually uses -- - had no `stack`, `probe`, `late-deps`, `reclaim`, `bootstrap` or `env`. - `stack` is the primary command of this tool. Two new groups now: "a whole - environment at once" and "features, not files". -- The module docstring kept a **Roadmap** marking check, search, - dependencies, install, remove and reload-pkg-list as `[TODO]` -- all six - implemented releases ago -- and none of the twenty commands added since - appeared at all. Replaced by a pointer to `--help` and the two design - questions that really are open. -- `lfs-helper --help` was missing `undone`, `sort-users` and `check-login`, - and said nothing about the chroot build's own steps (init-accounts, - establish-ownership, seal-install-dirs, install-profile, - repair-adoption). They are listed as what they are: dispatched by - build-all, and `lfs-helper list` names them. - -Block 375 now asks ARGPARSE for the command list -- it prints one when given -a bad command -- and requires every name to appear in the grouped help. -A list of commands maintained anywhere but beside the commands goes stale -silently; this one now cannot. - -## Previous session (1.14.174) -- audiornnoise needs no second git entry - -It is in gst-plugins-rs already: the element lives in **gst-plugin-audiofx**, -not in a plugin of its own, so it is one more `-p` on the entry added in -1.14.172 rather than a new checkout. Its noise model is the nnnoiseless -crate, vendored by cargo, so there is no system library to install first. - -The probe grew with it -- `webrtcsink && audiornnoise` -- and `check webrtc` -lists audiornnoise among the optional rust elements, with a line saying what -it is for (noise suppression for a microphone in a call). - -Three assertions in block 376 that came out of writing this: - -- an element named in a probe must be asked for by some `-p`, or the check - tests something nothing builds; -- `audiornnoise` must NOT appear as `-p gst-plugin-audiornnoise`, which is - the mistake that looks right and fails before anything builds; -- the build and install lines must ask for the SAME plugin set -- a plugin - compiled and not installed is the most confusing outcome available - (checked by removing audiofx from the install line and watching it fail). - -## Previous session (1.14.173) -- a plugin is a shared object - -The first real gst-plugins-rs run compiled everything (5m40s, four plugins) -and died installing a STATIC library: - - failed to copy `.../libgstrsrtp.a` to `/usr/lib/gstreamer-1.0/libgstrsrtp.a` - Permission denied (os error 13) - -Two separate faults, and neither was the permission: - -- **cargo-c installs a .a as well as a .so by default.** Nothing loads a - static GStreamer plugin. `--library-type cdylib` on both the build and the - install line, and the file that failed is never produced. -- **The repair ran ZERO rounds** ("gate: recognised, rounds run: 0") because - cargo names the destination in BACKTICKS, on the line above the reason -- - no quotes, and no "Permission denied" on the same line, so the path - extractor found nothing while the box still said "grant access and retry". - It now recognises `failed to copy ... to \`<path>\`` and offers the - destination's directory; asserted against that exact text. - -/usr/lib/gstreamer-1.0 is `p_gstreamer10:nimgnu_gst`, so the .so still needs -the group -- which the repair can now grant by itself, in one round. - - packagemanager stack sway --only gst-plugins-rs --run - -The compile is cached, so the retry is the install step only. - -## Previous session (1.14.172) -- gst-plugins-rs, and what it is honest about - -Added to sway.stack at entry 124: after gst10-plugins-bad and gst10-libav, -which it builds against, and before webkitgtk. cargo-c is already installed -(librsvg pulled it in), so nothing new is needed to build it. - -It is NOT a BLFS page, so three things that a book page would settle are -ours, and each is stated where the entry is -- the suite checks that the -statements are there: - -- **`ref=main` is not a pin.** Upstream tags 0.14.x for GStreamer 1.28, but - that pairing is unverified from here, so the entry follows main rather - than claiming a tag that may not match. Pin it once a build succeeds: the - ref that worked is the one to write down. -- **It downloads crates at build time.** There is no offline mode, so this - entry cannot be part of a build that has no network. -- **The plugin list is a choice.** Four are named -- webrtc (webrtcsink and - webrtcsrc), rtp, dav1d, gtk4 (gtk4paintablesink). All of gst-plugins-rs is - enormous and mostly irrelevant here. Add with another `-p`. - -`have=/usr/lib/gstreamer-1.0/libgstrswebrtc.so`, and -`expect="gst-inspect-1.0 webrtcsink"` -- an element again, not a file. - -`check webrtc` reports webrtcsink, webrtcsrc and gtk4paintablesink as -OPTIONAL (`->`, not `!!`): the plugin set is not a BLFS page and not -everyone builds it, so its absence must not fail a stack. Verified by -deleting the network warning from the stack and watching the suite fail. - -## Previous session (1.14.171) -- the advice was a loop - -`stack sway --only sdl3 --run` on an INSTALLED package runs `install`, which -answers "Nothing to do -- everything is installed and up to date". The expect -probe then failed and the tool recommended... the same command. Four bugs in -one screen: - -- **The suggested command could not work.** A stack entry does not rebuild an - installed package. The expect= failure now prints - `packagemanager update <pkg> --reinstall --regenerate --run --yes` - (`_rebuild_cmd`, the one place that knows this) and says plainly that - re-running the entry will not help. `probe` had the same mistake for the - installed case. -- **And it named a stack that does not exist**: `stack sway.stack --only ...` - -- `os.path.basename(path)` without dropping the suffix, while the probe - path next to it stripped it correctly. -- **"NOT recorded as done, so the next run rebuilds it" was false** for an - installed package. Removed; the truth is in its place. -- **The newer-stack notice argued against itself**: `(149 entries, this file - has 149)` for two files differing in every entry's options. It reports - added/removed lines now, and says when the entry count is unchanged so the - difference is inside the entries. Also `--take-new` said "149 entries now" - two lines above a header reading "(62 entries)" -- one counts file lines, - the other the buildable plan. One measure, one word. - -For sdl3 on the machine, after `make install` and taking the stack again -(the glob fix is in the shipped file): - - packagemanager probe sdl3 - packagemanager update sdl3 --reinstall --regenerate --run --yes - -## Previous session (1.14.170) -- four probes pass; both failures were misreported - -With the stacks taken, `probe` runs: **ffmpeg has ALSA, opusenc, v4l2src, -vp8enc, webrtcbin, dtlssrtpenc and srtpenc are all there** -- the WebRTC -chain and the earlier ALSA work came out right. The two failures were both -described wrongly: - -- **gst10-libav is not installed at all** (it was missing from the stack - until 1.14.167), and the report said "the feature is NOT in the installed - package" -- which sends the reader looking for a build option. A probe - failure now checks for `pkg.lst` and says NOT INSTALLED, with "build it" - rather than "rebuild it". -- **The sdl3 probe named one path.** SDL3 installs `libSDL3.so.0.<n>` and - the bare `libSDL3.so` symlink is not always there, so the probe could fail - for a reason that has nothing to do with ALSA. It globs now. A failing - probe also prints its own first line of output, because a probe can be - wrong about a path as easily as a package can be wrong about a feature. - -On the machine: - - packagemanager stack sway --only gst10-libav --run # not built yet - packagemanager probe sdl3 # after make install - # if it still fails, sdl3 really was built before alsa-lib: - packagemanager stack sway --only sdl3 --run - -Note the take-new also brought `cfg power` and `check net-wifi` into the -stacks as unbuilt entries -- both are read-only or root-only steps and can -be run with `--only`. - -## Previous session (1.14.169) -- "no probes" was the wrong half of the answer - -`packagemanager probe` on the machine: *no expect= probes in the stacks*. -True, and useless -- the probes are in the SHIPPED stack files, and an -edited stack is never overwritten, so they arrived as -`/etc/pkgusr/stacks/sway.stack.new`. The tool carried probes no installed -stack had, and the message read as "nothing to check". - -It now looks for `*.stack.new` carrying `expect=` and says so, with the -command to take each one. Failing that, it says probes live in stack files -as `expect="..."` and that an older stack has none -- rather than leaving the -reader to conclude everything is fine. - -**On the machine**, before probing anything: - - packagemanager stack sway --take-new # then network, services - packagemanager probe - -The whole `.new` mechanism remains on the deferred list -- three functions -move stack files and it still surprises people. This is the second time a -feature was invisible because the stack carrying it was sitting beside the -one in use. - -## Previous session (1.14.168) -- "+0 day(s)" was the whole bug, filed as fine - -`late-deps ffmpeg --why` came back clean on every encoder (x264, x265, -libvpx, aom, svt-av1, opus, lame, fdk-aac, dav1d -- all days earlier) and -showed one line that should have stopped the eye: - - recommended alsa-lib-1.2.15.3 -> alsa-lib, built +0 day(s) relative to it - -A day of slack was chosen because pkg.lst is rewritten by reload-pkg-list -and --clean, so packages from one run differ by minutes. But alsa-lib a few -HOURS after ffmpeg is the original bug exactly, and the slack filed it as -fine. An hour covers the rewrite noise; the report prints hours below a day, -and marks a gap under six as "same session: only a probe can settle it", -because two packages built in one session cannot be ordered by mtimes at -all. - -`packagemanager probe` is that answer: it runs every `expect=` in the -installed stacks against what is installed, and a failure names the rebuild. -Timestamps are evidence, a probe reads the artifact. - - packagemanager probe # all of them - packagemanager probe ffmpeg # one - -## Previous session (1.14.167) -- gst10-libav was in no stack - -Right: nothing built it. It carries the ffmpeg-backed decoders (avdec_h264, -avdec_aac and the rest), so without it H.264 -- most video on the web -- is -decoded by nothing, and the symptom is a black frame rather than an error. - -`book gst10-libav` now sits after ffmpeg (entry 1b) and gst10-plugins-base, -which it needs, named as an entry rather than left to a dependency plan so -that stays true. Resulting order: ffmpeg 90, base 116, bad 122, **libav -123**, webkitgtk 125, mpv 126. - -Its probe is an element, for the same reason as the others: the package -installs libgstlibav.so either way, and whether `avdec_h264` registers -depends on what ffmpeg was built with -- the thing that was wrong about -ffmpeg and ALSA. `check webrtc` reports it too, so a missing decoder shows -up beside the camera and the microphone. - -Block 376 asserts both the order and the probe, and names it as MISSING when -the entry is absent (checked by deleting the line). - -On the machine: `packagemanager stack sway --only gst10-libav --run` after -the reclaim and the rebuild pass. - -## Previous session (1.14.166) -- repair the chown, not one package at a time - -The rebuild pass stopped on glib2 with the SAME cause as gst-plugins-bad: -`/usr/include/glib-2.0` is `p_linux-headers`, from lfs-kernel's old -`chown -R "$_hacct" /usr/include`. The whole include tree is that package's, -so every package that ships a header fails to reinstall. The tool fix -(1.14.158) stops it happening again; the machine needs the ownership put -back. - -`packagemanager reclaim` was the right repair but took a named list, and -finding the victims means finding them first. It takes `--all` now: every -installed package, checked against its own pkg.lst, and a package with no -manifest is skipped quietly rather than reported as an error ninety times. - -**The repair, in order:** - - packagemanager reclaim --all --run # ownership back, from manifests - lfs-helper verify --fix # groups, modes, install dirs - packagemanager late-deps --rebuild --run # resume; glib2 first again - -Tested against the exact shape: a directory and a file under -/usr/include/glib-2.0 owned by p_linux-headers but recorded in p_glib2's -manifest -- both returned, directory included. - -Also: `_slice_def` (slicing a python def for the suite's assertions) was -defined inside one block and used by another, where it returned nothing and -made two assertions pass vacuously. It is in the preamble now, beside -`_slice_fn`. - -## Previous session (1.14.165) -- the optional edge is the whole point - -The clusters came out right (freetype2+harfbuzz before fontconfig and cairo, -libdrm+libxcb before mesa, base before bad), and one thing was still -backwards: `mpg123` was rebuilt BEFORE `pulseaudio`, and mpg123's missing -feature IS pulseaudio. The order graph was required+recommended, which are -the edges that break a BUILD -- while the edges this whole pass exists for -are the optional ones. - -The rebuild order now uses required + recommended + optional. That makes the -graph much more cyclic, which is honest rather than convenient, and the SCC -pass already handles it; inside a component the strong edges still decide, -because those are the ones that fail a build rather than a feature. - -`--why` earned its keep immediately: libxcb RECOMMENDS mesa and mesa -REQUIRES xorg7-lib and libdrm, so libxcb/mesa are one genuine cycle -- the -plan puts libdrm and libxcb before mesa, which is the best an ordering can -do there. It also showed the unresolved names are what they should be: -Doxygen, libunwind, lm-sensors, Valgrind -- packages that are not installed. - -## Previous session (1.14.164) -- order the clusters, not the nodes - -Still wrong: freetype2 and harfbuzz came out LAST, after fontconfig and -cairo, which both require freetype2. The cycle-break picked one node and then -went back to the ready scan, which strands the rest of the cluster at the -end -- picking fontconfig out of the cluster made everything that depended -on it ready, and freetype2 was left over. - -A cycle has to be ordered as a UNIT. Tarjan's strongly connected components, -computed iteratively (this graph has 400 nodes, so recursion is not an -option), condensed into a DAG that sorts by construction; inside a component -the rule is unchanged, least dependent and most depended on. The -freetype2/harfbuzz component now comes out before fontconfig and cairo, -asserted in block 375 on that exact shape. - -`late-deps --why <pkg>` prints the edges actually used for a package -- what -each dependency resolved to, how many days apart they were built, and which -names resolved to nothing. When a position still looks wrong, that is the -output to read: an edge that does not resolve is invisible to the sort. - -## Previous session (1.14.163) -- a cycle broken alphabetically is not an order - -Still wrong, and visibly: `gst10-plugins-bad` before `gst10-plugins-base`, -`cairo` and `fontconfig` before `freetype2`. Those are CLUSTERS with no entry -point -- base needs mesa, mesa needs libxcb, bad needs base -- so the -topological sort ran out of ready nodes and the fallback was -`sorted(left)[0]`. Alphabetical order produced exactly the mistake the plan -exists to prevent. - -The pick is now "least dependent, most depended on": fewest unmet -dependencies inside the cluster, then the most packages waiting for it. base -beats bad (bad needs base, nothing needs bad); libxcb beats mesa. And the -cycle is NAMED in the output, with the note that a cycle can need two passes --- breaking one silently makes the second pass look like a failure. - -Block 375 asserts it on the failing shape itself: three packages in a real -cycle, and base must come out before bad. - -Also: "103 dependency names could not be matched" read as 103 problems. It -now says what it means -- a name that matches no INSTALLED package is -usually one that is not installed, and a package that is not there cannot -have been built after anything. A name you know IS installed appearing there -is the interesting case, and worth reporting. - -## Previous session (1.14.162) -- the order was wrong, and the 119 explain why - -"Are we sure about the ordering?" No, and the tool had already said why in -its own last line: *119 dependency names could not be matched to a package*. -An edge that does not resolve is an edge that does not exist, so the -topological sort ran on what was left -- and put `gst10-plugins-bad` before -`gst10-plugins-base`, which it REQUIRES, and `fontconfig` before -`freetype2`. - -The mapping was being guessed by normalising strings: a dep is written as -the page TITLE (`gst-plugins-base-1.28.1`, `FreeType-2.14.1`), a package -directory is the page ID (`gst10-plugins-base`, `freetype2`), and no amount -of lowercasing bridges that. Every script already records `name_version` -- -the tarball base, which IS the title. `_dep_index` builds the mapping from -that, and both cases now resolve (asserted in block 375 against a real -resolve, not a grep). - -Two more, from the same run: - -- **`--reinstall` is not a question about versions.** With no recorded - version, `update curl --reinstall` answered "nothing to update, 1 it - cannot judge" and did nothing -- for 28 packages whose records predate the - tool. `--reinstall` now plans the rebuild regardless of what is recorded. -- **The rebuild order uses required + recommended.** mesa RECOMMENDS libxcb, - so ordering by `required` alone rebuilds mesa against the old libxcb. - Cycles are expected on that graph and are broken by picking one. - -Re-run `late-deps` after installing this: the order will differ, the -unmatched count should drop a long way, and what is left in it is worth -reading. - -## Previous session (1.14.161) -- there is no host any more - -The machine boots on its own now, and packagemanager still printed, before -every single command: - - These tools differ from the copies on the host: blfs, lfs, ... - On the HOST, run: lfs build-system sync-tools --run - -`/usr/share/lfs/tool-stamps.json` is written by `lfs build-system -install-tools`, which runs on a host against a chroot. After the system -boots, `make install` on the machine itself replaces the tools and nobody -updates the stamp -- so all seven "differ", and the advice names a host that -no longer exists. - -- `make install` writes the stamp now: it records what is installed, so - whoever installs it owns it. -- The check returns early unless it is actually in a chroot - (`/proc/1/root` is not this root). No outside, nothing to be behind. - -On the machine: `make install` from the repo once, and the warning goes for -good. `sync-tools` remains correct for the case it was written for -- editing -the tools on a host and refreshing a chroot. - -## Previous session (1.14.160) -- the rebuild pass rebuilt nothing - -Two bugs in 1.14.159, both visible in one run on the machine: - -- **`install` does not rebuild.** The plan emitted - `packagemanager install <pkg> --run --regenerate`, which answered - "Nothing to do -- everything is installed and up to date" 28 times in a - row -- correctly, and in a way that read like a successful pass. The verb - that rebuilds at the same version is - `packagemanager update <pkg> --reinstall --regenerate --run --yes`, and - that is what `_rebuild_cmd` emits now, in the plan and in the per-package - hint. -- **A recommended edge is still a dependency.** The cycle test walked only - `required=`, so the report advised `put mesa before libxcb` -- and mesa - needs libxcb; BLFS lists that edge as recommended. Same for cairo -> glib2, - which is why glib2 showed up as orderable. Advice that cannot be taken is - worse than none, so the "does an order exist" walk uses - required + recommended. - -Also widened the build-tool list (mako, setuptools, wheel, hatchling, -pytest, gi-docgen, gperf, unifdef) -- those were showing as missing features. - -## Previous session (1.14.159) -- three answers, not one list - -`late-deps` on the real machine returned 40 packages in one list, and they -are not one problem: - -- **An order exists** (ffmpeg/alsa-lib, gst-plugins-bad/libnice, mpg123, - libheif, pipewire, sdl3, v4l-utils): the dependency does not need the - package, so it can be built first. Each line now says - `in the stack: put alsa-lib, sdl2 before ffmpeg` -- a stack edit, fixed - for the next machine. -- **No order exists** (glib2/cairo+dbus, libxcb/mesa, xwayland/mesa, - xorg7-lib/dbus): the dependency REQUIRES the package. cairo cannot precede - glib2. The only fix is a second pass, and saying "move it earlier" would - be advice that cannot be taken. Detected by walking each package's own - `required=` list. -- **Documentation and build tools** (graphviz against six packages, - docutils, ruby, git, fcron, cython): rebuilding gains documentation, not - features. Summarised in one line; `--include-docs` to see them. - -`late-deps --rebuild` prints the plan in DEPENDENCY order -- rebuilding a -library after the thing that links it means doing it twice -- and -`--rebuild --run` carries it out, stopping at the first failure with what is -left. A finished stack run now ends by naming the question, because a second -pass is part of the job rather than an afterthought. - -For the machine: `alsa-lib` before `ffmpeg`/`sdl3`/`v4l-utils`/`pipewire` is -already the stack's order since 1.14.156, so those entries only need the -rebuild. The circular ones stay circular -- they need the second pass on -every fresh build, which is what the reminder is for. - -## Previous session (1.14.158) -- one line owned every header on the system - -gst-plugins-bad could not install `mse-enumtypes.h` into -`/usr/include/gstreamer-1.0/gst/mse`, because that directory belonged to -**p_linux-headers**. The repair handed it one file per round; after three -rounds the bundled-copy heuristic concluded meson was vendoring -linux-headers. Every step of that reasoned correctly from a wrong fact, and -the wrong fact was one line in `lfs-kernel`: - - chown -R "$_hacct:$_hacct" /usr/include - -A package owns what it INSTALLED, not the directory it installed into. -`make headers` output is now listed with `find -printf '%P\n'` before the -copy and only those paths are chowned; `/usr/include` itself stays root's, -because every package that ships a header shares it. - -**Repair on the machine** (the ownership is already wrong there): - - packagemanager reclaim gstreamer10 --run - packagemanager reclaim gst10-plugins-base --run - packagemanager reclaim gst10-plugins-bad --run - packagemanager install gst10-plugins-bad --run --regenerate - -`lfs-helper fix-ownership --run` is the wider net if other packages' headers -were taken too -- everything under /usr/include that is not a kernel header -is suspect on that machine. - -### packagemanager late-deps - -The audit the last two sessions kept needing. Every generated script records -the page's own required/recommended/optional lists; every pkg.lst is written -when the install finished. So: which packages were built before something -they can use? - - packagemanager late-deps # all of them - packagemanager late-deps ffmpeg # one - packagemanager late-deps --kind recommended,optional,required - -It reports the gap in days and the rebuild command, ignores gaps under a day -(pkg.lst is rewritten by reload-pkg-list, so packages from one run differ by -minutes), and counts the dependency names it could NOT match to a package -- -a silent miss would under-report exactly what is being looked for. Tested -against a synthetic tree reproducing both the ALSA and the WebRTC cases. - -## Previous session (1.14.157) -- a library is not a plugin - -The browser: "GstWebRTC plugin not found." gst-plugins-bad *is* installed, -`verify` is clean, and `libgstwebrtc-1.0.so` is in its file list -- because -that library is always built. The **webrtcbin element**, which is what a -browser loads, is compiled only if libnice was there at configure time. Their -`p_gst10-plugins-bad/install_last` says `generated by blfs 1.14.43`: built -long before the WebRTC block was added to the stack, so libnice did not -exist yet. Same shape as the ALSA bug, one level further in -- the artifact -that proves the feature is an entry in the element registry, not a file. - -- `opus` and `libvpx` are stack entries now, before gst10-plugins-base and - -good: a call negotiates Opus and VP8, and both plugins are autodetected. -- `expect=` probes on all three gst entries, by element: - `gst-inspect-1.0 webrtcbin && dtlssrtpenc && srtpenc` for -bad, - `opusenc` for -base, `v4l2src && vp8enc` for -good. -- **`check webrtc`** (`stacks/webrtc.sh`): the whole chain in one report -- - eight elements with the library each needs and the command that rebuilds - it, the .pc files, /dev/video* and the video group, what - `p_webkitgtk/install_last` was built with, and a warning when a package's - script is OLDER than `nice.pc` (built before its dependency existed -- - exactly this bug, detectable from mtimes). -- machine.conf gained a commented `[gst10-plugins-bad]` block: setting - `webrtc = enabled` turns silent autodetect into a build failure. Commented - because the option NAMES are machine-checkable, not memorable -- it says - how to confirm them against the source on disk. - -Fix on the machine: `packagemanager stack sway --only gst10-plugins-bad --run` -(libnice, libsrtp and usrsctp are installed now, so it picks them up), then -`--only webrtc --run` to confirm. webkitgtk does not need rebuilding for the -element -- the plugin is loaded at run time. - -## Previous session (1.14.156) -- ffmpeg and SDL3 without ALSA - -Both autodetect ALSA, and both were built before alsa-lib existed. `book -alsa-lib` sat at entry 7, while ffmpeg is built inside **mesa's** plan -(entry 2) and SDL3 inside gstreamer's. They configured cleanly, installed -cleanly, passed `have=`, and had no sound. Nothing failed, so nothing said -so. - -The stack already stated the rule, for harfbuzz: an optional dependency of -something built inside a plan can only be placed by ENTRY ORDER, because a -plan builds its own tree in the book's order, not the stack's. It was simply -not applied to audio. - -- `book alsa-lib` is entry 1b now, before mesa. `ffmpeg` and `sdl3` are - named as entries there too, so they are built in this order rather than - wherever a plan happens to reach them. -- **`expect="<shell>"` on a stack entry**: run after the entry installs, and - it must succeed. A failing probe does NOT record the entry as done, so the - next run rebuilds it. Two are shipped: - `ffmpeg -hide_banner -devices | grep -qw alsa`, and for SDL3 (which - dlopens libasound rather than linking it, so ldd says nothing) - `strings /usr/lib/libSDL3.so | grep -q libasound`. -- Fix on the machine: `packagemanager stack sway --only alsa-lib --run` - (already built -- it will say so), then - `--only ffmpeg --run` and `--only sdl3 --run`. Anything else that wanted - ALSA and was built early -- mpv, pipewire, gstreamer -- is worth the same - treatment; `--only <name> --run` each. - -Block 375 of the suite asserts the order out of the stack file itself, and -fails when alsa-lib is moved back after ffmpeg (checked both ways). - -## Previous session (1.14.155) -- a package is not a service - -The network stack ran on a second machine and stopped at 9 of 9: - -- **`install-service-dhcpcd` was missing from blfs-bootscripts.** The dhcpcd - package installs no service script; `/lib/services/dhcpcd` comes from that - target, so `SERVICE=dhcpcd` and the wpa hand-off had nothing to run. - Adding `book dhcpcd` in 1.14.153 was half the job. Re-run - `packagemanager stack services --only blfs-bootscripts --run`. -- **`check net-wifi` failed the stack for the user's own files.** No - /etc/sysconfig/ifconfig.wlan0 is not something the stack was ever going to - write -- it holds an SSID. Two levels now: `!!` is the stack's and exits 1, - `->` is yours and does not. The step completes and still lists the work. -- **Two contradictory lines about one file.** It said - `wpa_supplicant-__wifi0__.conf has no psk` and, two lines later, that it - `holds a key and is world-readable`. The permission warning is now tied to - actually holding a key, and a `__placeholder__`/example/template name is - reported as a template to copy rather than a broken config. - -## Previous session (1.14.154) -- power as a group, and two dead udev rules - -The reference machine delegated halt/poweroff/reboot/pm-suspend by hand: -`4550 root:nimgnu_power`, plus a boot script re-applying modes on -/sys/power/state because /sys does not survive a reboot. That is the same -shape as `seatd-<user>` and it was in no stack. It is `cfg power` in -services.stack now (`stacks/power.sh`): the group (from `collector_prefix`, -or `POWER_GROUP`), the desktop user's membership read back afterwards, the -setuid bits with the mode confirmed, and **one** boot script in -`/etc/rc.d/init.d/power-perms` symlinked into `rcsysinit.d` -- the machine -had a second full copy under `rc3.d`, which is a second place to edit. - -Two pieces of that setup could never have worked, and both looked like -configuration that did: - -- `SUBSYSTEM=="power", KERNEL=="state", MODE="4550"` -- /sys/power/state is a - kernel attribute with no uevent, so udev is never asked about it. And the - mode is wrong twice: setuid is meaningless on a sysfs file, and 550 has no - write bit, so a group member could not write `mem` to it even if the rule - had applied. The boot script is what does the work, and it uses 0660. -- `ACTION=="add", KERNEL=="bootlog", OWNER="nimgnu_bootlog"` -- /run/bootlog - is an ordinary file the bootscripts write, not a device. No device, no - match, no owner ever applied: **the `nimgnu_bootlog` account is not needed - and never was.** Where the boot log goes and who may read it is `BOOTLOG` - in /etc/sysconfig/rc.site. - -`power.sh` reports both if it finds them in /etc/udev/rules.d, and does not -delete anything. - -## Also this session (1.14.153) -- dhcpcd was never in the stack - -`wlp3s0` exists, so 7.2.4 booted and the driver found its firmware. The link -would not go anywhere because **nothing in any stack installs a DHCP -client** -- the network stack had wpa_supplicant, libnl, libmnl, libnftnl, -iptables and net-tools, and `lfs config network dhcp` deliberately writes no -static interface files *because* a client from BLFS is supposed to follow. -Nothing followed. `book dhcpcd` is in `network.stack` now. - -Beside it, `check net-wifi` (`stacks/net-wifi.sh`): a read-only report of -the five separate things that each produce "wifi does not come up" -- -missing client, missing `ifconfig.<iface>`, a file named for the wrong -interface, missing per-SSID config, or a `SERVICE=` no installed script -implements. It reads the wireless interface names out of `/sys/class/net` -(a file called `ifconfig.wifi0` does nothing for `wlp3s0`) and the hand-off -variable out of `/lib/services/wpa` rather than remembering which release -uses which. It also flags a world-readable psk file. - -The new script needed no list edits: the Makefile and the chroot refresh -both walk `stacks/` (the "derive lists" rule, already applied there). - -## Also this session -- the regulatory database, and a latent failure - -The run came back healthy: 686 of 723 amdgpu declarations installed, 32 of -50 iwlwifi, the four embedded blobs present, `iwlwifi-cc-a0-77.ucode` flat -where the loader wants it. One item left, and it was homework rather than a -fix: - -- **wireless-regdb is now installed by the tool.** `_install_regdb` reads - kernel.org's listing, takes the newest release, unpacks it as - `p_linux-firmware` and installs `regulatory.db` + `.p7s` into - /lib/firmware -- one directory, one owner. `REGDB=yes` in kernel.conf - (default), and `--fix` does it. Tested against a local http mirror; the - sed that parses **kernel.org's real listing is unverified** (no network to - kernel.org from the authoring sandbox), and if it comes back empty the - command says so and prints the two manual commands. -- **A latent failure in the shipped config.** 1.14.147 put - `regulatory.db*` in FIRMWARE_ONLY. A pattern that matches nothing in the - repository is an error, so the next full `lfs-kernel firmware` would have - failed outright, blaming the pattern. Removed from the shipped value, and - `_install_firmware` now recognises an entry that lives elsewhere and - installs it from there instead of failing. -- **A message that contradicted the note above it.** The candidate section - said the boot list "is the reliable one" two lines under a note saying a - chroot's boot list is the host's. In a chroot it now says to boot this - kernel and run it again there. - -## Previous session (1.14.152) -- the chroot host is another machine - -`ls` settled the wifi question: `/lib/firmware/iwlwifi-cc-a0-77.ucode` is -there, flat, 1368100 bytes -- the file 7.2.4 asked for. It is in place, and -`intel/iwlwifi/` has the -50 to -77 series beside it. Nothing more to -install for the AX200; **boot 7.2.4 and look**. - -Three things the report was getting wrong about its own authority: - -- **A satisfied request was skipped in silence**, so a run with everything - in place looked identical to one that had not looked. Boot and config - lines now print `ok ... installed now`, which in a chroot is the only - reassuring thing on the page. -- **The candidate summary counted only what is absent.** It says "1 of 2 - declared file(s) not installed" now, so a module with most of its - declarations present reads as such. -- **The chroot note was too weak.** The host may be another distribution on - other hardware -- it is here -- in which case its boot log can neither - confirm nor deny anything about this tree. And no static check can - promise a driver will find its firmware: iwlwifi derives the name from - the RF and CRF ids it reads off the card at run time. The report says - both. - -Still open for the machine: `regulatory.db` (wireless-regdb, not -linux-firmware) and the boot itself. - -## Previous session (1.14.151) -- say what IS there - -The 7.2.0 boot logs no iwlwifi firmware failure, so that kernel's wifi -firmware loaded: a cc-a0 blob is on this disk somewhere, and 7.2.4's driver -would not take it. The report said nothing about either fact. It now checks, -for every proven miss, the two things that change what you do next: - -- **the same file under another path.** A `FIRMWARE_ONLY` entry keeps the - repository's path, so `intel` installs - `/lib/firmware/intel/iwlwifi/iwlwifi-cc-a0-77.ucode`, and the loader asks - for `/lib/firmware/iwlwifi-cc-a0-77.ucode` and nothing else. `--fix` - copies it into place, no network and no repository needed. -- **an older firmware API installed.** `iwlwifi-cc-a0-72.ucode` present - while the driver requires 77: the number is the API the driver demands and - it does not fall back, so the answer is a newer checkout, not a different - path. The line names the sibling that is there, and the newest three the - checkout has. - -`_fw_neighbours` returns the path in `_FW_LOCAL` rather than on stdout -- -the first version had the caller's `$(...)` swallow the whole explanation -along with it, so the report said only "installed". - -## Previous session (1.14.150) -- two bugs the chroot run exposed - -The modules scan worked and printed 56 MISSING lines. Both problems were in -how that list was produced and presented. - -- **`git ls-files` lies under a sparse index.** Git collapses every excluded - directory into a single entry, so the listing was `intel/` and every blob - under it looked absent: 56 files reported as "this checkout has no file by - that name" while they were all in the repository. `_fw_index` now uses - `git ls-tree -r --name-only HEAD`, which reads the commit and no sparse - setting touches. (Reproduced here: `ls-files --sparse` prints `intel/`.) -- **A declaration is not a failure.** `modinfo -F firmware amdgpu` lists - firmware for every chip amdgpu supports -- Vega, Sienna Cichlid, - Aldebaran, gc_12_1_0 -- and iwlwifi lists five RF families this laptop - does not have. All true, and the conclusion ("56 missing") was wrong: a - driver loads what its own chip needs. `modules` lines are now collected - per module and summarised in one line each as CANDIDATES; only `boot` and - `config` lines are treated as things to fix. `--fix --all` installs the - declared ones anyway. - -The `boot` list stays the reliable one, which is why booting 7.2.4 and -re-running is still worth doing: it named `iwlwifi-cc-a0-77.ucode` exactly. - -## Previous session (1.14.149) -- answer in the chroot - -"Why not make me able to run in chroot?" -- fair. Refusing to answer where -the work happens is not an answer, and dmesg was the wrong source anyway: it -describes the boot that is running, and firmware is requested at boot, so it -can only ever report the PREVIOUS kernel's experience. - -`--missing` now derives the list from three sources and labels every line -with the one that named it: - -- `boot` -- the failed loads in dmesg (in a chroot, the host's; said so) -- `modules` -- every modalias under /sys/devices resolved against - `/lib/modules/<ver>/modules.alias` (`modprobe -S <ver> -R`), then - `modinfo -k <ver> -F firmware`. This machine's hardware, this tree's - kernel, no boot required. -- `config` -- `CONFIG_EXTRA_FIRMWARE`, which a running kernel never - complains about because the image already carries it. - -Which kernel's modules: `VERSION` from kernel.conf first, then `uname -r`, -then the newest under /lib/modules -- and the report names it. `uname -r` in -a chroot is the host's. - -Two more things the output showed: - -- **`regulatory.db` is not in linux-firmware at all.** It is released as - wireless-regdb, so "update the checkout" was advice that could never - work. There is now a small table of firmware that lives elsewhere, with - the URL and the two commands. -- **A nested blob with a flat request cannot be expressed in - FIRMWARE_ONLY.** An entry is copied to the same relative path, so - suggesting `intel/iwlwifi` for a driver asking for - `iwlwifi-cc-a0-77.ucode` would install it where the loader never looks. - The suggestion is now only made when the paths match, and the mismatch is - named. Relatedly, `_install_firmware` copied a nested entry with - `cp -a $src/$d /lib/firmware/`, dropping the directory (the deferred item - from 1.14.147): it uses `cp -a --parents` from inside the checkout now. - -## Previous session (1.14.148) -- three bugs in 1.14.147's firmware command - -All three were found by the user's first real `firmware --missing --fix`, and -the first two are old rules re-broken: - -- **A swallowed error became a confident wrong answer.** `git -C "$src" - ls-files 2>/dev/null` returns nothing and rc 128 when the repository - belongs to `p_linux-firmware` and the command runs as root (`fatal: - detected dubious ownership`). The empty result was printed as - "linux-firmware does not have a file by that name at all", about - `regulatory.db`, which is in the repository. There is now one `_fw_index`, - it passes `-c safe.directory=$src`, and a failed lookup reports UNKNOWN - with git's message rather than "absent". -- **A check that did not know what it was measuring.** `dmesg` is a syscall - against the running kernel, so in a chroot it works and answers about the - host's boot. `--missing` now detects the chroot and says the list is the - host's. -- **`local a="$2" d="/lib/firmware/$a"`** reads `$a` before that same - `local` assigns it; `set -u` killed `_fw_place` on its first real call. - Split into two statements. - -Also: `--fix` no longer touches the sparse checkout at all. It read one blob -with `git show HEAD:<path>`, because the old path widened the checkout and -fell back to `sparse-checkout disable` -- which in cone mode would have -materialised all ten gigabytes to install one 2 MB file. And an unchanged -`FIRMWARE_ONLY` is no longer printed as though it were a fix. - -Tested here: all four `_fw_place` branches (on disk, index-only, -nested-repo-path to flat-request, absent), the ownership failure, and blocks -109, 374, 375. - -## Previous session (1.14.147) - -- `lfs-helper setid [--run]`: the setid ledger is no longer emptied by the - one notice that prints it. `setid` lists what was refused, checks a small - on-disk list (su, passwd, chfn, chsh, gpasswd, newgrp, newuidmap, - newgidmap, dbus-daemon-launch-helper) for the case where nothing was - recorded, and `--run` chowns to root, sets the mode, and reads it back. - `verify` now counts outstanding requests as a problem. -- The **install** wrapper records refused setid modes, against the file that - actually landed, and only after the install succeeded. The chmod wrapper - had done this since 1.14.82; the same rule, one of two places. -- `lfs-kernel firmware` (install firmware with no kernel build) and - `lfs-kernel firmware --missing [--fix]`: reads the failed loads out of - dmesg, finds each file in linux-firmware's git *index* (a sparse checkout - has removed the very file being looked for), installs it at the path the - driver asked for, widening the sparse checkout by that one path, and - confirms the copy. The firmware functions were hoisted above the build to - make this reachable without one. -- `lfs-kernel --help` derives its own slice from the header instead of a line - range that a new command pushes out of view. -- Suite: `--list/--from/--to/--only`, a `skip()` counter, rc.site skips with - a reason, and two new blocks (374, 375). Blocks 109, 293, 373, 374, 375, - 204 run green here; the rest of the suite was not run. - -## Previous session (1.14.146) +## This session (1.14.146) - `BUILD.md` written; README tightened (kernel caveat was stale: it said nothing builds a kernel, `lfs-kernel` does). @@ -3125,6 +205,8 @@ nested-repo-path to flat-request, absent), the ownership failure, and blocks None of this blocks the machine. +- The regression suite exceeds one execution slot; make it splittable + (`--from N`, or per-tool) before it grows further. - Three functions move stack files (Makefile, chroot refresh, `--take-new`); they should be one, and the whole `.new` mechanism deserves rethinking around a single source of truth. @@ -3136,14 +218,9 @@ None of this blocks the machine. the stack, not a package installed by hand. - `--check-refs` cannot see tar URLs. A `have=`-style check for tarballs would close the gap libnftnl exposed. -- The preamble of the suite (everything before block 1) always runs, and a - block that needs a variable an earlier block computed dies on `set -u` - rather than skipping. Only tool-path assignments are hoisted. -- The `modules` source needs `modprobe`/`modinfo` (kmod) and a mounted - /sys; without either it falls back to the boot list silently apart from a - header line. It was tested with stubs here, never against real kmod. - The "busy tree" test is flaky: it races a `sleep` in the process table and should use a private marker. +- The rc.site test should say it needs a SysV book rather than failing. - The whole-book invariants (954 scripts, 0 syntax errors, 0 empty install phases) have caught or confirmed nearly every filter change. Extend them to build phases, listing the known-empty pages by name. diff --git a/Makefile b/Makefile index df793ac..5d7850f 100644 --- a/Makefile +++ b/Makefile @@ -23,7 +23,7 @@ SYSCONFDIR ?= $(DESTDIR)/etc # The four tools, the book merger, and the phase runner every generated # install script sources (it goes in bin so `. lfs-phases` finds it on PATH, # on the host, in the chroot and on the built system alike). -TOOLS := lfs lfs-helper packagemanager blfs lfs-sysvbook lfs-phases lfs-kernel lfs-backup +TOOLS := lfs lfs-helper packagemanager blfs lfs-sysvbook lfs-phases lfs-kernel COMPLETION := lfs-completion.bash TESTS := test_lfs_crosschain.sh @@ -60,10 +60,6 @@ install: rm -f "$(BINDIR)/packagemanager_install" \ && echo " removed retired packagemanager_install"; \ fi - @# lfs-config is retired (1.14.291): lfs-backup replaces it. - @if [ -e "$(BINDIR)/lfs-config" ]; then \ - rm -f "$(BINDIR)/lfs-config" && echo " removed retired lfs-config (now lfs-backup)"; \ - fi @for t in $(TOOLS); do \ if [ -f "$$t" ]; then \ $(INSTALL) -m 755 "$$t" "$(BINDIR)/$$t" && echo " $$t"; \ @@ -71,16 +67,6 @@ install: echo " !! missing: $$t" >&2; exit 1; \ fi; \ done - @# THE STAMP RECORDS WHAT IS INSTALLED, so whoever installs writes it. - @# It was written only by `lfs build-system install-tools`, which runs - @# on a host against a chroot -- so on a BOOTED system every later - @# `make install` left the stamp describing the old copies, and - @# packagemanager announced "these tools differ from the copies on the - @# host" and told the reader to run sync-tools on a host that no longer - @# exists. - @$(MKDIR) $(DESTDIR)/usr/share/lfs - @{ echo '{'; first=1; for t in $(TOOLS); do [ -f "$$t" ] || continue; m=$$(md5sum "$$t" | cut -d' ' -f1); [ $$first = 1 ] || echo ','; first=0; printf ' "%s": "%s"' "$$t" "$$m"; done; echo; echo '}'; } > $(DESTDIR)/usr/share/lfs/tool-stamps.json - @echo " recorded $(DESTDIR)/usr/share/lfs/tool-stamps.json" @echo "==> stacks -> $(DESTDIR)/etc/pkgusr/stacks" $(MKDIR) $(DESTDIR)/etc/pkgusr/stacks @# A STACK FILE IS THE USER'S. This overwrote it unconditionally, so @@ -120,17 +106,6 @@ install: @# kernel.conf is a file to EDIT BEFORE building, so it is installed @# rather than written on first run -- and, like a stack file, an @# edited one is kept and the new one lands beside it. - @# backup.conf: what lfs-backup carries. Yours is kept, like kernel.conf. - @if [ -f backup.conf ]; then \ - $(MKDIR) "$(DESTDIR)/etc/pkgusr"; \ - d="$(DESTDIR)/etc/pkgusr/backup.conf"; \ - if [ ! -e "$$d" ] || cmp -s backup.conf "$$d"; then \ - $(INSTALL) -m 644 backup.conf "$$d" && echo " backup.conf"; \ - else \ - $(INSTALL) -m 644 backup.conf "$$d.new" \ - && echo " backup.conf -> KEPT YOURS ($$d.new)"; \ - fi; \ - fi @if [ -f kernel.conf ]; then \ d="$(DESTDIR)/etc/pkgusr/kernel.conf"; \ if [ ! -e "$$d" ] || cmp -s kernel.conf "$$d"; then \ @@ -209,7 +184,7 @@ uninstall: @echo "Left alone (they are yours):" @echo " $(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile" @echo " /usr/share/lfs books, settings, snapshots" - @echo " /etc/pkgusr packagemanager settings, kernel.conf, backup.conf" + @echo " /etc/pkgusr packagemanager settings" check: @bash $(TESTS) ./lfs diff --git a/README.md b/README.md index b23a159..4c79fc7 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,6 @@ lfs build-system session # the interview: partition, prefixes, locale, logi lfs run # everything up to the chroot, then enter it lfs-helper build-all # inside: chapters 7-9 packagemanager bootstrap --run # wget, Python modules, BLFS book, certificates -packagemanager stack # list the stacks packagemanager stack sway --run # then network, services lfs-kernel # the kernel, from /etc/pkgusr/kernel.conf ``` @@ -195,28 +194,6 @@ packagemanager user create firefox --shared --share-dir --launcher The session group is read from the runtime directory, not assumed. Set it up once for your own account; every shared user then joins whatever is there. -Many accounts at once come from `/etc/pkgusr/stacks/app-users.conf`, one line -per launcher, applied by `packagemanager stack services --only app-users ---run`. Undoing them: - -| Command | Removes | -|---|---| -| `packagemanager user remove-launcher <name> --run` | one launcher, its menu entry and icon | -| `packagemanager user delete <account> --run` | the account; its home is archived to `/root` first | -| `packagemanager user purge --run` | all of them, plus the session group, the profile fragments and the PAM wiring | - -A home with a browser profile in it is archived to `/root/u_<name>-home-<date>.tar.gz` -(0600) before the account goes — logins, cookies and downloads that nothing -else can give back. A home that was never used is not archived. Drop the -archives when you no longer want them: `rm /root/u_*-home-*.tar.gz`. - -Each is a dry run without `--run`. `$SHARED_*` comes from -`/etc/profile.d/shared-dirs.sh`. `/etc/profile.d` is read by **login** shells -only, so a terminal opened inside your session shows what it inherited at -login. The step also sources the fragment from `/etc/bashrc`; any shell can -refresh itself with `. /etc/profile.d/shared-dirs.sh`, which also drops the -variables whose directory is gone. - --- ## When ownership starts @@ -247,13 +224,6 @@ Everything is dry-run by default. `--run` applies. | `lfs-helper which-package <path>` | Which package installed this | | `lfs-helper manifests` | What each package installed | | `lfs-helper sort-users --run` | Sort passwd and group by id | -| `lfs-helper setid [--run]` | Setuid/setgid bits the build refused (`su`, `passwd`, ...) | -| `lfs-backup create --run` | Back up the files in `/etc/pkgusr/backup.conf` (modes, owners) | -| `lfs-backup restore <archive> [--skip accounts,ssh] [--run]` | Put them on a new system: creates missing files, writes `.new` for the rest | -| `lfs-backup plan <archive> [file]` | A restore plan to edit: skip files, or send them elsewhere | -| `lfs-backup restore <archive> --plan <file> [--run]` | Restore by that plan | -| `lfs-backup merge <archive>` | Go through the `.new` files: diff, keep live, take backup, or edit | -| `lfs-backup clean <archive> [--run]` | Remove those `.new` files after merging (edited ones are kept) | --- @@ -327,7 +297,6 @@ lfs-kernel list # what kernel.org offers lfs-kernel # build the version in /etc/pkgusr/kernel.conf lfs-kernel 6.12.40 # that version, once lfs-kernel verify # is everything installed? -lfs-kernel firmware --missing [--fix] # firmware this machine wants and lacks ``` Everything it does is configured in `/etc/pkgusr/kernel.conf`, which is @@ -355,19 +324,15 @@ enter it. ## Tests ```sh -bash test_lfs_crosschain.sh ./lfs # all of it -bash test_lfs_crosschain.sh --list # numbered blocks -bash test_lfs_crosschain.sh --from 300 --to 340 ./lfs -bash test_lfs_crosschain.sh --only firmware ./lfs +bash test_lfs_crosschain.sh ./lfs ``` Each test encodes a bug that actually happened, with a comment explaining what broke. -Run it from a directory holding all the tools — about 100 tests skip without -them. A selection re-executes a generated subset (preamble + chosen blocks + -summary); a block that needs something an earlier one computed says so -instead of passing quietly. +Run it from a directory holding all the tools — about 100 tests skip +without them. It no longer fits one sandbox slot; on a real machine run it +whole, or one block at a time (each starts `# ---- <title>`). --- diff --git a/blfs b/blfs index 0646127..dbfacf1 100755 --- a/blfs +++ b/blfs @@ -128,7 +128,7 @@ CACHE_VERSION = 56 # v56: a skipped build variant does not keep its install st # tool + generated-script format versions. Bump SCRIPT_VERSION when the shape # of generated install scripts changes (phases, dispatcher, ...), and # CACHE_VERSION when the parsed/cached data structure changes. -BLFS_VERSION = "1.14.299" +BLFS_VERSION = "1.14.146" def _build_id(): @@ -838,17 +838,8 @@ def cache_dir(): def _book_stamp(path): - # THE CODE THAT EXTRACTED THE DATA IS PART OF THE KEY. - # - # The key was the book's size and mtime plus CACHE_VERSION, a number to - # bump by hand. Three fixes to the extractors -- 1.14.212, both halves - # of the poppler pair-skip, and the first try at shadow's pam.d blocks -- - # were each "proved" by regenerating a script that came out byte-identical - # and then reverted, because the script was rendered from a cache built - # by the OLD code. The edits never ran. _build_id changes whenever this - # file does, so an edited extractor re-indexes once, by construction. st = os.stat(path) - return f"{st.st_size}:{int(st.st_mtime)}:v{CACHE_VERSION}:{_build_id()}" + return f"{st.st_size}:{int(st.st_mtime)}:v{CACHE_VERSION}" def _cache_file_for(path): @@ -999,30 +990,6 @@ def build_index(soup): anchor = a.get("id") or a.get("name") if anchor and anchor not in idx: idx[anchor] = header - # THE ID IS NOT ALWAYS INSIDE THE HEADER. - # - # A page whose anchor sits on the section DIV rather than on an <a> inside - # its title header is invisible to the loop above -- it is not "missing - # from the book", it is missing from the index, and the two look identical - # from outside: `packagemanager stack network` said NOT IN BOOK for - # dhcpcd, whose page is right there in basicnet. - # - # Only ids that name a section carrying a Package Information block are - # added, and only when nothing already claims the anchor, so this can add - # pages and can never move an existing one. - for div in soup.select("div.sect1[id], div.sect2[id]"): - anchor = div.get("id") - if not anchor or anchor in idx: - continue - if div.find("div", class_="package", recursive=False) is None: - continue - # the DIV is what gets indexed, not its header: for this shape the - # header sits INSIDE the section, so the "titlepage then sibling - # sect" walk that resolves the common shape finds div.package next - # and gives up. resolve_package takes a section div directly. - if div.find(lambda t: t.name in H_TAGS - and (_header_classes(t) & set(HEADER_CLASSES))) is not None: - idx[anchor] = div return idx @@ -1046,18 +1013,6 @@ def resolve_package(soup, anchor): header = get_index(soup).get(anchor) if header is None: return None - # A SECTION DIV THAT CARRIES THE ID IS ALREADY THE CONTENT. build_index - # adds these (see there); they are whole pages whose anchor is an - # attribute of the section rather than an <a> in its title. - if getattr(header, "name", None) == "div": - h = _content_title_header(header) - title = " ".join(h.get_text().split()) if h is not None else anchor - if header.find("div", class_="package", recursive=False) is None: - return None - if not extract_link(header).strip(): - return None - return {"anchor": anchor, "title": title, "content": header, - "is_module": "sect1" not in _header_classes(header)} classes = _header_classes(header) title = " ".join(header.get_text().split()) @@ -1102,7 +1057,7 @@ def resolve_package(soup, anchor): # This rejects in-package subsection anchors (foo-kernel "Kernel # Configuration", foo-config "Configuring foo", ...) that would otherwise # resolve to their enclosing package's data. - if sect_anchor(content) != anchor and content.get("id") != anchor: + if sect_anchor(content) != anchor: return None return {"anchor": anchor, "title": title, "content": content, "is_module": is_module} @@ -1441,28 +1396,6 @@ def extract_commands(sec, selector): if not elements: return "" out = [] - _emit_command_elements(elements, out, nested=False) - return "\n".join(out) - - -# A SUB-SECTION IS PART OF THE SECTION. -# -# Shadow's configure phase came out as prose and a bare `:` -- no -# /etc/pam.d/login, passwd, chage or the ten admin tools -- so `passwd` said -# `pam_start() failed, error 26` and login `PAM Failure, aborting`. Every one -# of those blocks is on the page, one level down: -# div.configuration > div.sect3 > div.sect4 > pre.root -# and this walk read the children of sect3 and dropped any div that was not a -# note. linux-pam's four system-* files sit directly in its sect3, which is -# why those four existed and shadow's sixteen did not. -# -# So a nested div.sect4/sect5 is walked with the same rules, and its title is -# kept as a comment. The TOP level is unchanged byte for byte: its own -# titlepage divs were never emitted and still are not. -_NESTED_SECT_CLASSES = {"sect4", "sect5"} - - -def _emit_command_elements(elements, out, nested): for el in elements: name = getattr(el, "name", None) if name == "h2": @@ -1486,7 +1419,7 @@ def _emit_command_elements(elements, out, nested): # A conditional block is conditional wherever it appears. _g = _conditional_guard(el) if _g: - out.append(SKIP_MARK + _g) + out.append("## OPTIONAL, skipped -- the book says: " + _g) out.append(_commented(text)) else: out.append(text) @@ -1509,13 +1442,7 @@ def _emit_command_elements(elements, out, nested): lines.append(child) out.append("## " + " ".join(lines).replace(" , ", ", ")) out.append("") - elif _NESTED_SECT_CLASSES & set(classes): - _emit_command_elements(el.find_all(True, recursive=False), - out, nested=True) - elif nested and "titlepage" in classes: - title = " ".join(el.get_text().split()) - if title: - out.append("\n\n## " + title) + return "\n".join(out) # The book's conditionals live in PROSE: "If you wish to build and install @@ -1543,53 +1470,8 @@ _COND_GUARD_RE = re.compile(r"^(if\b|optionally\b|should you\b)", re.I) # a failing command was ignored; the moment the dispatcher started honouring # set -e it would fail every such package. Tests belong to `test` # (PM_TEST=1), not to a build. -# ...AND THE BOOK DOES NOT ALWAYS START THE SENTENCE WITH "To test". -# numpy's is -# "The installation can be tested with the following commands:" -# which this matched none of, so a venv, `pip3 install hypothesis attrs` -# and `numpy.test()` were generated into the INSTALL phase. numpy itself -# installed (1320 files) and then the package failed on -# ModuleNotFoundError: No module named 'pytest' -# -- a test dependency, in an install. So these are searched for anywhere -# in the introducing sentence, not just at its start. _TEST_BLOCK_RE = re.compile( - r"^(to test\b|to check\b|the test suite\b|tests? (can|may) be run\b)" - r"|\bcan be tested\b|\bto test the (results|installation|build)\b" - r"|\bto run the test", re.I) - -# THE COMMANDS SAY WHAT THEY ARE, AND PROSE IS A SECOND-HAND ACCOUNT. -# -# 1.14.211 widened the sentence match so numpy's "The installation can be -# tested with the following commands" would be recognised -- and the -# regenerated script STILL had the venv, `pip3 install hypothesis attrs` -# and `numpy.test()` in its install phase, because the book being read on -# that machine is 13.0 and words or nests it differently again. -# -# Chasing the prose is a losing game: `numpy.test()` IS a test suite -# whatever sentence introduces it. These patterns match the block's own -# commands, so a rewording upstream cannot bring the bug back. -_TEST_CONTENT_RE = re.compile( - r"""(?mx) - ^\s*(make|ninja)\s+(-\w+\s+)*(check|test)\b # make check / ninja test - | ^\s*meson\s+test\b - | ^\s*ctest\b - | ^\s*py\.test\b | ^\s*pytest\b - | ^\s*python3?\s+-m\s+(pytest|unittest)\b - | \.test\(\) # numpy.test() - | ^\s*python3?\s+-m\s+venv\b # a throwaway test venv - | ^\s*source\s+\S*testenv/bin/activate - """) - - -def _test_content_reason(pre): - """Why this block is a test suite, judged by its commands, or None.""" - txt = pre.get_text() - m = _TEST_CONTENT_RE.search(txt) - if not m: - return None - hit = " ".join(m.group(0).split()) - return "it IS the test suite -- these commands run it (`%s`)" % hit - + r"^(to test\b|to check\b|the test suite\b|tests? (can|may) be run\b)", re.I) def _conditional_guard(pre): """The conditional sentence introducing this <pre>, or None if the block @@ -1607,16 +1489,12 @@ def _conditional_guard(pre): Mixing local libtiff with shared libjpeg not supported "optional" wins over both: an optional tool is exactly what we do not install.""" - # content first: it does not depend on the book's wording or nesting - _by_content = _test_content_reason(pre) - if _by_content: - return _by_content prev = pre.find_previous(["p", "pre", "h3", "h4"]) if prev is None or prev.name != "p": return None txt = " ".join(prev.get_text().split()) - if _TEST_BLOCK_RE.search(txt): - return "the book says: " + txt # a test, never part of installing + if _TEST_BLOCK_RE.match(txt): + return txt # a test, never part of installing if not _COND_GUARD_RE.match(txt): return None low = txt.lower() @@ -1628,7 +1506,7 @@ def _conditional_guard(pre): return None if "optional" not in low and ("recommended" in low or "required" in low): return None # our contract satisfies it - return "the book says: " + txt + return txt def _commented(text): @@ -1707,7 +1585,7 @@ def extract_commands_phased(sec): is_root = "root" in classes and not guard stream = install if (is_root or install) else build if guard: - stream.append(SKIP_MARK + guard) + stream.append("## OPTIONAL, skipped -- the book says: " + guard) stream.append(_commented(text)) continue stream.append(text) @@ -2622,61 +2500,6 @@ _CONFIG_TEST_INTRO_RE = re.compile( r"|test your installation", re.I) -# A SITE CONFIGURATION FILE IS NOT AN INSTALL STEP. -# -# The dhcpcd page ends with an example interface file: -# cat > /etc/sysconfig/ifconfig.eth0 << "EOF" -# ONBOOT="yes" -# IFACE="eth0" -# SERVICE="dhcpcd" -# ... -# Run as a configure phase by p_dhcpcd it died with -# /etc/sysconfig/ifconfig.eth0: Permission denied -# -- correctly: /etc/sysconfig is drwxrwxr-t root:install and that file is -# root's. Granting the package user a collector group for /etc/sysconfig -# would "fix" it by letting a package write the machine's network -# configuration, which is worse than the failure. -# -# And the content is wrong anyway: `eth0` is the book's placeholder, and this -# machine's interfaces are `enp5s0` and `wlp3s0`. A file naming an interface -# that does not exist does nothing, or shadows the one you meant. -_SITE_CONFIG_RE = re.compile( - r"/etc/sysconfig/(ifconfig\.|network-devices|wpa_supplicant)" - r"|/etc/wpa_supplicant" - r"|/etc/sysconfig/console|/etc/sysconfig/clock", re.I) - - -def _comment_site_config_writes(cmds, name=""): - """Comment out writes to files that hold THIS machine's settings.""" - out, skipping, heredoc = [], False, None - for line in (cmds or "").splitlines(): - st = line.strip() - # PROSE IS NOT CODE: a comment mentioning the path is not a write. - if st.startswith("#") and not heredoc: - out.append(line) - continue - if heredoc is not None: - out.append("# " + line if skipping else line) - if st == heredoc or st == heredoc.strip('"\''): - heredoc, skipping = None, False - continue - if _SITE_CONFIG_RE.search(st) and re.search(r">|tee |install |cp |mv ", st): - out.append("## SKIPPED -- this writes a file that belongs to the" - " MACHINE, not to the package:") - out.append("## %s" % st[:120]) - out.append("## The interface name in the book is a placeholder, and" - " /etc/sysconfig is root's.") - out.append("## Write yours by hand; what is missing is reported by:") - out.append("## packagemanager stack network --only net-wifi --run") - out.append("# " + line) - m = re.search(r"<<-?\s*([\"']?)([A-Za-z_][A-Za-z0-9_]*)\1", st) - if m: - heredoc, skipping = m.group(2), True - continue - out.append(line) - return "\n".join(out) - - def _comment_config_tests(cmds): out, skipping = [], False for line in (cmds or "").splitlines(): @@ -3217,21 +3040,6 @@ def _drop_orphaned_variant_install(cmds): return "\n".join(out) -# ONE MARKER, ONE CONSTANT. `_enable_one_alternative` looked for -# "## OPTIONAL, skipped" and 1.14.212 renamed the marker to "## SKIPPED --" -# -- so the picker silently stopped firing, and webkitgtk's script came out -# with BOTH build blocks commented: a build phase of `:` and an install -# phase that runs `ninja install` against a directory with no build.ninja. -# Two strings for one fact, drifting apart, exactly as the rule says. -SKIP_MARK = "## SKIPPED -- " - - -def _is_skip_line(line): - """A skipped-block header, under either spelling this tool has used.""" - st = line.strip() - return st.startswith(SKIP_MARK.strip()) or st.startswith("## OPTIONAL, skipped") - - def _enable_one_alternative(cmds): """A page that offers a CHOICE must still build something. @@ -3250,7 +3058,8 @@ def _enable_one_alternative(cmds): if any(l.strip() and not l.strip().startswith("#") and l.strip() != ":" for l in lines): return cmds - starts = [i for i, l in enumerate(lines) if _is_skip_line(l)] + starts = [i for i, l in enumerate(lines) + if l.strip().startswith("## OPTIONAL, skipped")] if not starts: return cmds i = starts[0] @@ -3258,11 +3067,8 @@ def _enable_one_alternative(cmds): out = lines[:i] out.append("## Every build block on this page is conditional, so none " "would have run.") - _why = lines[i] - for _sep in ("the book says:", SKIP_MARK.strip()): - if _sep in _why: - _why = _why.split(_sep, 1)[-1] - out.append("## ENABLED HERE: %s" % _why.strip()[:110]) + out.append("## ENABLED HERE: %s" + % lines[i].split("the book says:", 1)[-1].strip()[:110]) out.append("## If you want one of the page's other variants instead, " "comment these") out.append("## lines and uncomment that block.") @@ -3325,7 +3131,6 @@ def render_script(data, source_url, book_version=""): config_cmds = _comment_profile_sourcing(data.get("config_cmds") or "") config_cmds = _comment_bootscript_installs(config_cmds, data.get("name")) config_cmds = _comment_config_tests(config_cmds) - config_cmds = _comment_site_config_writes(config_cmds, data.get("name")) data["config_cmds"] = config_cmds build_cmds = _comment_profile_sourcing(build_cmds) install_root = _comment_profile_sourcing(install_root) @@ -4011,69 +3816,18 @@ def cmd_debug(args): anchor = anchor_from_input(args.target) print(f"input : {args.target}") print(f"anchor : {anchor}") - # WHICH BOOK AM I EVEN READING. The first question when a page that - # plainly exists upstream is "not in the book" is whether this is the - # book you think it is, and how much of it parsed. - try: - _st = os.stat(book.path) - print("book : %s" % book.path) - print(" %.1f MB, modified %s" - % (_st.st_size / 1048576.0, - time.strftime("%Y-%m-%d %H:%M", time.localtime(_st.st_mtime)))) - except OSError: - print("book : %s (cannot stat it)" % book.path) - print("pages parsed : %d" % len(book.packages)) anchor = resolve_anchor(book, anchor) data = book.get(anchor) if data is None: - # NAME THE TEST THAT FAILED. "not an installable package (no package - # block + download link)" covered three different causes with three - # different fixes, and told you which of them applied: none. + # not a package in the cache -- parse the soup to explain why soup = book.soup - in_index = anchor in get_index(soup) node = soup.find(attrs={"id": anchor}) or soup.find("a", attrs={"name": anchor}) - print("in page index : %s" % ("yes" if in_index else "no")) if node is None: - print("resolve : NOT FOUND -- no element in this book carries " - "that id or name") - near = sorted({(e.get("id") or e.get("name")) - for e in soup.find_all(attrs={"id": True}) - if anchor in (e.get("id") or "")} - | {a.get("name") for a in soup.find_all("a", attrs={"name": True}) - if anchor in (a.get("name") or "")}) - near = [n for n in near if n and n != anchor][:8] - if near: - print("similar ids : %s" % ", ".join(near)) - print("hint : the page is there under another anchor; " - "use one of those") - elif re.search(re.escape(anchor), soup.get_text(" ", strip=True)[:4000000], - re.IGNORECASE): - print("hint : the NAME appears in the text but no anchor " - "matches -- likely a different book edition") - else: - print("hint : this book really does not have it. A tar or " - "git entry with build=/install= is the way in.") + print("resolve : NOT FOUND (no element with this id/name)") else: - print("resolve : found <%s id=%s>, and it is not accepted as a " - "package. Which test fails:" % (node.name, node.get("id"))) - content = node if node.name == "div" else node.find_parent( - "div", class_=["sect1", "sect2", "sect3", "sect4", "sect5"]) - if content is None: - print(" content : no enclosing section div") - else: - pkg_div = content.find("div", class_="package", recursive=False) - print(" package block: %s" % ("present" if pkg_div is not None - else "MISSING (a direct child " - "<div class=package> is required)")) - link = extract_link(content).strip() if pkg_div is not None else "" - if pkg_div is not None: - print(" download link: %s" % (link or "MISSING (no upstream tarball url)")) - sa = sect_anchor(content) - print(" titles it : %s" % ("yes" if (sa == anchor - or content.get("id") == anchor) - else "no -- %r titles that section, so " - "this is a subsection anchor" % sa)) + print(f"resolve : found <{node.name} id={node.get('id')}> but it is " + f"not an installable package (no package block + download link)") print("hint : blfs search " + anchor) return diff --git a/kernel.conf b/kernel.conf index e779b54..f9074e2 100644 --- a/kernel.conf +++ b/kernel.conf @@ -54,23 +54,6 @@ CONFIG_FROM=running # Either way the tool reports how many lines changed. CONFIG_REFRESH=olddefconfig -# CONFIG_SET: options THIS MACHINE always wants, applied on top of whatever -# CONFIG_FROM gave, after the refresh -- so `running` + a new kernel version -# cannot quietly lose them. SYMBOL=VALUE, space separated, CONFIG_ optional: -# y / m / n enable, module, disable -# "text" a string option -# 123 a number -# Each is read back from the final .config; one the kernel refuses (a missing -# dependency, a symbol this version does not have) is named and stops the build. -# -# LEGACY_TIOCSTI=n no program may type into a terminal it does not own. -# With it allowed, shadow's su (4.20+) detaches the shell -# of every `su - <user>` run by root -- "cannot set -# terminal process group ... no job control" -- because -# that user could otherwise inject commands into root's -# terminal. Off, su keeps the terminal and job control. -CONFIG_SET="LEGACY_TIOCSTI=n" - # ---- firmware ------------------------------------------------------------ # # Most drivers need firmware blobs: amdgpu will not bring up a display @@ -103,33 +86,12 @@ FIRMWARE_GIT=https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firm # older releases kept iwlwifi-*.ucode at the top level, # which is what a FILE pattern like iwlwifi-cc-* is for. # rtl_bt Realtek Bluetooth -# iwlwifi-*.ucode THE INTEL WIRELESS BLOBS THEMSELVES. The driver asks -# the loader for a FLAT path -- iwlwifi-cc-a0-77.ucode, -# not intel/iwlwifi/... -- so `intel` above does not -# satisfy it, and the AX200 comes up with -# Direct firmware load for iwlwifi-cc-a0-77.ucode failed -# no suitable firmware found! -# and no wlan interface at all. Since 1.14.279 the -# pattern matches at any depth, and the flat names are -# made as links from linux-firmware's WHENCE -- so -# `intel` alone gets them too; the pattern does no harm. -# NOT here: regulatory.db. cfg80211 asks for it, but linux-firmware does -# not carry it -- it is released on its own as wireless-regdb -- and naming -# it here fails the whole firmware install. REGDB below fetches it. # Find what your machine asks for, once it is running, with: -# lfs-kernel firmware --missing # and --fix to install them # dmesg | grep -i firmware # (quoted: the value has spaces and a wildcard, and the config is sourced by # the shell -- unquoted, `intel` was run as a command and the whole # assignment came out empty, which meant "install everything") -FIRMWARE_ONLY="amdgpu amd-ucode amd intel iwlwifi-*.ucode" - -# REGDB: cfg80211's regulatory database, which is wireless-regdb rather than -# part of linux-firmware. yes installs the newest release from kernel.org -# into /lib/firmware, owned by the firmware package like everything else -# there. Without it: "failed to load regulatory.db" at boot, and the most -# conservative channels and transmit power the driver has. -REGDB=yes +FIRMWARE_ONLY="amdgpu amd-ucode amd intel" # ---- headers ------------------------------------------------------------- # diff --git a/lfs b/lfs index 655993f..cd534f8 100755 --- a/lfs +++ b/lfs @@ -60,7 +60,7 @@ except ImportError: " packagemanager pip install beautifulsoup4\n") sys.exit(2) -LFS_VERSION = "1.14.299" +LFS_VERSION = "1.14.146" def _build_id(): @@ -235,11 +235,11 @@ LAYOUT_DEFAULTS = { # the system being built "target_pkgusr_home": "/usr/src/pkgusr", "target_cfguser_home": "/usr/src/cfg", - "target_appuser_home": "/home/shared_user", # one subdir, not /home itself + "target_appuser_home": "/usr/src", # or /home/shared_users # this machine "pkgusr_home": "/usr/src/pkgusr", "cfguser_home": "/usr/src/cfg", - "appuser_home": "/home/shared_user", + "appuser_home": "/usr/src", } @@ -266,10 +266,7 @@ def _accounts_root(kind): target=True).rstrip("/") _c = {} try: - # load_config, the function that exists: `read_config` did not, and - # the NameError was swallowed below -- a configured pkgusr_subdir / - # cfguser_subdir was silently ignored in favour of the default - _c = load_config() or {} + _c = read_config() or {} except Exception: _c = {} sub = str(_c.get("pkgusr_subdir" if kind == "pkgusr" else "cfguser_subdir") @@ -1480,7 +1477,7 @@ _SESSION_KEYS = [ "/usr/src"), ("target_appuser_home", "where application users live on the BUILT system", - "/home/shared_user"), + "/usr/src"), ("locale", "system locale (see 'locale -a')", "en_US.UTF-8"), ("paper_size", "paper size for groff: A4 or letter", "A4"), # Everything below is a decision the build used to discover halfway @@ -3299,38 +3296,6 @@ def _handover_build_user_only(lfs, run=True): uid = _build_user_uid() if uid is None: return 0 - # THE HOST'S lfs UID CAN BE A PACKAGE USER'S UID IN THE TREE. - # - # This selects files by NUMBER, because that is the only thing that means - # the same on both sides of the chroot. Book 4.3 pins no uid for `lfs`, - # so the host's is whatever was free -- 10753 on one machine -- and - # package users here are created in the 10000-20000 range. If those - # numbers ever meet, this hands a package user's files to root and the - # package silently loses every one of them. - # - # Ask the tree's own passwd before touching anything. - _clash = [] - _pw = os.path.join(lfs, "etc", "passwd") - try: - with open(_pw, encoding="utf-8", errors="replace") as f: - for line in f: - parts = line.split(":") - if len(parts) > 2 and parts[2].strip() == str(uid): - _clash.append(parts[0]) - except OSError: - pass - if _clash: - warn("REFUSING to hand files back: the host's build user is uid %d," - % uid) - warn("and that is also %s in %s." % (", ".join(_clash), _pw)) - note(" Selecting by uid cannot tell them apart, so this would give") - note(" that account's files to root and the package would lose them.") - note(" See what it owns:") - note(" find %s -xdev -uid %d | head" % (lfs, uid)) - note(" Then renumber the HOST's lfs account -- it exists only to") - note(" build, so nothing of yours depends on its number:") - note(" usermod -u <free-uid> lfs # on the host, then retry") - return 0 dirs = [os.path.join(lfs, d) for d in LFS_HANDOVER_DIRS if os.path.exists(os.path.join(lfs, d))] if not dirs: diff --git a/lfs-helper b/lfs-helper index 98985e0..2e69f94 100755 --- a/lfs-helper +++ b/lfs-helper @@ -49,7 +49,7 @@ set -u # Bump on every change that alters behaviour. `lfs-helper --version` makes it # possible to tell at a glance whether the copy inside the chroot is the one # that was just fixed -- guessing at that has wasted a lot of time. -LFS_HELPER_VERSION="1.14.299" +LFS_HELPER_VERSION="1.14.146" # A fingerprint of this file's own contents. A hand-maintained version number # goes stale the moment someone forgets to bump it -- and the reason for @@ -980,7 +980,7 @@ cmd_add_user() { pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" if user_exists "$name"; then - [ "${_QUIET_EXISTING:-0}" = 1 ] || say "# user '$name' already exists" + say "# user '$name' already exists" else if have_shadow_tools; then # Let Shadow allocate the ids. Choosing them by hand means racing @@ -1056,20 +1056,14 @@ load_snapshot() { have_su() { command -v su >/dev/null 2>&1; } snapshot_dirs() { - # PRUNE, DO NOT FILTER. `-not -path "$SRCROOT/*"` hides a path AFTER find - # has walked into it, so every scan still descended /usr/src -- every - # package's unpacked source and build tree, LLVM's and WebKit's included - # -- and threw the answer away. Each phase ended with five such walks: - # nvtop's two-second install sat for minutes on "scanning the tree". - # -prune does not descend. The pruned directory itself is still listed, - # exactly as the filter listed it. local r="${SNAP_ROOT%/}" - find "${SNAP_ROOT:-/}" -xdev \ - \( -path "$SRCROOT" -o -path "$r/dev" -o -path "$r/proc" \ - -o -path "$r/sys" -o -path "$r/run" -o -path "$r/tmp" \ - -o -path "$r/sources" -o -path "$r/build" \ - -o -path "$r/root" -o -path "$r/home" \) -type d -print -prune \ - -o -type d -print \ + find "${SNAP_ROOT:-/}" -xdev -type d \ + -not -path "$SRCROOT/*" \ + -not -path "$r/dev/*" -not -path "$r/proc/*" \ + -not -path "$r/sys/*" -not -path "$r/run/*" \ + -not -path "$r/tmp/*" -not -path "$r/sources/*" \ + -not -path "$r/build/*" \ + -not -path "$r/root/*" -not -path "$r/home/*" \ 2>/dev/null | LC_ALL=C sort } @@ -1084,13 +1078,13 @@ snapshot() { # since comm cannot tell. A package user's profile sets LC_ALL=POSIX and # root's shell may not, so the two sides really can differ. local r="${SNAP_ROOT%/}" - # pruned, not filtered -- see snapshot_dirs - find "${SNAP_ROOT:-/}" -xdev \ - \( -path "$SRCROOT" -o -path "$r/dev" -o -path "$r/proc" \ - -o -path "$r/sys" -o -path "$r/run" -o -path "$r/tmp" \ - -o -path "$r/sources" -o -path "$r/build" \ - -o -path "$r/root" -o -path "$r/home" \) -type d -prune \ - -o \( -type f -o -type l \) -print \ + find "${SNAP_ROOT:-/}" -xdev \( -type f -o -type l \) \ + -not -path "$SRCROOT/*" \ + -not -path "$r/dev/*" -not -path "$r/proc/*" \ + -not -path "$r/sys/*" -not -path "$r/run/*" \ + -not -path "$r/tmp/*" -not -path "$r/sources/*" \ + -not -path "$r/build/*" \ + -not -path "$r/root/*" -not -path "$r/home/*" \ 2>/dev/null | LC_ALL=C sort } @@ -2498,64 +2492,6 @@ unwritable_dirs_from_log() { | sort -u)" fi - # CARGO NAMES THE DESTINATION IN BACKTICKS, ON THE LINE ABOVE THE REASON. - # - # Error: CliError { error: Some(failed to copy `.../libgstrsrtp.a` to - # `/usr/lib/gstreamer-1.0/libgstrsrtp.a` - # Caused by: - # Permission denied (os error 13) - # - # No quotes, no "Permission denied" on the same line, and the path that - # matters is the SECOND backticked one -- so nothing matched, the repair - # ran zero rounds, and the box still said "grant access and retry". - local _cargodirs="" - if grep -qE "failed to copy \`|os error 13" "$log" 2>/dev/null; then - _cargodirs="$(tail -c 524288 "$log" 2>/dev/null \ - | grep -aoE 'to `/[^`]+`' \ - | sed 's/^to `//; s/`$//' \ - | sed 's|/[^/]*$||' \ - | sort -u)" - fi - - # AN ABSOLUTE PATH IN THE MESSAGE IS THE ANSWER, AND NOBODY READ IT. - # - # install: cannot create regular file - # '/usr/lib/udev/rules.d/90-backlight.rules': Permission denied - # - # The file does not exist, so the handover has nothing to give away; the - # directory is another package's and not writable, so the grant is what - # is needed -- and the `cd`-based extractor below only fires when the - # build printed a `cd` and then a RELATIVE path. With an absolute path - # right there in the message, nothing was extracted and light's udev - # rule stopped the install. - # CMAKE WRAPS ITS MESSAGE. - # - # file INSTALL cannot copy file - # "/usr/src/pkgusr/p_nvtop/.../io.github.syllo.nvtop.metainfo.xml" - # to "/usr/share/metainfo/io.github.syllo.nvtop.metainfo.xml": Permission - # denied. - # "Permission" and "denied" on two lines, so no line matched and the second - # round found nothing to grant: nvtop stopped after one round with a plain - # permission error on the screen. Join each `file INSTALL` block (up to - # the blank line CMake ends it with) and read the destination from that. - local _cmakedirs="" - _cmakedirs="$(tail -c 524288 "$log" 2>/dev/null | awk ' - /file INSTALL cannot/ { blk = $0; inb = 1; next } - inb && /^[[:space:]]*$/ { print blk; inb = 0; next } - inb { blk = blk " " $0 } - END { if (inb) print blk }' \ - | tr -s ' ' \ - | grep -aE 'Permission denied|Operation not permitted' \ - | grep -aoE '(to|directory) "/[^"]+"' | sed 's/^[a-z]* "//; s/"$//' \ - | while IFS= read -r _cp; do - if [ -d "$_cp" ]; then echo "$_cp"; else echo "${_cp%/*}"; fi - done | sort -u)" - local _instdirs="" - _instdirs="$(tail -c 524288 "$log" 2>/dev/null \ - | grep -aoE "cannot create (regular file|directory) '/[^']+'" \ - | sed "s/.*'\(\/[^']*\)'.*/\1/" \ - | sed 's|/[^/]*$||' | sort -u)" - # (plain awk: match() with a capture array is a gawk extension) _cdpaths="$(tail -c 524288 "$log" 2>/dev/null | awk ' /cd .\/|cd \x27\// { @@ -2570,9 +2506,6 @@ unwritable_dirs_from_log() { }' 2>/dev/null | sort -u)" { printf '%s\n' "$_mesondirs" - printf '%s\n' "$_cargodirs" - printf '%s\n' "$_instdirs" - printf '%s\n' "$_cmakedirs" printf '%s\n' "$_cdpaths" printf '%s\n' "$_tail" | grep -aoE "'[^']+'|\"[^\"]+\"" | tr -d "'\"" printf '%s\n' "$_tail" | grep -aoE "/[A-Za-z0-9._/+-]{6,}"; } \ @@ -2951,7 +2884,7 @@ meson_takeover_from_log() { # meson's chmod succeeds. Only files it is actually installing (named on # meson's own "Installing SRC to DIR" lines) and only where the current # owner is another PACKAGE USER: root's files are never taken. - local log="$1" user="$2" took=0 src dest f owner _t_owner="" _t_src="" + local log="$1" user="$2" took=0 src dest f owner grep -qE "Unhandled python OSError|\[Errno 13\]|\[Errno 1\]" "$log" 2>/dev/null || return 1 while IFS= read -r line; do src="${line#Installing }"; src="${src%% to *}" @@ -2966,47 +2899,8 @@ meson_takeover_from_log() { esac pkgusr_kind "$owner" >/dev/null 2>&1 || true say " $f belongs to $owner -- handing it to $user (the installer owns what it installs)" - soft "could not hand $f to $user" -- real_chown "$user:$user" "$f" \ - && { took=$((took+1)); _t_owner="$owner"; _t_src="$src"; } + soft "could not hand $f to $user" -- real_chown "$user:$user" "$f" && took=$((took+1)) done <<< "$(tail -c 2097152 "$log" 2>/dev/null | grep '^Installing .* to /')" - # THE WHOLE INSTALL PLAN, NOT THE LINE IT STOPPED ON. - # - # libglvnd replaces Mesa's GL, GLES, EGL and KHR headers and its libGL: - # a legitimate move of dozens of files from p_mesa to p_libglvnd. meson - # stops at the first one, so the log named one file per round, and after - # three the round limit called it a bundled subproject -- it was not. - # meson writes its complete plan to meson-info/intro-installed.json - # (source -> destination) at setup, so once one file has had to change - # hands, every destination still owned by that same package changes - # hands in this round. Same guards: package users only, never root. - # Not for a subproject's file: that is the bundled-copy case below, and - # handing its whole plan over would hide it. - if [ "$took" -ge 1 ] && [ -n "$_t_owner" ] \ - && [ "${_t_src#*/subprojects/}" = "$_t_src" ]; then - local _plan="" _dir _home _n=0 _d _o - # walk up from the file meson was installing to the tree that holds - # meson-info, and no further than the package user's home - _home="$(getent passwd "$user" | cut -d: -f6)" - _dir="$(dirname "$_t_src")" - while [ -n "$_dir" ] && [ "$_dir" != / ]; do - _plan="$(find "$_dir" -maxdepth 3 -path '*/meson-info/intro-installed.json' 2>/dev/null | head -1)" - [ -n "$_plan" ] && break - [ "$_dir" = "$_home" ] && break - _dir="$(dirname "$_dir")" - done - if [ -n "$_plan" ] && command -v python3 >/dev/null 2>&1; then - while IFS= read -r _d; do - [ -e "$_d" ] && [ ! -L "$_d" ] || continue - _o="$(stat -c %U "$_d" 2>/dev/null)" || continue - [ "$_o" = "$_t_owner" ] || continue - real_chown "$user:$user" "$_d" 2>/dev/null && _n=$((_n+1)) - done <<< "$(python3 -c 'import json,sys; print("\n".join(json.load(open(sys.argv[1])).values()))' "$_plan" 2>/dev/null)" - if [ "$_n" -gt 0 ]; then - say " ...and $_n more file(s) of $_t_owner that meson's install plan puts there" - took=$((took + _n)) - fi - fi - fi # ONE FILE PER ROUND IS A LOOP, NOT PROGRESS. # # meson stops at the FIRST file it may not replace, so its log names only @@ -3015,9 +2909,8 @@ meson_takeover_from_log() { # .mo file per round. When a package keeps taking a single file from the # same other package, it is installing a BUNDLED COPY of it, and the fix # is upstream of the ownership question. - # ...and only a file that comes FROM a subproject is a bundled copy. - if [ "$took" = 1 ] && [ -n "$_t_owner" ] && [ "${_t_src#*/subprojects/}" != "$_t_src" ]; then - LAST_SINGLE_HANDOVER="$_t_owner" + if [ "$took" = 1 ] && [ -n "${owner:-}" ]; then + LAST_SINGLE_HANDOVER="$owner" else LAST_SINGLE_HANDOVER="" fi @@ -3025,255 +2918,6 @@ meson_takeover_from_log() { } -# A RULE ON DISK IS NOT A RULE IN FORCE. -# -# light installs 90-backlight.rules, and until udevd is told, nothing has -# changed: /sys/class/backlight/*/brightness keeps its old group, light still -# cannot write it, and the only symptom is "it does not work" -- on a system -# where every file is exactly where it should be. A reboot fixes it, which -# is how this kind of thing stays hidden for a week. -# -# Reload always (cheap, and cannot break a running device), then re-trigger -# ONLY the subsystems the new rules actually mention. A bare `udevadm -# trigger` re-adds every device on the machine; on a desktop that can reset -# input devices and drop a session, which is not a thing a package install -# should do. -_udev_reload_if_rules_changed() { # <manifest> - local man="$1" rules subs - [ -r "$man" ] || return 0 - rules="$(grep -E '/udev/rules\.d/[^/]+\.rules$' "$man" 2>/dev/null)" - [ -n "$rules" ] || return 0 - if ! command -v udevadm >/dev/null 2>&1; then - note "# installed a udev rule, and there is no udevadm here to load it" - return 0 - fi - # IN A CHROOT THERE IS NO udevd TO TELL. Saying so beats a confusing - # "Failed to send reload request". - if [ "$(stat -c %d:%i /proc/1/root/. 2>/dev/null)" \ - != "$(stat -c %d:%i / 2>/dev/null)" ]; then - note "# a udev rule was installed; this is a chroot, so nothing was" - note "# reloaded -- it takes effect when you boot this tree" - return 0 - fi - say "# a udev rule changed -- telling udevd:" - if udevadm control --reload 2>/dev/null; then - say "# rules reloaded" - else - warn "udevadm control --reload failed; the rule is on disk but not in force" - return 0 - fi - # THE ACTION MATTERS AS MUCH AS THE SUBSYSTEM. - # - # `udevadm trigger` sends **change** unless told otherwise, and light's - # rule is `ACTION=="add", SUBSYSTEM=="backlight", RUN+="chgrp video ..."` - # -- so the trigger fired, udevd re-examined the device, the rule did not - # match, and /sys/class/backlight/*/brightness stayed root:root. Reload - # plus trigger LOOKED like it had done the job. - # - # So take the actions out of the rules too, and trigger each one. - acts="$(printf '%s\n' "$rules" | while IFS= read -r r; do - [ -r "$r" ] || continue - sed -n 's/.*ACTION=="\([a-z]*\)".*/\1/p' "$r" - done | sort -u | tr '\n' ' ')" - [ -n "$acts" ] || acts="change" - subs="$(printf '%s\n' "$rules" | while IFS= read -r r; do - [ -r "$r" ] || continue - sed -n 's/.*SUBSYSTEM=="\([a-z0-9_-]*\)".*/\1/p' "$r" - done | sort -u | tr '\n' ' ')" - if [ -z "$subs" ]; then - say "# no SUBSYSTEM== in the new rule(s), so nothing was re-triggered." - say "# If a device needs it: udevadm trigger --action=add --subsystem-match=<what>" - return 0 - fi - for sub in $subs; do - for act in $acts; do - if udevadm trigger --action="$act" --subsystem-match="$sub" 2>/dev/null; then - say "# re-triggered '$sub' with action '$act'" - else - warn "could not re-trigger '$sub' ($act); by hand:" - warn " udevadm trigger --action=$act --subsystem-match=$sub" - fi - done - done - udevadm settle --timeout=10 >/dev/null 2>&1 || true -} - -_missing_py_module() { # _missing_py_module <log> -> module name or "" - grep -aoE "Cannot import '[a-zA-Z0-9_.]+'|No module named '[a-zA-Z0-9_.]+'" \ - "$1" 2>/dev/null | head -n1 | cut -d"'" -f2 -} - -# A TEST'S DEPENDENCY IS NOT A BUILD'S DEPENDENCY, and installing it would -# have hidden a generator bug for two releases. numpy's install phase -# wrongly contained its test suite; the missing module was `pytest`, and an -# eager "install what is missing" would have installed pytest, run a -# multi-minute test suite inside an install, and left the real fault -- -# a test block in the wrong phase -- undiscovered. -_PY_TEST_ONLY="pytest py.test hypothesis attrs pytest_xdist pytest-xdist - nose coverage tox mock flaky" - -_py_module_provider() { # _py_module_provider <module> -> BLFS page or "" - case "$1" in - mesonpy) echo meson_python ;; - setuptools_scm) echo setuptools_scm ;; - Cython|cython) echo Cython ;; - numpy) echo NumPy ;; - flit_core) echo flit_core ;; - hatchling) echo hatchling ;; - scikit_build_core) echo scikit-build-core ;; - poetry.core|poetry_core) echo poetry-core ;; - wheel) echo wheel ;; - setuptools) echo setuptools ;; - *) echo "" ;; - esac -} - -# INSTALL A MISSING BUILD BACKEND, IF ASKED TO. Opt-in (--auto-modules or -# PM_AUTO_MODULES=1): a build that cannot proceed without a python backend -# is a dependency problem the tool can fix, but installing packages nobody -# asked for is not something to do by default -- and never for a module -# only the TEST suite wants. -_install_missing_module() { # _install_missing_module <log> - local mod prov - mod="$(_missing_py_module "$1")" - [ -n "$mod" ] || return 1 - case " $_PY_TEST_ONLY " in - *" $mod "*) - warn "'$mod' is a TEST dependency, not a build one -- NOT installing it" - note " a test suite in an install phase is the real fault here." - note " Regenerate the script and the test moves to \`test\`:" - note " packagemanager regenerate-script $(unprefix_pkg_user "${name:-}") --overwrite" - return 1 ;; - esac - prov="$(_py_module_provider "$mod")" - if [ -z "$prov" ]; then - warn "no known BLFS page provides the python module '$mod'" - note " find it: packagemanager search $mod" - return 1 - fi - say "# '$mod' is missing; installing $prov (asked for with --auto-modules) ..." - if packagemanager install "$prov" --run --yes --recursive; then - printf '%s\n' "$prov ($mod)" >> "${_AUTO_MODULES_LOG:-/dev/null}" - return 0 - fi - warn "installing $prov failed; $mod is still missing" - return 1 -} - -_take_cmake_dest() { # _take_cmake_dest <log> <package user> - # CMAKE NAMES THE DESTINATION ACROSS THREE LINES. - # - # CMake Error at tools/llvm-ar/cmake_install.cmake:52 (file): - # file INSTALL cannot copy file - # ".../build/bin/llvm-ar" to - # "/usr/bin/llvm-ar": Permission denied. - # - # /usr/bin is drwxrwxr-t root:install -- writable by the group and - # STICKY, so only the file's owner may replace it, and /usr/bin/llvm-ar - # belonged to p_rust. No grant fixes that: the directory was never the - # problem. The meson handover next door reads "Installing SRC to DIR" - # lines and needs "Unhandled python OSError", so it saw nothing either, - # and the repair ran zero rounds. - # - # Ownership follows the installer. llvm-ar is llvm's file; take it, and - # only from another PACKAGE USER -- root's files are never taken. - # the log is joined into one line before matching, because the message - # is three lines and the destination is on the third - local log="$1" user="${2:-}" dest owner took=0 - [ -r "$log" ] || return 1 - [ -n "$user" ] || return 1 - # ...AND A SYMLINK IT WILL NOT OVERWRITE AT ALL. - # - # file INSTALL cannot duplicate symlink - # ".../build/libtiff/libtiff.so.6" - # at - # "/usr/lib/libtiff.so.6" - # because: File exists - # - # Round 1 handed the .pc file over and the retry then died on this: the - # ownership was fine, cmake simply refuses to replace an existing - # symlink. Handing it over changes nothing -- it has to GO. Only - # another package user's, never root's, and only after saying whose it - # was. - # THE GATE HAS TO KNOW THE WORDING TOO. The extractor learned meson's - # "Operation not permitted: '<path>'" and this gate did not, so the - # function returned before looking -- the same shape as the 1.14.222 - # cmake detector: two places, one fact. - grep -qE "file INSTALL cannot (copy file|duplicate symlink)|Operation not permitted: '" \ - "$log" 2>/dev/null || return 1 - while IFS= read -r dest; do - [ -n "$dest" ] || continue - case "$dest" in /*) ;; *) continue ;; esac - [ -e "$dest" ] || continue - owner="$(stat -c %U "$dest" 2>/dev/null)" || continue - [ "$owner" = "$user" ] && continue - case "$owner" in - root|"") - # A ROOT FILE THIS PACKAGE'S MANIFEST CLAIMS (a setid bit we - # applied) is handed over BEFORE the phase by the lease, and - # restored if the phase fails. 1.14.261 did it here, after - # the failure, which needed this log's wording to match. - # Nothing root owns is taken here any more. - note "# $dest is root's -- left alone"; continue ;; - esac - if [ -L "$dest" ]; then - # cmake will not overwrite a symlink whatever it is owned by - say " $dest is $owner's symlink -- removing it ($user installs it now)" - rm -f "$dest" 2>/dev/null || { - warn "could not remove $dest"; continue; } - took=1 - continue - fi - say " $dest belongs to $owner -- handing it to $user (the installer owns what it installs)" - chown "$user:$user" "$dest" 2>/dev/null || { - warn "could not hand $dest to $user"; continue; } - took=1 - done <<EOF -$(tail -c 262144 "$log" 2>/dev/null | tr '\n' ' ' \ - | grep -aoE 'file INSTALL cannot copy file[[:space:]]+"[^"]+"[[:space:]]+to[[:space:]]+"[^"]+"' \ - | sed 's/.*[[:space:]]to[[:space:]]*"//; s/"$//' | sort -u -tail -c 262144 "$log" 2>/dev/null | tr '\n' ' ' \ - | grep -aoE 'cannot duplicate symlink[[:space:]]+[^[:space:]]+[[:space:]]+at[[:space:]]+[^[:space:]]+' \ - | awk '{print $NF}' | sort -u -# meson/python say it differently, and EPERM on a sticky directory is the -# same problem: "Operation not permitted: '/usr/sbin/unix_chkpwd'" -tail -c 262144 "$log" 2>/dev/null \ - | grep -aoE "Operation not permitted: '[^']+'" \ - | sed "s/.*'\([^']*\)'.*/\1/" | sort -u) -EOF - [ "$took" = 1 ] -} - -_clear_stale_dest() { # _clear_stale_dest <log> <package user> - # `mv`/`cp` refusing an existing destination is not a permission problem - # and no grant will fix it. Remove the stale copy IF the package being - # installed owns it, and say so; anything else is left alone, because a - # destination owned by someone else is a real conflict. - local log="$1" user="${2:-}" path owner cleared=0 - [ -r "$log" ] || return 1 - [ -n "$user" ] || return 1 - while IFS= read -r path; do - [ -n "$path" ] || continue - case "$path" in /*) ;; *) continue ;; esac - [ -e "$path" ] || continue - owner="$(stat -c %U "$path" 2>/dev/null)" - if [ "$owner" != "$user" ]; then - warn "$path exists and belongs to $owner, not $user --" - warn " leaving it alone: that is a conflict, not stale output" - continue - fi - note "# $path is $user's own output from a previous install;" - note "# removing it so this install can replace it" - rm -rf "$path" || { warn "could not remove $path"; continue; } - cleared=1 - done <<EOF -$(tail -c 262144 "$log" 2>/dev/null \ - | grep -aoE "cannot overwrite '[^']+'|cannot move '[^']+' to '[^']+': Directory not empty" \ - | grep -aoE "'[^']+'" | tr -d "'" | sort -u) -EOF - [ "$cleared" = 1 ] -} - auto_grant_from_log() { local log="$1" user="$2" need d granted=0 need="$(unwritable_dirs_from_log "$log" "$user")" @@ -3666,181 +3310,6 @@ pkg_users_list() { done < <(pkgusr_roots) } -# --------------------------------------------------------------------------- -# setuid/setgid bits: recorded during the build, applied by root afterwards. -# -# A PACKAGE USER OWNS FILES, NOT IDENTITY. The install and chmod wrappers -# refuse setid modes, which is right -- a package user CAN set setuid on its -# own file, and the result runs as the PACKAGE USER, not as root. That is -# how `su` came to say -# setgid: Operation not permitted -# instead of working: shadow's `install -m 4755 su` was downgraded to 755, -# and nothing set it afterwards. -# -# So: chown to root FIRST, then the mode. Both, or neither. -# --------------------------------------------------------------------------- - -# What upstream installs with a setid bit, for the case where no request was -# recorded (an older build, or a package installed by hand). -# path|mode|group|who asks for it -- checked only if the file exists, and -# always reported with the mode and owner actually found. -# -# `-` IN THE GROUP COLUMN MEANS KEEP WHAT IS THERE, and for a setUID file -# that is the right answer on this system: only the owner has to be root for -# the bit to mean root, and the group records which package installed the -# file. These all said `root`, so this table ran immediately after the -# request table and undid it -- `/usr/bin/su is now 4755 root:p_shadow` -# followed by `/usr/bin/su is now 4755 root:root`, seventeen changes to -# reach the wrong one. -# -# A setGID entry is different: THERE the group is the whole point -# (dbus-daemon-launch-helper must be group messagebus, fcrontab group -# fcron), so those keep a named group. -_setid_known() { - # ONE TABLE: setid.conf. This was a second list, embedded here, and it - # disagreed with setid.conf on the mode -- 4755 here, 4711 there -- so - # `cfg shadow-pam` set 4711 and the next `lfs-helper setid --run` set - # 4755, each correcting the other (the 1.14.260 shape again). Now the - # config file is the list and this reads it. Only root-owned lines are - # setid policy. Output: path|mode|group|why - local conf - if ! conf="$(_setid_conf)"; then - warn "no setid.conf found (looked in \$SETID_CONF, ${PKGUSR_ETC:-/etc/pkgusr}/stacks, beside lfs-helper)" - return 0 - fi - awk -F'|' -v src="$(basename "$conf")" ' - /^[[:space:]]*(#|$)/ { next } - { f = $1; m = $2; o = $3; g = $4 - gsub(/[[:space:]]/, "", f); gsub(/[[:space:]]/, "", m) - gsub(/[[:space:]]/, "", o); gsub(/[[:space:]]/, "", g) - if (f == "" || m == "") next - if (o != "" && o != "root") next - print f "|" m "|" (g == "" ? "-" : g) "|" src }' "$conf" -} - -_setid_state() { # -> "<mode> <owner>:<group>" - stat -c '%a %U:%G' "$1" 2>/dev/null -} - -_setid_apply() { # <mode> <group> <path> (root; reads the result back) - local mode="$1" grp="$2" f="$3" got="" - chown "root:$grp" "$f" || { warn " chown root:$grp $f failed"; return 1; } - chmod "$mode" "$f" || { warn " chmod $mode $f failed"; return 1; } - # CONFIRM YOUR OWN EFFECTS: chmod returns 0 on a nosuid mount too, and - # the bit is then not there. - got="$(_setid_state "$f")" - case "$got" in - "${mode#0} root:$grp"|"$mode root:$grp") - ok " $f is now $got"; return 0 ;; - esac - warn " $f still reads '$got', not '$mode root:$grp'" - warn " (a nosuid mount cannot hold a setid bit -- check: findmnt -T $f)" - return 1 -} - -cmd_setid() { - need_root - local run=0 a - for a in "$@"; do case "$a" in --run) run=1 ;; esac; done - local outstanding=0 applied=0 known=0 line mode grp why got home u f - - say "Setuid/setgid bits refused during the build (a package user cannot" - say "set them; root can):" - say "" - while IFS= read -r u; do - home="$(pkgusr_home_for "$u" 2>/dev/null)" || continue - # a ledger left root-owned by an older release blocks the NEXT - # build, not this one, so repair it whenever we are here anyway - for _lf in "$home/.pkgusr-setid.requests" "$home/.pkgusr-setid.applied"; do - [ -e "$_lf" ] || continue - [ "$(stat -c %U "$_lf" 2>/dev/null)" = "$u" ] && continue - chown "$u:$u" "$_lf" 2>/dev/null \ - && note "gave $u back $(basename "$_lf")" - done - [ -s "$home/.pkgusr-setid.requests" ] || continue - say " $u:" - while IFS= read -r line; do - [ -n "$line" ] || continue - # the recorded line is literally `chmod <mode> <path> [path ...]` - set -- $line - [ "${1:-}" = chmod ] || continue - mode="${2:-}"; shift 2 - for f in "$@"; do - case "$f" in -*) continue ;; esac - if [ ! -e "$f" ]; then - warn " $f: recorded, but not on disk (package removed?)" - continue - fi - got="$(_setid_state "$f")" - say " $f is $got, asked for $mode" - outstanding=$((outstanding+1)) - # KEEP THE GROUP THE FILE ALREADY HAS. Only the USER has - # to be root for a setuid bit to mean root; the group is how - # this system records which package owns a file, and - # root:root threw that away -- `/usr/bin/su is now 4755 - # root:root` when it should read root:p_shadow. setid.conf - # uses `-` for exactly this; the request path did not. - _grp_now="$(stat -c '%G' "$f" 2>/dev/null)" - [ -n "$_grp_now" ] || _grp_now=root - # A PATH setid.conf NAMES IS DECIDED THERE. The package asked - # for 4755 on su, the table says 4711; applying both flipped - # the file twice in one run. - if _setid_policy_for "$f" >/dev/null; then - say " (setid.conf decides this one -- see below)" - continue - fi - [ "$run" = 1 ] && { _setid_apply "$mode" "$_grp_now" "$f" \ - && applied=$((applied+1)); } - done - done < "$home/.pkgusr-setid.requests" - if [ "$run" = 1 ]; then - # moved aside, not deleted: what was applied stays answerable - cat "$home/.pkgusr-setid.requests" >> "$home/.pkgusr-setid.applied" 2>/dev/null - : > "$home/.pkgusr-setid.requests" 2>/dev/null || true - # ...and hand both back: they live in the package user's home and - # the next build appends to them as that user - chown "$u:$u" "$home/.pkgusr-setid.requests" \ - "$home/.pkgusr-setid.applied" 2>/dev/null || true - fi - done < <(pkg_users_list) - [ "$outstanding" = 0 ] && say " (none recorded)" - - say "" - say "Programs upstream installs with a setid bit, as found on disk:" - say "" - while IFS='|' read -r f mode grp why; do - [ -n "$f" ] || continue - [ -e "$f" ] || continue - got="$(_setid_state "$f")" - # `-` MEANS KEEP THE GROUP THE FILE HAS. With `root` hard-coded - # here, this table undid the request table one line later. - if [ "$grp" = "-" ]; then - grp="$(stat -c '%G' "$f" 2>/dev/null)" - [ -n "$grp" ] || grp=root - fi - [ "$got" = "${mode#0} root:$grp" ] && continue - known=$((known+1)) - say " $f" - say " is $got" - say " want $mode root:$grp ($why)" - [ "$run" = 1 ] && { _setid_apply "$mode" "$grp" "$f" \ - && applied=$((applied+1)); } - done < <(_setid_known) - [ "$known" = 0 ] && say " (each one is already root-owned with its bit)" - - say "" - if [ "$run" = 1 ]; then - say "Applied $applied change(s). Try what was broken: su -" - return 0 - fi - if [ $((outstanding + known)) -gt 0 ]; then - say "Nothing was changed. Apply it: lfs-helper setid --run" - return 1 - fi - ok "every setid bit this system needs is in place" - return 0 -} - cmd_find_user_site() { need_root local run=0; [ "${1:-}" = "--run" ] && run=1 @@ -4045,22 +3514,7 @@ EOF # install: cannot change permissions of '/usr/sbin': Operation not permitted # The directory is already there and correctly set up, so there is nothing to do. real=/usr/bin/install - -# CHECK BEFORE REDIRECTING. `printf >> file 2>/dev/null || echo ...` does -# not work: the shell itself reports the failed redirection ("Permission -# denied") before printf ever runs, and 2>/dev/null on printf cannot -# suppress it -- which is how the recorder's own message came to fail an -# install that had finished. -_sid_record() { # _sid_record <line> - local f="$HOME/.pkgusr-setid.requests" - if [ -w "$f" ] || { [ ! -e "$f" ] && [ -w "$HOME" ]; }; then - printf '%s\n' "$1" >> "$f" - else - echo "*** (could not record: $1)" >&2 - echo "*** apply it as root instead: lfs-helper setid --run" >&2 - fi -} -args=(); skipped=""; _setid_mode="" +args=(); skipped="" # BUNDLED SHORT OPTIONS FIRST, or none of the rules below see what they are # looking for. Everything here matched `-d` only as a whole argument, and @@ -4135,16 +3589,7 @@ while [ $# -gt 0 ]; do -m|--mode) case "$2" in [2467][0-7][0-7][0-7]) # setuid/setgid/sticky requested - # WRITE DOWN WHAT WAS REFUSED -- the chmod wrapper does, - # and this is the same refusal. shadow installs su with - # `install -m 4755`, so the bit was dropped here and - # nothing recorded it; months later `su -` failed with - # setgid: Operation not permitted - # which names neither su's mode nor its owner. The - # targets are known after the parse loop, so the mode is - # remembered here and written at the end. - _setid_mode="$2" - echo "*** install -m $2 (using 755 instead; apply it as root: lfs-helper setid --run)" >&2 + echo "*** install -m $2 (using 755 instead; set the bit as root if you want it)" >&2 args+=(-m 755); shift 2 ;; *) args+=("$1" "$2"); shift 2 ;; esac ;; @@ -4210,32 +3655,7 @@ case "$_dest" in exit 0 fi ;; esac -# RECORD THE REFUSED BIT AGAINST THE FILE THAT ACTUALLY LANDED. -# -# Only after the install succeeded: a request naming a file that was never -# installed sends root to chmod something that is not there. -if [ -z "$_setid_mode" ]; then - exec "$real" "${args[@]}" -fi -"$real" "${args[@]}" || exit $? -_f="$HOME/.pkgusr-setid.requests" -_last="${args[$((${#args[@]} - 1))]}" -_srcs2=(); _n2=${#args[@]} -for ((_i = 0; _i < _n2 - 1; _i++)); do - case "${args[$_i]}" in -*) continue ;; esac - if [ "$_i" -gt 0 ]; then - case "${args[$((_i - 1))]}" in -m|--mode|-o|--owner|-g|--group|-t|-S) continue ;; esac - fi - _srcs2+=("${args[$_i]}") -done -if [ -d "$_last" ]; then - for _s in ${_srcs2[@]+"${_srcs2[@]}"}; do - _sid_record "chmod $_setid_mode ${_last%/}/$(basename "$_s")" - done -else - _sid_record "chmod $_setid_mode $_last" -fi -exit 0 +exec "$real" "${args[@]}" EOF # A SERVICE ACCOUNT IS ROOT'S JOB. @@ -4286,21 +3706,6 @@ EOF #!/bin/bash # Allow everything except setuid/setgid, which a package user must not set. real=/usr/bin/chmod - -# CHECK BEFORE REDIRECTING. `printf >> file 2>/dev/null || echo ...` does -# not work: the shell itself reports the failed redirection ("Permission -# denied") before printf ever runs, and 2>/dev/null on printf cannot -# suppress it -- which is how the recorder's own message came to fail an -# install that had finished. -_sid_record() { # _sid_record <line> - local f="$HOME/.pkgusr-setid.requests" - if [ -w "$f" ] || { [ ! -e "$f" ] && [ -w "$HOME" ]; }; then - printf '%s\n' "$1" >> "$f" - else - echo "*** (could not record: $1)" >&2 - echo "*** apply it as root instead: lfs-helper setid --run" >&2 - fi -} me="$(id -un)" for a in "$@"; do case "$a" in @@ -4315,8 +3720,7 @@ for a in "$@"; do # its setgid bit at install time. A refusal nobody records is a # decision that has to be re-derived from its consequences. echo "*** chmod $* (skipped: refuses to set setuid/setgid; do it as root if needed)" >&2 - # A RECORDER THAT CANNOT WRITE IS A NOTE, NOT A FAILURE. - _sid_record "chmod $*" + printf 'chmod %s\n' "$*" >> "$HOME/.pkgusr-setid.requests" 2>/dev/null exit 0 ;; esac done @@ -5012,26 +4416,13 @@ _claim_candidates() { # <owner> <manifest> ; paths on stdout printf '%s\n' "$r$p" done < "$man" > "$tmp" # stat in batches: a path that does not exist simply produces no line - # A SETUID FILE IS NEVER CLAIMED. - # - # ~ > su - - # su: Authentication service cannot retrieve authentication info - # - # This ran before EVERY phase, configure included, and shadow's manifest - # lists /usr/bin/su: "gave 'p_shadow' 16 path(s)" was su, passwd, chage - # and the rest going 4711 root -> 755 p_shadow, because chown clears the - # bit. Nothing put them back -- the lease only sees a file someone - # ELSE owns, and after this p_shadow owned them. Root's setid files are - # the lease's to hand over: it copies them aside, restores them on - # failure and applies setid.conf on success. - xargs -d '\n' -r -a "$tmp" stat -c '%U %A %n' -- 2>/dev/null | awk -v o="$owner" ' + xargs -d '\n' -r -a "$tmp" stat -c '%U %n' -- 2>/dev/null | awk -v o="$owner" ' { - u = $1; a = $2 - i = index($0, " "); rest = substr($0, i + 1) - j = index(rest, " "); path = substr(rest, j + 1) + u = $1 + i = index($0, " ") + path = substr($0, i + 1) if (u == o) next if (u != "root" && u != "lfs" && u != "UNKNOWN") next - if (substr(a, 4, 1) ~ /[sS]/ || substr(a, 7, 1) ~ /[sS]/) next print path }' rm -f "$tmp" @@ -5076,30 +4467,8 @@ _claim_earlier_stages() { detail "# claiming files from earlier stages of $owner ..." local chownbin; chownbin="$(_real_tool chown)" local batch; batch="$(mktemp)" || return 0 - # AN EARLIER STAGE IS EARLIER THAN THIS PACKAGE'S OWN INSTALLS. - # - # Chapter 7 builds gettext, perl, Python ... as root under the same names - # chapter 8 uses, so the step's OWN manifest holds root-owned leftovers - # the first package-user build must take over. After that first - # successful install, a root-owned entry in its own manifest is not a - # leftover -- it is something recorded by mistake: LLVM's listed - # /usr/bin/lfs, /usr/bin/packagemanager and /etc/pam.d/su_u_user (root - # wrote them during its hours-long install window), and this step - # "gave 'p_llvm' 13 path(s)", making a script root runs on every su - # writable by p_llvm. The -tmp/-pass/libstdcpp stages are always earlier. - local _home_done="" - _home_done="$(pkgusr_home_for "$(unprefix_pkg_user "$owner")" 2>/dev/null)" - local _installed_before=0 - [ -n "$_home_done" ] && { [ -s "$_home_done/.pkgusr-installed" ] || [ -s "$_home_done/VERSION" ]; } \ - && _installed_before=1 for base in $seen; do man="$MANIFESTS/$base.files" - if [ "$_installed_before" = 1 ]; then - case "$base" in - *-tmp|*-pass1|*-pass2|libstdcpp|libstdc++) ;; - *) continue ;; - esac - fi if [ -s "$man" ]; then _claim_candidates "$owner" "$man" > "$batch" if [ -s "$batch" ]; then @@ -5272,354 +4641,6 @@ EOF return 0 } -# --------------------------------------------------------------------------- -# THE LEASE: files a phase will overwrite that someone else owns. (1.14.263) -# -# The old way was fail, read the log, repair, retry. Every tool words the -# refusal differently -- cmake, meson, python, a bash redirect -- and each -# needed its own parser. Shadow's configure died on -# line 153: /etc/pam.d/su: Permission denied -# which no parser recognised, so the round granted the DIRECTORY, which was -# never the problem, and stopped. And the setid pass and the 1.14.261 -# take-back were two halves of one cycle: root gets the file for its bit, -# the next install cannot replace it, a repair hands it back. -# -# So, around every install/configure phase, as root: -# before copy each file the phase will write that another account owns, -# with its owner and mode, and hand it to the installer. COPY, -# never move: the phase itself starts with `su - p_<pkg>`, and -# su, its pam file and unix_chkpwd must exist at that moment. -# failure put every copy back, owner and mode included, and read it back. -# success a setid.conf file gets its policy; a file the phase did not -# change goes back to its owner; a file it rewrote belongs to the -# installer (the rule everywhere else). -# -# WHICH FILES, before anything runs: -# * install/all/update: the package's own manifest -- ROOT-owned entries -# only. Those are files it installed that root took for a setid bit. -# * configure/all/update: the paths configure_pkg writes, read from the -# script (script_write_targets). -# A root-owned file is leased ONLY when this package's manifest claims it. -# The script is editable by the package user; without that rule, one edited -# line could hand it /etc/shadow. Files of non-package accounts (the main -# user, u_*) are never leased. The first install of a file another package -# owns cannot be known in advance, so the repair loop stays as the fallback. -# --------------------------------------------------------------------------- -LEASES="$STATE/leases" - -# script_write_targets <script> [function ...] -> absolute paths, one a line -# -# What a shell function WRITES, read from its text: redirect targets -# (> >> and tee), the destination of install/cp/mv/ln, the files of sed -i. -# A `for VAR in a b c` loop expands ${VAR}/$VAR; any other expansion, a -# glob or a brace is dropped rather than guessed. Comments and heredoc -# BODIES are skipped -- a heredoc body is data, and "prose is not code". -script_write_targets() { - local script="$1"; shift - [ -r "$script" ] || return 1 - [ $# -gt 0 ] || set -- configure_pkg - local fn - for fn in "$@"; do - sed -n "/^${fn}() {/,/^}/p" "$script" - done | awk ' - function emit(p, n, i, x, v, out) { - gsub(/^["\047]|["\047]$/, "", p) - if (p !~ /^\//) return - if (p ~ /[*?[{}]/) { - # ${VAR} is the one brace we understand - if (p !~ /\$\{[A-Za-z_][A-Za-z_0-9]*\}/) return - tmp = p; gsub(/\$\{[A-Za-z_][A-Za-z_0-9]*\}/, "", tmp) - if (tmp ~ /[*?[{}]/) return - } - if (p ~ /\$/) { - for (v in lists) { - if (index(p, "${" v "}") || p ~ ("\\$" v "([^A-Za-z_0-9]|$)")) { - n = split(lists[v], x, " ") - for (i = 1; i <= n; i++) { - out = p - gsub("\\$\\{" v "\\}", x[i], out) - gsub("\\$" v, x[i], out) - if (out !~ /\$/) emit(out) - } - return - } - } - return - } - if (p ~ /^\/(dev|proc|sys|tmp|run)(\/|$)/) return - print p - } - # skip a heredoc body until its delimiter - hd != "" { d = $0; if (strip) sub(/^\t+/, "", d); if (d == hd) hd = ""; next } - /^[[:space:]]*#/ { next } - { - buf = buf $0 - if (sub(/\\$/, "", buf)) next - line = buf; buf = "" - # the opener line still counts; its BODY does not - if (match(line, /<<-?[[:space:]]*["\047]?[A-Za-z_][A-Za-z_0-9]*["\047]?/)) { - h = substr(line, RSTART, RLENGTH) - strip = (h ~ /^<<-/) - gsub(/^<<-?[[:space:]]*|["\047]/, "", h) - hd = h - line = substr(line, 1, RSTART - 1) " " substr(line, RSTART + RLENGTH) - } - # redirect targets (not 2> / &> / <>, and not >&2) - l = line - while (match(l, /(^|[^0-9&<>])>>?[[:space:]]*[^[:space:];|&<>()]+/)) { - t = substr(l, RSTART, RLENGTH); l = substr(l, RSTART + RLENGTH) - sub(/^[^>]*>>?[[:space:]]*/, "", t) - if (t !~ /^&/) emit(t) - } - nseg = split(line, seg, /&&|\|\||;|\|/) - for (s = 1; s <= nseg; s++) { - # drop empty fields: a segment ending in a space splits into - # a trailing "" that would become the "last word" - nn = split(seg[s], ww, /[[:space:]]+/); n = 0 - for (j = 1; j <= nn; j++) if (ww[j] != "") w[++n] = ww[j] - k = 1; while (k <= n && (w[k] ~ /^(do|then|else|elif|!|sudo|\{|\()$/)) k++ - if (k > n) continue - cmd = w[k] - if (cmd == "for" && k + 2 <= n && w[k+2] == "in") { - v = w[k+1]; lists[v] = "" - for (j = k + 3; j <= n; j++) { x1 = w[j]; sub(/;$/, "", x1); if (x1 != "" && x1 != "do") lists[v] = lists[v] " " x1 } - continue - } - if (cmd ~ /^(install|cp|mv|ln)$/) { - last = ""; isdir = 0; tdir = "" - for (j = k + 1; j <= n; j++) { - if (w[j] ~ /^>/) break - if (w[j] == "-t" && j < n) { tdir = w[j+1]; j++; continue } - if (cmd == "install" && w[j] ~ /^-[A-Za-z]*d/) isdir = 1 - if (w[j] ~ /^-/) { - # install -m 644 / -o x / -g y take a value - if (cmd == "install" && w[j] ~ /^-[mog]$/) j++ - continue - } - last = w[j] - } - if (!isdir && tdir == "" && last != "") emit(last) - continue - } - if (cmd == "tee") { - for (j = k + 1; j <= n; j++) if (w[j] !~ /^-/) emit(w[j]) - continue - } - if (cmd == "sed") { - inplace = 0; havescript = 0 - for (j = k + 1; j <= n; j++) { - if (w[j] ~ /^-i/ || w[j] == "--in-place") { inplace = 1; continue } - if (w[j] == "-e" || w[j] == "-f") { havescript = 1; j++; continue } - if (w[j] ~ /^-/) continue - if (!havescript) { havescript = 1; continue } - if (inplace) emit(w[j]) - } - } - } - }' | awk '!seen[$0]++' -} - -cmd_script_targets() { - local script="${1:-}" - case "$script" in - ""|-h|--help) - say "lfs-helper script-targets <install script> [function ...]" - say "" - say "Lists the absolute paths the named functions write (default" - say "configure_pkg): redirects, tee, install/cp/mv/ln destinations," - say "sed -i. for-loops are expanded; other expansions are dropped." - say "This is the list the lease reads before a phase runs." - [ -n "$script" ]; return ;; - esac - shift - [ -r "$script" ] || die "script-targets: cannot read $script" - script_write_targets "$script" "$@" -} - -# the single policy table: setid.conf, wherever this install keeps it -_setid_conf() { - local c - for c in "${SETID_CONF:-}" "${PKGUSR_ETC:-/etc/pkgusr}/stacks/setid.conf" \ - "$(dirname "$(readlink -f "$0" 2>/dev/null || echo "$0")")/stacks/setid.conf"; do - [ -n "$c" ] && [ -r "$c" ] && { printf '%s\n' "$c"; return 0; } - done - return 1 -} - -# _setid_policy_for <path> -> "mode|owner|group" from setid.conf, or nothing -_setid_policy_for() { - local conf; conf="$(_setid_conf)" || return 1 - awk -F'|' -v p="$1" ' - /^[[:space:]]*(#|$)/ { next } - { f = $1; gsub(/[[:space:]]/, "", f) - if (f == p) { m = $2; o = $3; g = $4 - gsub(/[[:space:]]/, "", m); gsub(/[[:space:]]/, "", o); gsub(/[[:space:]]/, "", g) - print m "|" (o == "" ? "root" : o) "|" (g == "" ? "-" : g); found = 1; exit } } - END { exit !found }' "$conf" -} - -_lease_dir() { printf '%s/%s\n' "$LEASES" "$(unprefix_pkg_user "$1")"; } - -# _lease_candidates <owner> <script> <phase> -_lease_candidates() { - local owner="$1" script="$2" phase="$3" pkg man - pkg="$(unprefix_pkg_user "$owner")" - man="$MANIFESTS/$pkg.files" - case "$phase" in - install|all|update) - if [ -r "$man" ]; then - # A PACKAGE RECLAIMS ITS OWN FILES. Every manifest entry - # someone else owns now -- root (a setid bit applied) or - # another package (a later install took it) -- is leased. - # Only root's took this path, so shadow's own file held by - # another package still failed on a bash redirect or cp, - # whose wording no repair parses. lease_files keeps the - # guards: package accounts and manifest-claimed root only. - grep -v '^[[:space:]]*$' "$man" | xargs -r -d '\n' stat -c '%U|%n' 2>/dev/null \ - | awk -F'|' -v o="$owner" '$1 != o { sub(/^[^|]*\|/, ""); print }' - fi ;; - esac - case "$phase" in - configure|all|update) script_write_targets "$script" configure_pkg ;; - esac -} - -# lease_files <owner> <script> <phase> (root; before the phase) -lease_files() { - local owner="$1" script="$2" phase="$3" dir ledger n=0 p u g m accounts - [ "$(id -u)" = 0 ] || return 0 - case "$phase" in install|configure|all|update) ;; *) return 0 ;; esac - user_exists "$owner" || return 0 - dir="$(_lease_dir "$owner")"; ledger="$dir/ledger" - # A LEDGER LEFT BEHIND means the last phase never got to settle (killed, - # power cut). Its copies are the only record of what was there: restore - # them first, then lease afresh. - if [ -s "$ledger" ]; then - note "# $owner: a lease from an interrupted run is still open -- restoring it first" - lease_restore "$owner" || { - warn "!! could not restore the old lease in $dir -- not leasing again" - return 1; } - fi - local man="$MANIFESTS/$(unprefix_pkg_user "$owner").files" - accounts="$(pkg_users_list 2>/dev/null)" - mkdir -p "$dir/files" 2>/dev/null && chmod 700 "$LEASES" "$dir" 2>/dev/null - : > "$ledger" || { warn "!! cannot write $ledger -- no lease"; return 1; } - local _cfg_targets="" - case "$phase" in - configure|all|update) _cfg_targets="$(script_write_targets "$script" configure_pkg 2>/dev/null)" ;; - esac - while IFS= read -r p; do - [ -n "$p" ] || continue - [ -e "$p" ] || [ -L "$p" ] || continue - [ -d "$p" ] && [ ! -L "$p" ] && continue - grep -qF "$p|" "$ledger" 2>/dev/null && continue - # NO SYMLINK IN THE WAY. If a directory on the path is a link a - # package user controls, the name the manifest vouches for is not - # the file that would be handed over. - if [ "$(cd "$(dirname "$p")" 2>/dev/null && pwd -P)/$(basename "$p")" != "$p" ]; then - detail "# $p goes through a symlinked directory -- left alone" - continue - fi - u="$(stat -c %U "$p" 2>/dev/null)" || continue - [ "$u" = "$owner" ] && continue - if [ "$u" = root ]; then - if ! grep -qxF "$p" "$man" 2>/dev/null; then - detail "# $p is root's and $owner's manifest does not claim it -- left alone" - continue - fi - # A MANIFEST CLAIM IS NOT ENOUGH FOR ROOT'S FILES. Manifests are - # built by time window, and a long phase swallowed what root - # wrote meanwhile -- LLVM's claimed /usr/bin/lfs and - # /etc/pam.d/su_u_user, and the lease handed them to p_llvm. A - # package's file is root's for two reasons only: a setid bit - # (the policy put it there) or its own configure commands write - # it. Anything else root owns is not the package's to borrow. - if ! { [ -u "$p" ] || [ -g "$p" ] || _setid_policy_for "$p" >/dev/null 2>&1 \ - || printf '%s\n' "$_cfg_targets" | grep -qxF "$p"; }; then - detail "# $p is root's, not setid and not written by $owner's configure -- left alone" - continue - fi - elif ! printf '%s\n' "$accounts" | grep -qxF "$u"; then - detail "# $p belongs to $u, not a package account -- left alone" - continue - fi - g="$(stat -c %G "$p")"; m="$(stat -c %a "$p")" - n=$((n + 1)) - cp -a "$p" "$dir/files/$n" 2>/dev/null || { - warn "!! could not copy $p aside -- not leasing it"; continue; } - printf '%s|%s|%s|%s|%s\n' "$p" "$m" "$u" "$g" "$n" >> "$ledger" - if chown -h "$owner:$owner" "$p" 2>/dev/null; then - detail "# leased $p ($m $u:$g) to $owner for this phase" - else - warn "!! could not hand $p to $owner" - fi - done < <(_lease_candidates "$owner" "$script" "$phase") - [ -s "$ledger" ] || rm -rf "$dir" 2>/dev/null - return 0 -} - -# lease_restore <owner> -- failure: every copy back, owner and mode, read back -lease_restore() { - local owner="$1" dir ledger p m u g n bad=0 got want - dir="$(_lease_dir "$owner")"; ledger="$dir/ledger" - [ -s "$ledger" ] || return 0 - while IFS='|' read -r p m u g n; do - [ -n "$p" ] || continue - rm -f "$p" 2>/dev/null - if cp -a "$dir/files/$n" "$p" 2>/dev/null; then - chown -h "$u:$g" "$p" 2>/dev/null - [ -L "$p" ] || chmod "$m" "$p" 2>/dev/null - fi - got="$(stat -c '%a %U:%G' "$p" 2>/dev/null)" - want="$m $u:$g" - [ -L "$p" ] && { got="${got#* }"; want="$u:$g"; } - if [ "$got" = "$want" ] && cmp -s "$dir/files/$n" "$p" 2>/dev/null \ - || { [ -L "$p" ] && [ "$(readlink "$p")" = "$(readlink "$dir/files/$n")" ]; }; then - detail "# restored $p ($want)" - else - warn "!! $p did not come back as it was ($got, wanted $want)" - warn " the original is still at $dir/files/$n" - bad=1 - fi - done < "$ledger" - [ "$bad" = 0 ] && rm -rf "$dir" - return "$bad" -} - -# lease_settle <owner> -- success -lease_settle() { - local owner="$1" dir ledger p m u g n pol pm po pg - dir="$(_lease_dir "$owner")"; ledger="$dir/ledger" - [ -s "$ledger" ] || return 0 - while IFS='|' read -r p m u g n; do - [ -n "$p" ] || continue - if [ ! -e "$p" ] && [ ! -L "$p" ]; then - note "# $p was leased and the phase removed it" - continue - fi - if pol="$(_setid_policy_for "$p")"; then - IFS='|' read -r pm po pg <<<"$pol" - [ "$pg" = "-" ] && pg="$(stat -c %G "$p")" - # _setid_apply is root:<group> + mode, read back; the owner - # column is root for every setid line, and anything else is - # not a setid policy this code should apply - if [ "$po" = root ]; then - _setid_apply "$pm" "$pg" "$p" >/dev/null || warn "!! $p: setid.conf policy did not apply" - continue - fi - fi - if cmp -s "$dir/files/$n" "$p" 2>/dev/null; then - # the phase did not change it: the lease was not needed - chown -h "$u:$g" "$p" 2>/dev/null - [ -L "$p" ] || chmod "$m" "$p" 2>/dev/null - detail "# $p unchanged -- back to $u:$g" - else - say "# $p rewritten by $owner -- it owns it now (was $u:$g)" - fi - done < "$ledger" - rm -rf "$dir" -} - report_setid_requests() { # Printed after the phase, by root, because only root can act on it. local owner="${1:-}" home f @@ -5627,32 +4648,6 @@ report_setid_requests() { home="$(pkgusr_home_for "$owner" 2>/dev/null)" f="$home/.pkgusr-setid.requests" [ -s "$f" ] || return 0 - # APPLY THEM, DO NOT RECITE THEM. - # - # This function runs AS ROOT, right after the install -- the one moment - # when the thing that is missing can simply be done. It printed nine - # chmod lines for shadow and left `su` non-setuid until somebody typed - # them, which is the same "warn and walk away" that - # /etc/pam.d/su_u_user and the mode of that file were both guilty of. - # - # `lfs-helper setid --run` is the routine that already knows how: it - # applies each recorded request and READS THE MODE BACK. Call it, and - # report what it did rather than what the reader should do. - if [ "$(id -u)" = 0 ]; then - warn "" - warn "This package asked for setuid/setgid bits, which a package user" - warn "cannot set -- applying them now (root):" - while IFS= read -r line; do - [ -n "$line" ] && warn " $line" - done < "$f" - if cmd_setid --run; then - warn " applied. 'lfs-helper setid' lists anything still outstanding." - else - warn " SOME COULD NOT BE APPLIED -- see above; the requests are kept." - warn " retry at any time: lfs-helper setid --run" - fi - return 0 - fi warn "" warn "This package asked for setuid/setgid bits, which a package user" warn "cannot set. Nothing else refuses them; the binaries are installed" @@ -5662,14 +4657,7 @@ report_setid_requests() { while IFS= read -r line; do [ -n "$line" ] && warn " $line" done < "$f" - warn "" - warn "All of them at once, at any later time:" - warn " lfs-helper setid # what is outstanding" - warn " lfs-helper setid --run # apply it" - # NOT CLEARED HERE. Printing once and emptying the file made this a - # notice you had to be watching for: six hours of build output later - # there was no way to ask what had been refused. `setid --run` is what - # clears a line, once it has applied it and read the result back. + : > "$f" 2>/dev/null || true } run_phase_as() { @@ -5708,13 +4696,6 @@ run_phase_as() { case "$phase" in all|update) clear_phases "$track" ;; esac fi mkdir -p "$(dirname "$log")" 2>/dev/null || true - # THE LEASE (1.14.263): files this phase will overwrite that root or - # another package owns are handed to the installer first, and settled or - # restored below. Root writes anything, so a root run needs none. - local _leased=0 - if [ "$as_root" != 1 ] && have_su && user_exists "$owner"; then - lease_files "$owner" "$script" "$phase" && _leased=1 - fi if [ "$as_root" = 1 ] || ! have_su || ! user_exists "$owner"; then ( [ -n "$cwd" ] && cd "$cwd"; eval "env $envpass bash '$script' '$phase'" ) \ 2>&1 | tee "$log" @@ -5724,22 +4705,6 @@ run_phase_as() { su - "$owner" -c "env $envpass bash '$script' '$phase'" 2>&1 | tee "$log" rc=${PIPESTATUS[0]} fi - # A SINGLE-PHASE RUN KNOWS WHICH PHASE FAILED: ITS OWN. - # - # LAST_FAILED_PHASE was only set in the all|update branch below, and it - # is cleared at the top of this function -- so the FIRST retry (which - # runs just `install`) left it empty, and cmd_build's second round fell - # back to `$phase` = all. opencv's round 2 therefore re-cloned and - # rebuilt a tree that had reached 100%, to redo an install that needed - # one more directory grant. Rounds 1 and 2 printed the difference: - # round 1 ... (phase install only -- the build is done) - # round 2 ... (phase all) - if [ "$rc" != 0 ]; then - case "$phase" in - all|update) ;; # parsed from the log below - *) LAST_FAILED_PHASE="$phase" ;; - esac - fi if [ -n "$track" ]; then local _p case "$phase" in @@ -5809,23 +4774,7 @@ run_phase_as() { fi if [ "$rc" = 0 ] && [ -n "$log" ] && [ -f "$log" ]; then local _swallowed="" - # CMAKE SAYS IT DIFFERENTLY, AND IT SAID IT. - # - # CMake Error at CMakeLists.txt:6 (set): - # There is an unterminated variable reference. - # -- Configuring incomplete, errors occurred! - # - # ...and the run printed "llvm: done (all phases complete)". The - # pattern below knows make and ninja; cmake fails before either runs, - # so nothing matched and a configure error became a finished build. - # That is the exact failure this check was written for, in another - # tool's words. - # - # And the `ninja` in the first alternative never matched anything: - # ninja prints `ninja: build stopped: subcommand failed.` with no - # `***`, so it needs its own branch -- the one place this check was - # supposed to work, and did not. - _swallowed="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^ninja: build stopped|^install: cannot create|: Permission denied$|^CMake Error|^-- Configuring incomplete|^configure: error:|^meson\.build:[0-9]+:[0-9]+: ERROR:|^ERROR: Unhandled python|^error: could not compile|^fatal: could not read Username|^fatal: repository .* not found|^fatal: Authentication failed|BackendUnavailable: Cannot import|^ModuleNotFoundError: No module named" \ + _swallowed="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^install: cannot create|: Permission denied$" \ "$log" 2>/dev/null)" if [ -n "$_swallowed" ]; then warn "!! this phase exited 0, but its log says otherwise:" @@ -5835,17 +4784,6 @@ run_phase_as() { rc=1 fi fi - # SETTLE ONLY AFTER THE LOG HAS HAD ITS SAY: a zero exit with "Permission - # denied" in the log is a failure, and a failure puts every leased file - # back. - if [ "$_leased" = 1 ]; then - if [ "$rc" = 0 ]; then - lease_settle "$owner" - else - lease_restore "$owner" \ - || warn "!! some leased files could not be restored -- see above" - fi - fi return "$rc" } @@ -5929,37 +4867,6 @@ _version_from_script() { | sed 's/^.*-\([0-9][0-9.]*\)$/\1/' } -# WHAT IS INSTALLED IS WHAT FINISHED -- ON EVERY PATH THAT FINISHES. -# -# `update` reads the installed version from .pkgusr-installed. It was -# written in ONE branch: a single-phase run completing the last phase. A -# normal `--phase all` install (every packagemanager install) took the other -# branch and never wrote it; and VERSION, beside it, parsed only -# name_version="..." while the generator writes name_version='...'. So no -# record was written on success, the reader fell back to install_last (staged -# BEFORE the build), and a failed build advanced the version. 1.14.273 then -# seeded the marker with the OLD version before a build -- and with nothing -# updating it on success, LLVM 22.1.8 installed and kept reading 21.1.8, and -# every stack run planned it again. -# -# Called after every phase has succeeded, whichever branch got there. -_record_installed() { # <name> <owner> <script> - local name="$1" owner="$2" script="$3" nv home - nv="$(sed -n "s/^name_version=['\"]\([^'\"]*\)['\"].*/\1/p" "$script" 2>/dev/null | head -n1)" - [ -n "$nv" ] || return 0 - home="$(pkgusr_home_for "$owner" 2>/dev/null)" - [ -n "$home" ] && [ -d "$home" ] || home="$(pkgusr_home_for "$name" 2>/dev/null)" - [ -n "$home" ] && [ -d "$home" ] || return 0 - { - echo "$nv" - echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper" - [ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script" - } > "$home/VERSION" - printf '%s\n' "$nv" > "$home/.pkgusr-installed" - # benign: notes in the package's own home, rewritten every install - real_chown "$owner:$owner" "$home/VERSION" "$home/.pkgusr-installed" 2>/dev/null || true -} - _write_pkgusr_info() { local name="$1" owner="$2" source="${3:-lfs}" version="${4:-}" local home="$(pkgusr_home_for "$name")" @@ -6090,8 +4997,6 @@ _package_already_owns_files() { cmd_build() { need_root local name="" phase="all" force=0 run_tests=0 jobs="" auto_fix=1 - local auto_modules="${PM_AUTO_MODULES:-0}" - local _AUTO_MODULES_LOG="" local _force_stage=auto script_override="" as_root=0 while [ $# -gt 0 ]; do case "$1" in @@ -6108,7 +5013,6 @@ cmd_build() { --tests) run_tests=1; shift ;; --jobs|-j) jobs="${2:?--jobs needs a number}"; shift 2 ;; --no-auto-fix) auto_fix=0; shift ;; - --auto-modules) auto_modules=1; shift ;; # A CONFIGURATION STEP THAT NEEDS ROOT (1.14.79) told people to run # lfs-helper build fcron --phase configure --force --as-root # and cmd_build did not take --as-root: the flag existed on @@ -6296,9 +5200,7 @@ cmd_build() { else pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" owner="$(pkg_owner_name "$name")" - # an existing account is the normal case for every build after the - # first -- say nothing about it (`lfs-helper add-user` still does) - _QUIET_EXISTING=1 cmd_add_user "$owner" + cmd_add_user "$owner" stagedir="$(pkgusr_home_for "$owner")" staged="$stagedir/install_$name" # ALREADY THERE IS ALREADY STAGED. @@ -6381,20 +5283,7 @@ cmd_build() { mkdir -p "$(dirname "$GRANTED")" 2>/dev/null || true : > "$GRANTED" 2>/dev/null || true create_service_accounts_from_script "$staged" - # ROOT MUST NOT LEAVE A FILE THE PACKAGE USER CANNOT WRITE. - # - # This truncation runs as root, so the ledger came out root:root -- and - # then the install wrapper, running as the package user, could not - # append to it: - # /usr/lib/pkgusr/install: .pkgusr-setid.requests: Permission denied - # The swallowed-failure check saw "Permission denied" and failed the - # build, correctly, over the RECORDER rather than the package. Every - # root write to this file hands it back. - _sidf="$(pkgusr_home_for "$owner" 2>/dev/null)/.pkgusr-setid.requests" - if [ -n "${_sidf:-}" ]; then - : > "$_sidf" 2>/dev/null || true - chown "$owner:$owner" "$_sidf" 2>/dev/null || true - fi + : > "$(pkgusr_home_for "$owner" 2>/dev/null)/.pkgusr-setid.requests" 2>/dev/null || true say "# building $name ..." local rc=0 make_wrappers @@ -6528,32 +5417,6 @@ cmd_build() { && ! printf '%s' "$envpass" | grep -q "MAKEFLAGS="; then envpass="$envpass MAKEFLAGS='$MAKEFLAGS'" fi - # THREE BUILD SYSTEMS, THREE VARIABLES, ONE NUMBER. - # - # make reads MAKEFLAGS; cmake reads CMAKE_BUILD_PARALLEL_LEVEL; cargo - # reads CARGO_BUILD_JOBS; ninja reads NOTHING and defaults to nproc+2, - # which is why machine.conf patches its -j into the script. Setting - # MAKEFLAGS alone limited one of the four, so `--jobs 13` was not a - # limit -- a cmake or rust package still ran wide and got OOM-killed. - # READ THE VALUE THAT WILL ACTUALLY BE PASSED. - # - # This looked at ${MAKEFLAGS:-} -- the AMBIENT one, in root's shell -- - # while the number lives in /etc/pkgusr/build.env, which is already in - # $envpass by now. In a chroot root shell MAKEFLAGS is unset, so - # `MAKEFLAGS=-j13` in build.env produced no CARGO_BUILD_JOBS at all and - # rust built on every core. - _jn="$jobs" - [ -n "$_jn" ] || _jn="$(printf '%s' "$envpass" \ - | grep -oE "MAKEFLAGS='?-j ?[0-9]+" \ - | grep -oE '[0-9]+' | head -n1)" - [ -n "$_jn" ] || _jn="$(printf '%s' "${MAKEFLAGS:-}" \ - | grep -oE '\-j ?[0-9]+' | grep -oE '[0-9]+' | head -n1)" - if [ -n "$_jn" ]; then - printf '%s' "$envpass" | grep -q "CMAKE_BUILD_PARALLEL_LEVEL=" \ - || envpass="$envpass CMAKE_BUILD_PARALLEL_LEVEL='$_jn'" - printf '%s' "$envpass" | grep -q "CARGO_BUILD_JOBS=" \ - || envpass="$envpass CARGO_BUILD_JOBS='$_jn'" - fi # Keep pip out of the package user's ~/.local. PIP_USER=0 stops it # installing there, and PYTHONNOUSERSITE=1 stops it *seeing* an existing # per-user copy and concluding the requirement is "already satisfied" -- @@ -6667,10 +5530,9 @@ cmd_build() { # and it then reports a diff that is simply wrong. The sort and the # comparison have to agree; pinning either one alone guarantees they do not. newfiles="$(LC_ALL=C comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after"))" - local _after_dirs; _after_dirs="$(snapshot_dirs)" # one walk, used twice - newdirs="$(LC_ALL=C comm -13 <(printf '%s\n' "$before_dirs") <(printf '%s\n' "$_after_dirs"))" + newdirs="$(LC_ALL=C comm -13 <(printf '%s\n' "$before_dirs") <(snapshot_dirs))" printf '%s\n' "$after" > "$SNAPSHOT" - printf '%s\n' "$_after_dirs" > "$SNAPDIRS" + printf '%s\n' "$(snapshot_dirs)" > "$SNAPDIRS" # Everything this build wrote: created OR modified since the stamp. # Always work out what the build TOUCHED, even when we are not going to @@ -6707,23 +5569,20 @@ cmd_build() { # knew that; this one did not. Same rule, one more place. local _bld="${BUILD_ROOT:-$r/build}" say "# scanning the tree for what this step touched (a fresh tree takes a moment) ..." - # ONE pruned walk for both lists (it was two unpruned ones). A - # pruned directory is reported only as itself, as before: $SRCROOT, - # sources, dev, proc, sys, run and tmp when newer; never the build root. - local _tw - _tw="$(find "${r:-/}" -xdev \ - \( -path "$_bld" \) -prune \ - -o \( -path "$SRCROOT" -o -path "$r/sources" -o -path "$r/dev" \ - -o -path "$r/proc" -o -path "$r/sys" -o -path "$r/run" \ - -o -path "$r/tmp" \) -type d -newer "$stamp" -printf 'd %p\n' -prune \ - -o \( -path "$SRCROOT" -o -path "$r/sources" -o -path "$r/dev" \ - -o -path "$r/proc" -o -path "$r/sys" -o -path "$r/run" \ - -o -path "$r/tmp" \) -prune \ - -o -newer "$stamp" \( -type d -printf 'd %p\n' \ - -o \( -type f -o -type l \) -printf 'f %p\n' \) \ - 2>/dev/null)" - touched_dirs="$(printf '%s\n' "$_tw" | sed -n 's/^d //p' | sort -u)" - touched="$(printf '%s\n' "$_tw" | sed -n 's/^f //p' | sort -u)" + touched_dirs="$(find "${r:-/}" -xdev -type d -newer "$stamp" \ + -not -path "$SRCROOT/*" \ + -not -path "$r/sources/*" -not -path "$_bld/*" \ + -not -path "$_bld" -not -path "$r/dev/*" \ + -not -path "$r/proc/*" -not -path "$r/sys/*" \ + -not -path "$r/run/*" -not -path "$r/tmp/*" \ + 2>/dev/null | sort -u)" + touched="$(find "${r:-/}" -xdev \( -type f -o -type l \) -newer "$stamp" \ + -not -path "$SRCROOT/*" \ + -not -path "$r/sources/*" -not -path "$_bld/*" \ + -not -path "$r/dev/*" \ + -not -path "$r/proc/*" -not -path "$r/sys/*" \ + -not -path "$r/run/*" -not -path "$r/tmp/*" \ + 2>/dev/null | sort -u)" fi rm -f "$stamp" @@ -6731,38 +5590,14 @@ cmd_build() { # Record what was touched regardless of whether ownership changed -- a # config step that rewrites /etc/hosts installs no NEW file but has very # much done its job. - # A PACKAGE USER CANNOT HAVE WRITTEN A ROOT-OWNED FILE. - # - # Attribution is by time window. LLVM's install phase ran for hours - # (its compile is generated into install_pkg), and in that window root - # ran `make install` of these tools and `cfg app-users` -- so - # /usr/bin/lfs, /usr/bin/packagemanager and /etc/pam.d/su_u_user went - # into LLVM's manifest. The lease then treated them as LLVM's own - # root-owned files and handed su_u_user, a script root runs on every - # su, to p_llvm for the length of each install round. - # When the phase ran AS the package user, it could only have written - # files it can write: whatever is root's now was not its doing. (A - # file it rewrote under a lease is its own by now -- lease_settle ran - # before this scan.) Root-run phases and chown-after builds still record - # root files, which is what they produce. - local _root_ok=0 - { [ "${as_root:-0}" = 1 ] || [ "${chown_after:-0}" = 1 ]; } && _root_ok=1 - { [ -f "$man" ] && cat "$man" - { printf '%s\n' "$newfiles"; printf '%s\n' "$touched"; } \ - | while IFS= read -r _tf; do - [ -n "$_tf" ] || continue - if [ "$_root_ok" != 1 ] && [ "$(stat -c %u "$_tf" 2>/dev/null)" = 0 ]; then - continue - fi - printf '%s\n' "$_tf" - done; } \ + { [ -f "$man" ] && cat "$man"; printf '%s\n' "$newfiles" + printf '%s\n' "$touched"; } \ | grep -v '^$' | sort -u \ | _drop_other_packages_files "$owner" > "$man.tmp" \ && mv "$man.tmp" "$man" { [ -f "$mand" ] && cat "$mand"; printf '%s\n' "$newdirs"; } \ | grep -v '^$' | sort -u > "$mand.tmp" && mv "$mand.tmp" "$mand" say "# tracked $(wc -l < "$man") file(s) for '$name'" - _udev_reload_if_rules_changed "$man" # ...and the owner-scanned record too, once ownership makes it meaningful. # One door: cmd_record_install decides which strategy applies (item 6). @@ -6849,31 +5684,8 @@ cmd_build() { # that stops because the USER must do something -- "mount your EFI System # Partition first" -- must never be answered by handing directories away. local _perm_ish=0 - # "file INSTALL cannot": CMake wraps "Permission denied" over two lines. - grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|couldn't copy|read-only file system|file INSTALL cannot" \ + grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|couldn't copy|read-only file system" \ "$log" 2>/dev/null && _perm_ish=1 - # A BOOK COMMAND THAT WORKS ONCE. - # - # libtiff's page ends by moving its html docs into a versioned - # directory. The first install creates - # /usr/share/doc/libtiff-4.7.1/tiff; the second cannot: - # mv: cannot overwrite '/usr/share/doc/libtiff-4.7.1/tiff': Directory not empty - # Nothing is wrong with the system -- the destination is this package's - # own output from last time, and a REINSTALL means replacing it. The - # gate did not recognise it, so 38 rebuilds stopped on a directory the - # package itself had put there. - # - # Only ever the exact path the message names, only when it belongs to - # THIS package, and only for a phase that is reinstalling. - # THE GATE DECIDES; THE LOOP ACTS. - # - # This used to call _clear_stale_dest here. It removed the directory -- - # correctly, and said so -- and then the retry loop ran zero rounds, - # because the loop only counts what IT fixed. The repair happened and - # nothing re-ran the phase, so the package failed anyway with the - # obstacle already gone. Detect here, act in the loop. - grep -qE "cannot overwrite '[^']+'|: Directory not empty" "$log" 2>/dev/null \ - && _perm_ish=1 # "cannot make directory ...: No such file or directory" is not a # permission problem and granting will never fix it -- but it IS # automatically fixable, so it must reach the retry loop. @@ -6916,18 +5728,6 @@ cmd_build() { && _fixed_something=1 fi create_missing_dirs_from_log "$log" "$owner" && _fixed_something=1 - # the package's own leftovers from a previous install of the same - # version: removing them IS the fix, and it counts as one, so the - # phase is retried - _clear_stale_dest "$log" "${owner:-}" && _fixed_something=1 - # a destination file another package owns, in a sticky directory: - # handing it over is the fix, and it counts as one - _take_cmake_dest "$log" "${owner:-}" && _fixed_something=1 - if [ "${auto_modules:-0}" = 1 ]; then - [ -n "$_AUTO_MODULES_LOG" ] \ - || _AUTO_MODULES_LOG="$(mktemp -t pm-auto-modules.XXXXXX)" - _install_missing_module "$log" && _fixed_something=1 - fi # a file another package owns, which THIS package is installing: # ownership follows the installer (granting cannot fix EPERM) takeover_files_from_log "$log" "$owner" && _fixed_something=1 @@ -6968,21 +5768,11 @@ cmd_build() { # THAT phase; the sources and the build tree are untouched and still # there, which is exactly why the grant is all that was missing. local _retry_phase="$phase" - # `update` IS A COMPOUND PHASE TOO. - # - # This mapped only `all`, so a failure inside `update` retried - # `update` -- and poppler's retry, after a grant for - # /usr/share/gir-1.0, re-unpacked the tarball and re-ran cmake from - # scratch. 145 files were already installed; the one that needed - # doing again was the install step. The log says which inner phase - # failed (`!! phase 'install' FAILED`); both compound phases should - # use it. - case "$phase" in - all|update) - case "${LAST_FAILED_PHASE:-}" in - install|configure) _retry_phase="$LAST_FAILED_PHASE" ;; - esac ;; - esac + if [ "$phase" = all ]; then + case "${LAST_FAILED_PHASE:-}" in + install|configure) _retry_phase="$LAST_FAILED_PHASE" ;; + esac + fi say "" if [ "$_retry_phase" != "$phase" ]; then say "# retrying $name (phase $_retry_phase only -- the build is done), round $_round ..." @@ -7076,42 +5866,6 @@ cmd_build() { fail "! auto-repair: $([ "$auto_fix" = 1 ] && echo enabled || echo DISABLED)," \ "gate: $([ "$_perm_ish" = 1 ] && echo "recognised" || echo "did not recognise this failure")," \ "rounds run: ${_round:-0}" - # A PYTHON BUILD BACKEND IS A PACKAGE, AND ITS NAME IS NOT THE - # MODULE'S. numpy died on - # BackendUnavailable: Cannot import 'mesonpy' - # and the page that provides `mesonpy` is called meson_python -- not - # guessable, and the traceback is 60 lines of pip internals with the - # one useful word at the bottom. - local _pymod _pyprov - _pymod="$(grep -aoE "Cannot import '[a-zA-Z0-9_.]+'|No module named '[a-zA-Z0-9_.]+'" \ - "$log" 2>/dev/null | head -n1 | cut -d"'" -f2)" - if [ -n "$_pymod" ]; then - case "$_pymod" in - mesonpy) _pyprov="meson_python" ;; - setuptools_scm) _pyprov="setuptools_scm" ;; - Cython|cython) _pyprov="Cython" ;; - numpy) _pyprov="NumPy" ;; - flit_core) _pyprov="flit_core" ;; - hatchling) _pyprov="hatchling" ;; - scikit_build_core) _pyprov="scikit-build-core" ;; - poetry.core|poetry_core) _pyprov="poetry-core" ;; - *) _pyprov="" ;; - esac - if [ -n "${_AUTO_MODULES_LOG:-}" ] && [ -s "$_AUTO_MODULES_LOG" ]; then - fail "! installed on the way (--auto-modules), not asked for:" - while IFS= read -r _am; do fail "! $_am"; done < "$_AUTO_MODULES_LOG" - fi - fail "! a python module is missing, not a permission: '$_pymod'" - if [ -n "$_pyprov" ]; then - fail "! the BLFS page that provides it is '$_pyprov':" - fail "! packagemanager install $_pyprov --run --recursive" - else - fail "! find the page that provides it:" - fail "! packagemanager search $_pymod" - fi - fail "! then rebuild this package. A plain \`install\` builds ONE" - fail "! package: --recursive is what builds what it needs." - fi local _md_now _md_now="$(missing_dirs_from_log "$log" 2>/dev/null | tr '\n' ' ')" if [ -n "$_md_now" ]; then @@ -7899,13 +6653,27 @@ EOF fi fi - # Record what is installed, in the package user's own home: VERSION and - # .pkgusr-installed, both in _record_installed (below), for the branch - # that finished every phase. + # Record what is installed, in the package user's own home. + # + # The scripts are regenerated from the book whenever it moves on, so their + # name_version tells you what the BOOK has, not what is on disk. Without a + # record of the latter, "is this up to date?" is unanswerable. + local _nv _home + _nv="$(sed -n 's/^name_version="\(.*\)"$/\1/p' "$s" 2>/dev/null | head -n1)" + _home="$(pkgusr_home_for "$owner")" + if [ -n "$_nv" ] && [ -d "$_home" ]; then + { + echo "$_nv" + echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper" + [ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script" + } > "$_home/VERSION" + # benign: a note in the package's own home, rewritten every install + real_chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true + fi + if [ "$phase" = "all" ] || [ "$phase" = "update" ]; then _write_pkgusr_info "$name" "$owner" "${LFS_PKG_SOURCE:-lfs}" \ "$(_version_from_script "$staged")" - _record_installed "$name" "$owner" "$s" mark_done "$name" ok "# $name: done" report_granted_groups @@ -7928,18 +6696,7 @@ EOF # running again on every build-all. [ -f "$MANIFESTS/$name.files" ] && \ _mark_adopted "$name" "$(_manifest_size "$MANIFESTS/$name.files")" - _record_installed "$name" "$owner" "$s" ok "# $name: done (all phases complete)" - # AND WHAT IT INSTALLED ON THE WAY, on the SUCCESS path too: the - # whole point of reporting it is that the run changed more than - # the package that was asked for, and a successful run is - # exactly when nobody goes looking. - if [ -n "${_AUTO_MODULES_LOG:-}" ] && [ -s "$_AUTO_MODULES_LOG" ]; then - say "# installed on the way (--auto-modules), not asked for:" - while IFS= read -r _am; do say "# $_am"; done \ - < "$_AUTO_MODULES_LOG" - say "# they are real package users now: packagemanager info <name>" - fi report_granted_groups fi fi @@ -8176,153 +6933,6 @@ cmd_rename_group() { ok "renamed: $old -> $new" } -# REGROUP ONE DIRECTORY. -# -# /etc/pam.d p_bash-completion:nimgnu_bash-completion drwxrwxr-x -# -# A directory granted through the wrong group -- here one named after a -# package that has nothing to do with pam -- cannot be fixed with chgrp -# alone: every member of the old group keeps write access to it, and the -# packages that DO install there have to join the new one. So: -# -# 1. who installs INTO it: every package whose manifest or pkg.lst lists a -# direct child, every account owning an entry in it, and its owner -# 2. the new group's name, asked the way a grant asks (owner-named, -# directory-named, or typed) -# 3. which members of the OLD group still need it: those installing into -# another directory that keeps the old group (or owning one) -# 4. the plan, then yes/no -- nothing changes before the answer -# -# The old group itself is left as it is (even when empty): deleting a group -# is a separate decision, and the plan says when it has become unused. -cmd_regroup_dir() { - need_root - local dir="${1:?usage: lfs-helper regroup-dir <directory>}" - dir="$(readlink -f "$dir")" || die "no such directory: $1" - [ -d "$dir" ] || die "not a directory: $dir" - local owner old - owner="$(stat -c %U "$dir")"; old="$(stat -c %G "$dir")" - [ "$owner" = root ] && die "$dir is owned by root -- that is not a package's directory to share" - - say "$dir ($(stat -c '%A %U:%G' "$dir"))" - - # 1. installers into THIS directory (direct children only) - local installers - installers="$( - { for man in "$MANIFESTS"/*.files; do - [ -f "$man" ] || continue - if awk -v d="$dir" '{ p = $0; sub(/\/[^\/]*$/, "", p); if (p == d) { f = 1; exit } } END { exit !f }' "$man"; then - pkg_owner_name "$(basename "${man%.files}")" - fi - done - for _r in "$PKGUSR_ROOT" "$CFGUSR_ROOT"; do - [ -d "$_r" ] || continue - for _l in "$_r"/*/pkg.lst; do - [ -f "$_l" ] || continue - awk -v d="$dir" '{ p = $0; sub(/\/[^\/]*$/, "", p); if (p == d) { f = 1; exit } } END { exit !f }' "$_l" \ - && basename "$(dirname "$_l")" - done - done - find "$dir" -mindepth 1 -maxdepth 1 -printf '%u\n' 2>/dev/null - echo "$owner" - } | while IFS= read -r _u; do - [ -n "$_u" ] || continue - # package and config accounts only: never root, a login or a u_ account - case "$_u" in - "${PKGUSR_PREFIX:-p_}"?*|"${CFGUSR_PREFIX:-cfg_}"?*) ;; - *) continue ;; - esac - getent passwd "$_u" >/dev/null 2>&1 && echo "$_u" - done | sort -u)" - - # 2. the new name - local by_owner by_dir new ans - by_owner="$(sanitise_group_name "$(collector_group_for "$owner")")" - by_dir="$(sanitise_group_name "${COLLECTOR_PREFIX}_$(basename "$dir")")" - say "" - say " installs into it: $(echo $installers)" - say " current group: $old (members: $(getent group "$old" | cut -d: -f4))" - say "" - say " New group for $dir:" - say " 1) ${C_B}${by_dir}${C_OFF} (named after the directory)" - say " 2) ${C_B}${by_owner}${C_OFF} (named after its owner)" - say " or type another name, or q to cancel" - printf ' [1/2/name/q] (default 1): ' > /dev/tty - read -e -r ans < /dev/tty || ans=q - case "$ans" in - ""|1) new="$by_dir" ;; - 2) new="$by_owner" ;; - q|Q) say "cancelled -- nothing changed"; return 0 ;; - *) new="$(sanitise_group_name "$ans")" - [ -n "$new" ] || die "'$ans' is not a usable group name -- nothing changed" ;; - esac - [ "$new" = "$old" ] && { say "$dir already has group $new -- nothing to do"; return 0; } - - # 3. who still needs the OLD group, for the directories that keep it - local others keep="" drop="" m d2 needs - # EVERY real filesystem, not just /'s: a directory of $old on another - # mount that is not seen here would cost its installers their membership. - local _mnts - _mnts="$(findmnt -rno TARGET -t ext2,ext3,ext4,xfs,btrfs,f2fs,jfs,reiserfs 2>/dev/null | tr '\n' ' ')" - [ -n "$_mnts" ] || _mnts="/" - others="$(find $_mnts -xdev \( -path "$SRCROOT" -o -path /proc -o -path /sys -o -path /dev \ - -o -path /run -o -path /tmp \) -prune \ - -o -type d -group "$old" -print 2>/dev/null | sort -u | grep -vxF "$dir")" - for m in $(getent group "$old" | cut -d: -f4 | tr ',' ' '); do - needs="" - while IFS= read -r d2; do - [ -n "$d2" ] || continue - if [ "$(stat -c %U "$d2")" = "$m" ] \ - || find "$d2" -mindepth 1 -maxdepth 1 -user "$m" -print -quit 2>/dev/null | grep -q .; then - needs="$d2"; break - fi - for man in "$MANIFESTS/$(unprefix_pkg_user "$m").files" "$PKGUSR_ROOT/$m/pkg.lst" "$CFGUSR_ROOT/$m/pkg.lst"; do - [ -f "$man" ] || continue - if awk -v d="$d2" '{ p = $0; sub(/\/[^\/]*$/, "", p); if (p == d) { f = 1; exit } } END { exit !f }' "$man"; then - needs="$d2"; break 2 - fi - done - done <<< "$others" - if [ -n "$needs" ]; then keep="$keep $m"; else drop="$drop $m"; fi - done - - # 4. the plan - say "" - say " PLAN" - say " chgrp $new $dir (mode g+rwx kept for the installers)" - group_exists "$new" && say " group $new exists (gid $(getent group "$new" | cut -d: -f3))" \ - || say " CREATE group $new (collector range)" - say " add to $new: $(echo $installers)" - [ -n "$drop" ] && say " remove from $old: $(echo $drop) (no other directory of $old is theirs)" - [ -n "$keep" ] && say " stay in $old: $(echo $keep) (they still install into a directory of $old)" - [ -z "$others" ] && say " $old is then on no directory at all -- delete it yourself if unused: groupdel $old" - printf ' apply? [y/N]: ' > /dev/tty - read -e -r ans < /dev/tty || ans=n - case "$ans" in [Yy]*) ;; *) say "cancelled -- nothing changed"; return 0 ;; esac - - ensure_group "$new" || die "could not create group $new -- nothing changed" - for m in $installers; do add_to_group "$m" "$new" || warn "could not add $m to $new"; done - real_chgrp "$new" "$dir" || die "chgrp $new $dir failed" - real_chmod g+rwx "$dir" 2>/dev/null - for m in $drop; do - gpasswd -d "$m" "$old" >/dev/null 2>&1 || warn "could not remove $m from $old" - done - # the records a later grant consults - local rel="${dir#${SNAP_ROOT%/}}" - if [ -f "$COLLECTOR_MAP" ]; then - grep -v "^$rel|" "$COLLECTOR_MAP" > "$COLLECTOR_MAP.tmp" 2>/dev/null; mv "$COLLECTOR_MAP.tmp" "$COLLECTOR_MAP" - fi - mkdir -p "$(dirname "$COLLECTOR_MAP")"; echo "$rel|$new" >> "$COLLECTOR_MAP" - for m in $installers; do - grep -qxF "$new|$m|$owner" "$GRANTED" 2>/dev/null || echo "$new|$m|$owner" >> "$GRANTED" - done - say "" - ok "$dir is now $(stat -c '%A %U:%G' "$dir")" - say " $new: $(getent group "$new" | cut -d: -f4)" - say " $old: $(getent group "$old" | cut -d: -f4)" - say " (membership changes apply to NEW logins and builds; a running shell keeps its groups)" -} - # Which package installed this path? # # The manifests are the record of who installed what, so a file that should @@ -9777,21 +8387,6 @@ cmd_verify() { ok "every file has a real owner" fi - # ---- 3c. setid bits the build refused --------------------------------- # - # A CHECK MUST SAY WHAT IS MISSING RATHER THAN WAIT TO BE ASKED. These - # were printed once, hours earlier, in the middle of a build log. - local _sid=0 _sf _su - while IFS= read -r _su; do - _sf="$(pkgusr_home_for "$_su" 2>/dev/null)/.pkgusr-setid.requests" - [ -s "$_sf" ] && _sid=$((_sid + $(grep -c . "$_sf"))) - done < <(pkg_users_list) - if [ "$_sid" -gt 0 ]; then - warn "$_sid setuid/setgid bit(s) were refused during the build and never applied" - warn " what and why: lfs-helper setid" - warn " apply them: lfs-helper setid --run" - problems=$((problems+1)) - fi - detail "# checking homes, the build user and the uid order ..." # ---- 4b. the passes the dead copy had and this one did not ------------- # # homes owned by their own user, the build user's leftovers, and the uid @@ -10259,24 +8854,6 @@ cmd_pm_install() { warn " used -- move your edits into $base (or delete the old file)" fi fi - # RECORD WHAT IS INSTALLED BEFORE OVERWRITING THE RECORD. - # - # `update` reads the installed version from .pkgusr-installed, written - # only after a successful install -- and, for a package built before that - # marker existed, from install_last, which is replaced HERE, before the - # build. LLVM 21.1.8 (no marker) failed on 22.1.8's patch, and the next - # plan dropped it as current: spirv-llvm-translator then asked for LLVM 22 - # and found 21.1.8. So the first time through, the old install_last's - # version becomes the marker, and a failure leaves it there. - if [ ! -s "$home/.pkgusr-installed" ] && [ -f "$home/install_last" ] \ - && ! [ "$install_script" -ef "$home/install_last" ]; then - local _was - _was="$(sed -n "s/^name_version=['\"]\{0,1\}\([^'\"]*\).*/\1/p" "$home/install_last" 2>/dev/null | head -n1)" - if [ -n "$_was" ]; then - printf '%s\n' "$_was" > "$home/.pkgusr-installed" \ - && chown "$user_name:" "$home/.pkgusr-installed" 2>/dev/null || true - fi - fi if ! [ "$install_script" -ef "$home/install_last" ]; then rm -f "$home/install_last" 2>/dev/null cp "$install_script" "$home/install_last" @@ -11208,29 +9785,11 @@ EOF [ -z "${INPUTRC:-}" ] && [ -f /etc/inputrc ] && export INPUTRC=/etc/inputrc EOF - # THE LOCALE COMES FROM A FILE, NOT FROM A LINE YOU EDIT INTO A SCRIPT. - # - # This used to write `: # export LANG=en_US.UTF-8` -- commented out, on - # the grounds that an unset LANG is a choice. It is not a choice anyone - # makes: nothing else on the system sets LANG either, so every machine - # built this way sits in the C locale until somebody notices a terminal - # complaining ("foot: 'C' is not a UTF-8 locale"), and the only way to - # fix it is to hand-edit a generated file. BLFS's own version has the - # same shape and the same outcome -- a placeholder nobody fills in. - # - # So it reads /etc/locale.conf, which is where LFS puts the locale and - # what `cfg shell-env` writes from `lfs config locale`. With no - # locale.conf this does nothing, exactly as before. _write_if_absent "$d/i18n.sh" <<'EOF' -# written by lfs-helper (shell startup files) -- safe to replace -# Locale. LANG lives in /etc/locale.conf, one file for the whole system: -# echo 'LANG=en_US.UTF-8' > /etc/locale.conf -# `packagemanager stack services --only shell-env --run` writes it for you, -# from `lfs config locale`, and checks the locale really exists. -if [ -r /etc/locale.conf ]; then - . /etc/locale.conf - if [ -n "${LANG:-}" ]; then export LANG; fi - if [ -n "${LC_ALL:-}" ]; then export LC_ALL; fi +# Locale. Set LANG here (e.g. en_US.UTF-8); an unset LANG means the C +# locale, which is a choice, not an accident. +if [ -z "${LANG:-}" ] && command -v locale >/dev/null 2>&1; then + : # export LANG=en_US.UTF-8 fi export LESSCHARSET="${LESSCHARSET:-utf-8}" EOF @@ -11851,11 +10410,6 @@ Builds one package as its own package user, tracking every file it installs. expect some failures) --jobs N override MAKEFLAGS for this build --no-auto-fix do not grant collector-group access on a permission failure - --auto-modules on a missing python build backend, install the BLFS page - that provides it (--recursive) and retry. Never for a - module only a TEST needs (pytest, hypothesis, ...): a test - suite in an install phase is a script bug, not a missing - package. Off by default; PM_AUTO_MODULES=1 also works. EOF ;; strip) @@ -11916,21 +10470,10 @@ Ownership and permissions (the package-user model): lfs-helper verify check the tree against what the manifests say it should be lfs-helper verify --fix and repair what it can - lfs-helper setid [--run] setuid/setgid bits the build refused - (su, passwd, dbus-launch-helper, ...) lfs-helper fix-perms <name> --run grant one package the access it needs lfs-helper grant-dir <dir> <user> --run lfs-helper rename-group <old> <new> --run # fix a mistyped group name - lfs-helper regroup-dir <directory> # move a directory to another group (asks, shows the plan) give one user access to one directory - lfs-helper undone steps of the chroot build not yet done - lfs-helper sort-users --run sort passwd and group by id - lfs-helper check-login can root and the main user log in - - The chroot build's own steps -- init-accounts, establish-ownership, - seal-install-dirs, install-profile, repair-adoption -- are dispatched by - `build-all` and listed by `lfs-helper list`. They are not wrong to run by - hand, but nothing needs to. Collector groups (shared directories): lfs-helper export-groups [<file>] save the decisions for the next build @@ -11964,8 +10507,6 @@ Look around / tidy up: run one phase as the package user (the one runner both drivers use) lfs-helper find-user-site --run find pip installs hidden in ~/.local - lfs-helper script-targets <script> [fn ...] - the files a phase writes (the lease list) Any command: -h explains it. Dry run is the default; --run applies. State: $STATE @@ -11996,8 +10537,6 @@ case "${1:-}" in manifests) shift; cmd_manifests "$@" ;; fix-perms) shift; cmd_fix_perms "$@" ;; clean-sources) shift; cmd_clean_sources "$@" ;; - setid) shift; cmd_setid "$@" ;; - script-targets) shift; cmd_script_targets "$@" ;; find-user-site) shift; cmd_find_user_site "$@" ;; check-toolchain) shift; cmd_check_toolchain "$@" ;; verify) shift; cmd_verify "$@" ;; @@ -12011,7 +10550,6 @@ case "${1:-}" in which-package) shift; cmd_which_package "$@" ;; grant-dir) shift; cmd_grant_dir "$@" ;; rename-group) shift; cmd_rename_group "$@" ;; - regroup-dir) shift; cmd_regroup_dir "$@" ;; make-wrappers) shift; cmd_make_wrappers "$@" ;; wrapper-dir) shift; cmd_wrapper_dir "$@" ;; export-groups) shift; cmd_export_groups "$@" ;; diff --git a/lfs-kernel b/lfs-kernel index bcd52d9..5a4c91b 100755 --- a/lfs-kernel +++ b/lfs-kernel @@ -7,14 +7,6 @@ # lfs-kernel latest the newest stable, once # lfs-kernel detect-esp print the EFI directory this would use # lfs-kernel verify [ver] check that everything is installed -# lfs-kernel firmware install what FIRMWARE_ONLY names, no kernel build -# lfs-kernel firmware --missing [--fix] [--all] -# firmware this machine asks for and has not got, -# from the boot log, this tree's modules and the -# config; --fix installs it at exactly the path -# the driver asked for, --all includes the files -# a module merely DECLARES (other chips' -- large -# and mostly unused) # # Settings: /etc/pkgusr/kernel.conf, INSTALLED by `make install` so it can be # edited before the first run. The build commands are in one block below, @@ -56,25 +48,8 @@ case "${1:-}" in printf '%s\n' "$(_detect_esp)"; exit 0 ;; verify) shift; _VERIFY=1; VERSION="${1:-}" ;; - firmware) - # firmware alone, with no kernel build: the blobs are a separate - # package and a missing one is a run-time problem, so fixing it must - # not mean rebuilding a kernel that is already installed and booted. - shift; _FWONLY=1; _FWARG=""; _FWFIX=0; _FWALL=0 - for _a in "$@"; do - case "$_a" in - --fix) _FWFIX=1 ;; - --all) _FWALL=1 ;; - --missing|missing) _FWARG="--missing" ;; - *) echo "!! lfs-kernel firmware: unknown argument '$_a'"; exit 1 ;; - esac - done - [ "$_FWFIX" = 1 ] || [ "$_FWALL" = 1 ] && _FWARG="--missing" ;; -h|--help) - # the slice is the header above, which grew: derive the end from - # the header itself rather than hard-coding a line number that a - # new command silently pushes out of range. - sed -n '2,/^# Settings:/p' "$0" | sed '$d' | sed 's/^# \?//'; exit 0 ;; + sed -n '2,14p' "$0" | sed 's/^# \?//'; exit 0 ;; "") ;; -*) echo "!! unknown option: $1 (try: lfs-kernel --help)"; exit 1 ;; *) VERSION="$1" ;; @@ -339,6 +314,91 @@ if [ "${_VERIFY:-0}" = 1 ]; then exit 1 fi +echo "# kernel $VERSION -> $ESP (tree $tree, $JOBS)" + +# -------------------------------------------------------------------------- +# THE COMMANDS. Everything above chooses the version and checks the paths; +# everything below is the build, and is meant to be read. +# -------------------------------------------------------------------------- +install -d "$SRC" +if [ ! -d "$tree" ]; then + [ -f "$tarball" ] || { + echo "# fetching $MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" + wget -q --show-progress -O "$tarball.part" \ + "$MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" \ + && mv "$tarball.part" "$tarball" \ + || { rm -f "$tarball.part" + echo "!! download failed -- KERNEL_LIST=1 shows what exists"; exit 1; } + } + tar -xf "$tarball" -C "$SRC" || exit 1 + chown -R "$acct:$acct" "$tree" "$tarball" 2>/dev/null || true +fi +[ -d "$tree" ] || { echo "!! $tree is missing"; exit 1; } + +# boot logo (optional) +if [ -n "${BOOTLOGO:-}" ]; then + logo="$SRC/$(basename "$BOOTLOGO")" + [ -f "$logo" ] || { wget -qO "$logo" "$BOOTLOGO" && chown "$acct:$acct" "$logo"; } || logo="" + [ -n "$logo" ] && _as_pkg "cp -v '$logo' drivers/video/logo/logo_linux_clut224.ppm" +fi + +# .config +case "$CONFIG_FROM" in + running) _as_pkg "gunzip < /proc/config.gz > .config" || { + echo "!! /proc/config.gz unreadable -- set CONFIG_FROM=file:<path>"; exit 1; } ;; + file:*) _as_pkg "cp -v '${CONFIG_FROM#file:}' .config" || exit 1 ;; + keep) [ -f "$tree/.config" ] || { echo "!! no .config in $tree"; exit 1; } ;; + *) echo "!! CONFIG_FROM=$CONFIG_FROM not understood"; exit 1 ;; +esac +# ---- refreshing the config for a NEW kernel ------------------------------ +# +# A config from one kernel is never complete for another: every version adds +# symbols the old file has no answer for. Something must decide them, and +# which something matters: +# +# olddefconfig take the DEFAULT for each new symbol, no questions. The +# right choice for an unattended run, and what this tool has +# always done -- but the defaults are upstream's, not yours. +# oldconfig ask about each new symbol, one at a time. Slow, and needs +# someone at the terminal, but nothing is decided silently. +# menuconfig open the editor and decide everything at once. +# none use the .config exactly as it is; `make` will then ask, or +# fail, depending on what is missing. +# +# It also says how many symbols the refresh had to answer, because "47 new +# symbols were set to their defaults" is worth knowing before a boot. +: "${CONFIG_REFRESH:=olddefconfig}" +_cfg_before="$(mktemp)"; cp "$tree/.config" "$_cfg_before" 2>/dev/null || true +case "$CONFIG_REFRESH" in + none) echo "# not refreshing the config (CONFIG_REFRESH=none)" ;; + olddefconfig|oldconfig|menuconfig|*config) + echo "# refreshing the config for $VERSION (make $CONFIG_REFRESH) ..." + _as_pkg "make $CONFIG_REFRESH" || exit 1 ;; + *) echo "!! CONFIG_REFRESH=$CONFIG_REFRESH is not understood"; exit 1 ;; +esac +if [ -s "$_cfg_before" ] && [ "$CONFIG_REFRESH" != none ]; then + _newsym="$(diff <(sort "$_cfg_before") <(sort "$tree/.config") 2>/dev/null \ + | grep -c '^>' || true)" + [ "${_newsym:-0}" -gt 0 ] \ + && echo "# $_newsym config line(s) differ from the config you started from" +fi +rm -f "$_cfg_before" + +# what a Wayland compositor needs; a warning, not a refusal +for opt in CONFIG_DRM CONFIG_DRM_FBDEV_EMULATION CONFIG_INPUT_EVDEV CONFIG_SYSVIPC; do + grep -qE "^$opt=(y|m)" "$tree/.config" || echo "!! $opt is not set (sway needs it)" +done +# Drivers that need firmware are no use without the driver. These are the +# ones a laptop misses most often; the firmware for each is in FIRMWARE_ONLY. +for opt in CONFIG_IWLWIFI:"Intel wireless (AX200 and friends)" \ + CONFIG_IWLMVM:"the AX200's own MVM firmware layer" \ + CONFIG_BT_HCIBTUSB:"USB Bluetooth (the AX200's other half)" \ + CONFIG_DRM_AMDGPU:"AMD graphics"; do + _o="${opt%%:*}"; _why="${opt#*:}" + grep -qE "^$_o=(y|m)" "$tree/.config" \ + || echo "!! $_o is not set -- no $_why" +done + # ---- firmware, as its own package ---------------------------------------- # # The drivers need it at RUN time and CONFIG_EXTRA_FIRMWARE needs it at BUILD @@ -366,9 +426,6 @@ _sparse_args() { out="$out $e" fi done - # WHENCE lists the flat-name links (see _fw_whence_links). Cone mode - # always checks out top-level files; pattern mode only what it is told. - [ -n "$mode" ] && out="$out /WHENCE" printf '%s' "${out# }" } @@ -390,11 +447,7 @@ _firmware_not_found() { # the question actually being asked. local _hits="" if [ -d "$src/.git" ]; then - _hits="$(_fw_index "$src" | grep -i -- "$stem" | head -12)" - if [ -n "$_FW_INDEX_ERR" ]; then - echo " (could not read the repository index, so this is what is on" - echo " disk only: $_FW_INDEX_ERR)" - fi + _hits="$(git -C "$src" ls-files 2>/dev/null | grep -i -- "$stem" | head -12)" fi [ -n "$_hits" ] || _hits="$(find "$src" -maxdepth 2 -iname "*$stem*" \ -printf '%P\n' 2>/dev/null | head -12)" @@ -413,64 +466,6 @@ _firmware_not_found() { fi } -# THE REGULATORY DATABASE IS NOT IN linux-firmware. -# -# cfg80211 asks the loader for regulatory.db, linux-firmware does not carry -# it, and no FIRMWARE_ONLY entry can conjure it -- worse, an entry naming it -# makes the whole firmware install FAIL, because a pattern matching nothing -# in the repository is an error. It is released on its own as -# wireless-regdb, and it belongs to the same package user as the rest of -# /lib/firmware: one directory, one owner. -REGDB_BASE="${REGDB_BASE:-https://www.kernel.org/pub/software/network/wireless-regdb}" -_install_regdb() { - local acct=p_linux-firmware home tar url ver work - case "${REGDB:-yes}" in yes|1|true) ;; *) return 0 ;; esac - [ -e /lib/firmware/regulatory.db ] && [ "${1:-}" != force ] && return 0 - id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 - home="$(getent passwd "$acct" | cut -d: -f6)" - # ASK, DO NOT ASSUME A VERSION: the directory listing names the releases, - # and a hard-coded date goes stale between them. - ver="$(wget -qO- "$REGDB_BASE/" 2>/dev/null \ - | sed -n 's/.*wireless-regdb-\([0-9.]*\)\.tar\.xz.*/\1/p' \ - | sort -V | tail -n1)" - if [ -z "$ver" ]; then - echo "!! could not read $REGDB_BASE/ -- no network, or the layout moved." - echo " Fetch a wireless-regdb tarball by hand, then:" - echo " su - $acct -c 'cp regulatory.db regulatory.db.p7s /lib/firmware/'" - return 1 - fi - url="$REGDB_BASE/wireless-regdb-$ver.tar.xz" - tar="$home/src/wireless-regdb-$ver.tar.xz" - install -d -o "$acct" -g "$acct" -m 755 "$home/src" 2>/dev/null - # CONFIRM THE DIRECTORY BEFORE DOWNLOADING INTO IT. `install -d -o` on - # an account that does not exist fails quietly, and the next line then - # reports a download failure for a URL that was never the problem. - [ -d "$home/src" ] || { - echo "!! $home/src does not exist and could not be created" - echo " (does the account $acct have that home? lfs-helper fix-home $acct --run)" - return 1; } - echo "# wireless-regdb $ver (the regulatory database, not in linux-firmware)" - [ -s "$tar" ] || su - "$acct" -c "wget -qO '$tar' '$url'" || { - echo "!! could not download $url"; rm -f "$tar"; return 1; } - work="$home/src/wireless-regdb-$ver" - su - "$acct" -c "cd '$home/src' && tar -xf '$tar'" || return 1 - [ -s "$work/regulatory.db" ] || { - echo "!! $tar unpacked without a regulatory.db in it"; return 1; } - install -d -m 755 /lib/firmware 2>/dev/null - [ "$(stat -c %U /lib/firmware 2>/dev/null)" = "$acct" ] \ - || chown "$acct:$acct" /lib/firmware 2>/dev/null - su - "$acct" -c "cp -a '$work/regulatory.db' '$work/regulatory.db.p7s' /lib/firmware/" \ - || { echo "!! could not install the database as $acct"; return 1; } - if [ -s /lib/firmware/regulatory.db ]; then - echo "# installed /lib/firmware/regulatory.db ($(stat -c %s /lib/firmware/regulatory.db) bytes)" - echo "# /lib/firmware/regulatory.db.p7s" - packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true - return 0 - fi - echo "!! /lib/firmware/regulatory.db is absent after the copy" - return 1 -} - _install_firmware() { local acct="p_linux-firmware" src="" only="${FIRMWARE_ONLY:-}" id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 @@ -543,49 +538,14 @@ _install_firmware() { su - "$acct" -c "cp -a '$src'/* /lib/firmware/" || return 1 else for d in $only; do - # AN ENTRY FOR SOMETHING THIS REPOSITORY DOES NOT HAVE IS NOT AN - # ERROR WHEN WE KNOW WHERE IT DOES LIVE. `regulatory.db*` in - # FIRMWARE_ONLY failed the entire firmware install, and the - # message blamed the pattern. - _skip=0 - while IFS='|' read -r _ep _epkg _eurl; do - [ -n "${_ep:-}" ] || continue - case "$_ep" in - $d) _skip=1 ;; - esac - done < <(_fw_elsewhere) - if [ "$_skip" = 1 ]; then - echo "# '$d' is not in linux-firmware -- it is wireless-regdb;" - echo "# installing that instead (REGDB=${REGDB:-yes})" - _install_regdb || echo "# ...which did not work; see above" - continue - fi # a pattern matches files at the top level; a name is a directory case "$d" in *[*?[]*) - # A PATTERN MATCHES AT ANY DEPTH, AND A LINK NAME COUNTS. - # - # linux-firmware moved iwlwifi-*.ucode into intel/iwlwifi/ - # and keeps the flat names the driver asks for as `Link:` - # lines in WHENCE. The sparse checkout (gitignore rules) - # already fetched them at their new depth; this looked - # only at the top level and stopped the kernel build. - local _hits _links - _hits="$(cd "$src" && find . -name .git -prune -o \( -type f -o -type l \) -name "$d" -print 2>/dev/null | sed 's|^\./||')" - _links="$(_fw_whence_link_names "$src" | while IFS= read -r _ln; do - case "$_ln" in $d) echo "$_ln" ;; esac; done)" - if [ -z "$_hits" ] && [ -z "$_links" ]; then + if ! su - "$acct" -c "ls -d $src/$d >/dev/null 2>&1"; then _firmware_not_found "$src" "$d"; return 1 fi - echo "# installing firmware: $d ($(printf '%s\n' "$_hits" | grep -c .) file(s)$([ -n "$_links" ] && printf ', %s flat link(s) from WHENCE' "$(printf '%s\n' "$_links" | grep -c .)"))" - # --parents, and from inside the checkout: `cp -a - # $src/intel/iwlwifi /lib/firmware/` dropped the intel/ - # and landed as /lib/firmware/iwlwifi, where nothing - # looks for it. An entry keeps its path. - if [ -n "$_hits" ]; then - printf '%s\n' "$_hits" \ - | su - "$acct" -c "cd '$src' && xargs -r -d '\\n' cp -a --parents -t /lib/firmware/" || return 1 - fi ;; + echo "# installing firmware: $d" + su - "$acct" -c "cp -a $src/$d /lib/firmware/" || return 1 ;; *) # ...the same help for a NAME. Only the pattern case # searched, so `iwlwifi` (a directory that does not @@ -593,637 +553,14 @@ _install_firmware() { # the reader exactly where the pattern had. [ -e "$src/$d" ] || { _firmware_not_found "$src" "$d"; return 1; } echo "# installing firmware: $d" - su - "$acct" -c "cd '$src' && cp -a --parents '$d' /lib/firmware/" || return 1 ;; + su - "$acct" -c "cp -a '$src/$d' /lib/firmware/" || return 1 ;; esac done fi - [ -n "$only" ] && _fw_whence_links "$src" "$acct" - _install_regdb || true packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true return 0 } -# WHENCE's `Link: <name> -> <target>` lines are how linux-firmware keeps the -# names drivers ask for after moving a blob (iwlwifi-cc-a0-77.ucode -> -# intel/iwlwifi/iwlwifi-cc-a0-77.ucode). Its own copy-firmware.sh makes -# them; a selective install has to as well. The target is relative to the -# link's directory. -_fw_whence_link_names() { - [ -r "$1/WHENCE" ] || return 0 - sed -n 's/^Link:[[:space:]]*\([^[:space:]]*\)[[:space:]]*->.*/\1/p' "$1/WHENCE" -} -_fw_whence_links() { - local src="$1" acct="$2" name target made=0 - [ -r "$src/WHENCE" ] || { echo "# (no WHENCE in the checkout -- flat firmware names not linked)"; return 0; } - while read -r name target; do - [ -n "$name" ] && [ -n "$target" ] || continue - # only for blobs that were installed, and never over a real file - [ -e "/lib/firmware/$(dirname "$name")/$target" ] || continue - [ -e "/lib/firmware/$name" ] || [ -L "/lib/firmware/$name" ] && continue - su - "$acct" -c "mkdir -p '/lib/firmware/$(dirname "$name")' && ln -s '$target' '/lib/firmware/$name'" \ - && made=$((made + 1)) - done < <(sed -n 's/^Link:[[:space:]]*\([^[:space:]]*\)[[:space:]]*->[[:space:]]*\([^[:space:]]*\).*/\1 \2/p' "$src/WHENCE") - [ "$made" -gt 0 ] && echo "# linked $made flat firmware name(s) from WHENCE" - return 0 -} - -# REPORT WHAT YOU FOUND, NOT WHAT YOU EXPECTED. -# -# FIRMWARE_ONLY="... intel" installed intel/, and the AX200 still had no -# firmware, because the driver asks for a FLAT path: -# iwlwifi 0000:03:00.0: Direct firmware load for iwlwifi-cc-a0-77.ucode failed -# iwlwifi 0000:03:00.0: no suitable firmware found! -# Four releases went into guessing linux-firmware's layout from the outside. -# The running kernel already knows the answer -- it printed the exact path it -# wanted -- and the checkout's git index knows where that file lives. So ask -# both, instead of reasoning about directory names. -_fw_requests() { - # every firmware path THIS boot asked for and did not get - dmesg 2>/dev/null | sed -n \ - -e 's/.*Direct firmware load for \([^ ]*\) failed.*/\1/p' \ - -e 's/.*firmware: failed to load \([^ ]*\).*/\1/p' \ - | sed 's/[,:]$//' | sort -u -} - -# ASK THE INDEX -- AND NOTICE WHEN THE ASKING FAILED. -# -# `git -C "$src" ls-files 2>/dev/null` returned nothing and rc 128, because -# the repository belongs to p_linux-firmware and this runs as root: -# fatal: detected dubious ownership in repository at '...' -# The empty output was then reported as a FACT -- "linux-firmware does not -# have a file by that name at all" -- about a file that is right there. An -# error hidden behind 2>/dev/null becomes a confident wrong answer. -_FW_INDEX_ERR="" -_fw_index() { # _fw_index <repo> [<basename to match>] - local src="$1" want="${2:-}" out="" err="" - _FW_INDEX_ERR="" - if [ -d "$src/.git" ]; then - # -c safe.directory: the checkout is the firmware account's and this - # runs as root, which git refuses to touch without being told. - # the scratch file goes in /tmp, not in the repository: leaving - # untracked files in someone else's checkout is not this tool's business - local tmp; tmp="$(mktemp)" - # ls-TREE, not ls-files. With a sparse index git collapses every - # excluded directory into one entry, so `ls-files` answered `intel/` - # and every nested blob under it looked absent -- 56 files reported - # as "this checkout has no file by that name" while they were all - # right there. ls-tree reads the commit, which no sparse setting - # touches. - err="$(git -c "safe.directory=$src" -C "$src" ls-tree -r --name-only HEAD 2>&1 >"$tmp")" - if [ -n "$err" ] && [ ! -s "$tmp" ]; then - _FW_INDEX_ERR="$err" - rm -f "$tmp" - return 1 - fi - out="$(cat "$tmp")" - rm -f "$tmp" - else - out="$(cd "$src" && find . -type f -printf '%P\n' 2>/dev/null)" - fi - if [ -n "$want" ]; then - printf '%s\n' "$out" | grep -- "\(^\|/\)$want$" - else - printf '%s\n' "$out" - fi -} - -_fw_repo() { # the linux-firmware checkout, or empty - local home - home="$(getent passwd p_linux-firmware 2>/dev/null | cut -d: -f6)" - case "$FIRMWARE_FROM" in - copy:*) printf '%s' "${FIRMWARE_FROM#copy:}" ;; - *) [ -n "$home" ] && printf '%s' "$home/src/linux-firmware" ;; - esac -} - -# WHICH KERNEL'S MODULES? In a chroot the running kernel is the host's, so -# `uname -r` is the wrong answer. Prefer the version this tool is configured -# for, then whatever is installed under /lib/modules, and SAY which one. -_fw_modver() { - local v - for v in "${VERSION:-}" "$(uname -r)"; do - [ -n "$v" ] && [ -d "/lib/modules/$v" ] && { printf '%s' "$v"; return 0; } - done - ls -1d /lib/modules/*/ 2>/dev/null | sed 's|.*/modules/||; s|/$||' \ - | sort -V | tail -n1 -} - -# Firmware that is NOT in linux-firmware, so "not in the checkout" is the -# right answer and updating the checkout will never help. -_fw_elsewhere() { - cat <<'ELSE' -regulatory.db|wireless-regdb|https://kernel.org/pub/software/network/wireless-regdb/ -regulatory.db.p7s|wireless-regdb|https://kernel.org/pub/software/network/wireless-regdb/ -ELSE -} - -# WHAT WILL BE ASKED FOR -- from sources that also work in a chroot. -# -# dmesg answers "what did the kernel that is running ask for", which in a -# chroot is the host's boot: it reported one missing file for 7.2.0 while the -# kernel of interest was 7.2.4, and said nothing about the wireless at all. -# The tree can be asked directly instead: the modules it installed declare -# their firmware (modinfo -F firmware), and the hardware is right here in -# /sys, so the two together give the list for THIS machine and THIS kernel -# without booting it. -# -# Prints "<source>|<path>" so every line can say where the fact came from. -_fw_wanted() { # _fw_wanted <modules-version> - local ver="${1:-}" a m f cfg - # 1. the boot that is running -- _fw_requests is the one dmesg parser - _fw_requests | sed 's/^/boot|/' - # 2. this tree's modules, for the hardware that is actually present - if [ -n "$ver" ] && [ -d "/lib/modules/$ver" ] \ - && command -v modinfo >/dev/null 2>&1; then - find /sys/devices -name modalias -print 2>/dev/null \ - | xargs -r cat 2>/dev/null | sort -u \ - | while IFS= read -r a; do - [ -n "$a" ] || continue - # modprobe resolves an alias through the TREE's - # modules.alias when told which version to use - modprobe -S "$ver" -R -- "$a" 2>/dev/null || true - done | sort -u \ - | while IFS= read -r m; do - [ -n "$m" ] || continue - # the MODULE's name travels with the request: a module - # declares firmware for every chip it supports, so these - # are candidates, and which module they came from is the - # only thing that makes them readable - modinfo -k "$ver" -F firmware -- "$m" 2>/dev/null \ - | sed "s|^|modules:$m\||" - done - fi - # 3. what the config embeds -- those must be on disk to BUILD, and are - # the one part a running kernel never complains about - for cfg in "$ESP/config-$ver" "/boot/config-$ver" /proc/config.gz; do - [ -r "$cfg" ] || continue - case "$cfg" in - *.gz) zcat "$cfg" 2>/dev/null ;; - *) cat "$cfg" ;; - esac | sed -n 's/^CONFIG_EXTRA_FIRMWARE="\(.*\)"$/\1/p' \ - | tr ' ' '\n' | sed '/^$/d; s/^/config|/' - break - done -} - -# A MODULE DECLARES FIRMWARE FOR EVERY CHIP IT SUPPORTS. -# -# The first version of the modules scan printed 56 lines of "MISSING", each -# with its own remedy: amdgpu blobs for Vega, Sienna Cichlid and Aldebaran, -# iwlwifi blobs for five RF families this laptop does not have. Every one of -# them was a fact -- `modinfo -F firmware amdgpu` really does list them -- -# and the conclusion drawn from them ("56 missing") was wrong: the driver -# loads what its own chip needs and ignores the rest. So a boot failure and -# a module declaration are not the same kind of statement and must not be -# printed the same way. -# WHAT IS THERE, NEXT TO WHAT IS NOT. -# -# `iwlwifi-cc-a0-77.ucode` missing, on a machine whose wifi worked under the -# previous kernel, means one of two things -- the file is on disk under -# another directory, or an OLDER API version of it is installed and the new -# driver will not take it -- and the report said neither. Both are one -# `find` and one glob away, and both change what you do next. -_FW_LOCAL="" -_fw_neighbours() { # _fw_neighbours <requested-path> [<repo>] - # THE EXPLANATION GOES TO THE READER, THE PATH GOES IN A VARIABLE. - # Returning the path on stdout meant the caller's `$(...)` swallowed - # every line of the explanation with it, and the report said only - # "installed" with no word about where the file had been. - local req="$1" src="${2:-}" base stem ext found sib _api - _FW_LOCAL="" - base="$(basename "$req")" - # 1. the same file, somewhere else under /lib/firmware. FIRMWARE_ONLY - # entries keep the repository's path, and a driver asks the loader for - # its own -- so `intel` installs intel/iwlwifi/x.ucode while the - # driver wants x.ucode at the top. - found="$(find /lib/firmware -name "$base" -printf '%P\n' 2>/dev/null \ - | grep -v "^$req$" | head -3)" - if [ -n "$found" ]; then - echo " it IS on disk, at another path:" - printf ' /lib/firmware/%s\n' $found - echo " the loader asks for /lib/firmware/$req and nothing else," - echo " so --fix puts a copy there." - _FW_LOCAL="$(printf '%s' "$found" | head -n1)" - return 0 - fi - # 2. an API-versioned sibling: iwlwifi-cc-a0-72.ucode when -77 is asked - # for. The number is the firmware API the driver demands, so an older - # one on disk is the whole story: this kernel needs a newer blob. - case "$base" in - *-[0-9][0-9].*|*-[0-9][0-9][0-9].*) - ext="${base##*.}" - stem="${base%-*}" - sib="$(ls -1 "/lib/firmware/$(dirname "$req")/$stem"-*."$ext" \ - 2>/dev/null | xargs -r -n1 basename | tr '\n' ' ')" - [ -n "$sib" ] && { - echo " installed already, for the same device but an older" - echo " firmware API: $sib" - _api="${base##*-}"; _api="${_api%.*}" - echo " This kernel's driver requires API $_api and will not" - echo " fall back, so the checkout needs to be newer."; } - if [ -n "$src" ] && [ -d "$src" ]; then - sib="$(_fw_index "$src" | grep -- "\(^\|/\)$stem-[0-9]*\.$ext$" \ - | xargs -r -n1 basename | sort -V | tail -3 | tr '\n' ' ')" - [ -n "$sib" ] && echo " the checkout has: $sib" - fi ;; - esac - return 1 -} - -_firmware_missing() { - local fix="${1:-0}" all="${2:-0}" src req f hits n=0 gone=0 fixed=0 failed=0 - local sugg="" ver source chroot=0 _rr _tr seen="" cand=0 cdir mod - src="$(_fw_repo)" - ver="$(_fw_modver)" - cdir="$(mktemp -d)" - _rr="$(stat -c %d:%i /proc/1/root/. 2>/dev/null || true)" - _tr="$(stat -c %d:%i / 2>/dev/null || true)" - [ -n "$_rr" ] && [ -n "$_tr" ] && [ "$_rr" != "$_tr" ] && chroot=1 - - echo "### firmware this machine will ask for, and has not got" - echo "# modules scanned: ${ver:-(none installed under /lib/modules)}" - echo "# running kernel: $(uname -r)$([ "$chroot" = 1 ] && printf ' (the HOST kernel -- this is a chroot)')" - if [ "$chroot" = 1 ]; then - echo "# in a chroot, 'boot' lines describe the HOST's boot -- possibly" - echo "# another distribution on other hardware, in which case they can" - echo "# neither confirm nor deny anything about this tree. 'modules'" - echo "# and 'config' lines are this tree's." - fi - [ -z "$ver" ] && echo "# no modules to scan, so only the running boot can be asked." - echo - while IFS='|' read -r source req; do - [ -n "${req:-}" ] || continue - case "$req" in */) continue ;; esac - case " $seen " in *" $req "*) continue ;; esac - seen="${seen:-} $req" - n=$((n + 1)) - if [ -e "/lib/firmware/$req" ]; then - # SAY WHAT IS SATISFIED, especially here. A boot line that is - # now present is the answer to "will this work when I boot it", - # and it was skipped silently -- so a run with everything in - # place looked identical to one that had not looked at all. - case "$source" in - modules:*) printf '%s\n' "$req" >> "$cdir/${source#modules:}.ok" ;; - *) echo " ok $req ($source) -- installed now" ;; - esac - continue - fi - # A DECLARATION IS A CANDIDATE, NOT A FAILURE. Collected per module - # and summarised below; only boot and config lines are treated as - # things that must be fixed. - case "$source" in - modules:*) - mod="${source#modules:}" - printf '%s\n' "$req" >> "$cdir/$mod" - cand=$((cand + 1)) - [ "$all" = 1 ] || continue ;; - esac - gone=$((gone + 1)) - echo " MISSING /lib/firmware/$req ($source)" - while IFS='|' read -r _ep _epkg _eurl; do - [ "$_ep" = "$req" ] || continue - echo " not part of linux-firmware: it is released as $_epkg" - echo " $_eurl" - if [ "$fix" = 1 ] && [ "$_epkg" = wireless-regdb ]; then - _install_regdb force && fixed=$((fixed + 1)) - else - echo " install it: lfs-kernel firmware --missing --fix" - echo " (without it cfg80211 falls back to the most" - echo " conservative channels and power -- wifi still works)" - fi - continue 2 - done < <(_fw_elsewhere) - # WHAT IS THERE, before what is not: a copy under another path, or an - # older API version, answers the question without the repository. - _fw_neighbours "$req" "$src" - if [ -n "$_FW_LOCAL" ] && [ -e "/lib/firmware/$_FW_LOCAL" ]; then - if [ "$fix" = 1 ]; then - _fw_place "/lib/firmware" "$req" "$_FW_LOCAL" \ - && fixed=$((fixed + 1)) - fi - continue - fi - if [ -z "$src" ] || [ ! -d "$src" ]; then - echo " no linux-firmware checkout to look in" - echo " (FIRMWARE_FROM=$FIRMWARE_FROM; run lfs-kernel firmware first)" - continue - fi - hits="$(_fw_index "$src" "$(basename "$req")" | head -4)" - if [ -n "$_FW_INDEX_ERR" ]; then - echo " could not read the repository, so where this file" - echo " lives is UNKNOWN -- not absent:" - printf ' %s\n' "$_FW_INDEX_ERR" - failed=$((failed + 1)) - continue - fi - if [ -z "$hits" ]; then - echo " this checkout has no file by that name." - echo " su - p_linux-firmware -c 'git -C $src pull'" - continue - fi - printf ' the repository has it at: %s\n' $hits - for f in $hits; do - if [ "$f" = "$req" ]; then - case "$f" in - */) ;; - */*) sugg="$sugg ${f%/*}" ;; - *) sugg="$sugg $f" ;; - esac - else - echo " no FIRMWARE_ONLY entry can express this: the" - echo " repository keeps it at '$f', the driver asks for" - echo " '$req'. --fix installs it where the loader looks." - fi - done - [ "$fix" = 1 ] && _fw_place "$src" "$req" "$(echo $hits | cut -d" " -f1)" \ - && fixed=$((fixed + 1)) - done < <(_fw_wanted "$ver") - - [ "$gone" = 0 ] && [ "$n" -gt 0 ] \ - && echo " nothing that has actually been asked for is missing." - - # ---- the candidates, per module, in a few lines rather than fifty ---- - if [ "$cand" -gt 0 ] && [ "$all" != 1 ]; then - echo - echo "Declared but not installed -- CANDIDATES, not failures. A module" - echo "declares firmware for every chip it supports; yours loads only" - echo "the ones it needs, and never asks for the others:" - for f in "$cdir"/*; do - [ -f "$f" ] || continue - case "$f" in *.ok) continue ;; esac - mod="$(basename "$f")" - echo " $mod: $(grep -c . "$f") of $(( $(grep -c . "$f") \ - + $( [ -f "$cdir/$mod.ok" ] && grep -c . "$cdir/$mod.ok" || echo 0) )) \ -declared file(s) not installed, e.g. $(head -n1 "$f")" - done - if [ "$chroot" = 1 ]; then - echo " If a driver you need is silent, boot THIS kernel and run this" - echo " again there: the 'boot' list above is the host's, and says" - echo " nothing about the driver you are asking about." - else - echo " If a driver you need is silent, look at what it actually asked" - echo " for -- the 'boot' list above, which is this kernel's." - fi - echo " To install every declared file anyway (large, mostly unused):" - echo " lfs-kernel firmware --missing --fix --all" - fi - - # WHICH FILE A DRIVER PICKS IS DECIDED AT RUN TIME. iwlwifi derives the - # name from the RF and CRF ids it reads off the card, so no amount of - # looking at modules and directories can promise it will find one. Say - # that, rather than implying a green run is a working driver. - if [ "$cand" -gt 0 ] || [ "$gone" -gt 0 ]; then - echo - echo " Only a boot of THIS kernel settles it: a driver derives the" - echo " firmware name from the ids it reads off the card, so what is" - echo " installed can be checked here, and what will be asked for" - echo " cannot." - fi - rm -rf "$cdir" - echo - if [ "$n" = 0 ]; then - echo "Nothing asks for firmware here: no failed load in this boot, and" - echo "no module under /lib/modules${ver:+/$ver} declares any." - return 0 - fi - if [ "$gone" = 0 ]; then - return 0 - fi - if [ "$fix" = 1 ]; then - echo "Placed $fixed of $gone file(s) at the exact path the driver asked for." - if [ -n "$sugg" ]; then - echo "Make it survive the next full run -- in $CONF:" - echo " FIRMWARE_ONLY=\"$FIRMWARE_ONLY$(printf '%s' "$sugg")\"" - fi - [ "$fixed" -gt 0 ] && { - echo "Then load the driver again (or reboot):" - echo " modprobe -r <driver> # iwlwifi, amdgpu, ..."; } - if [ "$failed" -gt 0 ]; then - echo "$failed lookup(s) failed, so nothing is known about those files." - return 1 - fi - [ "$fixed" = "$gone" ] && return 0 - echo "Still missing: $((gone - fixed))." - return 1 - fi - echo "Nothing was changed. Install what linux-firmware has:" - echo " lfs-kernel firmware --missing --fix" - if [ -n "$sugg" ]; then - echo "or add them to FIRMWARE_ONLY in $CONF and re-run lfs-kernel firmware:" - echo " FIRMWARE_ONLY=\"$FIRMWARE_ONLY$(printf '%s' "$sugg")\"" - fi - return 1 -} - -# Install ONE blob at the path the driver actually asked for. The driver's -# path and the repository's path are not the same thing (iwlwifi asks for a -# flat name; the repository may keep it under a directory), and it is the -# driver's that has to exist. -# -# ONE FILE MEANS ONE FILE. The first version widened the sparse checkout, -# and fell back to `sparse-checkout disable` when the add was refused -- in -# cone mode a file path is refused, so the fallback would have materialised -# the whole of linux-firmware: over ten gigabytes, to fix one 2 MB blob. -# `git show HEAD:<path>` reads exactly one blob out of the repository and -# needs no sparse mode at all (in a partial clone it fetches that blob and -# nothing else). -_fw_place() { # <src-repo> <requested-path> <repo-path> - local src="$1" req="$2" rel="$3" acct=p_linux-firmware - # SEPARATE STATEMENT. `local a="$2" d="/lib/firmware/$a"` reads $a before - # this local has assigned it, and `set -u` kills the function: - # lfs-kernel: line ...: req: unbound variable - local dst="/lib/firmware/$req" - id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 - # /lib/firmware is this package's directory (the full run hands it over); - # the account has to be able to write the file it is about to own. - install -d -m 755 /lib/firmware 2>/dev/null - [ "$(stat -c %U /lib/firmware 2>/dev/null)" = "$acct" ] \ - || chown "$acct:$acct" /lib/firmware 2>/dev/null - case "$req" in - */*) install -d -o "$acct" -g "$acct" -m 755 "/lib/firmware/$(dirname "$req")" \ - 2>/dev/null || true ;; - esac - if [ -e "$src/$rel" ]; then - local _cerr - _cerr="$(su - "$acct" -c "cp -a '$src/$rel' '$dst'" 2>&1)" || { - echo " !! could not copy it as $acct:" - printf ' %s\n' "${_cerr:-(no message)}"; return 1; } - elif [ -d "$src/.git" ]; then - echo " reading it out of the repository (it is in the index," - echo " not in the sparse checkout)" - local _gerr - # git's own message, not a guess about what went wrong: "does not - # exist in HEAD" and "unable to read sha1 file" (a partial clone with - # no network) need different answers from the reader. - _gerr="$(su - "$acct" -c "git -c 'safe.directory=$src' -C '$src' show 'HEAD:$rel' > '$dst'" 2>&1)" || { - echo " !! git could not produce $rel:" - printf ' %s\n' "${_gerr:-(no message)}" - echo " a partial clone fetches the blob on demand, so" - echo " this needs the network; check with:" - echo " su - $acct -c 'git -C $src fetch --depth 1'" - rm -f "$dst"; return 1; } - else - echo " !! $rel is not in $src" - return 1 - fi - # CONFIRM YOUR OWN EFFECT: a failed `git show` can leave an empty file - # behind, which looks exactly like a successful install. - if [ -s "$dst" ]; then - echo " installed $dst ($(stat -c %s "$dst") bytes)" - packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true - return 0 - fi - echo " !! $dst is empty or absent after the copy" - rm -f "$dst" - return 1 -} - -# ---- firmware without a kernel build -------------------------------------- -if [ "${_FWONLY:-0}" = 1 ]; then - case "${_FWARG:-}" in - --missing|missing) - _firmware_missing "${_FWFIX:-0}" "${_FWALL:-0}"; exit $? ;; - "") - [ "$FIRMWARE_FROM" = none ] \ - && { echo "!! FIRMWARE_FROM=none in $CONF -- nothing to install."; exit 1; } - _install_firmware || exit 1 - echo "# firmware installed; no kernel was built or touched." - exit 0 ;; - *) echo "!! lfs-kernel firmware takes --missing [--fix] or nothing"; exit 1 ;; - esac -fi - -echo "# kernel $VERSION -> $ESP (tree $tree, $JOBS)" - -# -------------------------------------------------------------------------- -# THE COMMANDS. Everything above chooses the version and checks the paths; -# everything below is the build, and is meant to be read. -# -------------------------------------------------------------------------- -install -d "$SRC" -if [ ! -d "$tree" ]; then - [ -f "$tarball" ] || { - echo "# fetching $MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" - wget -q --show-progress -O "$tarball.part" \ - "$MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" \ - && mv "$tarball.part" "$tarball" \ - || { rm -f "$tarball.part" - echo "!! download failed -- KERNEL_LIST=1 shows what exists"; exit 1; } - } - tar -xf "$tarball" -C "$SRC" || exit 1 - chown -R "$acct:$acct" "$tree" "$tarball" 2>/dev/null || true -fi -[ -d "$tree" ] || { echo "!! $tree is missing"; exit 1; } - -# boot logo (optional) -if [ -n "${BOOTLOGO:-}" ]; then - logo="$SRC/$(basename "$BOOTLOGO")" - [ -f "$logo" ] || { wget -qO "$logo" "$BOOTLOGO" && chown "$acct:$acct" "$logo"; } || logo="" - [ -n "$logo" ] && _as_pkg "cp -v '$logo' drivers/video/logo/logo_linux_clut224.ppm" -fi - -# .config -case "$CONFIG_FROM" in - running) _as_pkg "gunzip < /proc/config.gz > .config" || { - echo "!! /proc/config.gz unreadable -- set CONFIG_FROM=file:<path>"; exit 1; } ;; - file:*) _as_pkg "cp -v '${CONFIG_FROM#file:}' .config" || exit 1 ;; - keep) [ -f "$tree/.config" ] || { echo "!! no .config in $tree"; exit 1; } ;; - *) echo "!! CONFIG_FROM=$CONFIG_FROM not understood"; exit 1 ;; -esac -# ---- refreshing the config for a NEW kernel ------------------------------ -# -# A config from one kernel is never complete for another: every version adds -# symbols the old file has no answer for. Something must decide them, and -# which something matters: -# -# olddefconfig take the DEFAULT for each new symbol, no questions. The -# right choice for an unattended run, and what this tool has -# always done -- but the defaults are upstream's, not yours. -# oldconfig ask about each new symbol, one at a time. Slow, and needs -# someone at the terminal, but nothing is decided silently. -# menuconfig open the editor and decide everything at once. -# none use the .config exactly as it is; `make` will then ask, or -# fail, depending on what is missing. -# -# It also says how many symbols the refresh had to answer, because "47 new -# symbols were set to their defaults" is worth knowing before a boot. -: "${CONFIG_REFRESH:=olddefconfig}" -_cfg_before="$(mktemp)"; cp "$tree/.config" "$_cfg_before" 2>/dev/null || true -case "$CONFIG_REFRESH" in - none) echo "# not refreshing the config (CONFIG_REFRESH=none)" ;; - olddefconfig|oldconfig|menuconfig|*config) - echo "# refreshing the config for $VERSION (make $CONFIG_REFRESH) ..." - _as_pkg "make $CONFIG_REFRESH" || exit 1 ;; - *) echo "!! CONFIG_REFRESH=$CONFIG_REFRESH is not understood"; exit 1 ;; -esac -if [ -s "$_cfg_before" ] && [ "$CONFIG_REFRESH" != none ]; then - _newsym="$(diff <(sort "$_cfg_before") <(sort "$tree/.config") 2>/dev/null \ - | grep -c '^>' || true)" - [ "${_newsym:-0}" -gt 0 ] \ - && echo "# $_newsym config line(s) differ from the config you started from" -fi -rm -f "$_cfg_before" - -# ---- CONFIG_SET: this machine's own answers, applied last ---------------- -# -# The config comes from the running kernel (or a file), then a refresh -# answers the new symbols. An option you decided on has to survive both, so -# it is applied AFTER them, resolved once more (olddefconfig -- a forced -# symbol can pull or drop others), and READ BACK: `scripts/config` writes -# what it is told, and the resolve then silently reverts anything whose -# dependencies are not met. -if [ -n "${CONFIG_SET:-}" ]; then - _cs_args="" - for _cs in $CONFIG_SET; do - _sym="${_cs%%=*}"; _val="${_cs#*=}"; _sym="${_sym#CONFIG_}" - [ "$_sym" != "$_cs" ] || { echo "!! CONFIG_SET entry '$_cs' has no '=' -- SYMBOL=VALUE"; exit 1; } - case "$_val" in - y) _cs_args="$_cs_args --enable $_sym" ;; - m) _cs_args="$_cs_args --module $_sym" ;; - n) _cs_args="$_cs_args --disable $_sym" ;; - [0-9]*) _cs_args="$_cs_args --set-val $_sym $_val" ;; - *) _cs_args="$_cs_args --set-str $_sym ${_val//\"/}" ;; - esac - done - echo "# applying CONFIG_SET: $CONFIG_SET" - _as_pkg "./scripts/config $_cs_args && make olddefconfig" >/dev/null || exit 1 - _cs_bad="" - for _cs in $CONFIG_SET; do - _sym="${_cs%%=*}"; _val="${_cs#*=}"; _sym="CONFIG_${_sym#CONFIG_}" - case "$_val" in - n) grep -qE "^$_sym=(y|m)" "$tree/.config" && _cs_bad="$_cs_bad $_sym=n" ;; - y|m) grep -qx "$_sym=$_val" "$tree/.config" || _cs_bad="$_cs_bad $_sym=$_val" ;; - *) grep -qx "$_sym=$_val" "$tree/.config" \ - || grep -qx "$_sym=\"${_val//\"/}\"" "$tree/.config" \ - || _cs_bad="$_cs_bad $_sym=$_val" ;; - esac - done - if [ -n "$_cs_bad" ]; then - echo "!! CONFIG_SET did not take -- the kernel reverted these:" - for _cs in $_cs_bad; do - echo " $_cs (now: $(grep -E "^(# )?${_cs%%=*}[= ]" "$tree/.config" | head -1))" - done - echo " a dependency is missing, or $VERSION has no such symbol." - echo " Look it up: cd $tree && make menuconfig (then / and the name)" - exit 1 - fi - echo "# all $(echo $CONFIG_SET | wc -w) set, read back from .config" -fi - -# what a Wayland compositor needs; a warning, not a refusal -for opt in CONFIG_DRM CONFIG_DRM_FBDEV_EMULATION CONFIG_INPUT_EVDEV CONFIG_SYSVIPC; do - grep -qE "^$opt=(y|m)" "$tree/.config" || echo "!! $opt is not set (sway needs it)" -done -# Drivers that need firmware are no use without the driver. These are the -# ones a laptop misses most often; the firmware for each is in FIRMWARE_ONLY. -for opt in CONFIG_IWLWIFI:"Intel wireless (AX200 and friends)" \ - CONFIG_IWLMVM:"the AX200's own MVM firmware layer" \ - CONFIG_BT_HCIBTUSB:"USB Bluetooth (the AX200's other half)" \ - CONFIG_DRM_AMDGPU:"AMD graphics"; do - _o="${opt%%:*}"; _why="${opt#*:}" - grep -qE "^$_o=(y|m)" "$tree/.config" \ - || echo "!! $_o is not set -- no $_why" -done - # FIRMWARE TROUBLE SHOULD NOT COST YOU THE KERNEL. # # This ran before the build and exited on failure, so a wrong FIRMWARE_ONLY @@ -1398,32 +735,12 @@ case "$HEADERS" in # what makes the files the headers package's. Same shape as the ESP # copies and the setuid bit. echo "# installing headers into /usr/include (as root, owned by $_hacct) ..." - # WHAT IT INSTALLED, NOT WHERE IT INSTALLED. - # - # `chown -R $_hacct /usr/include` gave this package every header on - # the system. /usr/include/gstreamer-1.0/gst/mse then belonged to - # p_linux-headers, so gst-plugins-bad could not install into its own - # directory, the repair handed it one file per round, and after three - # rounds the bundled-copy heuristic concluded meson was vendoring - # linux-headers. Every step of that was correct reasoning from a - # wrong fact. - # - # So: copy, remember exactly which paths came out of the tree, and - # chown only those. - _hlist="$(mktemp)" - ( cd "$tree/usr/include" && find . -mindepth 1 -printf '%P\n' ) > "$_hlist" cp -r "$tree/usr/include" /usr || { echo "!! could not install the headers into /usr/include" echo " if some of them belong to a third package:" echo " packagemanager reclaim $HEADERS_PKG --run" - rm -f "$_hlist"; exit 1; } - ( cd /usr/include && tr '\n' '\0' < "$_hlist" \ - | xargs -0 -r chown -h "$_hacct:$_hacct" ) 2>/dev/null || true - # the directory itself stays root's: it is shared by every package - # that ships a header, and one owner for it is what caused this - echo "# $(grep -c . "$_hlist") path(s) now owned by $_hacct;" - echo "# /usr/include itself left as $(stat -c %U /usr/include)" - rm -f "$_hlist" + exit 1; } + chown -R "$_hacct:$_hacct" /usr/include 2>/dev/null || true packagemanager reload-pkg-list "$HEADERS_PKG" >/dev/null 2>&1 || true echo "# headers installed and recorded for $_hacct" ;; *) echo "# headers: skipped (HEADERS=$HEADERS)" ;; diff --git a/lfs-phases b/lfs-phases index 0d79363..ced43ca 100755 --- a/lfs-phases +++ b/lfs-phases @@ -34,7 +34,7 @@ # scripts are also SOURCED by lfs-helper to read their metadata, and this file # comes along -- it must not shadow anything in the sourcing shell. -PKGUSR_LIB_VERSION="1.14.299" +PKGUSR_LIB_VERSION="1.14.146" _pp_build_id() { local f="${BASH_SOURCE[0]}" @@ -189,30 +189,12 @@ _pp_mirrors_for() { } _pp_fetch() { - # SAVE IT AS THE NAME THE SCRIPT ASKS FOR. - # - # A GitHub archive URL ends in the TAG, not the project: - # .../cisco/openh264/archive/refs/tags/v2.6.0.tar.gz - # so wget wrote v2.6.0.tar.gz, the unpack looked for the declared - # pkg=openh264-2.6.0.tar.gz, and the run said "no - # openh264-2.6.0.tar.gz ... and nowhere to fetch it from" one line under - # "'v2.6.0.tar.gz' saved [60302243]". Both statements were true. - # - # `pkg` is what every later phase uses, and two projects can both - # publish a v2.6.0.tar.gz, so the file has to land under that name. - local url="$1" try u out="${2:-${pkg:-}}" + local url="$1" try u for try in 1 2 3; do while IFS= read -r u; do [ -n "$u" ] || continue [ "$u" = "$url" ] || _pp_say "trying mirror: $u" - if [ -n "$out" ]; then - wget -4 -c -O "$out" "$u" && return 0 - # -O leaves an empty file behind on a 404, and -c would then - # "resume" it for ever - [ -s "$out" ] || rm -f "$out" - else - wget -4 -c "$u" && return 0 - fi + wget -4 -c "$u" && return 0 done <<< "$(_pp_mirrors_for "$url" | awk '!seen[$0]++')" [ "$try" = 3 ] && break _pp_say "every mirror failed (try $try/3) -- retrying in $((try * 15))s" @@ -258,32 +240,10 @@ _pp_unpack() { if [ "${is_git:-}" = 1 ]; then pkg_dir="$name_version" - # GIT MUST NEVER ASK A QUESTION IN A BUILD. - # - # A URL that is not there gets a 401 from GitHub (it will not admit - # a repository is missing), so git asks for a username and the build - # ends with - # fatal: could not read Username for 'https://github.com' - # ...which reads like a credentials problem on a public repository - # and, on a terminal, would simply hang. Refuse the prompt and say - # what it means. - export GIT_TERMINAL_PROMPT=0 GIT_ASKPASS=/bin/true GIT_SSH_COMMAND="ssh -oBatchMode=yes" if [ -d "$pkg_dir/.git" ]; then git -C "$pkg_dir" fetch --all --tags --prune else - rm -rf "$pkg_dir" - if ! git clone --recurse-submodules "$link" "$pkg_dir"; then - _pp_fail "could not clone $link" - echo " git was refused, and was not allowed to ask for a" >&2 - echo " password (this is a build). On a public URL that" >&2 - echo " almost always means the PATH IS WRONG: GitHub answers" >&2 - echo " 401 for a repository that does not exist rather than" >&2 - echo " admitting it is missing." >&2 - echo " Check it, from here, before editing anything else:" >&2 - echo " git ls-remote $link | head -3" >&2 - echo " Then fix url= in the stack entry." >&2 - return 1 - fi + rm -rf "$pkg_dir" && git clone --recurse-submodules "$link" "$pkg_dir" fi # A REF THAT DOES NOT EXIST IS A TYPO IN THE STACK, NOT A BUILD ERROR. # @@ -296,22 +256,10 @@ _pp_unpack() { _pp_fail "'$git_ref' is not a tag, branch or commit in $link" echo " The stack entry (or this script) asks for a ref the" >&2 echo " repository does not have. Recent tags:" >&2 - # AN EMPTY LIST UNDER A HEADING IS A RIDDLE. A fork with no - # tags printed "Recent tags:" and then nothing, which reads like - # the command failed rather than "there are none -- use a - # branch". - _pp_tags="$(git -C "$pkg_dir" tag --sort=-v:refname 2>/dev/null | head -8)" - if [ -n "$_pp_tags" ]; then - printf '%s\n' "$_pp_tags" | sed 's/^/ /' >&2 - else - echo " (none at all -- this repository is not tagged," >&2 - echo " so a branch is the only thing to ask for)" >&2 - fi + git -C "$pkg_dir" tag --sort=-v:refname 2>/dev/null | head -8 \ + | sed 's/^/ /' >&2 echo " Branches:" >&2 - # HEAD and the bare remote name are not branches anyone can ask - # for: the list printed `origin` above `origin/main`. git -C "$pkg_dir" branch -r --format='%(refname:short)' 2>/dev/null \ - | grep -vE '^[^/]+$|/HEAD$' | sed 's|^[^/]*/||' \ | head -5 | sed 's/^/ /' >&2 echo " Fix ref= in the stack entry for '${name:-$name_version}'," >&2 echo " or set git_ref= in this script, then run it again." >&2 @@ -339,36 +287,13 @@ _pp_unpack() { && _pp_say "# the build resumes in $_pp_prepared_in" return 0 fi - _pp_stage_file "$pkg" || { [ -n "${link:-}" ] && _pp_fetch "$link" "$pkg"; } - if [ ! -f "$pkg" ]; then - _pp_fail "no $pkg in $BUILD_ROOT and nowhere to fetch it from" - # SAY WHAT DID ARRIVE. This fired one line under a successful - # download and read as "the fetch failed", when the fetch had - # worked and only the NAME differed. - local _got - _got="$(ls -1t *.tar.* *.tgz *.zip 2>/dev/null | head -3)" - if [ -n "$_got" ]; then - echo " what IS here, newest first:" >&2 - printf ' %s\n' $_got >&2 - echo " if one of those is it, the entry's pkg= does not match the" >&2 - echo " name it arrived under -- a GitHub archive URL ends in the" >&2 - echo " TAG, not the project." >&2 - fi - return 1 - fi + _pp_stage_file "$pkg" || { [ -n "${link:-}" ] && _pp_fetch "$link"; } + [ -f "$pkg" ] || { _pp_fail "no $pkg in $BUILD_ROOT and nowhere to fetch it from"; return 1; } _pp_check_md5 || return 1 local li f - # EACH EXTRA UNDER ITS OWN NAME. _pp_fetch saves to its second - # argument, else to $pkg -- and this call passed none, so LLVM's patch - # was fetched with `wget -c -O llvm-project-22.1.8.src.tar.xz`. The - # tarball was bigger, wget answered 416 "already fully retrieved", and - # the build stopped on - # patch: **** Can't open patch file ../llvm-22.1.8-upstream_fix-1.patch - # (An extra LARGER than the tarball would have been appended onto it.) for li in "${additional_links[@]}"; do f="${li##*/}" - _pp_stage_file "$f" || _pp_fetch "$li" "$f" || return 1 - [ -s "$f" ] || { _pp_fail "fetched $li, and there is no $f"; return 1; } + _pp_stage_file "$f" || _pp_fetch "$li" || return 1 done if [ "${pkg_unpack:-yes}" = "no" ]; then _pp_say "fetched $pkg (the page's own commands unpack it)" diff --git a/packagemanager b/packagemanager index 4984503..2fc602f 100755 --- a/packagemanager +++ b/packagemanager @@ -27,23 +27,22 @@ installs ends up owned by that user, so `list_package <name>` records the exact manifest into /usr/src/<name>/pkg.lst. Collector groups (nimgnu_*) share directories between package-users; u_* users run GUI apps for the human. -The commands ------------- -`packagemanager --help` lists them, grouped by what you are doing, and that -list is generated beside the parsers themselves. - -THERE WAS A SECOND LIST HERE. A hand-kept "Roadmap" marking check, search, -dependencies, install, remove and reload-pkg-list as [TODO] -- all six had -been implemented for releases, and none of the twenty commands added since -appeared at all. A list of commands maintained anywhere but next to the -commands goes stale silently, and this file is the place a reader starts. - -Design decisions still open: - * should the CONFIGURE phase run as root? Every BLFS "Configuring" section - is written for root, and files it creates would be root-owned, which the - file tracking does not expect. - * the `.new` mechanism: three functions move stack files and it still - surprises people. +Roadmap (built incrementally, one command at a time) +---------------------------------------------------- + info [IMPLEMENTED] package-user info table (name/uid/gid/home/pkg_list) + check [TODO] report install state of a package + search [TODO] find a package in the book -> blfs search + dependencies [TODO] recursive dependency order -> blfs deps/order + install [TODO] resolve deps, build missing, run lfs-helper pm-install + remove [TODO] remove a package-user + its files (via pkg.lst) + reload-pkg-list [TODO] regenerate /usr/src/<name>/pkg.lst + list [TODO] list all package-users + files_with_broken_id [TODO] find files whose uid/gid don't resolve + +Design decisions still open (we'll settle these as we implement the commands): + * install: auto-run lfs-helper pm-install per package, or just print/queue? + * packagemanager_install: retired in 1.12.0 -- folded into lfs-helper pm-install + * fix `check`, implement `remove`; parameterize the hardcoded site values. """ import argparse @@ -118,35 +117,8 @@ BASE_DIR = os.environ.get("PKGUSR_BASE") or _base_dir_default() def appuser_home(): """Where application users live -- separate from package users, because - they are accounts a person uses rather than build trees. - - DEFAULT /home, NOT the package base. This returned BASE_DIR, so every - application user's home, shared directory and XDG profile link were - built under /usr/src -- `mkdir -p /usr/src/u_claude/Shared`, and a - ~/Shared/Claude link pointing into the package tree. A build tree is - not a home. Configurable as before (`appuser_home` in the config), but - the default is now the one that is right. - """ - return load_config().get("appuser_home") or APPUSER_HOME_DEFAULT - - -# ONE SUBDIRECTORY FOR ALL OF THEM. The user's layout, recorded on the -# machine as appuser_home=/home/shared_user -- and ignored: the account was -# moved to a hard-coded "/home/<user>", so /home filled up with u_* beside -# the real logins. The default is that subdirectory; the config key still -# overrides it. -APPUSER_HOME_DEFAULT = "/home/shared_user" - - -def _app_home(u): - """An application user's home: what passwd says, or where they go. - - passwd FIRST, because an account whose home has already been moved - (usermod -d) must not have its shared directory rebuilt somewhere else - -- which is exactly what happened: the homes were moved to /home and the - next `cfg app-users` run recreated Shared under /usr/src. - """ - return _home_of(u) or os.path.join(appuser_home(), u) + they are accounts a person uses rather than build trees.""" + return load_config().get("appuser_home") or BASE_DIR INSTALL_SCRIPTS_DIR = os.environ.get("PKGUSR_INSTALL_SCRIPTS", "/etc/pkgusr/install_scripts") TMP_DIR = "/tmp/packagemanager" @@ -156,7 +128,7 @@ TMP_DIR = "/tmp/packagemanager" CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf") # Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure # changes (so stale plans are ignored); TOOL_VERSION is the program version. -TOOL_VERSION = "1.14.299" +TOOL_VERSION = "1.14.146" def _build_id(): @@ -664,44 +636,7 @@ def find_blfs(): return ["blfs"] # rely on $PATH -# A PACKAGE MAY NEED A DIFFERENT BOOK, AND ONLY A WHOLE PAGE IS COHERENT. -# -# The systemd book's linux-pam page lists systemd as required; the 12.4 SysV -# book's linux-pam page has no Required section at all -- it wants only -# Shadow. Same package, two books, two dependency graphs, and this machine -# is SysV. -# -# NOT a merge. Taking the dependency list from one book and the commands -# from another produces a script neither book has ever tested: the versions -# differ (1.7.1 vs 1.7.2), the configure switches differ, the doc tarball -# differs. A whole page from the older book is something upstream shipped -# and someone ran. Half of each is not. -# -# machine.conf: -# [linux-pam] -# book = /usr/share/blfs/books/BLFS-BOOK-12_4-nochunks.html -# -# ...and everything that asks the book about THAT package -- versions, -# dependencies, the generated script -- asks the one named there. -_BOOK_FOR_PKG = {} - - -def _book_override(anchor): - """The book this package should be read from, or None.""" - if not anchor: - return None - if anchor not in _BOOK_FOR_PKG: - opts = _machine_opts_for(anchor) - _BOOK_FOR_PKG[anchor] = opts.get("book") or "" - return _BOOK_FOR_PKG[anchor] or None - - -def _blfs_book_flags(anchor=None): - over = _book_override(anchor) if anchor else None - if over: - if os.path.isfile(over): - return ["--book-file", over] - return ["--book", over] +def _blfs_book_flags(): if os.environ.get("BLFS_BOOK_FILE"): return ["--book-file", os.environ["BLFS_BOOK_FILE"]] if os.environ.get("BLFS_BOOK"): @@ -730,14 +665,7 @@ def run_blfs(args): Each call is a subprocess, and the first one for a given book INDEXES it, which takes many seconds. Announce what is happening: an install that sits silent for half a minute before its first output looks frozen.""" - # the package a call is ABOUT is its first non-flag argument, which is - # what decides whether an override applies - _anchor = next((a for a in args[1:] if not a.startswith("-")), None) \ - if len(args) > 1 else None - cmd = find_blfs() + _blfs_book_flags(_anchor) + list(args) - if _anchor and _book_override(_anchor): - printInfo(" %s: read from %s (machine.conf)" - % (_anchor, os.path.basename(_book_override(_anchor)))) + cmd = find_blfs() + _blfs_book_flags() + list(args) verb = args[0] if args else "" if verb in _BLFS_ACTIVITY: arg = args[1] if len(args) > 1 and not args[1].startswith("-") else "" @@ -804,29 +732,15 @@ def book_version(anchor): _VERSION_MAP = book_versions_all() or {} except Exception: _VERSION_MAP = {} - # THE PREFIX IS NOT PART OF THE BOOK'S NAME. - # - # `info webkitgtk` asks with the ACCOUNT name, `p_webkitgtk`, which no - # book anchor matches -- so every call missed the map and fell through to - # `blfs debug p_webkitgtk`, a subprocess that reads the book cache to - # answer one lookup and then fails. That is the pause after - # "looking up p_webkitgtk in the book", and the reason `info` showed no - # book version to compare against. - _tries = [anchor] - for _pre in (pkgusr_prefix(), user_prefix(), collector_prefix()): - if _pre and anchor.startswith(_pre): - _tries.append(anchor[len(_pre):]) - for _a in _tries: - hit = _VERSION_MAP.get(_a) - if hit: - return hit + hit = _VERSION_MAP.get(anchor) + if hit: + return hit if _VERSION_MAP: - for _a in _tries: - low = _a.lower() - for a, nv in _VERSION_MAP.items(): - if a.lower() == low: - return nv - rc, out, _ = run_blfs(["debug", _tries[-1]]) + low = anchor.lower() + for a, nv in _VERSION_MAP.items(): + if a.lower() == low: + return nv + rc, out, _ = run_blfs(["debug", anchor]) if rc != 0: return None m = re.search(r"^title\s*:\s*(.+)$", out, re.M) @@ -854,54 +768,9 @@ def book_rdeps(target, no_recommended=False, optional=False): return result -def _versions_cache_path(): - """Where the anchor -> version map is kept between runs.""" - d = os.path.join("/var", "cache", "pkgusr") - try: - os.makedirs(d, exist_ok=True) - except OSError: - d = "/tmp" - return os.path.join(d, "book-versions.json") - - def book_versions_all(): - """Map anchor -> name_version for the whole book. - - ONE CALL WAS NOT ENOUGH: IT IS ONE CALL PER PROCESS. `blfs versions` - re-reads the whole book to build this map, which is eighteen seconds on - a 12 MB book -- and `info` pays it to answer one question. The page - index is cached; this map was not, so every command that wanted a - version rebuilt it from scratch. - - Cached on disk against the book's own size and mtime: the moment the - book file changes the cache is ignored, so a stale map cannot outlive - the book it came from. - """ - # WHICH book: the explicit one if given, else whichever cached book file - # is newest -- the same file `blfs` would read. - _bf = os.environ.get("BLFS_BOOK_FILE") or "" - if not (_bf and os.path.isfile(_bf)): - _bf = "" - _books = sorted(glob.glob("/usr/share/blfs/books/*.html"), - key=lambda f: os.stat(f).st_mtime, reverse=True) - if _books: - _bf = _books[0] - _key = None - if _bf: - try: - _st = os.stat(_bf) - _key = "%s:%d:%d" % (_bf, _st.st_size, int(_st.st_mtime)) - except OSError: - _key = None - _cp = _versions_cache_path() - if _key: - try: - with open(_cp, encoding="utf-8") as f: - _c = json.load(f) - if _c.get("key") == _key and isinstance(_c.get("versions"), dict): - return _c["versions"] - except (OSError, ValueError): - pass + """Map anchor -> name_version for the whole book, in ONE blfs call (so update + doesn't spawn a process per package).""" rc, out, _ = run_blfs(["versions"]) versions = {} if rc == 0: @@ -911,19 +780,6 @@ def book_versions_all(): a, nv = a.strip(), nv.strip() if a and nv and " " not in a: versions[a] = nv - if versions and _key: - # CONFIRM IT CAN BE READ BACK, not just written: a cache that fails - # to load silently turns into the 18 seconds all over again, on - # every command, invisibly. - try: - with open(_cp, "w", encoding="utf-8") as f: - json.dump({"key": _key, "versions": versions}, f) - with open(_cp, encoding="utf-8") as f: - if json.load(f).get("key") != _key: - printWarning(" the version cache did not read back; " - "every command will rebuild it") - except (OSError, ValueError) as e: - printInfo(" (not caching the book version map: %s)" % e) return versions @@ -1017,20 +873,9 @@ def _parse_bash_vars(text, scalars=(), arrays=()): # trailing comment is allowed and ignored. out = {} for key in scalars: - # SINGLE QUOTES TOO, AND THE BOOK USES THEM. - # - # This read only key="value". The book-generated scripts write - # name_version='Rustc-1.93.1' - # so name_version was never parsed for any package built from the - # book -- and that is where the installed version comes from. - # `info webkitgtk` printed uid, home, 1727 files and every - # dependency, and not WHICH webkitgtk; `update` filed it under - # "cannot judge"; late-deps reported 28 packages with no recorded - # version. One quote character, three symptoms. - m = re.search(rf'^{key}=(?:"(?P<d>.*?)"|\x27(?P<s>.*?)\x27)[ \\t]*(#.*)?$', text, re.M) + m = re.search(rf'^{key}="(.*?)"[ \t]*(#.*)?$', text, re.M) if m: - out[key] = m.group("d") if m.group("d") is not None \ - else m.group("s") + out[key] = m.group(1) for key in arrays: m = re.search(rf'^{key}=\((.*?)\)[ \t]*(#.*)?$', text, re.M) if m: @@ -1226,26 +1071,6 @@ class User: @property def version(self): - # THE VERSION IS WHAT FINISHED INSTALLING. - # - # install_last is staged into the home BEFORE the build, so reading - # the version out of it meant a FAILED build advanced it: webkitgtk - # 2.52.5 died at configure and `update` then said "already up to - # date (2.52.5)" with 2.50.5's libraries still on disk. - # - # .pkgusr-installed is written by lfs-helper only after every phase - # succeeded. Where it is absent -- every package built before this - # release -- fall back to install_last, which is the old behaviour - # and no worse than it was. - if self.home: - try: - with open(os.path.join(self.home, ".pkgusr-installed"), - encoding="utf-8", errors="replace") as f: - done = f.read().strip() - if done: - return done - except OSError: - pass return self.meta.get("name_version") @@ -1302,46 +1127,8 @@ def print_user_details(u): head += f" ({u.version})" print(f"\n{colors.cyan}{head}{colors.normal} " f"[{_STATE_COLOR.get(state, '')}{state}{colors.normal}]") - # `why` EXISTS TO EXPLAIN A VERDICT THAT NEEDS EXPLAINING. - # - # For an installed package it was printing the routine sentence every - # time -- "recorded files present (the page names no targets to verify - # against)" above the very fields it is talking about -- so the one line - # that ever matters was indistinguishable from noise. Routine reasons - # are dropped; anything unusual is kept and labelled `note`, because for - # an installed package it is a note, not a justification. - _routine = ("all installed_* targets present", - "recorded files present", - "its script names no verifiable target") - _dull = any(r in (why or "") for r in _routine) - if why and not (state == "installed" and _dull): - _label = "note" if state == "installed" else "why" - print(f" {colors.dim}{_label:<12}{colors.normal}: {why}") - # THE VERSION IS THE FIRST THING ANYONE WANTS, and it was only in the - # header -- shown when `u.version` happened to be set, and silent - # otherwise. webkitgtk printed uid, home, 1727 files and every - # dependency, and not which webkitgtk. - # - # `name_version` is in the install script itself (WebKitGTK-2.50.2), so - # when the recorded version is missing there is still an answer; it just - # has to say which answer it is. And the BOOK's version belongs beside - # it: "is this current" is the question behind the question. - # JUST THE NUMBER. The header already says WebKitGTK-2.50.5, so - # repeating the name in the field made the two lines look like two - # different facts. - def _ver_only(nv): - m2 = re.search(r"-([0-9][0-9A-Za-z.\-_+]*)$", nv or "") - return m2.group(1) if m2 else (nv or "") - _nv = m.get("name_version") or "" - _inst = _ver_only(u.version or "") - if not _inst and _nv: - _inst = _ver_only(_nv) - if _inst: - _inst += " (from the install script, not recorded)" - _book = _ver_only(expected or "") + print(f" {colors.dim}why{colors.normal:<9}: {why}") rows = [ - ("version", _inst or "not recorded, and the script does not name one"), - ("book", _book if _book and _book not in _inst else None), ("uid/gid", f"{u.uid}/{u.gid}" if u.has_user else None), ("home", u.home), ("link", m.get("link")), @@ -1359,25 +1146,6 @@ def print_user_details(u): if label == "info": lines = [ln.strip() for ln in str(value).replace("\\n", "\n").split("\n") if ln.strip()] - # THE BOOK'S BLURB REPEATS THE DEPENDENCY LISTS, which are three - # rows further down: webkitgtk printed 18 required packages - # twice, in different spellings, and the description they were - # attached to scrolled away. Keep the prose, drop the lines that - # are just a dependency list. - _deps = set() - for _k in ("required", "recommended", "optional"): - for _d in m.get(_k, []): - _deps.add(_d.split("-")[0].lower()) - def _is_dep_list(ln): - parts = [x.strip() for x in ln.replace(" and ", ", ").split(",") - if x.strip()] - if len(parts) < 3: - return False - hit = sum(1 for x in parts - if x.split("-")[0].lower().replace(" ", "-") in _deps - or x.split("-")[0].lower() in _deps) - return hit >= max(2, len(parts) // 2) - lines = [ln for ln in lines if not _is_dep_list(ln)] if not lines: continue print(f" {label:<12}: {lines[0]}") @@ -1583,59 +1351,6 @@ def _prewarm_scripts(plan, script_dir, force_regen=False): printInfo(" %d written" % made) -def _report_stale_requires(names, _verbose=False): - """For a single-package update: which of its requirements are older here. - - The prereq check catches a missing TOOL. It says nothing about a - dependency that is present but older than the version the book's page - was written against -- and that is the real question behind "can I - update webkitgtk without updating glib2": - - webkitgtk 2.52.5 requires GLib-2.88.3; this system has GLib-2.86.4. - - That may build perfectly. It may also fail at configure, or worse, - build and misbehave. Nobody can tell from here, so this reports the - numbers and leaves the decision alone -- refusing would break the one - workflow it exists to serve, which is updating one package at a time. - """ - for name in names[:4]: - u = gather_user(name) - if not u.meta: - continue - rows = [] - for kind in ("required", "recommended"): - for dep in u.meta.get(kind, []): - a = _resolve_dep(dep, _pkg_dirs()) if "_resolve_dep" in globals() \ - else None - want = book_version(a or dep.rsplit("-", 1)[0]) or dep - have = None - if a: - have = gather_user(a).version or \ - (gather_user(a).meta or {}).get("name_version") - if have and want and not _same_version(have, want): - _row = (kind, a or dep, have, want) - if _row not in rows: # two deps can resolve alike - rows.append(_row) - if not rows: - continue - print() - printWarning(" %s's requirements that differ from the book here:" - % name) - _show = rows if _verbose else rows[:12] - for kind, a, have, want in _show: - printWarning(" %-11s %-22s installed %-24s book %s" - % (kind, a, have, want)) - if len(rows) > len(_show): - # A TRUNCATED LIST MUST SAY HOW TO SEE THE REST. "... and 5 - # more" with no way to ask is a dead end. - printInfo(" ... and %d more -- all of them: packagemanager " - "update %s -v" % (len(rows) - len(_show), name)) - printInfo(" This is not a refusal: one of them may be exactly what") - printInfo(" you do not want to touch. It is what the page was") - printInfo(" written against, so if the build fails at configure,") - printInfo(" start with these.") - - def _stop_if_prereqs_missing(args): """A plan that cannot finish should not start. @@ -1977,45 +1692,9 @@ def _read_conf_section(path, anchor): return {} -def _opt_sections(anchor, keys, first_only=False): - """Which sections actually define these keys -- [all], the package, or both. - - Both messages below used to name the package section whatever the truth - was: `machine.conf [rust]: x.py --jobs 13` for a file with no [rust] - section, and `[somepkg] and [all] set weird_option` where only one of - them did. A message that names a thing must be checked against the - thing, and here the thing is two files and two sections. - """ - keys = set(keys) - out = [] - for sec in (anchor, "all"): - for f in (os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), - os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES)): - if keys & set(_read_conf_section(f, sec)): - out.append(sec) - break - # the package section overrides [all], so for a value that was USED - # only the winner is worth naming: "[webkitgtk] and [all]: ninja -j8" - # credited a section whose 13 had just been overridden by 8. - if out and first_only: - break - return out - - def _machine_opts_for(anchor): - # [all] FIRST, THEN THE PACKAGE. - # - # `jobs` had to be written per package, because ninja ignores MAKEFLAGS - # and there was no way to say "never more than 13 on this machine" once. - # An [all] section in either file applies to every package, and a named - # section overrides it -- so [webkitgtk] jobs = 8 still wins over - # [all] jobs = 13. opts = _read_conf_section( - os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), "all") - opts.update(_read_conf_section( - os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), "all")) - opts.update(_read_conf_section( - os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), anchor)) + os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), anchor) opts.update(_read_conf_section( os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), anchor)) return opts @@ -2288,29 +1967,6 @@ def _check_script_prereqs(anchor, script_path): _warn_missing_build_tools(anchor, script_path) -def _has_command(body, cmd): - r"""Is `cmd` really invoked here -- not mentioned in a comment? - - THE BOOK PUTS IT AFTER `&&`. webkitgtk's page writes - mkdir -vp build && cd build && cmake -D CMAKE_BUILD_TYPE=Release \\ - and the detector used `^\s*cmake`, so it found no cmake line and - reported "machine.conf [webkitgtk] set USE_WOFF2 ... but the script has - no `meson setup` or `cmake` line -- NOT applied". Seven options, - silently dropped, on the one package where they matter most. - `_insert_on_command` had already learnt about `&&`; only the test that - decides whether to call it had not. - - Comment lines do not count: the same page opens with a NOTE about "the - cmake command", which is prose. - """ - for line in body.split("\n"): - if line.lstrip().startswith("#"): - continue - if re.search(r"(?:^|\s|&&|;)%s\b" % re.escape(cmd), line): - return True - return False - - def _insert_on_command(body, cmd, arg): """Add an argument to the first REAL invocation of `cmd`, skipping comments -- a book NOTE that mentions the command by name is prose.""" @@ -2324,68 +1980,9 @@ def _insert_on_command(body, cmd, arg): return "\n".join(out) -def _meson_option_names(anchor): - """The options this package's own source declares, if it is unpacked. - - meson fails the whole build on an unknown option, and a machine.conf - section is written by hand: `[gst10-plugins-bad] alsa = enabled` looks - perfectly reasonable and is wrong, because `alsa` is a - gst-plugins-BASE option. The answer is in the source tree, which is on - disk by the time anything is built. - """ - home = pkgusr_home(anchor) - src = os.path.join(home, "src") - if not os.path.isdir(src): - return None - names = set() - found = False - for d in sorted(os.listdir(src)): - for f in ("meson.options", "meson_options.txt"): - path = os.path.join(src, d, f) - if not os.path.isfile(path): - continue - found = True - try: - body = open(path, encoding="utf-8", errors="replace").read() - except OSError: - continue - names |= set(re.findall(r"""^\s*option\(\s*['"]([^'"]+)['"]""", - body, re.M)) - return names if found else None - - -def _warn_unknown_meson_opts(anchor, opts): - """Drop options the package does not have, and say which they are.""" - known = _meson_option_names(anchor) - if not known: - return opts # not unpacked yet, or not a meson package - # meson's own built-ins are not in meson.options and are always valid - builtin = {"wrap_mode", "buildtype", "prefix", "libdir", "default_library", - "c_args", "cpp_args", "c_link_args", "cpp_link_args", - "b_lto", "b_ndebug", "werror", "optimization", "strip", - "auto_features", "backend", "jobs", "configure_args"} - bad = {k: v for k, v in opts.items() if k not in known and k not in builtin} - if not bad: - return opts - for k in sorted(bad): - printWarning(" machine.conf [%s]: '%s' is not an option of this " - "package -- NOT passed" % (anchor, k)) - # the near misses are what makes this useful: `alsa` under - # [gst10-plugins-bad] belongs under [gst10-plugins-base] - near = sorted(n for n in known if k in n or n in k)[:4] - if near: - printInfo(" it does have: %s" % ", ".join(near)) - printInfo(" meson would have failed the whole build on this" - " (\"Unknown option\").") - printInfo(" the options it does have: grep -h \"^option(\" %s" - % os.path.join(pkgusr_home(anchor), "src", "*", - "meson.options")) - return {k: v for k, v in opts.items() if k not in bad} - - def apply_machine_opts(anchor, script_path): """Merge machine.conf's [anchor] options into the script's meson line.""" - opts = _warn_unknown_meson_opts(anchor, _machine_opts_for(anchor)) + opts = _machine_opts_for(anchor) if not opts or not script_path or not os.path.isfile(script_path): return try: @@ -2404,10 +2001,6 @@ def apply_machine_opts(anchor, script_path): # ERROR: Unknown option: "needs". # One file, two readers: each must ignore what belongs to the other. opts.pop("needs", None) - # Same hazard, same answer: expect= is a post-install feature probe read - # by the stack runner. It belongs in a stack line, but this file is - # where a key that is not a build option does damage, so drop it here too. - opts.pop("expect", None) # ...and REMOVE it if an earlier version already wrote it into the script. # 1.13.69 stopped adding -Dneeds=, but every script generated while the # bug existed still carries it, and merging is idempotent so nothing took @@ -2442,45 +2035,13 @@ def apply_machine_opts(anchor, script_path): # the failure it prevents arrives hours in, as a killed compiler. _jobs = opts.pop("jobs", None) if _jobs: - _n = str(_jobs).lstrip("-j") - # NINJA IS NOT THE ONLY ONE THAT IGNORES THE ENVIRONMENT. - # - # rust's own page says it: the build "can be mostly limited to a - # specified number of processors by a COMBINATION of adding the - # switch --jobs <N> on each invocation of ./x.py and using the - # CARGO_BUILD_JOBS environment variable". With only the variable - # set, `./x.py build` used every core. - # - # Each pattern refuses to touch a line that already names -j or - # --jobs, so an explicit choice in a script or a stack entry wins. - _subs = ( - # ninja: no environment variable at all - # ...and not only at the start of a line: `cmake .. && ninja` is - # how half the pages are written, and that ninja was untouched - (r"(?m)(^|&&[ \t]*|;[ \t]*)(ninja\b(?![^\n]*(\s-j|\s--jobs)))", - r"\1\2 -j" + _n, "ninja -j%s"), - # ./x.py build|install|test|dist - (r"(?m)^(\s*\.?/?x\.py\s+\w+(?![^\n]*(\s-j|\s--jobs)))", - r"\1 --jobs " + _n, "x.py --jobs %s"), - # cargo build / cbuild / install / cinstall - (r"(?m)^(\s*cargo\s+c?(?:build|install)\b(?![^\n]*(\s-j|\s--jobs)))", - r"\1 -j " + _n, "cargo -j %s"), - ) - _changed = [] - for _pat, _rep, _what in _subs: - _nb = re.sub(_pat, _rep, body) - if _nb != body: - body = _nb - _changed.append(_what % _n) - if _changed: + _nb = re.sub(r"(?m)^(\s*ninja\b(?![^\n]*\s-j))", + r"\1 -j" + str(_jobs).lstrip("-j"), body) + if _nb != body: + body = _nb with open(script_path, "w") as f: f.write(body) - printInfo(" machine.conf %s: %s" - % (" and ".join("[%s]" % w - for w in _opt_sections(anchor, ["jobs"], - first_only=True)) - or "[%s]" % anchor, - ", ".join(_changed))) + printInfo(" machine.conf [%s]: ninja -j%s" % (anchor, _jobs)) _wrap = opts.pop("wrap_mode", None) if _wrap: if "meson setup" not in body: @@ -2496,20 +2057,6 @@ def apply_machine_opts(anchor, script_path): with open(script_path, "w") as f: f.write(body) printInfo(" machine.conf [%s]: wrap-mode %s" % (anchor, _wrap)) - # KEYS THAT ADDRESS THE TOOL, NOT THE BUILD. - # - # `[linux-pam] book = .../BLFS-BOOK-12_4-nochunks.html` tells the tool - # WHICH BOOK to read that package from -- and it went straight onto the - # build line: - # meson setup -Dbook=/usr/share/blfs/books/... .. - # ERROR: Unknown option: "book". - # jobs and configure_args were already popped out one at a time; a - # reserved SET means the next such key cannot leak the same way, and - # `_warn_unknown_meson_opts` cannot save us here because at script - # generation time the source is not unpacked yet and it has no - # meson.options to check against. - for _reserved in ("book",): - opts.pop(_reserved, None) _conf_args = opts.pop("configure_args", None) if _conf_args: if "./configure" not in body: @@ -2565,26 +2112,11 @@ def apply_machine_opts(anchor, script_path): # and the person had nowhere to put -D ENABLE_SPELLCHECK=OFF. Both build # systems take -Dkey=value; only the command word differs. (autotools is # already handled above, by configure_args.) - # NOTHING LEFT TO APPLY IS NOT A PROBLEM. - # - # `[all] jobs = 13` was consumed above (x.py --jobs 13, printed and - # correct) and then this warned that rust's options had NOT been - # applied -- with nothing left in opts to apply. - if not opts: - return - _anchor_cmd = ("meson setup" if _has_command(body, "meson setup") - else ("cmake" if _has_command(body, "cmake") else "")) + _anchor_cmd = "meson setup" if "meson setup" in body else ( + "cmake" if re.search(r"^\s*cmake\b", body, re.M) else "") if not _anchor_cmd: - # ...AND NAME THE SECTION THAT ACTUALLY EXISTS. It said - # machine.conf has [rust] but ... - # about a file with no [rust] section at all: the keys had come from - # [all]. A message that names a thing must be checked against the - # thing. - _where = _opt_sections(anchor, opts) - printWarning(" machine.conf %s set %s for %s, but the script has no " - "`meson setup` or `cmake` line -- NOT applied" - % (" and ".join("[%s]" % w for w in _where) or "(unknown)", - ", ".join(sorted(opts)), anchor)) + printWarning(f" machine.conf has [{anchor}] but the script has no " + f"`meson setup` or `cmake` line -- options NOT applied") printWarning(" (an autotools package takes configure_args instead)") return new = body @@ -2738,26 +2270,14 @@ def install_targets(meta): """Collect concrete things the install script says it installs: programs, libraries, directories (from installed_program(s)/-libraries/ -director(y|ies)).""" - # ONE SCALAR, TWO PATHS. The book writes "Installed Directories: - # /usr/include/opus and /usr/share/doc/opus-1.6.1", and both ended up in - # a single `installed_directory=`. The check then looked for a - # directory literally called - # "/usr/include/opus /usr/share/doc/opus-1.6.1" - # which of course does not exist -- so opus read "unvalidated: missing: - # dir /usr/include/opus ..." with /usr/include/opus sitting right there. - # Same for a program or library field with two names in it. - def _paths(v): - return [x for x in str(v or "").split() if x] - progs, libs, dirs = [], [], [] - progs += _paths(meta.get("installed_program")) - for _p2 in meta.get("installed_programs", []): - progs += _paths(_p2) - for _l in meta.get("installed_libraries", []): - libs += _paths(_l) - dirs += _paths(meta.get("installed_directory")) - for _d in meta.get("installed_directories", []): - dirs += _paths(_d) + if meta.get("installed_program"): + progs.append(meta["installed_program"]) + progs += meta.get("installed_programs", []) + libs += meta.get("installed_libraries", []) + if meta.get("installed_directory"): + dirs.append(meta["installed_directory"]) + dirs += meta.get("installed_directories", []) # A STACK ENTRY'S OWN DECLARATION WINS OVER A STALE RECORD. # # If the stack says what proves this package installed, use that: it is @@ -2870,25 +2390,6 @@ def assess(u, expected=None, run_cmd=True): and missing[0] == f"program {_pkgname}"): return "unvalidated", ("its script names no verifiable target" " (rebuilt scripts declare one)") - # A DOC DIRECTORY NOBODY WAS GOING TO BUILD IS NOT A FAULT. - # - # /usr/share/doc/opus-1.6.1 exists only if Doxygen and texlive are - # installed, and this system deliberately has neither -- the - # generator comments those steps out. So the page names a directory - # that was never going to appear, and opus read "unvalidated" - # because of it while every header and library was in place. - # the path COMPONENT, not a /usr prefix: a doc directory is a doc - # directory under any prefix, and anchoring on /usr/share/doc made - # this untestable anywhere else - _docs = [x for x in missing - if x.startswith("dir ") - and ("/share/doc/" in x or "/share/gtk-doc" in x - or "/share/info" in x or "/share/man" in x)] - if _docs and len(_docs) == len(missing): - return "installed", ("docs not built (no doxygen/texlive here): %s" - % ", ".join(d[4:] for d in _docs[:2])) - if _docs: - missing = [x for x in missing if x not in _docs] if missing: # SAY WHERE THE EXPECTATION CAME FROM. "missing: dir # /usr/include/wlr" sent the user to look for a directory; the @@ -3022,7 +2523,7 @@ def cmd_verify(args): printWarning("the book named no file to check):") for n in _act: u = gather_user(n) - st, why = assess(u, None, run_cmd=False) + st, why = assess(u, versions.get(u.meta.get("name") or n)) printWarning(" %-24s %-14s %s" % (n, st, why)) # ONLY WHAT THIS COMMAND CAN ACTUALLY BUILD. # @@ -3084,30 +2585,15 @@ def cmd_info(args): if st in ("not installed", "unvalidated") and not args.no_book: print_book_info(u.name or u.search) - # THREE BUGS IN TWO LINES, and `info` crashed after printing everything - # it was asked for: - # NameError: name '_states' is not defined - # `_states` was never initialised here (it belongs to verify's loop, - # where this was copied from), `name` does not exist in this scope -- - # `anchor` does -- and the line was there twice. The dict is worth - # having: it is what names the packages that need attention. - counts, _states = {}, {} + counts = {} for u in users: anchor = u.meta.get("name") or u.name or u.search st, _ = assess(u, versions.get(anchor), run_cmd=False) counts[st] = counts.get(st, 0) + 1 - _states[anchor] = st + _states[name] = st + _states[name] = st print(f"\nTotal: {len(users)} " + " ".join(f"{k}: {v}" for k, v in sorted(counts.items()))) - # ...and say which ones, where the eye lands, rather than leaving the - # reader to match a count against the list above. - _act = [n for n, st in _states.items() - if st in ("not installed", "broken")] - if _act: - print() - printWarning("%d of these need attention: %s" - % (len(_act), ", ".join(sorted(_act)))) - printInfo(" what and why: packagemanager verify %s" % " ".join(sorted(_act))) # =========================================================================== # @@ -3126,13 +2612,6 @@ def cmd_info(args): COLLECTOR_GID_MIN = 90000 COLLECTOR_GID_MAX = 99000 -# Application users (u_*) get their OWN range too. They were created with -# the package-user range, so u_claude took 10349 and the next package user -# started after it -- an application account in the middle of the package -# accounts, uid and private gid alike. -APP_UID_MIN = 30000 -APP_UID_MAX = 39999 - def nimgnu_name(name): """The collector group for a package or directory name. @@ -3547,23 +3026,18 @@ def cmd_nimgnu_list(args): def cmd_nimgnu_create(args): g = nimgnu_name(args.name) - if _group_exists(g): - printInfo(f"{g}: already exists.") + if _group_exists(g) and _user_exists(g): + printInfo(f"{g}: already exists (group + user).") return printHeader(f"Create collector {g}") print() - # A GROUP, IN THE COLLECTOR RANGE -- no account. This ran - # the package-user helper with the package range, which made a login account - # named after the group (home /usr/src/pkgusr/p_nimgnu_python-modules) - # and a gid in the middle of the package users (10096). Every other - # collector group comes from _create_collector_group at 90000+. - if _create_collector_group(g): - printSuccess(f"{g} created (gid {grp.getgrnam(g).gr_gid if grp else '?'}).") + cmd = find_add_package_user() + [g, g, "10000", "20000", g, "10000", "20000"] + printInfo(f"Creating collector group + user: {' '.join(cmd)}") + if _run(cmd): + printSuccess(f"{g} created (group + collector user).") + report_created_user(g, kind="collector user") printInfo(f" assign directories to it with: " f"packagemanager add-dir-to-sysgroup {args.name} <dir>") - else: - printError(f"could not create {g}") - return 1 def cmd_nimgnu_add(args): @@ -3736,11 +3210,6 @@ SHARED_DIR_PATTERNS = [ "/usr/share/aclocal", "/usr/share/gtk-doc/html", "/usr/lib/systemd/system", "/usr/lib/udev/rules.d", "/etc/xdg/autostart", "/usr/libexec", - # /etc/profile.d is shared by nature: rust, bash-completion, gettext and - # the book's own config pages each drop ONE file in it. Without this the - # first package to create the directory owns it -- p_libnice here -- and - # every later package has to be granted it one at a time. - "/etc/profile.d", "/usr/share/terminfo/*", ] @@ -4177,10 +3646,7 @@ def install_tag(anchor, book_version_str): if u.state != "installed": return colors.yellow + "not installed" + colors.normal if u.version and book_version_str and u.version != book_version_str: - # "the book has moved on", not "this will be updated": nothing in a - # stack run changes an installed package. - return (colors.cyan + f"installed {u.version} (book is newer)" - + colors.normal) + return colors.cyan + f"installed {u.version} (update available)" + colors.normal return colors.green + f"installed {u.version or ''}".rstrip() + colors.normal @@ -4431,31 +3897,6 @@ def build_install_plan(args): install order (deps first); reason in {new, update, reinstall}. why maps anchor -> (kind, parent) so we can show 'required by <parent>'.""" ignore = {x for x in (args.ignore or "").split(",") if x} - # THE AVOID LIST IS A PROPERTY OF THE MACHINE, NOT OF A STACK RUN. - # - # `install linux-pam --run --recursive` planned **Systemd-261.2** on a - # SysV system. avoid.stack says `avoid systemd` and has since 1.14.224 -- - # but only `stack` ever read it, so a bare install walked the book's - # graph with no exclusions at all and offered to install the one thing - # this machine must never have. - # - # A package named ON THE COMMAND LINE is still built: asking for it - # explicitly is an instruction, and refusing that would be second- - # guessing. It is dependencies that are filtered. - _named = {a for a in (args.packages or [])} - _avoided = _machine_avoids() - _named - if _avoided: - ignore |= _avoided - # SAY IT, because an avoided REQUIRED dependency changes the build. - # linux-pam's page (in the systemd book) lists systemd as required; - # skipping it is the right call on a SysV machine and the reader - # still needs to know it happened. - printInfo(" %d package(s) this machine never builds are excluded " - "from the plan" % len(_avoided)) - printInfo(" (avoid.stack; a package you name yourself is still " - "built). If the plan") - printInfo(" then looks short, one of them was a dependency: " - "packagemanager deps <pkg>") plan, seen, missing_all, why = [], set(), [], {} # WHAT IS ALREADY DONE, counted as we go. # @@ -4705,35 +4146,12 @@ def cmd_install(args, _replanned=False): # because libass was never installed and was never going to be. The # plan knows; it just did not say. if not getattr(args, "recursive", False) and len(plan) == 1: - # A SWALLOWED EXCEPTION HERE IS SILENCE WHERE THE WARNING SHOULD BE. - # - # `except (SystemExit, Exception): _missing = []` made a failed - # dependency lookup indistinguishable from "no missing - # dependencies", and the trap this whole block exists to catch went - # uncaught -- numpy planned alone and died on - # BackendUnavailable: Cannot import 'mesonpy' - # with meson_python (a book page, and numpy's Required) never built. - _missing, _why_not = [], "" try: - # the SAME ignore set the plan was built with (build_install_plan - # has its own local; this referenced a name that does not exist - # here, so every non-recursive install printed "NameError: name - # 'ignore' is not defined" instead of its missing dependencies) - _ignore = {x for x in (getattr(args, "ignore", "") or "").split(",") if x} - _deps, _ = blfs_order_anchors(plan[0][0], ignore=_ignore) + _deps, _ = blfs_order_anchors(plan[0][0], ignore=ignore) _missing = [nv for a, nv, _k, parent in _deps if parent and gather_user(a).state != "installed"] - except SystemExit as e: - _why_not = "the book lookup exited (%s)" % e - except Exception as e: - _why_not = "%s: %s" % (type(e).__name__, e) - if _why_not: - printWarning(" could not work out what %s depends on -- %s" - % (plan[0][1], _why_not)) - printWarning(" so this plan builds ONLY %s, and a missing" - " dependency will not be" % plan[0][1]) - printWarning(" reported until the build fails on it. Safer:" - " --recursive") + except (SystemExit, Exception): + _missing = [] if _missing: printWarning(" %d dependenc%s of %s %s NOT installed:" % (len(_missing), "y" if len(_missing) == 1 else "ies", @@ -4929,16 +4347,6 @@ def _orphan_sweep(user, old_paths, since): if old_paths is None: return 0 protected = _PROTECTED_PREFIXES + (BASE_DIR.rstrip("/") + "/",) - # NEVER ANOTHER PACKAGE'S FILE. An old manifest can list a path that - # has since moved to someone else: switching Mesa to glvnd hands - # libGL.so.1 to libglvnd, which installs it BEFORE the Mesa rebuild -- - # so it is old, in Mesa's old list, and not rewritten by Mesa. mtime - # alone would delete the GL library. A package cleans only what it - # still owns. - try: - own_uid = pwd.getpwnam(user).pw_uid if pwd else None - except KeyError: - own_uid = None victims = [] for p in sorted(old_paths): if any(p == pre.rstrip("/") or p.startswith(pre) for pre in protected): @@ -4946,10 +4354,7 @@ def _orphan_sweep(user, old_paths, since): if not os.path.lexists(p): continue try: - st = os.lstat(p) - if own_uid is not None and st.st_uid != own_uid: - continue - if st.st_mtime < since: + if os.path.getmtime(p) < since: victims.append(p) except OSError: continue @@ -5096,13 +4501,9 @@ def _report_no_book(no_book, consult_lfs, no_record=(), unjudged=(), print() if no_record: printInfo(f" {len(no_record)} installed, but no recorded version " - f"here -- what the book carries is shown beside each:") - for _nr in sorted(no_record)[:12]: - printInfo(" " + _nr) - if len(no_record) > 12: - printInfo(" ... and %d more" % (len(no_record) - 12)) - printInfo(" the installed version, if the script still names it:" - " packagemanager info <name>") + f"(built before packagemanager, or its record was lost):") + printInfo(" " + ", ".join(sorted(no_record)[:12]) + + (" ..." if len(no_record) > 12 else "")) if unjudged: printInfo(f" {len(unjudged)} with no completed-install record here " f"(the LFS build tracks these in its own manifests):") @@ -5175,65 +4576,7 @@ def _systemctl_users(steps): return out -def _machine_avoids(): - """Everything this machine never builds, from avoid.stack. - - `avoid` entries were only consulted by `stack`, so `install` and - `update` walked the book's dependency graph with nothing excluded. The - exclusions describe the MACHINE -- that is why they were moved into - their own file in 1.14.224 -- so every plan should honour them. - """ - out = set() - d = STACK_DIR if "STACK_DIR" in globals() else "/etc/pkgusr/stacks" - for f in sorted(glob.glob(os.path.join(d, "*.stack"))): - try: - for r in _parse_stack_file(f): - if r["kind"] == "avoid": - out.add(r["target"]) - except (ValueError, OSError): - continue - return out - - -def _stack_targets(name): - """The packages a stack builds -- book/git/tar entries, not checks.""" - path = name if os.path.sep in name else os.path.join( - STACK_DIR if "STACK_DIR" in globals() else "/etc/pkgusr/stacks", - name if name.endswith(".stack") else name + ".stack") - if not os.path.isfile(path): - printError("no such stack: %s (%s)" % (name, path)) - return None - try: - rows = _parse_stack_file(path) - except ValueError as e: - printError(str(e)) - return None - return [r["target"] for r in rows - if r["kind"] in ("book", "git", "tar")] - - def cmd_update(args): - # "UPDATE THE MEDIA CHAIN" WITHOUT TYPING EIGHTEEN NAMES. - # - # `stack media --run` builds what is MISSING and never updates what is - # installed -- correct, and not what someone means by "update the media - # stack". The other end, `update webkitgtk`, is one package. Between - # them there was nothing but a list of names to type out. - if getattr(args, "stack", None): - _tg = _stack_targets(args.stack) - if _tg is None: - return 2 - if args.packages: - printInfo(" --stack %s and named packages: using the named ones" - % args.stack) - else: - args.packages = _tg - printInfo(" --stack %s: %d package(s) considered -- %s" - % (args.stack, len(_tg), ", ".join(_tg[:8]) - + (" ..." if len(_tg) > 8 else ""))) - printInfo(" only those the book has moved past will appear in " - "the plan.") - printInfo(" pick from them instead of taking all: --select") # --local: update a single package from a custom/local install script # (e.g. a git-based package). if getattr(args, "local", None): @@ -5281,7 +4624,6 @@ def cmd_update(args): cached = use_stored_plan("update", plan_key, assume_yes=args.yes) if args.run else None no_book, no_record, unjudged, uptodate = [], [], [], [] - git_tracked = [] _checked = 0 if cached is not None: @@ -5349,33 +4691,7 @@ def cmd_update(args): # same test as the comparison, where it came out falsy -- # and the package was filed as up to date. Not knowing is # not the same as being current. - # - # BUT --reinstall IS NOT A QUESTION ABOUT VERSIONS. It says - # "build this again at the book's version", and the answer - # does not depend on what is recorded: with the version - # missing, `update curl --reinstall` reported "nothing to - # update, 1 it cannot judge" and did nothing, 28 times in a - # row, for packages whose records predate the tool. - if getattr(args, "reinstall", False): - plan.append((anchor, name, "(reinstall)", bv)) - else: - # ...AND SAY WHAT THE BOOK HAS. "no recorded version" - # answers half the question: the reader wants to know - # what they would get, and the plan already knows it. - no_record.append("%s (book: %s)" % (name, bv or "?")) - continue - # A GIT-TRACKED PACKAGE HAS NO VERSION TO COMPARE. - # - # gst-plugins-rs is a `git` stack entry following upstream, and - # its name_version is `gst-plugins-rs-git`. Against the book's - # 1.28.6 that read as "older", so an update plan offered to - # replace a cargo-built checkout with the book's page -- a - # different build entirely, silently. `-git` is not a version - # and cannot be behind one. - _is_git_tracked = (str(u.version or "").endswith("-git") - or (u.meta or {}).get("is_git") in ("1", 1)) - if _is_git_tracked: - git_tracked.append((name, u.version, bv)) + no_record.append(name) continue outdated = not _same_version(bv, u.version) if outdated: @@ -5395,22 +4711,6 @@ def cmd_update(args): printWarning(f"{name} is NOT installed (no completed install " f"found) -- use: packagemanager install {name}") - if git_tracked: - print() - printInfo(" %d package(s) follow GIT, not the book -- left alone:" - % len(git_tracked)) - for _n, _iv, _bv in git_tracked: - printInfo(" %-22s tracked as %-20s (the book has %s)" - % (_n, _iv, _bv or "no page")) - printInfo(" `-git` is not a version, so it cannot be behind" - " one. To move such a") - printInfo(" package, rebuild its stack entry: packagemanager" - " stack <name> --only <pkg> --run --force") - printInfo(" (that follows the ref in the stack file. Taking" - " the book's page instead") - printInfo(" would be a different build: " - "`packagemanager update <pkg> --reinstall --run`.)") - # named packages that are already current: offer to re-apply for anchor, name, iv, bv in uptodate: if name not in named and anchor not in named: @@ -5419,15 +4719,6 @@ def cmd_update(args): printInfo(f"{name} is already up to date ({bv}); skipping " f"(--no-reinstall).") continue - # A DRY RUN ASKS NOTHING. `update --stack media` prompted four - # times -- once per already-current package -- before printing a - # plan it was never going to carry out. A preview that needs - # answers is not a preview. - if not getattr(args, "run", False): - printInfo(f" {name} is already current ({bv}) -- not in the " - f"plan. To rebuild it anyway:") - printInfo(f" packagemanager update {name} --reinstall --run") - continue if args.yes or ask_yes_no(f"{name} is already up to date ({bv}). " f"Re-apply the update anyway?"): plan.append((anchor, name, "(reinstall)", bv)) @@ -5631,27 +4922,6 @@ def cmd_update(args): "--clean" if getattr(args, "clean", False) else "", "-e" if args.ignore_recommended else "", "--optional" if args.optional else "") if f] + ["--run"]) - _report_stale_requires([st[1] for st in steps], - _verbose=getattr(args, "verbose", False)) - # WHERE THE WHOLE LIST IS. Asking about one package tells you - # nothing about the rest, and the command for "everything the - # book has moved on from" is this one with no name after it. - # WHAT WILL ACTUALLY CHANGE, in one line. The requirement - # report above lists a dozen packages and reads like a plan; it - # is not one, and saying so beats hoping the header is noticed. - printInfo(" this plan updates %d package(s): %s" - % (len(steps), ", ".join(st[1] for st in steps[:10]) - + (" ..." if len(steps) > 10 else ""))) - printInfo(" nothing else is touched -- the requirements above" - " are a comparison, not a list of work.") - if args.packages: - printInfo(" other ways to choose:") - printInfo(" packagemanager update #" - " everything the book has moved past") - printInfo(" packagemanager update --stack media #" - " just that stack's packages") - printInfo(" packagemanager update --stack media --select" - " --run # pick from them") printInfo(f"Dry run -- this exact command runs THIS plan ({len(steps)} " f"package(s)):\n {exact}") printInfo(f"(cached under key '{plan_key}', so --run reuses/resumes it).") @@ -5873,578 +5143,6 @@ def cmd_plan(args): "clear all: packagemanager plan clear") - -# --------------------------------------------------------------------------- -# WHO WAS BUILT BEFORE WHAT. -# -# Every generated script records the page's own required/recommended/optional -# lists, and every package's pkg.lst is written when it finished installing. -# Those two facts together answer the question that cost this system its ALSA -# and its WebRTC: was anything a package can USE installed after that package -# was built? Nothing else has to be inferred, guessed or remembered. -# --------------------------------------------------------------------------- - -def _pkg_dirs(): - root = os.path.join(BASE_DIR, load_config().get("pkgusr_subdir", "pkgusr")) - if not os.path.isdir(root): - root = BASE_DIR - out = {} - for d in sorted(os.listdir(root)): - full = os.path.join(root, d) - if not os.path.isdir(full): - continue - if not os.path.isfile(os.path.join(full, "install_last")): - continue - out[re.sub(r"^" + re.escape(pkgusr_prefix()), "", d)] = full - return out - - -def _built_when(home): - """When this package last finished installing. - - pkg.lst is written after the install, so its mtime is the install time. - install_last is the fallback: generated just before the run. - """ - for f in ("pkg.lst", "install_last"): - p = os.path.join(home, f) - try: - return os.path.getmtime(p), f - except OSError: - continue - return None, "" - - -def _script_deps(home): - """required/recommended/optional, as the page declared them.""" - out = {} - try: - body = open(os.path.join(home, "install_last"), - encoding="utf-8", errors="replace").read() - except OSError: - return out - for kind in ("required", "recommended", "optional"): - m = re.search(r"^%s=\((.*?)\)$" % kind, body, re.M | re.S) - if not m: - continue - out[kind] = re.findall(r"'([^']*)'", m.group(1)) - return out - - -def _norm(s): - return re.sub(r"[^a-z0-9]", "", s.lower()) - - -_DEP_IDX = {} - - -def _dep_index(installed): - """title -> package, built from the packages themselves. - - A dep is written as the page's TITLE ('gst-plugins-base-1.28.1', - 'FreeType-2.14.1'); a package directory is the page's id - ('gst10-plugins-base', 'freetype2'). Normalising the strings cannot - bridge that, and 119 of them did not resolve -- which quietly broke the - REBUILD ORDER, because a dependency edge that does not resolve is an edge - that does not exist: gst10-plugins-bad sorted before gst10-plugins-base, - and fontconfig before freetype2. - - Every script already records `name_version` -- the tarball base, which is - the title -- so the mapping is in the data and does not have to be - guessed. - """ - key = tuple(sorted(installed)) - if _DEP_IDX.get("key") == key: - return _DEP_IDX["idx"] - idx = {} - for name, home in installed.items(): - idx.setdefault(_norm(name), name) - try: - body = open(os.path.join(home, "install_last"), - encoding="utf-8", errors="replace").read(4096) - except OSError: - continue - m = re.search(r"^name_version='([^']*)'", body, re.M) - if not m: - continue - base = re.sub(r"-[0-9][0-9a-z.]*$", "", m.group(1)) - idx.setdefault(_norm(base), name) - _DEP_IDX.update(key=key, idx=idx) - return idx - - -def _resolve_dep(dep, installed): - """A dep is 'Little-CMS-2.18'; a package dir is 'lcms2'. - - THE MAJOR VERSION IS PART OF THE NAME, sometimes. gtk3 and gtk4 both - record name_version 'GTK-<v>', so the base name 'GTK' matched whichever - was indexed first -- and `GTK-4.20.3` resolved to **gtk3**. webkitgtk - requires both, so it listed gtk3 twice and gtk4 not at all; the same - collision hides libsoup-3.x behind libsoup, sdl3 behind sdl, qt6 behind - qt. - - So try base+major BEFORE base: `GTK-4.20.3` -> gtk4, `libsoup-3.6.6` -> - libsoup3, and `Opus-1.6.1` -> opus1 does not exist, which falls through - to opus as before. - """ - idx = _dep_index(installed) - base = re.sub(r"-[0-9][0-9a-z.]*$", "", dep) - m = re.search(r"-([0-9]+)", dep[len(base):]) if len(dep) > len(base) else None - major = m.group(1) if m else "" - cands = [] - if major: - cands += [base + major, base.replace("-", "") + major, - base.split("-")[0] + major] - cands += [base, base.replace("-", ""), base.split("-")[0]] - for cand in cands: - n = _norm(cand) - if n and n in idx: - return idx[n] - return None - - -# DOCS AND TESTS ARE NOT FEATURES. -# -# graphviz appeared against wayland, llvm, ruby, libinput, libexif and -# libical; docutils against glib2 and libdrm. Rebuilding six packages to -# regenerate documentation nobody reads is not the point of this audit, and -# burying the real findings under it is how a report gets ignored. -_DOC_DEPS = {"graphviz", "docutils", "doxygen", "sphinx", "texlive", "asciidoc", - "asciidoctor", "dejagnu", "valgrind", "gtk-doc", "xmlto", "ruby", - "fop", "docbook-xsl", "docbook", "libxslt-docs", "dblatex"} -# ...and these are build-time conveniences, not linked features -_TOOL_DEPS = {"git", "cmake", "cython", "fcron", "meson", "ninja", "nasm", - "yasm", "expect", "which", "mako", "setuptools", "wheel", - "hatchling", "pytest", "gi-docgen", "gobject-introspection-docs", - "unifdef", "gperf"} - - -# INSTALL DOES NOT REBUILD. `packagemanager install <pkg> --run -# --regenerate` on an installed package answers "Nothing to do -- everything -# is installed and up to date" -- correctly, and 28 times in a row, so the -# whole rebuild pass did nothing and said so in a way that looked like -# success. The verb that rebuilds at the same version is `update -# --reinstall`. -def _rebuild_cmd(name): - return ["packagemanager", "update", name, "--reinstall", "--regenerate", - "--run", "--yes"] - - -def _required_graph(installed, kinds=("required",)): - """package -> the packages it depends on, from its own script.""" - g = {} - for name, home in installed.items(): - deps, d = set(), _script_deps(home) - for kind in kinds: - for dep in d.get(kind, []): - got = _resolve_dep(dep, installed) - if got and got != name: - deps.add(got) - g[name] = deps - return g - - -def _depends_on(g, a, b, seen=None): - """does a require b, directly or through anything?""" - if seen is None: - seen = set() - if a in seen: - return False - seen.add(a) - for d in g.get(a, ()): - if d == b or _depends_on(g, d, b, seen): - return True - return False - - -# A TIMESTAMP IS EVIDENCE; A PROBE IS AN ANSWER. -# -# Two packages built in the same session cannot be ordered by pkg.lst -# mtimes, and that is exactly where the interesting cases are (ffmpeg and -# alsa-lib, zero days apart). The stacks already carry `expect=` probes that -# read the built artifact -- run them. -def cmd_probe(args): - import glob as _glob # noqa: also used by the no-probes branch below - stacks = sorted(_glob.glob("/etc/pkgusr/stacks/*.stack")) - if not stacks: - printError("no stack files in /etc/pkgusr/stacks") - return 1 - n = ok = bad = 0 - for path in stacks: - for e in _parse_stack_file(path): - probe = e["opts"].get("expect") - if not probe: - continue - if args.package and e["target"] not in args.package: - continue - n += 1 - r = subprocess.run(["/bin/bash", "-c", probe], - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - stack_name = os.path.basename(path).replace(".stack", "") - if r.returncode == 0: - ok += 1 - printSuccess("%-22s %s" % (e["target"], probe)) - continue - bad += 1 - printError("%-22s %s" % (e["target"], probe)) - # A PROBE THAT FAILS BECAUSE THE PACKAGE IS NOT THERE IS NOT THE - # SAME FINDING. gst10-libav had never been built -- it was - # missing from the stack until now -- and the report said "the - # feature is NOT in the installed package", which sends the - # reader looking for a build option. - home = pkgusr_home(e["target"]) - installed = os.path.exists(os.path.join(home, "pkg.lst")) - if not installed: - printInfo(" that package is NOT INSTALLED here -- nothing" - " was built without the feature,") - printInfo(" there is nothing built at all. Build it:") - printInfo(" packagemanager stack %s --only %s --run" - % (stack_name, e["target"])) - else: - printInfo(" the package is installed and the feature is" - " not in it. Rebuild:") - # not `stack --only`: that runs install, which no-ops on an - # installed package - printInfo(" %s" % " ".join(_rebuild_cmd(e["target"]))) - # ...and what the probe itself said, because a probe can be wrong - # about a path as easily as a package can be wrong about a feature - first = (r.stdout or b"").decode("utf-8", "replace").strip() - if first: - printInfo(" the check said: %s" % first.splitlines()[0][:120]) - if n == 0: - printInfo("no expect= probes in the stacks" - + (" for %s" % ", ".join(args.package) if args.package else "")) - # SAY WHY THERE ARE NONE. A stack file you have edited is never - # overwritten -- the new one lands beside it as .stack.new -- so an - # installed tool can carry probes that no installed STACK has, and - # "no probes" reads as "nothing to check" rather than "your stacks - # predate them". - waiting = [] - for newf in sorted(_glob.glob("/etc/pkgusr/stacks/*.stack.new")): - try: - if "expect=" in open(newf, encoding="utf-8", - errors="replace").read(): - waiting.append(newf) - except OSError: - continue - if waiting: - print("") - printWarning("%d newer stack file(s) beside yours DO carry probes:" - % len(waiting)) - for newf in waiting: - base = os.path.basename(newf).replace(".stack.new", "") - printInfo(" %s" % newf) - printInfo(" take it: packagemanager stack %s --take-new" - % base) - printInfo(" Your copy was edited, so make install left it alone" - " and put the new one beside it.") - else: - printInfo(" Probes live in the stack files as expect=\"...\"." - " These stacks have none,") - printInfo(" which is what an older stack looks like:" - " compare with the shipped one.") - return 0 - print("") - if bad: - printWarning("%d of %d probe(s) failed -- see each line: a package that" - " is not installed and one" % (bad, n)) - printWarning(" built without a feature are different problems.") - return 1 - printSuccess("all %d probe(s) pass: the features they check are present." % ok) - return 0 - - -def cmd_late_deps(args): - installed = _pkg_dirs() - if not installed: - printError("no package users with an install_last under %s" % BASE_DIR) - return 1 - kinds = [k.strip() for k in (args.kind or "recommended,optional").split(",") - if k.strip()] - # WHY DID IT DECIDE THAT? The order is computed from edges, and an edge - # that does not resolve is invisible -- so when a position looks wrong, - # the useful thing is the graph it actually used for that package. - if getattr(args, "why", False): - for name in (args.package or sorted(installed)): - if name not in installed: - printWarning("%s is not installed here" % name) - continue - when, src = _built_when(installed[name]) - print("") - printHeader(name) - print(" built: %s (%s)" - % (time.strftime("%Y-%m-%d %H:%M", - time.localtime(when)) if when else "unknown", src)) - d = _script_deps(installed[name]) - for kind in ("required", "recommended", "optional"): - for dep in d.get(kind, []): - got = _resolve_dep(dep, installed) - if got is None: - print(" %-12s %-28s -> no installed package of that name" - % (kind, dep)) - continue - dw, _ = _built_when(installed[got]) - if dw and when: - diff = dw - when - rel = ("built %+d day(s) relative to it" % (diff // 86400) - if abs(diff) >= 86400 else - "built %+d hour(s) relative to it" % (diff // 3600)) - else: - rel = "no date" - print(" %-12s %-28s -> %s, %s" % (kind, dep, got, rel)) - return 0 - only = set(args.package or []) - problems, unresolved, nodate = [], set(), [] - for name, home in installed.items(): - if only and name not in only: - continue - when, src = _built_when(home) - if when is None: - nodate.append(name) - continue - late = [] - for kind in kinds: - for dep in _script_deps(home).get(kind, []): - got = _resolve_dep(dep, installed) - if got is None: - unresolved.add(dep) - continue - if got == name: - continue - dwhen, _ = _built_when(installed[got]) - if dwhen is None: - continue - # SLACK THAT HIDES THE CASE IS TOO MUCH SLACK. - # - # A day was chosen because pkg.lst is rewritten by - # reload-pkg-list and by --clean, so packages from one run - # differ by minutes. But `ffmpeg ... alsa-lib +0 day(s)` was - # then filed as fine, and alsa-lib a few HOURS after ffmpeg - # is the original bug exactly. An hour covers the rewrite - # noise; anything more is a gap. - if dwhen - when > 3600: - late.append((kind, got, dep, dwhen - when)) - if late: - problems.append((name, when, src, late)) - - # THREE DIFFERENT ANSWERS, and the report was giving one. - # - # ORDERABLE the dependency does not need this package, so it can simply - # be built first -- a stack line moves and the problem is - # gone for the next machine. - # CIRCULAR the dependency REQUIRES this package (cairo needs glib2, - # mesa needs libxcb). No order exists. The only fix is a - # second pass: build, build the dependency, rebuild. - # COSMETIC documentation and build tools. Nothing to gain. - g = _required_graph(installed) - # RECOMMENDED IS A DEPENDENCY TOO, for the purpose of asking whether an - # order exists. With `required` alone the report said - # in the stack: put mesa before libxcb - # and mesa cannot precede libxcb -- it needs it. BLFS lists that edge as - # recommended, not required, and cairo needs glib2 the same way. Advice - # that cannot be taken is worse than no advice. - g_order = _required_graph(installed, ("required", "recommended")) - orderable, circular, cosmetic = [], [], [] - for name, when, src, late in problems: - o, c, d = [], [], [] - for item in late: - dep_pkg = item[1] - if dep_pkg in _DOC_DEPS or dep_pkg in _TOOL_DEPS: - d.append(item) - elif _depends_on(g_order, dep_pkg, name): - c.append(item) - else: - o.append(item) - if o: - orderable.append((name, when, src, o)) - if c: - circular.append((name, when, src, c)) - if d: - cosmetic.append((name, when, src, d)) - - def _show(rows, header, note): - if not rows: - return - print("") - printHeader(header) - printInfo(" " + note) - for name, when, src, late in sorted(rows, key=lambda r: -len(r[3])): - print("") - printWarning("%s (built %s)" - % (name, time.strftime("%Y-%m-%d", time.localtime(when)))) - for kind, got, _dep, gap in late: - when_s = ("%d day(s)" % (gap // 86400) if gap >= 86400 - else "%d hour(s)" % (gap // 3600)) - print(" %-12s %-22s installed %s later%s" - % (kind, got, when_s, - " <- same session: only a probe can settle it" - if gap < 21600 else "")) - # SAY WHAT TO CHANGE, not just what is wrong. The orderable ones - # are a stack edit; without the line the reader has to work out - # which file and which position. - if rows is orderable: - print(" in the stack: put %s before %s" - % (", ".join(sorted({g for _k, g, _d, _x in late})), name)) - - if not problems: - printSuccess("no package was built before something it can use.") - _show(orderable, "built too early -- and an order exists", - "move these dependencies earlier in the stack, then rebuild these once.") - _show(circular, "built too early -- no order exists", - "the dependency requires this package, so it cannot go first." - " A second pass is the only fix.") - if args.include_docs: - _show(cosmetic, "documentation and build tools only", - "rebuilding gains you documentation, not features.") - elif cosmetic: - print("") - printInfo("%d package(s) differ only in documentation or build tools" - " (--include-docs to see them)" % len(cosmetic)) - - # THE REBUILD ORDER IS NOT THE REPORT ORDER. Rebuilding a library after - # the thing that links it means doing it twice. - todo = {r[0] for r in orderable} | {r[0] for r in circular} - if args.include_docs: - todo |= {r[0] for r in cosmetic} - if todo: - # ORDER BY EVERY EDGE THAT DECIDES A FEATURE, not just the required - # ones: mesa RECOMMENDS libxcb, so libxcb has to be rebuilt first or - # mesa is rebuilt against the old one. Cycles are expected here - # (cairo and glib2 recommend each other) and are broken below. - # ORDER THE CLUSTERS, NOT THE NODES. - # - # Picking one node out of a cycle and then going back to the ready - # scan strands the rest of the cluster at the end: freetype2 and - # harfbuzz (which recommend/require each other) came out LAST, after - # fontconfig and cairo -- both of which require freetype2. So the - # cycle has to be treated as one unit. - # - # Tarjan gives the strongly connected components; the components form - # a DAG by construction, so they sort cleanly. Inside a component - # there is no right answer, and the same rule as before applies: - # least dependent, most depended on. - nodes = set(todo) - # THE OPTIONAL EDGE IS THE WHOLE POINT OF THIS PASS. - # - # The order graph was required+recommended, so mpg123 was rebuilt - # BEFORE pulseaudio -- and mpg123's missing feature IS pulseaudio. - # Ordering a rebuild by the edges that do not decide features, while - # ignoring the ones that do, cannot be right. - # - # Optional edges make the graph much more cyclic; that is honest - # rather than inconvenient, and SCCs handle it. Inside a component - # the STRONG edges (required, recommended) still decide, because - # those are the ones that break a build rather than a feature. - g_all = _required_graph(installed, - ("required", "recommended", "optional")) - adj = {n: (g_all.get(n, set()) & nodes) for n in nodes} - strong = {n: (g_order.get(n, set()) & nodes) for n in nodes} - index, low, on_stack, stack, sccs, counter = {}, {}, set(), [], [], [0] - - def _strong(v): - # iterative: a 400-package graph would recurse past the limit - work = [(v, iter(sorted(adj[v])))] - index[v] = low[v] = counter[0]; counter[0] += 1 - stack.append(v); on_stack.add(v) - while work: - node, it = work[-1] - advanced = False - for w in it: - if w not in index: - index[w] = low[w] = counter[0]; counter[0] += 1 - stack.append(w); on_stack.add(w) - work.append((w, iter(sorted(adj[w])))) - advanced = True - break - if w in on_stack: - low[node] = min(low[node], index[w]) - if advanced: - continue - work.pop() - if work: - low[work[-1][0]] = min(low[work[-1][0]], low[node]) - if low[node] == index[node]: - comp = [] - while True: - w = stack.pop(); on_stack.discard(w); comp.append(w) - if w == node: - break - sccs.append(comp) - - for n in sorted(nodes): - if n not in index: - _strong(n) - - # Tarjan emits each component only after everything it depends on, - # so this is already dependency order. - comp_of = {n: i for i, c in enumerate(sccs) for n in c} - ordered, cycles = [], [] - - def _inside(c): - cs = set(c) - return sorted(c, key=lambda n: (len(strong[n] & cs), - len(adj[n] & cs), - -sum(1 for m in cs if n in strong[m]), - -sum(1 for m in cs if n in adj[m]), - n)) - - for comp in sccs: - if len(comp) > 1: - cycles.extend(_inside(comp)) - ordered.extend(_inside(comp)) - else: - ordered.extend(comp) - - print("") - printHeader("rebuild, in this order (%d)" % len(ordered)) - printInfo(" dependencies first, so nothing is rebuilt twice.") - for n in ordered: - print(" %s" % " ".join(_rebuild_cmd(n))) - if cycles: - print("") - printInfo(" %d of these are in dependency cycles, where no order" - " is completely right:" % len(cycles)) - printInfo(" " + ", ".join(cycles)) - printInfo(" each was placed by fewest unmet dependencies, then by" - " how many wait for it.") - printInfo(" If a feature is still missing afterwards, run" - " late-deps again: a cycle") - printInfo(" can need two passes, and the second one is short.") - if args.rebuild: - print("") - if not args.run: - printInfo("that is the plan; add --run to do it.") - return 1 - for i, n in enumerate(ordered, 1): - printHeader("rebuild %d/%d: %s" % (i, len(ordered), n)) - rc = subprocess.run(_rebuild_cmd(n)).returncode - if rc != 0: - printError("%s failed (exit %d) -- stopping here, with" - " %d left." % (n, rc, len(ordered) - i)) - printInfo(" fix it, then run late-deps --rebuild --run again:" - " what succeeded is recorded and will not repeat.") - return 1 - printSuccess("rebuilt %d package(s)." % len(ordered)) - return 0 - if unresolved and args.verbose: - print("") - printInfo("could not match to an installed package (so not checked): %s" - % ", ".join(sorted(unresolved))) - elif unresolved: - print("") - printInfo("%d dependency name(s) match no INSTALLED package" - " (--verbose lists them)." % len(unresolved)) - printInfo(" Mostly that means they are not installed, which is" - " nothing to fix -- a package") - printInfo(" that is not there cannot have been built after anything." - " A name you know IS") - printInfo(" installed appearing there is worth reporting: it is an" - " edge this cannot see.") - if nodate: - printInfo("%d package(s) have no pkg.lst and no install_last date" - % len(nodate)) - return 1 if problems else 0 - def cmd_errors(args): """Show which packages failed in the LAST install/update run, with their log locations. With --all, show every package's result.""" @@ -6494,30 +5192,16 @@ def _versioned_script_in(home): That is the file a person is told to edit, and the one the copy that actually runs is made from -- so a failure has to be able to name it. """ - # THE ONE THAT WAS LAST INSTALLED, NOT THE FIRST IN THE ALPHABET. - # - # A home keeps every version's script: p_mesa has install_Mesa-25.3.5 - # and install_Mesa-26.1.7. sorted() returned 25.3.5, so `script install - # mesa` ran the OLD version's install over a 26.1.7 build ("9" also sorts - # after "10"). install_last names the version last staged; take that - # file. Without it, the newest by modification time. - try: - cands = [f for f in os.listdir(home) - if f.startswith("install_") and not f.endswith((".edited", ".bak")) - and re.search(r"-\d[\w.]*$", f)] - except OSError: - return None - if not cands: - return None try: - m = re.search(r"^name_version=['\"]([^'\"]+)['\"]", - open(os.path.join(home, "install_last"), - encoding="utf-8", errors="replace").read(), re.M) - if m and ("install_" + m.group(1)) in cands: - return os.path.join(home, "install_" + m.group(1)) + for f in sorted(os.listdir(home)): + if not f.startswith("install_") or f.endswith((".edited", ".bak")): + continue + # install_wget-1.25.0 -- a version after the last dash + if re.search(r"-\d[\w.]*$", f): + return os.path.join(home, f) except OSError: pass - return max((os.path.join(home, f) for f in cands), key=os.path.getmtime) + return None def _unpacked_source_dir(home): @@ -8710,16 +7394,14 @@ def _bootstrap_next_steps(st): def cmd_user_create(args): - if _resolve_shared_level(args) is None: - return 2 u = _user_prefixed(args.name) if _user_exists(u): printInfo(f"{u}: already exists.") else: printHeader(f"Create shared user {u}") print() - cmd = find_add_package_user() + [u, u, str(APP_UID_MIN), str(APP_UID_MAX), - u, str(APP_UID_MIN), str(APP_UID_MAX)] + cmd = find_add_package_user() + [u, u, "10000", "20000", + u, "10000", "20000"] printInfo(f"Creating shared user: {' '.join(cmd)}") if not _run(cmd): return @@ -8728,84 +7410,11 @@ def cmd_user_create(args): if args.groups: _add_groups(u, [g for g in re.split(r"[,\s]+", args.groups) if g]) - # AN APPLICATION USER IS NOT A PACKAGE USER, AND ITS HOME IS NOT IN - # /usr/src. - # - # `add_package_user` is what creates the account -- it handles the uid - # range and the matching group -- and it puts the home where PACKAGE - # homes go: - # /usr/src/u_claude, /usr/src/u_browser-mini, ... - # A package user's home is a build tree. An application user's home is - # a home: it holds a browser profile, a .config, a Downloads directory. - # It belongs in /home, which is also where the shared-directory links - # and the XDG skeleton expect it. - _relocate_app_home(u) - # An account on its own is not usable: you cannot become it, its home is # world-readable, and nothing launches the program. Do the rest. _setup_application_user(u, args) -def _relocate_app_home(u): - """Move an application user's home from the package tree to /home. - - Only from the package-user base, only when the target does not already - exist, and with `usermod -m` so the contents come along. Anything else - is left exactly as it is: a home someone has already set up elsewhere - is not this function's business. - """ - cur = _home_of(u) or "" - root = appuser_home() - want = os.path.join(root, u) - if not cur or os.path.realpath(cur) == os.path.realpath(want): - return - # Moved from where these tools put it before, and nowhere else: the - # package tree, or /home/<user> (the old hard-coded place). A home - # someone chose elsewhere is not this function's business. - base = os.path.realpath(BASE_DIR) - rc = os.path.realpath(cur) - if not (rc.startswith(base + os.sep) - or rc == os.path.realpath(os.path.join("/home", u))): - printInfo(" %s's home is %s -- left alone (not a place these tools" - " put it)" % (u, cur)) - return - if os.path.exists(want): - printWarning(" %s already exists -- not moving %s's home there" - % (want, u)) - printInfo(" check it, then: usermod -d %s -m %s" % (want, u)) - return - if not os.path.isdir(root): - _run(["install", "-d", "-o", "root", "-g", "root", "-m", "0755", root]) - printInfo(" %s's home is %s -- application users live in %s" - % (u, cur, root)) - if _run(["usermod", "-d", want, "-m", u]): - _run(["chmod", "0750", want]) - printSuccess(" moved to %s (0750)" % want) - _repoint_shared_link(u, cur, want) - else: - printWarning(" could not move it (is %s logged in or running" - " something?); by hand:" % u) - printInfo(" usermod -d %s -m %s && chmod 0750 %s" % (want, u, want)) - - -def _repoint_shared_link(u, old_home, new_home): - """~/Shared/<Name> of the main user pointed into the old home.""" - mh = _home_of(main_user() or "") or "" - hub = os.path.join(mh, "Shared") if mh else "" - if not hub or not os.path.isdir(hub): - return - for n in os.listdir(hub): - p = os.path.join(hub, n) - if not os.path.islink(p): - continue - t = os.readlink(p) - if t.startswith(old_home.rstrip("/") + "/"): - nt = new_home.rstrip("/") + t[len(old_home.rstrip("/")):] - os.unlink(p) - os.symlink(nt, p) - printInfo(" %s -> %s" % (p, nt)) - - def _xdg_runtime_dir_of(user): """The main user's XDG runtime directory, however this system names it.""" try: @@ -8822,50 +7431,37 @@ def _xdg_runtime_dir_of(user): return None -def xdg_runtime_group(): - """The one group that opens the main user's session to shared accounts.""" - return collector_prefix() + "xdg-runtime" - - def _join_xdg_runtime_group(main, u): - """Put an application user -- and the main user -- in the session group. + """Put an application user in the group that owns the session. A "shared user" is one that may reach the main user's XDG runtime - directory: the display socket (and pipewire's) live there, so without it - a graphical program starts and then cannot draw anything. - - THE GROUP IS NAMED, NOT READ OFF THE DIRECTORY. This used to take - whatever group /tmp/xdg-<main> carried -- which is the main user's own - private group -- refuse, and say "set up a runtime group first", which - nothing did. <prefix>_xdg-runtime is created here (collector range), the - main user joins it (a user may only chgrp to a group it is in) and so does - every shared account. The directory is recreated at every boot, so its - group is applied by the session itself: /usr/bin/xdg-runtime-share and - /etc/profile.d/xdg-runtime-share.sh, written by `cfg app-users`. - """ - g = xdg_runtime_group() - if not _create_collector_group(g): - printWarning(f" could not create the session group {g} -- {u} will " - f"not reach the session") - return - for who in (main, u): - _run(["usermod", "-a", "-G", g, who]) - printInfo(f" {main} and {u} are in '{g}' (the session group)") + directory -- that is where the display socket, the session bus and the + audio socket live, so without it a graphical program starts and then + cannot draw anything. + + The group is read from the directory rather than assumed: it is set up + once for the main user, and every shared account afterwards simply joins + whatever is there.""" rt = _xdg_runtime_dir_of(main) if not rt: - printInfo(f" no runtime directory for {main} right now -- the session " - f"applies the group when it starts") + printWarning(f" no XDG runtime directory found for {main} -- " + f"{u} will not reach the session") + printInfo(" (looked for /run/user/<uid> and /tmp/xdg-<user>)") return try: import grp - cur = grp.getgrgid(os.stat(rt).st_gid).gr_name + grp_name = grp.getgrgid(os.stat(rt).st_gid).gr_name except (KeyError, ImportError, OSError): - cur = "" - if cur != g: - # root may do it now; the session does it on every later login - _run(["chgrp", g, rt]) - _run(["chmod", "0750", rt]) - printInfo(f" {rt}: group {g}, 0750 (was {cur or '?'})") + printWarning(f" cannot read the group on {rt}") + return + if grp_name in (main, "root"): + printWarning(f" {rt} is group '{grp_name}', which is not a shared " + f"group -- {u} was not added") + printInfo(" Set up a runtime group for the session first, then " + "re-run this.") + return + _run(["usermod", "-a", "-G", grp_name, u]) + printInfo(f" {u} joined '{grp_name}' (the group on {rt})") XDG_SKEL = "/etc/pkgusr/skel-u_xdg/.bash_profile" @@ -8909,9 +7505,7 @@ def _link_xdg_profile(main, u): printWarning(f" could not write {XDG_SKEL}: {e}") return - # the ACCOUNT's home, not a package home: this linked - # /usr/src/u_claude/.bash_profile - home = _app_home(u) + home = pkgusr_home(u) target = os.path.join(home, ".bash_profile") if os.path.islink(target) and os.path.realpath(target) == \ os.path.realpath(XDG_SKEL): @@ -8938,7 +7532,7 @@ def _setup_application_user(u, args): * an optional shared directory both accounts can write * an optional launcher in ~/bin so the program runs as that user """ - home = _app_home(u) + home = os.path.join(appuser_home(), u) main = main_user() if not main: printWarning("\nNo main user is set, so this account cannot be reached " @@ -8970,74 +7564,36 @@ def _setup_application_user(u, args): # 5. a shared directory, when asked for if getattr(args, "share_dir", False): - share = os.path.join(_app_home(u), "Shared") + share = os.path.join(home, "Shared") _run(["mkdir", "-p", share]) _run(["chown", f"{u}:{u}", share]) _run(["chmod", "2770", share]) # setgid: new files keep the group - # ONE DIRECTORY OF LINKS, NAMED AFTER THE ACCOUNT. These used to - # land as ~/<name>-shared, one per account, scattered through the - # home directory. ~/Shared/<Name> keeps them together and gives the - # env file below something to enumerate. - _mh = _home_of(main) or "" - link = "" - if _mh: - _hub = os.path.join(_mh, "Shared") - if not os.path.isdir(_hub): - _run(["mkdir", "-p", _hub]) - _run(["chown", f"{main}:{main}", _hub]) - link = os.path.join(_hub, args.name[:1].upper() + args.name[1:]) - if os.path.islink(link): - if os.path.realpath(link) != os.path.realpath(share): - printWarning(f" {link} already points at " - f"{os.path.realpath(link)} -- left alone") - link = "" - elif os.path.exists(link): - # A REAL DIRECTORY IS NOT OURS TO REPLACE. - printWarning(f" {link} exists and is not a symlink -- " - f"left alone") - printInfo(" move it aside yourself if you want the link") - link = "" - else: - _run(["ln", "-s", share, link]) + link = os.path.join(_home_of(main) or "", f"{args.name}-shared") + if link and not os.path.exists(link): + _run(["ln", "-s", share, link]) printInfo(f" shared directory: {share}") if link: printInfo(f" reachable from your account as {link}") - _old = os.path.join(_mh, f"{args.name}-shared") if _mh else "" - if _old and os.path.islink(_old): - printInfo(f" (the older link {_old} is still there; remove it " - f"when you are ready)") # 6. a launcher and a menu entry, when the program is really there app = getattr(args, "app", None) or getattr(args, "name", None) \ or u[len(user_prefix()):] import shutil as _sh - # AN APP CAN BE A COMMAND LINE. `--app "/bin/browser-mini - # https://claude.ai/new"` is the whole point of a per-account browser, - # and `which` on that string finds nothing -- so the launcher and the - # menu entry were silently skipped for exactly the accounts that needed - # them most. Look up the FIRST word; run the whole line. - _app_argv = app.split() - _app_cmd = _app_argv[0] if _app_argv else app - _app_base = os.path.basename(_app_cmd) want_launcher = getattr(args, "launcher", False) want_desktop = getattr(args, "desktop", False) if want_launcher or want_desktop: # Their original checked `which $app` first and skipped both when it # was missing -- a launcher for a program that is not installed is # just a broken command in your PATH. - if not (_sh.which(_app_cmd) or os.path.isfile(_app_cmd)): - printWarning(f" no '{_app_cmd}' on PATH -- no launcher or menu " - f"entry") + if not _sh.which(app): + printWarning(f" no '{app}' on PATH -- no launcher or menu entry") printInfo(f" (install it first, then: packagemanager user " - f"setup {args.name} --launcher --desktop)") + f"setup {app} --launcher --desktop)") else: - # the launcher is NAMED after the account, not after the command: - # two accounts can run the same browser with different arguments, - # and ~/bin/browser-mini cannot be both. if want_launcher: - _write_launcher(main, u, app, name=args.name) + _write_launcher(main, u, app) if want_desktop: - _write_desktop_entry(main, u, app, name=args.name) + _write_desktop_entry(main, u, app) def _home_of(user): @@ -9080,63 +7636,11 @@ def _allow_su_via_pam(u, main): try: with open(pam, "w") as f: f.writelines(out) - # IT IS A SCRIPT pam EXECUTES. Rewriting it must not drop the exec - # bit, and a file written 644 makes `su` ask for a password with - # nothing in the logs to say why. - _m = os.stat(pam).st_mode & 0o777 - if not _m & 0o111: - os.chmod(pam, 0o755) - printInfo(f" {pam} was not executable ({_m:o}) -- set to 755, " - f"pam_exec.so runs it") printInfo(f" {main} may now `su - {u}` without a password") - # ...AND pam ONLY READS IT IF /etc/pam.d/su SAYS SO. - _su = "/etc/pam.d/su" - _line = "auth sufficient pam_exec.so quiet %s" % pam - try: - _has = os.path.isfile(_su) and "su_u_user" in open(_su).read() - except OSError: - _has = False - if not _has: - printWarning(f" {_su} does not call it, so su will still ask for" - f" a password.") - printInfo(" add, above the other auth lines:") - printInfo(" %s" % _line) - printInfo(" `packagemanager stack services --only app-users " - "--run` does it for you.") except OSError as e: printWarning(f" could not write {pam}: {e}") -def _resolve_shared_level(args): - """--shared-slim / --shared-full on top of the original --shared. - - `--shared` joins the group on the XDG runtime directory -- the display, - the session bus and audio sockets live there, so without it a graphical - program cannot draw. Audio and video are SEPARATE groups, and were only - reachable by spelling them out in --groups. - - An account that runs one browser window does not need a microphone. So - the level is named: slim is the desktop alone, full adds audio and video. - """ - slim = getattr(args, "shared_slim", False) - full = getattr(args, "shared_full", False) - if slim and full: - printError("--shared-slim and --shared-full are opposites; pick one") - return None - if slim or full: - args.shared = True - if full: - have = [g for g in (getattr(args, "groups", "") or "").split(",") if g] - for g in ("audio", "video"): - if g not in have: - have.append(g) - args.groups = ",".join(have) - printInfo(" --shared-full: desktop + groups %s" % ", ".join(have)) - elif slim: - printInfo(" --shared-slim: desktop only (no audio, no video)") - return args - - def cmd_user_setup(args): """Apply the application-user setup to an account that already exists. @@ -9144,8 +7648,6 @@ def cmd_user_setup(args): earlier -- or made by hand -- can be brought up to the same state. Each step is idempotent, so running it again is safe and only fills in what is missing.""" - if _resolve_shared_level(args) is None: - return 2 u = _user_prefixed(args.name) if not _user_exists(u): printError(f"{u}: no such user.") @@ -9153,389 +7655,61 @@ def cmd_user_setup(args): sys.exit(1) printHeader(f"Set up {u}") print() - # an account made before its home had a place is brought there too - _relocate_app_home(u) if args.groups: _add_groups(u, [g for g in re.split(r"[,\s]+", args.groups) if g]) _setup_application_user(u, args) -# BROWSERS THAT TAKE --app-id. A window's app id is what sway matches rules -# and the taskbar groups on; without one every account's browser is the same -# "browser-mini" window. A line that does not give one gets its launcher name. -_APP_ID_PROGRAMS = ("browser-mini", "browser-big") +def _write_desktop_entry(main, u, app): + """A menu entry that launches the program as the application user. -# WHAT YOU TYPE, AND WHAT HAPPENS WHEN YOU TYPE NOTHING. -# -# {} the word is the arguments / a search term -# {https://site/} ...and this page when there are no arguments -# -# The braces are the placeholder; anything INSIDE them is the fallback. It -# cannot be spelled `{|url}`: `|` is app-users.conf's field separator. -_TEMPLATE_RE = re.compile(r"\{[^{}]*\}") - - -def _is_template(word): - """Does this word carry a {} placeholder (with or without a fallback)?""" - return bool(_TEMPLATE_RE.search(word)) - - -def _launcher_words(u, app, who): - """The command line as words, with ~/ made the ACCOUNT's home, and an - --app-id added for a browser that has none.""" - import shlex as _shlex - try: - words = _shlex.split(app) - except ValueError: - words = app.split() - home = _app_home(u) or "" - words = [(home + w[1:]) if (w.startswith("~/") and home) else w for w in words] - if words and os.path.basename(words[0]) in _APP_ID_PROGRAMS \ - and "--app-id" not in words: - words[1:1] = ["--app-id", who] - return words - - -def _recolor_svg(path, color): - """Paint a one-colour SVG in `color`. - - A brand icon is a single path with one fill (or none at all, inheriting - black). simpleicons.org used to do this server-side -- /openai/38BDF8 -- - but only for the icons it still carries, and it dropped `openai` and - `grok`. Doing it here works on any monochrome SVG, wherever it came - from. `fill="none"` is left alone: that is a hole, not a colour. A - multi-colour logo comes out flat, which is why this only happens when a - colour is asked for. - """ - try: - t = open(path, encoding="utf-8", errors="replace").read() - except OSError: - return False - if "<svg" not in t: - return False - new = re.sub(r'fill="(?!none\b)[^"]*"', 'fill="%s"' % color, t) - new = re.sub(r'fill:\s*(?!none\b)[^;"\']+', 'fill:%s' % color, new) - if not re.search(r'<svg\b[^>]*\bfill=', new): - new = re.sub(r'<svg\b', '<svg fill="%s"' % color, new, count=1) - if new == t: - return False - try: - open(path, "w").write(new) - except OSError: - return False - return True - - -def _fetch_icon(main, who, icon): - """The menu icon, fetched once into ~/.local/share/icons/lfs-apps. - - icon: an http(s) URL -> that file; a word -> the simpleicons.org slug of - that name; empty -> the launcher name as the slug; "-" -> no icon. A - trailing #RRGGBB paints a monochrome SVG that colour after it is - fetched. A failed download is not an error: the entry falls back to the - program's own icon name. - """ - if icon == "-": - return None - home = _home_of(main) - if not home: - return None - spec = icon or "" - # #RRGGBB AT THE END IS A COLOUR, NOT PART OF THE URL. No URL these - # icons come from uses a fragment, and a colour has to be expressible for - # a plain slug too (`openai#38BDF8`), not just for a URL. - color = "" - m = re.search(r"#([0-9A-Fa-f]{3}|[0-9A-Fa-f]{6})$", spec) - if m: - color = "#" + m.group(1) - spec = spec[:m.start()] - url = spec if re.match(r"^https?://", spec) else \ - "https://cdn.simpleicons.org/%s" % ((spec or who).lower()) - ext = os.path.splitext(url.split("?")[0])[1].lower() - ext = ext if ext in (".svg", ".png", ".xpm") else ".svg" - d = os.path.join(home, ".local", "share", "icons", "lfs-apps") - path = os.path.join(d, who + ext) - stamp = path + ".from" - # the stamp records the WHOLE spec, so changing only the colour refetches - want = url + color - if os.path.isfile(path) and os.path.getsize(path) > 0: - try: - if open(stamp).read().strip() == want: - return path - except OSError: - pass - os.makedirs(d, exist_ok=True) - tmp = path + ".part" - r = subprocess.run(["curl", "-fsSL", "--max-time", "20", url, "-o", tmp], - stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - if r.returncode != 0 or not os.path.isfile(tmp) or os.path.getsize(tmp) == 0: - try: - os.unlink(tmp) - except OSError: - pass - printWarning(f" no icon for {who} from {url} -- the menu uses the program's") - if "cdn.simpleicons.org" in url: - printInfo(" (simpleicons.org has dropped several brands, `openai` " - "and `grok` among them -- give the icon column a URL)") - return None - os.replace(tmp, path) - if color and ext == ".svg": - printInfo(f" icon {who}: painted {color}" if _recolor_svg(path, color) - else f" icon {who}: {color} changed nothing (not a one-colour SVG)") - open(stamp, "w").write(want + "\n") - for f in (d, path, stamp): - _run(["chown", f"{main}:{main}", f]) - printInfo(f" icon: {path} ({url})") - return path - - -def _write_desktop_entry(main, u, app, name=None, icon=""): - """A menu entry that runs the launcher -- named after the LAUNCHER, so - one account can have several (google and duckduckgo run as u_browser).""" + Written next to the wrapper, so the program appears in the menu and starts + under the right account when clicked -- not just from a terminal.""" home = _home_of(main) if not home: return - who = os.path.basename(name or app.split()[0]) appdir = os.path.join(home, ".local", "share", "applications") os.makedirs(appdir, exist_ok=True) - path = os.path.join(appdir, f"{who}.desktop") - launcher = os.path.join(home, "bin", who) - words = _launcher_words(u, app, who) - prog = os.path.basename(words[0]) if words else who - icon_path = _fetch_icon(main, who, icon) - takes_args = any(_is_template(w) for w in words) - # A MENU NEEDS A CATEGORY. An entry with none is "Other" at best and - # missing at worst, depending on the menu -- and these are the entries a - # person actually clicks. Taken from the program, since that is what is - # known here. - if prog in _APP_ID_PROGRAMS: - cats = "Network;WebBrowser;" - elif prog in ("mpv", "vlc", "ffplay"): - cats = "AudioVideo;Video;Player;" - else: - cats = "Utility;" - text = ("[Desktop Entry]\n" - f"Name={who.capitalize()}\n" + path = os.path.join(appdir, f"{app}.desktop") + if os.path.exists(path) and not getattr(_write_desktop_entry, "_force", False): + printInfo(f" {path} already exists -- left alone") + return + launcher = os.path.join(home, "bin", app) + with open(path, "w") as f: + f.write( + "[Desktop Entry]\n" + f"Name={app.capitalize()}\n" f"Comment=Runs as the application user {u}\n" - f"Exec={launcher}{' %U' if takes_args else ''}\n" + f"Exec={launcher} %U\n" "Terminal=false\n" "Type=Application\n" - f"Categories={cats}\n" - f"Icon={icon_path or prog}\n" - f"StartupWMClass={who if prog in _APP_ID_PROGRAMS else prog.capitalize()}\n") - if os.path.exists(path): - try: - cur = open(path, encoding="utf-8", errors="replace").read() - except OSError: - cur = "" - if cur == text: - return - if "Comment=Runs as the application user " not in cur: - printInfo(f" {path} was not written by these tools -- left alone") - return - printInfo(f" menu entry {path} was stale -- rewritten") - with open(path, "w") as f: - f.write(text) + f"Icon={app}\n" + f"StartupWMClass={app.capitalize()}\n") os.chmod(path, 0o644) _run(["chown", f"{main}:{main}", path]) printInfo(f" menu entry: {path}") -_LAUNCHER_TEMPLATE = r'''#!/bin/bash -# Runs @APP@ as the application user @USER@. -# written by packagemanager user launcher -- rewritten when stale -T=(@ARR@) -enc() { # URL-encode, byte by byte; a space becomes + - local LC_ALL=C s="$*" o="" c i - for ((i = 0; i < ${#s}; i++)); do - c="${s:i:1}" - case "$c" in - [a-zA-Z0-9._~-]) o+="$c" ;; - " ") o+="+" ;; - *) printf -v c "%%%02X" "'$c"; o+="$c" ;; - esac - done - printf "%s" "$o" -} -cmd=() -for w in "${T[@]}"; do - if [ "$w" = "{}" ]; then - cmd+=("$@") # the arguments, one by one - elif [[ "$w" == *"{"*"}"* ]]; then - pre="${w%%\{*}"; rest="${w#*\{}" # ...{ inner }... - inner="${rest%%\}*}"; post="${rest#*\}}" - if [ $# -gt 0 ]; then - cmd+=("$pre$(enc "$*")$post") # what you typed, URL-encoded - elif [ -n "$inner" ]; then - cmd+=("$inner") # nothing typed: the default page - else - p="$pre"; case "$p" in *\?*) p="${p%\?*}" ;; esac - cmd+=("$p") # nothing typed, no default: cut at ? - fi - else - cmd+=("$w") - fi -done -# A MENU ENTRY HAS NO TERMINAL, AND su WANTS ONE. -# -# From a terminal this is the whole story: su authenticates through -# /etc/pam.d/su_u_user and execs the program. Started from a .desktop entry, -# a hotkey or a launcher menu there is no tty at all -- su fails, its message -# goes to a stderr nobody is reading, and the entry looks like it "does -# nothing". script(1) gives it a pty and nothing else; the typescript goes -# to /dev/null and -e returns the program's own exit status. -if [ -t 0 ] && [ -t 1 ]; then - exec su - @QUSER@ -c 'exec "$0" "$@"' -- "${cmd[@]}" -fi -script -qec "$(printf '%q ' su - @QUSER@ -c 'exec "$0" "$@"' -- "${cmd[@]}")" /dev/null -rc=$? -# ...and say so where it can be seen, since there is no terminal to print to -if [ "$rc" != 0 ] && command -v notify-send >/dev/null 2>&1; then - notify-send "@WHO@" "did not start (exit $rc) -- run @WHO@ in a terminal to see why" -fi -exit $rc -''' - - -def _write_launcher(main, u, app, name=None): - """A wrapper in ~/bin, named after the LAUNCHER, that runs the command - line as the application account. - - {} IN THE COMMAND LINE IS WHAT YOU TYPE. - a word that IS {} -> the arguments, one by one ("browser-mini {}") - {} inside a word -> the arguments as a URL-encoded search term - ("https://www.youtube.com/results?search_query={}") - and with no arguments the word is cut before - its ? (or before {}): the site's front page - {https://site/} in one -> the same, but with nothing typed the word - becomes exactly what is inside the braces. - Cutting at the ? gives /search/ -- a search - page for a search nobody made. This gives - the page you actually want to land on, and - it is per line: reddit's is the front page - today and may be your own profile tomorrow. - no {} at all -> arguments are NOT passed: `claude foo` must - not hand browser-mini a second address - - THE ARGUMENTS GO THROUGH AS ARGUMENTS. The old writer built - su -c "app $(printf %q-each "$@")", which with no arguments produced one - EMPTY argument ("a second address:"). Here the final list is built in - the launcher and handed to su after --: bash gets $0 and "$@" exactly. - """ +def _write_launcher(main, u, app): + """A wrapper in ~/bin that runs <app> as the application user.""" home = _home_of(main) if not home: return bindir = os.path.join(home, "bin") os.makedirs(bindir, exist_ok=True) - who = os.path.basename(name or app.split()[0]) - path = os.path.join(bindir, who) - import shlex as _shlex - words = _launcher_words(u, app, who) - arr = " ".join(_shlex.quote(w) for w in words) - # A PLAIN TEMPLATE, NOT %-FORMATTING. This body is bash full of `%02X` - # and `${w%%...}`, and every one of them had to be written `%%%%` here. - # One miscount and the launcher is subtly wrong, which is how the empty - # second address survived so long. Three named slots, .replace(), done. - text = _LAUNCHER_TEMPLATE.replace("@APP@", app).replace("@USER@", u) \ - .replace("@QUSER@", _shlex.quote(u)) \ - .replace("@WHO@", who) \ - .replace("@ARR@", arr) + path = os.path.join(bindir, app) if os.path.exists(path): - try: - cur = open(path, encoding="utf-8", errors="replace").read() - except OSError: - cur = "" - # OURS AND STALE IS REWRITTEN; ANYTHING ELSE IS YOURS. - if cur == text: - printInfo(f" launcher {path} is current") - return - if not re.search(r"^# Runs .* as the application user \S+\.$", cur, re.M): - # SAY WHAT TO DO ABOUT IT. A launcher from an older version of - # these tools is not recognised either, so "left alone" quietly - # kept the OLD behaviour -- no default page, no --app-id -- on - # exactly the launchers the conf had just changed. - printInfo(f" {path} exists and was not written by these tools -- left alone") - printInfo(f" to take the generated one: rm {path} and run this again") - return - _was = re.search(r"su - (\S+)", cur) - printInfo(f" launcher {path} was stale (ran as " - f"{_was.group(1) if _was else '?'}) -- rewritten") + printInfo(f" {path} already exists -- left alone") + return with open(path, "w") as f: - f.write(text) + f.write('#!/bin/bash\n' + '# Runs %s as the application user %s.\n' + 'exec su - %s -c "%s $(printf \'%%q \' "$@")"\n' % (app, u, u, app)) os.chmod(path, 0o755) _run(["chown", f"{main}:{main}", path]) printInfo(f" launcher: {path}") -def cmd_user_launcher(args): - """One launcher (and menu entry) for an EXISTING application account. - - Separate from `user setup` because an account can have several: the - `browser` account runs browser-mini, browser-big, google and duckduckgo. - """ - u = _user_prefixed(args.account) - if not _user_exists(u): - printError(f"{u}: no such account -- `user create {args.account}` first") - return 1 - main = main_user() - if not main: - printError("no main user set (packagemanager config --main-user <login>)") - return 1 - who = args.name - import shutil as _sh - words = _launcher_words(u, args.app, who) - if not words or not (_sh.which(words[0]) or os.path.isfile(words[0])): - printWarning(f" {who}: no '{words[0] if words else args.app}' on PATH " - f"-- no launcher or menu entry") - return 0 - _write_launcher(main, u, args.app, name=who) - if not args.no_desktop: - _write_desktop_entry(main, u, args.app, name=who, icon=args.icon or "") - return 0 - - -def _ours_for(path, marker): - try: - return marker in open(path, encoding="utf-8", errors="replace").read() - except OSError: - return False - - -def cmd_user_remove_launcher(args): - """Remove a launcher, its menu entry and icon -- only files these tools - wrote (a hand-made ~/bin/<name> is never touched).""" - main = main_user() - home = _home_of(main) if main else None - if not home: - printError("no main user set") - return 1 - who = os.path.basename(args.name) - files = [] - lp = os.path.join(home, "bin", who) - if os.path.isfile(lp) and _ours_for(lp, "as the application user"): - files.append(lp) - dp = os.path.join(home, ".local", "share", "applications", who + ".desktop") - if os.path.isfile(dp) and _ours_for(dp, "Comment=Runs as the application user "): - files.append(dp) - idir = os.path.join(home, ".local", "share", "icons", "lfs-apps") - if os.path.isdir(idir): - files += [os.path.join(idir, f) for f in os.listdir(idir) - if f.startswith(who + ".")] - if not files: - printInfo(f" {who}: no launcher, menu entry or icon of these tools") - return 0 - for f in files: - printInfo((" remove " if args.run else " would remove ") + f) - if args.run: - for f in files: - os.unlink(f) - else: - printInfo(" dry run -- add --run") - printInfo(f" and take its line out of /etc/pkgusr/stacks/app-users.conf, " - f"or the next `cfg app-users` writes it again") - return 0 - - def cmd_user_allow(args): user = args.user up = user_prefix() @@ -9560,420 +7734,41 @@ def cmd_user_disallow(args): printInfo(f"{args.user} is not in {g}.") -# WHAT AN ACCOUNT'S HOME HOLDS WHEN IT HAS NEVER RUN ANYTHING. -# `user create` puts a .bash_profile symlink there (and keeps whatever was -# there as .bash_profile.pkgusr), plus an empty Shared if the line asked for -# one. Nothing else is ours. -_HOME_SCAFFOLD = {".bash_profile", ".bash_profile.pkgusr", ".bashrc", - ".bash_logout", ".bash_history", ".profile", ".dircolors", - "pkg.lst"} - - -def _home_is_untouched(home): - """Is there anything in this home worth keeping? - - The archive exists because a home holds a browser profile: logins, - cookies, history, downloads -- months of state that `userdel -r` would - take with it and nothing could give back. A home that was created and - never used holds none of that, and a tarball of it is 200 bytes of - nothing in /root, one per account, fourteen more on the next purge. - """ - try: - entries = os.listdir(home) - except OSError: - return False - for n in entries: - if n in _HOME_SCAFFOLD: - continue - p = os.path.join(home, n) - if n == "Shared" and os.path.isdir(p) and not os.path.islink(p) \ - and not os.listdir(p): - continue - return False - return True - - -def _shared_hub(main_home): - """The main user's ~/Shared, where the per-account links live.""" - return os.path.join(main_home, "Shared") if main_home else "" - - -def _stale_shared_links(main_home, home=""): - """Links in ~/Shared that point into `home`, plus every link there that - leads nowhere at all. A dangling link is what keeps a SHARED_* variable - alive after the account behind it is gone.""" - hub = _shared_hub(main_home) - out = [] - if not hub or not os.path.isdir(hub): - return out - for f in sorted(os.listdir(hub)): - fp = os.path.join(hub, f) - if not os.path.islink(fp): - continue - if not os.path.exists(fp): # dangling: nothing to keep - out.append(fp) - elif home and os.readlink(fp).startswith(home.rstrip("/") + "/"): - out.append(fp) - return out - - -def _shared_vars_note(paths): - """Say what an open shell still thinks, and how to correct it. - - /etc/profile.d/shared-dirs.sh runs at LOGIN. Removing the link removes - tomorrow's variable; the shell you are typing in keeps today's until you - log in again -- so `cd $SHARED_BROWSER_MINI` still completes and still - fails, which reads like the removal did not work.""" - names = [os.path.basename(p).upper().replace("-", "_") for p in paths] - if not names: - return - printInfo(" shells that are already open still export " - + ", ".join("SHARED_" + n for n in names)) - printInfo(" they are read at login: log in again, or right now:") - printInfo(" unset " + " ".join("SHARED_" + n for n in names)) - - def cmd_user_delete(args): - """Remove an application account SAFELY. - - Only an application account: the user prefix, and a home under the - application-user root or the package tree. Refused while it runs - anything. Without --no-archive its home is packed first into - /root/<account>-home-<date>.tar.gz (0600). Then: its launchers, menu - entries and icons (the ones these tools wrote that run as it), the - ~/Shared/<Name> link, its line in /etc/pam.d/su_u_user, the account, its - group, and the home (kept with --keep-home). - """ u = _user_prefixed(args.name) - if not u.startswith(user_prefix()): - printError(f"{u}: not an application account ({user_prefix()}...) -- refused") - return 1 - exists = _user_exists(u) - home = (pwd.getpwnam(u).pw_dir if (exists and pwd) else "") or _app_home(u) - roots = [os.path.realpath(appuser_home()), os.path.realpath(BASE_DIR)] - if home and not any(os.path.realpath(home).startswith(r.rstrip("/") + "/") for r in roots): - printError(f"{u}: home {home} is not under {' or '.join(roots)} -- refused") - return 1 - main = main_user() - mh = _home_of(main) if main else None - # what goes - launchers = [] - if mh: - for d, marker in ((os.path.join(mh, "bin"), f"as the application user {u}."), - (os.path.join(mh, ".local", "share", "applications"), - f"Comment=Runs as the application user {u}\n")): - if os.path.isdir(d): - for f in sorted(os.listdir(d)): - fp = os.path.join(d, f) - if os.path.isfile(fp) and _ours_for(fp, marker): - launchers.append(fp) - icons = [] - if mh: - idir = os.path.join(mh, ".local", "share", "icons", "lfs-apps") - names = {os.path.splitext(os.path.basename(f))[0] for f in launchers} - if os.path.isdir(idir): - icons = [os.path.join(idir, f) for f in sorted(os.listdir(idir)) - if f.split(".")[0] in names] - # THE LINK, AND ANY LINK THAT NO LONGER LEADS ANYWHERE. - # - # A link was only removed when it pointed inside THIS home, so a home - # moved after the link was made (usermod -d, or the move to - # /home/shared_user) left ~/Shared/<Name> behind, dangling. Nothing - # then removed it -- and /etc/profile.d/shared-dirs.sh enumerates that - # directory, which is why $SHARED_BROWSER_MINI outlived the account. - links = _stale_shared_links(mh, home) - printHeader(f"Remove application account {u}") - print(f" account : {u}" + ("" if exists else " (already gone)")) - if home: - print(f" home : {home}" + (" (kept: --keep-home)" if args.keep_home else "")) - if os.path.isdir(home) and not args.no_archive: - if _home_is_untouched(home): - print(" archive : none -- nothing in this home but the files " - "these tools put there") - else: - print(f" archive : /root/{u}-home-<date>.tar.gz first (0600)") - for f in launchers + icons + links: - print(f" remove : {f}") - print(" pam : its line in /etc/pam.d/su_u_user") - if exists: - busy = subprocess.run(["pgrep", "-u", u], stdout=subprocess.DEVNULL).returncode == 0 - if busy: - printError(f" {u} is running something -- close it (pkill -u {u}) and try again") - return 1 + home = pkgusr_home(u) + has_home = os.path.isdir(home) + if not _user_exists(u) and not has_home: + printInfo(f"{u}: does not exist.") + return + if _user_exists(u): + pw_dir = pwd.getpwnam(u).pw_dir if pwd else "" + if not pw_dir.startswith(BASE_DIR.rstrip("/") + "/"): + printError(f"{u}: user home {pw_dir} is not under {BASE_DIR}; " + f"refusing to delete.") + return + printHeader(f"Delete shared user {u}") + print() + if _user_exists(u): + print(f" user : {u} (uid {pwd.getpwnam(u).pw_uid})") + if has_home: + print(f" home : {home}") if not args.run: - printInfo("\n dry run -- nothing removed. Remove: " - f"packagemanager user delete {args.name} --run") - printInfo(" and take its lines out of /etc/pkgusr/stacks/app-users.conf, " - "or the next `cfg app-users` creates it again") - return 0 - if not args.yes and not ask_yes_no(f"\nReally remove {u}?"): + printInfo("\nDry run -- re-run with --run to delete.") + return + if not args.yes and not ask_yes_no(f"\nReally delete {u}?"): printWarning("Aborted.") - return 1 - if home and os.path.isdir(home) and not args.no_archive \ - and not _home_is_untouched(home): - arch = f"/root/{u}-home-{time.strftime('%Y%m%d-%H%M%S')}.tar.gz" - old_umask = os.umask(0o077) - ok_arch = _run(["tar", "-C", os.path.dirname(home), "-czf", arch, - os.path.basename(home)]) - os.umask(old_umask) - if not ok_arch: - printError(f" could not archive {home} -- nothing removed") - return 1 - _run(["chmod", "600", arch]) - printInfo(f" archived {home} -> {arch}") - for f in launchers + icons + links: - try: - os.unlink(f) - printInfo(f" removed {f}") - except OSError as e: - printWarning(f" {f}: {e}") - pam = "/etc/pam.d/su_u_user" - if os.path.isfile(pam): - try: - txt = open(pam).read() - new = re.sub(r"^\s*%s\)[^\n]*\n" % re.escape(u), "", txt, flags=re.M) - if new != txt: - open(pam, "w").write(new) - printInfo(f" removed {u} from {pam}") - except OSError as e: - printWarning(f" {pam}: {e}") - if exists: + return + if _user_exists(u): _run(["userdel", u]) if _group_exists(u): _run(["groupdel", u]) - if home and os.path.isdir(home) and not args.keep_home: + if has_home and os.path.isdir(home): try: shutil.rmtree(home) - printInfo(f" removed {home}") except OSError as e: printError(f" rmtree {home}: {e}") - # the hub is ours only while it holds our links: an empty ~/Shared is - # left-over furniture, a ~/Shared with something else in it is yours. - hub = _shared_hub(mh) - if hub and os.path.isdir(hub) and not os.listdir(hub): - try: - os.rmdir(hub) - printInfo(f" removed {hub} (empty)") - except OSError: - pass - printSuccess(f"{u} removed.") - _shared_vars_note(links) - printInfo(" take its lines out of /etc/pkgusr/stacks/app-users.conf, " - "or the next `cfg app-users` creates it again") - return 0 - - -# =========================================================================== # -# command: user purge -- undo the whole app-users setup [IMPLEMENTED] -# =========================================================================== # -# -# `user delete` removes ONE account. What it cannot remove is everything the -# accounts share, because the next account still needs it: the session group, -# the two profile fragments, /usr/bin/xdg-runtime-share, the pam file and the -# line in /etc/pam.d/su that calls it. Left behind, those are what makes a -# "deleted" setup still visible -- a group you are in, variables in every new -# shell, an auth line pointing at a file that is gone. -# -# NOTHING HERE TOUCHES A FILE THESE TOOLS DID NOT WRITE. Each file is read -# first and only removed if it carries the marker `cfg app-users` writes. -# The home archives in /root are never removed: they are the undo of the undo. - -APP_USERS_MARKER = "# written by stacks/app-users.sh -- safe to delete, regenerated" -APP_USERS_CONF = "/etc/pkgusr/stacks/app-users.conf" - - -def _conf_accounts(path): - """The account names in app-users.conf: column 1, both file formats.""" - out = [] - try: - lines = open(path, encoding="utf-8", errors="replace").read().splitlines() - except OSError: - return out - for raw in lines: - s = raw.strip() - if not s or s.startswith("#"): - continue - a = s.split("|")[0].strip() - if a in ("", "name", "user", "account") or re.search(r"[^a-z0-9_-]", a): - continue - if a not in out: - out.append(a) - return out - - -def _ours(path, marker): - """A file that exists and says these tools wrote it.""" - if not os.path.isfile(path): - return False - try: - return marker in open(path, encoding="utf-8", errors="replace").read() - except OSError: - return False - - -def cmd_user_purge(args): - """Undo the application-account setup: the accounts, then what they share. - - Dry run unless --run. Homes are archived to /root first, exactly as - `user delete` does, and those archives stay. - """ - conf = args.conf or APP_USERS_CONF - main = main_user() - mh = _home_of(main) if main else "" - accounts = _conf_accounts(conf) - if args.accounts: - accounts = [a for a in re.split(r"[,\s]+", args.accounts) if a] - # ACCOUNTS THE FILE NO LONGER MENTIONS ARE STILL OURS TO REMOVE. That is - # exactly the browser-mini/browser-big case: the line was rewritten, the - # account stayed. Found in passwd, not by listing a directory, so an - # account whose home is already gone is found too. - strays = [] - if pwd: - root = os.path.realpath(appuser_home()) - for e in pwd.getpwall(): - if not e.pw_name.startswith(user_prefix()): - continue - a = e.pw_name[len(user_prefix()):] - if a in accounts or a in strays: - continue - if os.path.realpath(e.pw_dir or "").startswith(root.rstrip("/") + "/"): - strays.append(a) - strays.sort() - todo = [a for a in accounts + strays if _user_exists(_user_prefixed(a))] - - printHeader("Undo the application-account setup") - print() - print(" accounts : " + (", ".join(user_prefix() + a for a in todo) or "none left")) - shared = [] - for p in ("/etc/profile.d/shared-dirs.sh", "/etc/profile.d/xdg-runtime-share.sh", - "/usr/bin/xdg-runtime-share"): - if _ours(p, APP_USERS_MARKER): - shared.append(p) - elif os.path.exists(p): - printWarning(f" {p} is not ours -- left alone") - pam_file = "/etc/pam.d/su_u_user" - pam_su = "/etc/pam.d/su" - # an ACTIVE auth line, not a comment that happens to name the file - pam_line = bool(os.path.isfile(pam_su) and re.search( - r"^\s*auth\s+sufficient\s+pam_exec\.so.*su_u_user", - open(pam_su, encoding="utf-8", errors="replace").read(), re.M)) - for p in shared: - print(f" remove : {p}") - if os.path.isfile(pam_file): - print(f" remove : {pam_file}") - if pam_line: - print(f" edit : {pam_su} (drop the pam_exec line; backup kept)") - g = xdg_runtime_group() - if _group_exists(g): - print(f" group : {g} (removed; {main or 'you'} leaves it)") - print(" kept : /root/" + user_prefix() + "*-home-*.tar.gz (the archives)") - print(f" kept : {conf} -- delete the lines you no longer want, or the") - print(" next `cfg app-users` builds all of this again") - - if not args.run: - printInfo("\n dry run -- nothing removed. Undo it: " - "packagemanager user purge --run") - return 0 - if not args.yes and not ask_yes_no("\nReally undo all of it?"): - printWarning("Aborted.") - return 1 - - # 1. the accounts, through the command that already knows how - import argparse as _ap - gone = 0 - for a in todo: - rc = cmd_user_delete(_ap.Namespace( - name=a, run=True, yes=True, - keep_home=args.keep_home, no_archive=args.no_archive)) - if rc: - printWarning(f" {a}: not removed -- stopping before the shared parts") - printInfo(" (close what it is running, then run this again)") - return 1 - gone += 1 - - # 2. anything left in ~/Shared that leads nowhere, then the hub itself - dead = _stale_shared_links(mh) - for p in dead: - try: - os.unlink(p) - printInfo(f" removed {p}") - except OSError as e: - printWarning(f" {p}: {e}") - hub = _shared_hub(mh) - if hub and os.path.isdir(hub) and not os.listdir(hub): - try: - os.rmdir(hub) - printInfo(f" removed {hub} (empty)") - except OSError: - pass - idir = os.path.join(mh, ".local", "share", "icons", "lfs-apps") if mh else "" - if idir and os.path.isdir(idir) and not os.listdir(idir): - try: - os.rmdir(idir) - printInfo(f" removed {idir} (empty)") - except OSError: - pass - - # 3. the files every account shared - for p in shared: - try: - os.unlink(p) - printInfo(f" removed {p}") - except OSError as e: - printWarning(f" {p}: {e}") - if os.path.isfile(pam_file): - try: - os.unlink(pam_file) - printInfo(f" removed {pam_file}") - except OSError as e: - printWarning(f" {pam_file}: {e}") - # THE LINE IN /etc/pam.d/su GOES LAST AND WITH A BACKUP. It is the one - # edit here that can cost you `su` if it goes wrong. - if pam_line: - bak = f"{pam_su}.bak-{time.strftime('%Y%m%d-%H%M%S')}" - try: - txt = open(pam_su).read() - new = "\n".join(l for l in txt.splitlines() - if not re.match(r"^\s*auth\s+sufficient\s+pam_exec\.so.*su_u_user", - l)) + "\n" - if new != txt: - shutil.copy2(pam_su, bak) - open(pam_su, "w").write(new) - printInfo(f" {pam_su}: pam_exec line removed (backup {bak})") - except OSError as e: - printWarning(f" {pam_su}: {e}") - - # 3b. the directory the accounts lived in, if it is now empty - root = appuser_home() - if os.path.isdir(root) and not os.listdir(root): - try: - os.rmdir(root) - printInfo(f" removed {root} (empty)") - except OSError: - pass - - # 4. the session group -- last, when nothing is in it any more - if _group_exists(g): - _run(["groupdel", g]) - printInfo(f" removed group {g}") - - printSuccess(f"undone: {gone} account(s) and everything they shared.") - _shared_vars_note(dead) - printInfo(" your session still has the old group until you log in again") - # THE ARCHIVES ARE THE ONE THING THIS DOES NOT REMOVE, so say what they - # are and how to be rid of them. They only exist for a home that had - # something in it -- a browser profile, its logins and its downloads. - import glob as _glob - arch = sorted(_glob.glob(f"/root/{user_prefix()}*-home-*.tar.gz")) - if arch: - printInfo(f" {len(arch)} home archive(s) kept in /root -- browser profiles, " - f"nothing else can give them back") - printInfo(f" look: tar tzf {arch[-1]} | head") - printInfo(f" or drop: rm /root/{user_prefix()}*-home-*.tar.gz") - if mh and os.path.isfile(os.path.join(mh, ".config", "sway", "config")): - printInfo(" and `exec xdg-runtime-share` is still in your sway config " - "-- yours to remove") - return 0 + printSuccess(f"{u} deleted.") # =========================================================================== # @@ -10407,7 +8202,7 @@ def cmd_script_status(args): printWarning("the book named no file to check):") for n in _act: u = gather_user(n) - st, why = assess(u, None, run_cmd=False) + st, why = assess(u, versions.get(u.meta.get("name") or n)) printWarning(" %-24s %-14s %s" % (n, st, why)) # ONLY WHAT THIS COMMAND CAN ACTUALLY BUILD. # @@ -10807,108 +8602,7 @@ def _warn_env_companions(vals): % (other, _ENV_SUGGESTIONS.get(other, "..."))) -SYSTEM_ENV_FILE = "/etc/profile.d/pkgusr-system.sh" -SYSTEM_ENV_MARKER = ("# written by `packagemanager env --system` -- " - "regenerate, do not edit") - - -def _system_env_values(): - """This system's own facts, for anything that needs to ask. - - Scripts, launchers and one-off commands keep rediscovering the same few - things -- who the human is, what prefixes this machine uses, where the - shared directories are -- and each rediscovery is a different guess: - `cat /etc/pkgusr/desktop-user` in one place, a hard-coded `u_` in - another, `$HOME/Shared` in a third. - - NOT the PKGUSR_ namespace. The tools READ PKGUSR_BASE, PKGUSR_CONFIG - and PKGUSR_LIB as overrides, so exporting that namespace system-wide - would reconfigure every tool in every shell -- a login that quietly - points the package manager at another tree. PKGUSR_SYS_ is inert: - nothing consumes it, so writing it cannot change what anything does. - """ - main = main_user() or "" - vals = { - "PKGUSR_SYS_MAIN_USER": main, - "PKGUSR_SYS_USER_PREFIX": user_prefix(), - "PKGUSR_SYS_PKG_PREFIX": pkgusr_prefix(), - "PKGUSR_SYS_COLLECTOR_PREFIX": collector_prefix(), - } - home = _home_of(main) if main else None - if home: - vals["PKGUSR_SYS_HOME"] = home - vals["PKGUSR_SYS_SHARED"] = os.path.join(home, "Shared") - return vals - - -def cmd_env_system(args): - """Write, or show, the system-fact variables as a profile fragment.""" - vals = _system_env_values() - missing = [k for k, v in vals.items() if not v] - body = [SYSTEM_ENV_MARKER, - "# This system's own facts. Read them; do not set them here --", - "# `packagemanager config` is where they live and this file is", - "# generated from it.", - ""] - for k in sorted(vals): - if vals[k]: - body.append('export %s="%s"' % (k, vals[k])) - text = "\n".join(body) + "\n" - - if not getattr(args, "run", False): - print(text, end="") - printInfo(" (dry run: --run writes %s)" % SYSTEM_ENV_FILE) - if missing: - printWarning(" not known yet, so not exported: %s" - % ", ".join(missing)) - printInfo(" packagemanager config --main-user <your-login>") - return 0 - - d = os.path.dirname(SYSTEM_ENV_FILE) - if not os.path.isdir(d): - printError("no %s -- nowhere to put a login-shell fragment" % d) - printInfo(" the values are above; export them from your own shell") - return 1 - # NEVER OVERWRITE A FILE WE DID NOT WRITE. - if os.path.exists(SYSTEM_ENV_FILE): - try: - had = open(SYSTEM_ENV_FILE, encoding="utf-8", - errors="replace").read() - except OSError: - had = "" - if SYSTEM_ENV_MARKER not in had: - printError("%s exists and is not ours -- left alone" - % SYSTEM_ENV_FILE) - printInfo(" move it aside, or export these yourself:") - print(text, end="") - return 1 - with open(SYSTEM_ENV_FILE, "w") as f: - f.write(text) - os.chmod(SYSTEM_ENV_FILE, 0o644) - # ROOT WROTE IT, SO ROOT OWNS IT. Writing into an existing file keeps - # that file's owner, and this one was first created inside a package - # user's build window -- so it stayed p_gstreamer10's, a package owning a - # file no package installed, in a directory every login shell sources. - # Re-asserted on every write; a chown that cannot run is not fatal. - try: - if os.stat(SYSTEM_ENV_FILE).st_uid != 0: - _run(["chown", "root:root", SYSTEM_ENV_FILE]) - except OSError: - pass - printSuccess("wrote %s" % SYSTEM_ENV_FILE) - for k in sorted(vals): - if vals[k]: - printInfo(" %-28s %s" % (k, vals[k])) - if missing: - printWarning(" not known yet: %s" % ", ".join(missing)) - printInfo(" login shells read /etc/profile, which reads that directory.") - printInfo(" in this shell, now: . %s" % SYSTEM_ENV_FILE) - return 0 - - def cmd_env(args): - if getattr(args, "system", False): - return cmd_env_system(args) vals = _read_build_env() if args.set: for pair in args.set: @@ -11037,63 +8731,17 @@ def _find_stack_file(name): "stacks", c)) for c in out: if os.path.isfile(c): - found = os.path.abspath(c) - _warn_shadowed_stack(found) - return found + return os.path.abspath(c) return None -def _warn_shadowed_stack(found): - """A stack file found BEFORE the installed one, that differs from it. - - The path as given is tried first, so `packagemanager stack services` run - from ~/lfs read ~/lfs/services.stack -- and a `cfg` entry runs - <that directory>/<name>.sh. An old shadow-pam.sh there ran instead of - the 1.14.262 one in /etc/pkgusr/stacks, printed the old "su and - system-auth are in place", and nothing said which file had run. Say it - when the two differ; a checkout that matches is not worth a word.""" - etc = "/etc/pkgusr/stacks" - if os.path.dirname(found) == etc: - return - inst = os.path.join(etc, os.path.basename(found)) - if not os.path.isfile(inst): - return - try: - import filecmp - differ = not filecmp.cmp(found, inst, shallow=False) - # the cfg scripts beside it are what actually run - d = os.path.dirname(found) - scripts = [f for f in sorted(os.listdir(d)) if f.endswith(".sh") - and os.path.isfile(os.path.join(etc, f)) - and not filecmp.cmp(os.path.join(d, f), - os.path.join(etc, f), shallow=False)] - except OSError: - return - if not differ and not scripts: - return - printWarning(" NOTE: using %s, not the installed %s" % (found, inst)) - if scripts: - printWarning(" cfg scripts that differ from the installed ones and " - "will run from %s: %s" % (d, ", ".join(scripts))) - printWarning(" run from another directory, or give the full path, " - "to use the installed stack") - - def _parse_stack_file(path): """-> list of {kind, target, opts{}} in file order. Raises ValueError with a line number on anything malformed -- a plan half-understood is worse than one refused.""" import shlex entries = [] - kinds = ("book", "git", "tar", "check", "cfg", "avoid", "include") - # THE "INCLUDE ONCE" SET BELONGS TO THE WHOLE PARSE, not to one include - # line. Owning it per line meant sway.stack's `include avoid` finished - # and cleared it, so `include media` started fresh and media's own - # `include avoid` spliced the 87 exclusions a second time -- and a third - # for the nesting: 332 entries where there are 158. - _top_parse = getattr(_parse_stack_file, "_spliced", None) is None - if _top_parse: - _parse_stack_file._spliced = {os.path.abspath(path)} + kinds = ("book", "git", "tar", "check", "cfg", "avoid") # A LINE MAY CONTINUE. # # Every entry had to be one line, so sway.stack's are 300 characters wide @@ -11116,8 +8764,7 @@ def _parse_stack_file(path): if _buf: raise ValueError("%s:%d: line ends with a backslash and nothing follows" % (path, _start)) - try: - for n, raw in _joined: + for n, raw in _joined: line = raw.strip() if not line or line.startswith("#"): continue @@ -11133,58 +8780,6 @@ def _parse_stack_file(path): if kind not in kinds: raise ValueError("%s:%d: unknown entry kind %r (one of %s)" % (path, n, kind, "/".join(kinds))) - # ONE LIST, TWO ENTRY POINTS. - # - # The media chain -- codecs, gstreamer, the WebRTC pieces, - # webkitgtk -- is worth updating on its own without replaying a - # whole desktop. Copying those entries into a second stack file - # would give two lists of the same packages, and two lists drift: - # gst10-libav was added to one of them for three releases before - # anyone noticed the other. - # - # `include <name>` splices another stack's entries in at this - # point, in order. The included file is a stack in its own right, - # so `packagemanager stack media --run` builds just that part. - if kind == "include": - if len(parts) != 2: - raise ValueError("%s:%d: include takes one stack name" - % (path, n)) - _inc = parts[1] - _p = _inc if os.path.sep in _inc else os.path.join( - os.path.dirname(os.path.abspath(path)), - _inc if _inc.endswith(".stack") else _inc + ".stack") - if not os.path.isfile(_p): - raise ValueError("%s:%d: include %s: no such stack (%s)" - % (path, n, _inc, _p)) - if os.path.abspath(_p) == os.path.abspath(path): - raise ValueError("%s:%d: include %s includes itself" - % (path, n, _inc)) - # A CYCLE WOULD RECURSE FOR EVER, and the error a reader gets - # from a RecursionError names none of the files involved. - _seen = getattr(_parse_stack_file, "_including", set()) - if os.path.abspath(_p) in _seen: - raise ValueError("%s:%d: include %s: circular include" - % (path, n, _inc)) - # INCLUDE ONCE PER COMPOSITION. - # - # sway.stack includes `avoid` AND `media`, and media.stack - # includes `avoid` too -- so the 87 exclusions were spliced - # three times: 332 entries where there are 158, and every - # avoided package listed three times in the plan. A second - # include of the same file adds nothing by definition, so - # skip it. - _sp = _parse_stack_file._spliced - if os.path.abspath(_p) in _sp: - continue # already in this plan - _sp.add(os.path.abspath(_p)) - _seen = _seen | {os.path.abspath(path)} - _parse_stack_file._including = _seen - try: - entries.extend(_parse_stack_file(_p)) - finally: - _parse_stack_file._including = \ - _seen - {os.path.abspath(path)} - continue if len(parts) < 2: raise ValueError("%s:%d: %r needs a target" % (path, n, kind)) target = parts[1] @@ -11207,11 +8802,8 @@ def _parse_stack_file(path): if kind == "tar" and ("url" not in opts or "pkg" not in opts): raise ValueError("%s:%d: a tar entry needs url= and pkg=" % (path, n)) entries.append({"kind": kind, "target": target, "opts": opts}) - if not entries: + if not entries: raise ValueError("%s: the stack file has no entries" % path) - finally: - if _top_parse: - _parse_stack_file._spliced = None return entries @@ -11398,97 +8990,7 @@ def _check_stack_refs(rows): return 1 if bad else 0 -def _list_stacks(): - """`packagemanager stack` with no name: what stacks there are. - - Every *.stack in /etc/pkgusr/stacks (and stacks/ beside this tool, when - that is a different directory), with its one-line description, how many - entries it has and how many the cache records as done, and whether a - newer shipped copy (.new) is waiting. avoid.stack is a list of what is - never built, not something to run, and says so. - """ - dirs = [] - for d in ("/etc/pkgusr/stacks", - os.path.join(os.path.dirname(os.path.abspath(__file__)), "stacks")): - if os.path.isdir(d) and os.path.realpath(d) not in [os.path.realpath(x) for x in dirs]: - dirs.append(d) - if not dirs: - printError("no stack directory (/etc/pkgusr/stacks) -- `make install` puts them there") - return 1 - any_behind = [] - # DONE IS WHAT ANY STACK RECORDED. sway includes media, so media's - # packages were built -- and recorded -- under sway, and `media` said - # "1/19 done" with all of it installed. - done_all = set() - for d in dirs: - for f in os.listdir(d): - if f.endswith(".stack"): - done_all |= _stack_done_set(os.path.join(d, f)) - # INSTALLED IS NOT CURRENT. The cache says an entry was built once; it - # never looks again. Compare each book entry's installed version with - # the book, so "79/79 done" cannot hide a sway that is three releases - # behind. One map for the whole book (cached); no book, no judgement. - try: - _bv = book_versions_all() or {} - except Exception: - _bv = {} - for d in dirs: - files = sorted(f for f in os.listdir(d) if f.endswith(".stack")) - printHeader("stacks in %s" % d) - if not files: - printInfo(" (none)") - continue - for f in files: - path = os.path.join(d, f) - name = f[:-len(".stack")] - desc = "" - try: - first = open(path, encoding="utf-8", errors="replace").readline() - if " -- " in first: - desc = first.split(" -- ", 1)[1].strip() - except OSError: - pass - try: - # what it BUILDS: the included avoid list is not an entry - entries = [e for e in _parse_stack_file(path) if e["kind"] != "avoid"] - except (OSError, ValueError) as ex: - print(" %-10s (does not parse: %s)" % (name, ex)) - continue - behind = "" - if name == "avoid": - state = "never built" - else: - done = len(done_all - & {"%s %s" % (e["kind"], e["target"]) for e in entries}) - state = "%d/%d done" % (done, len(entries)) - if _bv: - old_n = 0 - for e in entries: - if e["kind"] != "book": - continue - want = _bv.get(e["target"]) or _bv.get(e["target"].lower()) - have = gather_user(e["target"]).version if want else None - if want and have and not _same_version(want, have): - old_n += 1 - if old_n: - behind = " %d entr%s behind the book" % (old_n, "y" if old_n == 1 else "ies") - any_behind.append(name) - new = " (.new waiting: --take-new)" if os.path.isfile(path + ".new") else "" - print(" %-10s %-15s %s%s%s" % (name, state, desc, behind, new)) - print() - printInfo(" plan one: packagemanager stack <name>") - printInfo(" where it is, and what is behind (dependencies too):" - " packagemanager stack <name> --status") - if not _bv: - printInfo(" (no book on disk: versions not compared)") - printInfo(" (\"done\" means built once -- a stack run never updates; " - "packagemanager update --stack <name> does)") - return 0 - - def cmd_stack(args): - if not args.name: - return _list_stacks() path = _find_stack_file(args.name) if not path: printError("no stack file for %r (looked in ., /etc/pkgusr/stacks, " @@ -11550,40 +9052,15 @@ def cmd_stack(args): sys.exit(1) printSuccess("took the shipped stack file; yours is %s" % _bak) entries = _parse_stack_file(path) - # THE SAME MEASURE THE HEADER USES. This said "149 entries - # now" while the header two lines later said "(62 entries)": - # one counts every line of the file, the other the buildable - # plan after `avoid`. Two numbers for one file, both called - # entries. - printInfo(" %d line(s) in the file; the plan below is what" - " will be built." % len(entries)) + printInfo(" %d entries now." % len(entries)) print() _new = None if _new and os.path.isfile(_new): - # WHAT DIFFERS, NOT HOW MANY. The notice printed - # (149 entries, this file has 149) - # for two files that differ in every entry's options: the counts - # were equal, so its own evidence argued there was nothing to - # take. An entry count cannot see a changed expect= or a - # reordered library. printWarning("a newer version of this stack ships beside it:") - printWarning(" %s" % _new) - try: - import difflib as _dl - a = open(path, encoding="utf-8", errors="replace").readlines() - b = open(_new, encoding="utf-8", errors="replace").readlines() - added = sum(1 for l in _dl.unified_diff(a, b, n=0) - if l.startswith("+") and not l.startswith("+++")) - removed = sum(1 for l in _dl.unified_diff(a, b, n=0) - if l.startswith("-") and not l.startswith("---")) - _same = (_n_entries == len(entries)) - printWarning(" %d line(s) added, %d removed%s" - % (added, removed, - "; the entry count is the same, so the change" - " is in the entries themselves" - if _same and (added or removed) else "")) - except OSError: - pass + printWarning(" %s%s" % (_new, + "" if _n_entries is None else + " (%d entries, this file has %d)" + % (_n_entries, len(entries)))) printInfo(" diff %s %s" % (path, _new)) printInfo(" ...and take what you want; this run uses YOUR file.") print() @@ -11712,23 +9189,6 @@ def cmd_stack(args): # (entries are already filtered to --only above; the old `only` # variable was removed with the duplicate implementation, and this # reference outlived it -- NameError on every stack run.) - # A CACHED cfg ENTRY IS RE-CHECKED WHEN IT CAN BE. - # - # app-users puts the pam_exec line into /etc/pam.d/su; shadow's - # configure phase later rewrote su without it, and every stack run - # after that said `cfg app-users done (stack cache)` -- so - # `su - u_browser-mini` asked for a password and nothing re-ran the - # step that fixes it. A cfg entry's expect= states what it keeps - # true; when that no longer holds, the cache is wrong, not the check. - if (e["_key"] in donekeys and e["kind"] == "cfg" - and e["opts"].get("expect") - and subprocess.run(["/bin/bash", "-c", e["opts"]["expect"]], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL).returncode != 0): - donekeys.discard(e["_key"]) - if not getattr(args, "status", False): - printWarning(" cfg %s is cached as done, but its check fails" - " now -- running it again" % e["target"]) if e["_key"] in donekeys: rows.append((e, "done (stack cache)")) if not getattr(args, "status", False): @@ -11775,29 +9235,11 @@ def cmd_stack(args): notinbook = [e["target"] for e, st in rows if st == "NOT IN BOOK"] if notinbook: printWarning("\nnot in the book: " + ", ".join(notinbook) - + "\n WHY is the first question -- the page may be there under" - "\n another anchor, or this book may be a different one than" - "\n you think:" - + "".join("\n blfs debug %s" % t for t in notinbook) - + "\n then: rename the entry to the book's anchor, or make it a" - "\n tar/git entry with build=/install= commands.") - if args.run and not getattr(args, "skip_unknown", False): - # ONE BAD LINE MUST NOT COST THE OTHER EIGHT. Refusing the whole - # plan is right by default -- a plan that silently drops entries - # is how a stack looks finished while a package is missing -- but - # an entry the book has no page for blocked the eight that build - # and the checks that report, with no way past it but editing the - # file. + + "\n fix the entry (rename to the book's anchor, or make it" + "\n a git entry with build=/install= commands)") + if args.run: printError("refusing to run a plan with unknown targets.") - printInfo(" to run the rest and come back to it:" - "\n packagemanager stack %s --run --skip-unknown" - % os.path.basename(path)) sys.exit(1) - if args.run: - printWarning(" --skip-unknown: leaving %d entr%s out of this run" - % (len(notinbook), "y" if len(notinbook) == 1 else "ies")) - entries = [e for e, st in rows if st != "NOT IN BOOK"] - rows = [(e, st) for e, st in rows if st != "NOT IN BOOK"] missing = [e["target"] for e, st in rows if st == "MISSING SCRIPT"] if missing: @@ -11839,31 +9281,11 @@ def cmd_stack(args): pkgs.append((anchor, nv)) done_pkgs = [(a, v) for a, v in pkgs if "not installed" not in install_tag(a, v)] todo = [(a, v) for a, v in pkgs if (a, v) not in set(done_pkgs)] - # DONE BY ANY STACK: media's entries are built through sway, which - # includes it, and recorded in sway's cache -- "1 of 19 done" with - # 189 of 215 packages installed. - _done_any = set() - for _sd in {os.path.dirname(path), "/etc/pkgusr/stacks"}: - if os.path.isdir(_sd): - for _sf in os.listdir(_sd): - if _sf.endswith(".stack"): - _done_any |= _stack_done_set(os.path.join(_sd, _sf)) - done_n = sum(1 for e, st in rows - if st == "done (stack cache)" or e.get("_key") in _done_any - or ("%s %s" % (e["kind"], e["target"])) in _done_any) - _tags = {(a, v): install_tag(a, v) for a, v in done_pkgs} - behind = [a for (a, v), t in _tags.items() if "book is newer" in t] + done_n = sum(1 for _, st in rows if st == "done (stack cache)") printHeader("stack %s" % args.name) print(" entries : %d of %d done" % (done_n, len(rows))) if pkgs: print(" packages: %d of %d installed" % (len(done_pkgs), len(pkgs))) - if behind: - # INSTALLED IS NOT CURRENT, and --run never updates. Say how many, - # and the one command that does. - print(" behind : %d installed older than the book (%s%s)" - % (len(behind), ", ".join(behind[:5]), " ..." if len(behind) > 5 else "")) - print(" update them, with what depends on them: " - "packagemanager update --stack %s" % args.name) if skipped_by_avoid: print(" avoided : %d named in the stack (%s%s)" % (len(skipped_by_avoid), @@ -11872,20 +9294,8 @@ def cmd_stack(args): nxt_entry = next((e["target"] for e, st in rows if st != "done (stack cache)"), None) print(" next : %s" % (nxt_entry or "nothing left to do")) if todo: - # SAY WHAT A RUN WOULD DO, because the list above reads like a - # plan and is not one. `--run` builds what is MISSING; it never - # updates a package that is installed, however far behind the - # book it is. On a system built from an older book that is half - # the tree marked "(update available)", and it looked like - # `stack media --run` was about to rebuild glib2. - print(" to build (missing): " + ", ".join(a for a, _ in todo[:6]) + print(" building next: " + ", ".join(a for a, _ in todo[:6]) + (" ..." if len(todo) > 6 else "")) - print(" %d missing in total. Installed packages are NOT" - " touched by --run," % len(todo)) - print(" whatever the book now carries: \"book is newer\" is a" - " report, not a plan.") - print(" one package, deliberately: packagemanager update" - " <name> --run") if getattr(args, "full", False): print() for a, v in pkgs: @@ -12029,38 +9439,9 @@ def cmd_stack(args): install_local(target, spath, True, args.yes, "install") else: # check / cfg -- a root script through the one runner sp = _stack_root_script(e, stack_dir) - # THE SETTINGS COME FROM THE RESOLVER, NOT FROM EACH SCRIPT. - # power.sh parsed packagemanager.conf itself, with the lfs - # config's JSON syntax, on a key=value file -- so on the booted - # machine it found nothing and stopped the stack. The resolver - # knows every source (lfs config, this conf, the collector - # groups on the system); a cfg script reads the environment. - # Order: environment, lfs config, packagemanager.conf (both via - # load_config, which lets the lfs config win), then what the - # system shows. A built-in default is never exported. - _cenv = dict(os.environ) - _known = _adopt_existing_settings() - _file_keys = set() - try: - with open(CONFIG_PATH) as _cf: - _file_keys = {l.split("=", 1)[0].strip() for l in _cf - if "=" in l and not l.lstrip().startswith("#")} - except OSError: - pass - for _key, _var in (("collector_prefix", "LFS_COLLECTOR_PREFIX"), - ("main_user", "LFS_MAIN_USER")): - if _cenv.get(_var): - continue - if _key in _known and _known[_key][1] == "the lfs config": - _cenv[_var] = _known[_key][0] - elif _key in _file_keys and load_config().get(_key): - _cenv[_var] = str(load_config()[_key]).rstrip("_") - elif _key in _known: - _cenv[_var] = _known[_key][0] rc = subprocess.run(["lfs-helper", "run-script", "--user", "root", "--as-root", "--script", sp, - "--phase", "all", "--name", ""], - env=_cenv).returncode + "--phase", "all", "--name", ""]).returncode if rc != 0: printError("stack stopped at %s %r (exit %d, %d of %d done)." % (kind, target, rc, done, len(rows))) @@ -12072,146 +9453,14 @@ def cmd_stack(args): # installed (prog=/have=); if that is still absent after a build the # tools said succeeded, say so and leave the entry unrecorded rather # than hiding it forever. - # EXPECT: A FEATURE IS NOT PROVED BY A FILE EXISTING. - # - # ffmpeg and SDL3 autodetect ALSA. Built before alsa-lib existed -- - # which is what happens when a library is listed AFTER a package - # whose dependency plan pulls its consumers in -- they configure - # cleanly, install cleanly, pass have=, and have no sound. Nothing - # in a build log says so, because nothing went wrong. - # - # expect="<shell>" runs after the entry and must succeed. It is the - # only thing here that checks what was built rather than whether a - # build happened. - _expect = e["opts"].get("expect") - _expect_ok = True - if _expect: - printInfo(" checking what %s was built with: %s" % (target, _expect)) - _erc = subprocess.run(["/bin/bash", "-c", _expect], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL).returncode - if _erc != 0: - _expect_ok = False - printError("%s installed, but the feature check failed:" % target) - printError(" %s" % _expect) - if e["kind"] == "cfg": - # A cfg STEP HAS NO OPTIONAL DEPENDENCIES. Its check is - # what the step keeps true; after running it, that is - # still false -- the step's own output above says why. - printWarning(" This is a configuration step: it ran, and what it" - " keeps true is still not true.") - printWarning(" Read its output above for the part it could not" - " do, then run it again:") - printInfo(" packagemanager stack %s --only %s --run" - % (os.path.basename(path).replace(".stack", ""), - target)) - _expect_msg_done = True - else: - _expect_msg_done = False - if _erc != 0 and not _expect_msg_done: - printWarning(" The package is built and installed -- it is" - " MISSING A FEATURE, which almost always means") - printWarning(" its optional dependency was not installed yet" - " when it configured.") - printInfo(" What the check says is missing is usually fixed by:") - printInfo(" packagemanager install <the dependency> --run") - # A STACK ENTRY DOES NOT REBUILD AN INSTALLED PACKAGE. - # - # This used to suggest `stack <file>.stack --only <pkg> --run` - # -- with the .stack suffix, which is not a stack name -- and - # that command runs `install`, which answers "everything is - # installed and up to date" and changes nothing. Re-running - # it can never fix the feature, so recommending it was a loop. - # ...AND THE RIGHT COMMAND DEPENDS ON THE KIND OF ENTRY. - # - # `update --reinstall --regenerate` regenerates from the - # BOOK, and a git or tar entry has no book page -- for - # gst-plugins-rs that advice could not work at all. Those - # are rebuilt by the stack itself, with --force. - if e["kind"] == "book": - printInfo(" %s" % " ".join(_rebuild_cmd(target))) - printInfo(" Re-running this stack entry will NOT rebuild" - " it: the package is installed,") - printInfo(" so the entry's install step has nothing to" - " do. The line above does.") - else: - printInfo(" packagemanager stack %s --only %s --run" - " --force" - % (os.path.basename(path).replace(".stack", ""), - target)) - printInfo(" This is a %s entry, not a book page: there is" - " nothing to regenerate" % e["kind"]) - printInfo(" from the book, and --force is what rebuilds" - " it. (`update --reinstall`") - printInfo(" would look for a book page that does not" - " exist.)") - # AND SAY WHAT have= PROVED, WHICH IS LESS THAN IT LOOKS. - # A part-installed package satisfies it: gst-plugins-rs's - # first run installed libgstrswebrtc.so and then died on the - # next plugin, so have= said "installed" and audiornnoise - # was never there. - _have_f = e["opts"].get("have") - if _have_f: - printWarning(" NOTE: have=%s exists, which is why this" - " was skipped as installed --" % _have_f) - printWarning(" but one file is not the whole package." - " The probe is the better answer.") - - # A BOOK ENTRY WAS RECORDED DONE WITHOUT BEING CHECKED AT ALL. - # - # The guard below covered git and tar only. poppler -- a `book` - # entry -- came out `done (stack cache)` while `pkg-config - # --list-all | grep poppler` found nothing: not installed, and the - # stack skipped it on every later run, so zathura-pdf-poppler kept - # failing on a dependency the plan believed it had. - # - # Every kind gets the same question now: is it there AFTERWARDS? - # For a book package that is the package state, which is what - # `verify` reads -- an empty manifest is not an install. - _installed_now = True - _missing_what = "" - if e["kind"] in ("git", "tar"): - _installed_now = _git_installed(e) - _missing_what = (e["opts"].get("have") - or e["opts"].get("prog") or target) - elif e["kind"] == "book": - _u_now = gather_user(target) - _st_now, _ = assess(_u_now, None, run_cmd=False) - _installed_now = _st_now in ("installed", "unvalidated") - _missing_what = "%s (%s)" % (target, _st_now) - if not _expect_ok: - pass - elif not _installed_now: + if e["kind"] in ("git", "tar") and not _git_installed(e): printWarning("%s finished, but %s is still missing -- NOT recording" " it as done, so the next run tries again." - % (target, _missing_what)) + % (target, e["opts"].get("have") + or e["opts"].get("prog") or target)) else: _stack_mark_done(path, e["_key"]) done += 1 - # A SECOND PASS IS PART OF THE JOB, not an afterthought. - # - # Optional features are decided at configure time, and a stack builds - # things in one order -- so anything whose optional dependency came later - # in the same run is installed without it. Nothing fails, so this is the - # only place the question gets asked. - if done and not getattr(args, "status", False): - printInfo("") - printInfo("Optional features are decided when a package configures, so" - " anything built") - printInfo("before a dependency it can use was built WITHOUT it. " - " Check, and fix:") - printInfo(" packagemanager late-deps") - printInfo(" packagemanager late-deps --rebuild --run") - - # A RUN THAT LEFT SOMETHING OUT DID NOT FINISH. "all N entries done" with - # a skipped entry is exactly the sentence that makes a missing package - # invisible for a month. - if notinbook and getattr(args, "skip_unknown", False): - printWarning("\nstack %s: %d entries done, %d SKIPPED (not in the book): %s" - % (args.name, done, len(notinbook), ", ".join(notinbook))) - printInfo(" the stack is not complete. Why each one is missing:" - + "".join("\n blfs debug %s" % t for t in notinbook)) - sys.exit(1) printSuccess("\nstack %s: all %d entries done." % (args.name, done)) @@ -12265,18 +9514,6 @@ def cmd_reclaim(args): if os.geteuid() != 0: printError("reclaim changes ownership; run it as root.") return 1 - if getattr(args, "all", False): - # ONE ACCIDENT, EVERY PACKAGE. A tree-wide chown -- lfs-kernel's - # headers step gave p_linux-headers all of /usr/include -- is not a - # per-package problem, and naming the victims one at a time means - # finding them first. - args.packages = sorted(_pkg_dirs()) - printInfo("checking %d installed package(s) against their manifests" - % len(args.packages)) - elif not args.packages: - printError("name a package, or --all") - return 2 - rc = 0 for name in args.packages: acct = pkgusr_name(name) @@ -12284,11 +9521,6 @@ def cmd_reclaim(args): try: paths = [l.strip() for l in open(manifest) if l.strip()] except OSError as e: - # under --all a package with no manifest is not an error, it is - # a package that has not recorded one -- saying so 90 times - # buries the packages that DID lose files - if getattr(args, "all", False): - continue printError("%s: cannot read %s (%s)" % (acct, manifest, e)) rc = 1 continue @@ -12653,20 +9885,6 @@ commands, by what you are doing: fix-group-dir / fix-install repair permissions - a whole environment at once - stack <name> replay a stack file: packages, order, options - stack <name> --only <pkg> one entry of it - stack <name> --take-new take the shipped stack beside your edited one - bootstrap get a freshly booted system to where this works - env build-time variables the book expects exported - - features, not files - probe [<pkg>] run the stacks' expect= checks against what is - installed: is the FEATURE there, not the file - late-deps what was built before a dependency it can use - late-deps --rebuild --run rebuild those, in dependency order - late-deps <pkg> --why the edges it used, and which names resolved - settings and housekeeping config show or change settings paths where everything lives @@ -12674,8 +9892,6 @@ commands, by what you are doing: errors what went wrong in the last run plan what a run would do, before doing it reload-pkg-list refresh the cached package list - reclaim <pkg> | --all give a package back files another took from it - which-package <path> which package installed it blfs ... pass a command straight to blfs LFS base packages are the `lfs` tool's job; BLFS packages are handled here. @@ -12897,38 +10113,13 @@ def _file_md5(path): return None -def _in_chroot(): - """Is there a host at all? - - The whole point of the staleness warning is "the copy outside is newer - than the copy in here", and outside stops existing the moment the system - boots on its own. /proc/1/root is pid 1's root: in a chroot it is not - this root. - """ - try: - return os.stat("/proc/1/root/.").st_ino != os.stat("/").st_ino \ - or os.stat("/proc/1/root/.").st_dev != os.stat("/").st_dev - except OSError: - return False - - def warn_if_tools_are_stale(): - """Tell the user when the CHROOT is running older tools than the host. + """Tell the user when the chroot is running older tools than the host. The tools are copied into the tree; edit them outside and the chroot keeps running the old copy until they are copied again. That is a genuinely confusing failure mode -- behaviour that was fixed "yesterday" reappears -- - so say so plainly. - - ON A BOOTED SYSTEM THERE IS NO HOST. The stamp then records what was - copied in during the build, so every later `make install` on the machine - itself made all seven tools "differ" and this printed, before every - command, an instruction to run sync-tools on a host that no longer - exists. `make install` writes the stamp now; this says nothing unless - there is an outside to be behind. - """ - if not _in_chroot(): - return + so say so plainly.""" stamp = "/usr/share/lfs/tool-stamps.json" if not os.path.isfile(stamp): return @@ -13072,13 +10263,6 @@ def main(): p.add_argument("-R", "--dependents", action="store_true", help="also update everything that (recursively) depends on the " "named package(s) -- rebuild reverse deps too") - p.add_argument("-v", "--verbose", action="store_true", - help="show every requirement that differs, not the first 12") - p.add_argument("--stack", metavar="NAME", - help="only the packages of that stack (media, sway, " - "network, services, vision) -- 'what in the media " - "chain has the book moved past', without typing the " - "names. Combine with --select to pick from them.") p.add_argument("--select", action="store_true", help="interactively pick which packages of the plan to update " "(all selected by default; combine with --run to update the " @@ -13182,21 +10366,12 @@ def main(): p.add_argument("--refresh", action="store_true", help="drop scripts from an older blfs so the current " "questions can be seen") - p.add_argument("--system", action="store_true", - help="this system's own facts -- main user, the three " - "prefixes, the Shared directory -- as PKGUSR_SYS_* " - "in /etc/profile.d/pkgusr-system.sh, so a script can " - "ask instead of guessing") - p.add_argument("--run", action="store_true", - help="with --system: write the file (default: print it)") p.set_defaults(func=cmd_env) p = sub.add_parser( "stack", help="replay a stack file: one command, one whole environment") - p.add_argument("name", nargs="?", - help="stack name or path (e.g. sway, ./sway.stack); " - "without one, list the stacks there are") + p.add_argument("name", help="stack name or path (e.g. sway, ./sway.stack)") p.add_argument("--run", action="store_true", help="actually build (default: print the plan)") p.add_argument("--yes", "-y", action="store_true", @@ -13213,9 +10388,6 @@ def main(): p.add_argument("--ignore-prereqs", action="store_true", help="start entries even when a build tool or variable " "their scripts need is missing") - p.add_argument("--skip-unknown", action="store_true", - help="run the entries the book does have, and report the " - "others (exits non-zero: the stack is NOT complete)") p.add_argument("--only", action="append", metavar="ENTRY", help="run just these entries (repeatable or comma-separated); " "they are rebuilt even if the cache says done") @@ -13230,12 +10402,7 @@ def main(): p = sub.add_parser("reclaim", help="give a package back files another package took " "from it (ownership only; nothing is rebuilt)") - p.add_argument("packages", nargs="*") - # ONE ACCIDENT, EVERY PACKAGE. A tree-wide chown (lfs-kernel's headers - # step gave p_linux-headers all of /usr/include) is not a per-package - # problem, and naming the victims one at a time means finding them first. - p.add_argument("--all", action="store_true", - help="every installed package, not a named list") + p.add_argument("packages", nargs="+") p.add_argument("--run", action="store_true", help="actually change ownership (default: dry run)") p.set_defaults(func=cmd_reclaim) @@ -13287,31 +10454,6 @@ def main(): "(configs and source tree kept)") p.set_defaults(func=cmd_script) - p = sub.add_parser("probe", help="run the stacks' expect= feature checks " - "against what is installed") - p.add_argument("package", nargs="*", help="only these (default: all)") - p.set_defaults(func=cmd_probe) - - p = sub.add_parser("late-deps", help="packages built BEFORE a dependency " - "they can use (so its support was compiled out)") - p.add_argument("package", nargs="*", help="only these (default: all)") - p.add_argument("--kind", help="which dependency kinds to check " - "(default recommended,optional; add required to include it)") - p.add_argument("--verbose", action="store_true", - help="also list dependency names that could not be matched") - p.add_argument("--include-docs", action="store_true", - help="include dependencies that only affect documentation " - "or are build tools (graphviz, docutils, git ...)") - p.add_argument("--why", action="store_true", - help="show the dependency edges actually used for the named " - "package(s), and which names did not resolve") - p.add_argument("--rebuild", action="store_true", - help="print the rebuild plan in dependency order; with " - "--run, carry it out") - p.add_argument("--run", action="store_true", - help="with --rebuild: actually rebuild") - p.set_defaults(func=cmd_late_deps) - p = sub.add_parser("errors", help="show which packages failed in the last " "install/update run (with log locations)") p.add_argument("--all", action="store_true", @@ -13374,15 +10516,6 @@ def main(): formatter_class=argparse.RawDescriptionHelpFormatter) q.add_argument("name") q.add_argument("--groups", help="also add these groups, e.g. audio,video") - q.add_argument("--shared-slim", dest="shared_slim", action="store_true", - help="share the DESKTOP only: the account joins the group " - "on your XDG runtime directory, so it can draw -- and " - "gets no audio and no camera. For an account that " - "only ever runs a browser window.") - q.add_argument("--shared-full", dest="shared_full", action="store_true", - help="the desktop AND audio and video: --shared-slim plus " - "the audio and video groups, for anything that plays " - "sound or uses a camera.") q.add_argument("--shared", action="store_true", help="a SHARED user: joins the group on your XDG runtime " "directory, so it can reach the display, the session " @@ -13404,15 +10537,6 @@ def main(): formatter_class=argparse.RawDescriptionHelpFormatter) q.add_argument("name") q.add_argument("--groups", help="also add these groups, e.g. audio,video") - q.add_argument("--shared-slim", dest="shared_slim", action="store_true", - help="share the DESKTOP only: the account joins the group " - "on your XDG runtime directory, so it can draw -- and " - "gets no audio and no camera. For an account that " - "only ever runs a browser window.") - q.add_argument("--shared-full", dest="shared_full", action="store_true", - help="the desktop AND audio and video: --shared-slim plus " - "the audio and video groups, for anything that plays " - "sound or uses a camera.") q.add_argument("--shared", action="store_true", help="join the group on your XDG runtime directory") q.add_argument("--share-dir", action="store_true", @@ -13422,39 +10546,11 @@ def main(): q.add_argument("--desktop", action="store_true", help="a menu entry") q.add_argument("--app", help="program to launch (default: <name>)") q.set_defaults(func=cmd_user_setup) - q = usub.add_parser("delete", help="remove an application account safely: its " - "home archived first, launchers, pam line, account, group") + q = usub.add_parser("delete", help="delete an shared user + group + home") q.add_argument("name") - q.add_argument("--run", action="store_true", help="actually remove (default dry-run)") + q.add_argument("--run", action="store_true", help="actually delete (default dry-run)") q.add_argument("--yes", action="store_true", help="skip the confirmation prompt") - q.add_argument("--keep-home", action="store_true", help="leave the home directory") - q.add_argument("--no-archive", action="store_true", help="do not archive the home first") q.set_defaults(func=cmd_user_delete) - q = usub.add_parser("purge", help="undo the whole application-account setup: " - "every account, then the session group, the profile " - "fragments and the pam wiring they shared") - q.add_argument("--run", action="store_true", help="actually remove (default dry-run)") - q.add_argument("--yes", action="store_true", help="skip the confirmation prompt") - q.add_argument("--keep-home", action="store_true", help="leave the home directories") - q.add_argument("--no-archive", action="store_true", help="do not archive the homes") - q.add_argument("--accounts", help="only these accounts (comma separated), " - "instead of every one in app-users.conf") - q.add_argument("--conf", help=f"read the account list here (default {APP_USERS_CONF})") - q.set_defaults(func=cmd_user_purge) - q = usub.add_parser("launcher", help="a launcher + menu entry for an existing " - "application account (one account can have several)") - q.add_argument("account") - q.add_argument("name", help="the launcher's name: ~/bin/<name>, <name>.desktop") - q.add_argument("--app", required=True, help="command line; {} = what you type") - q.add_argument("--icon", default="", help="URL, simpleicons.org slug, or - for none " - "(default: the launcher name as the slug)") - q.add_argument("--no-desktop", action="store_true", help="no menu entry") - q.set_defaults(func=cmd_user_launcher) - q = usub.add_parser("remove-launcher", help="remove a launcher, its menu entry and " - "icon (only ones these tools wrote)") - q.add_argument("name") - q.add_argument("--run", action="store_true") - q.set_defaults(func=cmd_user_remove_launcher) q = usub.add_parser("allow", help="add supplementary group(s) to a user " "(e.g. audio video)") q.add_argument("user") diff --git a/stacks/kernel-sway.sh b/stacks/kernel-sway.sh index 25c30d9..2c61ca4 100755 --- a/stacks/kernel-sway.sh +++ b/stacks/kernel-sway.sh @@ -24,29 +24,6 @@ need CONFIG_SND "sound core" need CONFIG_SECCOMP "webkit's sandbox" need CONFIG_USER_NS "webkit's sandbox (bubblewrap)" need CONFIG_TMPFS "XDG_RUNTIME_DIR lives on tmpfs" -# THE SOFTWARE RENDERER NEEDS ITS OWN KERNEL OPTION. The book's mesa page: -# "For llvmpipe or softpipe, CONFIG_DRM_VGEM is required or X clients will -# fail". Without it the software fallback that exists to save you when the -# hardware path fails cannot be used either -- and the symptom is the same -# "failed to create dri2 screen". A note, not a failure: a machine that -# never falls back does not need it. -# `need` fails the stack; this one only reports, so it reads the config the -# same way rather than inventing a second reader. -want() { # want <config-option> <why> - local cfg="" opt="$1" why="$2" - for c in /proc/config.gz "/boot/config-$(uname -r)" /boot/config; do - [ -r "$c" ] && cfg="$c" && break - done - [ -n "$cfg" ] || return 0 - if case "$cfg" in - *.gz) zgrep -qE "^$opt=(y|m)" "$cfg" ;; - *) grep -qE "^$opt=(y|m)" "$cfg" ;; - esac - then echo "# $opt is set -- $why" - else echo "-> $opt is NOT set -- $why" - fi -} -want CONFIG_DRM_VGEM "the book: llvmpipe/softpipe need it or X clients fail" # runtime, not config: the render node must exist [ -e /dev/dri/renderD128 ] || { echo "!! no /dev/dri/renderD* render node"; fail=1; } [ "$fail" = 0 ] && echo "# kernel: everything sway + webkit need is present" diff --git a/stacks/machine.conf b/stacks/machine.conf index 6888b74..91052e4 100644 --- a/stacks/machine.conf +++ b/stacks/machine.conf @@ -2,103 +2,17 @@ # scripts' `meson setup` lines as -Dkey=value the moment the script is born. # The stack file stays portable; this file changes per computer. -# HOW MANY JOBS, ONCE. make reads MAKEFLAGS, cmake reads -# CMAKE_BUILD_PARALLEL_LEVEL, cargo reads CARGO_BUILD_JOBS -- and ninja reads -# nothing at all, defaulting to nproc+2, which is why the number has to be -# patched into its command line. [all] covers every package here; a named -# section below overrides it (webkitgtk keeps 8 for RAM reasons). -# -# [all] -# jobs = 13 -# -# `jobs` is patched into `ninja`, `./x.py` and `cargo build|install` lines -# when the script is generated -- rust's page says plainly that only the -# COMBINATION of --jobs on x.py and CARGO_BUILD_JOBS limits it. A package -# built before you set this needs --regenerate to pick it up. -# -# ...and for make/cmake/cargo, one line in /etc/pkgusr/build.env: -# MAKEFLAGS=-j13 -# (lfs-helper derives the cmake and cargo variables from it, or from --jobs.) - -# A HARDWARE-ONLY MESA HAS NOTHING TO FALL BACK TO. -# -# `gallium-drivers = radeonsi` alone means that when the hardware path -# cannot be set up for some EGL platform, there is no software renderer -# behind it and the app gets -# libEGL warning: egl: failed to create dri2 screen -# and then nothing. The book says it twice, in its own words: for -# gallium-drivers "You should probably add the softpipe or llvmpipe option -# as a backup driver", and for vulkan-drivers "You should probably add the -# swrast option as a backup driver unless you don't need Vulkan at all". -# -# llvmpipe (not softpipe) because llvm is enabled below: it is the faster of -# the two software rasterisers and it is built from the same LLVM this -# machine already has. -# READ ONE PACKAGE FROM A DIFFERENT BOOK. -# -# The systemd book's linux-pam page lists systemd as REQUIRED; the 12.4 -# SysV book's page has no Required section at all -- it wants only Shadow. -# Same package, two books, two dependency graphs, and this machine is SysV. -# `book =` points one package at another book, for everything: versions, -# dependencies and the generated script. -# -# Import the book once, then uncomment: -# blfs import /path/to/BLFS-BOOK-12_4-nochunks.html -# -# [somepackage] -# book = /usr/share/blfs/books/BLFS-BOOK-12_4-nochunks.html -# -# AND linux-pam IS NOT THE CASE FOR IT, on the evidence. Both pages were -# compared command by command: -# 13.1: Linux-PAM-1.7.2; systemd is in the REQUIRED list and in NO command -# 12.4: Linux-PAM-1.7.1; its commands include `rm -rf /usr/lib/systemd` -# So the newer page needs nothing from systemd to build -- the requirement -# is metadata -- and since 1.14.234 `install` drops avoided packages from a -# plan anyway. Pointing linux-pam at 12.4 would cost a version (1.7.2 -> -# 1.7.1) and buy nothing. Use `book =` where the COMMANDS differ, not -# where only the dependency list does. -# -# NOT a merge of two pages: taking the dependency list from one book and the -# commands from another gives a script neither book has tested -- the -# versions differ (1.7.1 vs 1.7.2), the switches differ, the doc tarball -# differs. A whole page from the older book is something upstream shipped -# and someone ran. - -# SHADOW: --disable-logind, because there is no libsystemd here. -# -# shadow 4.20 auto-detects logind and the book's page (written for a systemd -# host, where libsystemd is simply present) passes no switch either way, so -# configure stops: -# checking for sd_session_get_remote_host in -lsystemd... no -# configure: error: libsystemd not found. If logind integration is not -# required, disable it using the --disable-logind option. -# configure names the flag itself. This is the case `book =` exists for -- -# a page whose COMMANDS do not suit this machine -- but the 12.4 page would -# also mean shadow 4.18.0, and one switch is cheaper than four versions -# back. -[shadow] -configure_args = --disable-logind - [mesa] platforms = x11,wayland -gallium-drivers = radeonsi,llvmpipe -vulkan-drivers = amd,swrast +gallium-drivers = radeonsi +vulkan-drivers = amd video-codecs = all -# glvnd: the GL dispatcher (libglvnd, sway.stack, before mesa). Enabled so -# pkg-config `opengl` exists (libdecor's demo, some games and toolkits ask -# for it by that name) and a second GL vendor could be added. Changing it -# means rebuilding libglvnd and mesa together -- see the libglvnd entry. -glvnd = enabled +glvnd = disabled libunwind = disabled llvm = enabled valgrind = disabled egl = enabled opengl = true -# gbm: Mesa's default is auto, which on Linux means built -- and wlroots, -# sway and every Wayland compositor need it. Named so that a Mesa build -# which cannot provide it FAILS at configure, instead of finishing without -# it and leaving sway to fail at start. -gbm = enabled # Per-package build options. Same mechanism as [mesa]: every key is merged # into that package's generated `meson setup` line as -Dkey=value. @@ -109,70 +23,6 @@ gbm = enabled [iptables] configure_args = --enable-nftables -# GST_DEBUG=<level> ONLY WORKS WITH THIS. The book builds the core with -# -D gst_debug=false, which compiles the whole debug-log system out; this -# replaces that switch (the script's -D gst_debug=false becomes =true). -# Cost: a slightly larger library and one integer compare per log site while -# logging is off -- upstream's own default, what distributions ship. -# `gstreamer:gst_debug` is the same switch for a build that bundles the core -# as a meson SUBPROJECT; BLFS builds the core as its own package, so only -# this one applies (the subproject key would be dropped as unknown). -# -# THE PLUGINS MUST BE REBUILT AFTERWARDS. gstconfig.h carries -# GST_DISABLE_GST_DEBUG from the core's build, and each plugin package -# compiled against it has its own log calls compiled out. -[gstreamer10] -gst_debug = true - -# --------------------------------------------------------------------------- -# TURNING AUTODETECT INTO A REQUIREMENT. -# -# gst-plugins-bad decides at configure time whether to build each plugin, -# from what it can find. Built before libnice existed, it produced -# libgstwebrtc-1.0.so (the library, always built) and no webrtcbin element -- -# and said nothing, because nothing went wrong. `-Dwebrtc=enabled` makes the -# same situation a build FAILURE, which is the outcome worth having. -# -# COMMENTED OUT because these are option names, and an option name is -# machine-checkable rather than memorable. Confirm them against the source -# you actually have, then uncomment: -# -# grep -n '^option(' /usr/src/pkgusr/p_gst10-plugins-bad/src/*/meson_options.txt -# meson configure /usr/src/pkgusr/p_gst10-plugins-bad/src/*/build | less -# -# A wrong name is not dangerous -- meson stops with -# ERROR: Unknown option: "..." -# before anything is built -- but it costs you a run. -# -# # x264enc EXISTS ONLY WITH THIS. The book: "Without this switch, plugins -# with dependencies on (A)GPL licensed libraries are not built" -- and -# x264enc is the H.264 encoder every camera site wants for publishing. -[gst10-plugins-ugly] -gpl = enabled - -# alsa IS A -BASE OPTION, NOT A -BAD ONE. It reads perfectly well under -# [gst10-plugins-bad] and meson fails the whole build on it ("Unknown -# option: alsa"); the ALSA source/sink plugin lives in gst-plugins-base. -# (packagemanager checks this against the package's own meson.options now -# and drops a key the package does not have, with the near misses.) -[gst10-plugins-base] -alsa = enabled - -# CHECKED against gst-plugins-bad's own meson.options: webrtc, dtls, srtp, -# sctp and openh264 are all real options of THIS package. Enabled rather -# than left on auto, so a missing dependency fails the build instead of -# silently dropping the plugin. -[gst10-plugins-bad] -webrtc = enabled -dtls = enabled -srtp = enabled -sctp = enabled -openh264 = enabled -# -# [gst10-plugins-good] -# v4l2 = enabled -# vpx = enabled - # libdisplay-info builds v4l-utils as a SUBPROJECT for its edid-decode # comparison tests: it clones the repo and installs libdvbv5, # libv4lconvert and a dozen locale files -- every one of them already owned @@ -181,92 +31,3 @@ openh264 = enabled # downloads. nodownload blocks that. [libdisplay-info] wrap_mode = nodownload - -# WEBKITGTK: THE SWITCHES THE BOOK TURNS OFF. -# -# Each of these is off in the book's own cmake line, and each is invisible -# when missing -- no error, just a feature that does not work: -# -# USE_WOFF2=OFF a .woff2 icon font renders as a blank square -# ENABLE_WEB_RTC=OFF no camera or screen share in the browser -# ENABLE_MEDIA_STREAM the getUserMedia half of the same thing -# USE_GSTREAMER_WEBRTC which backend does the WebRTC -# -# woff2 must be INSTALLED first (sway.stack builds it just before -# webkitgtk), or the switch turns on support for a library that is not there. -# -# jobs = 8 rather than [all]'s 13: webkitgtk's link step is the biggest -# memory consumer in the stack, and the OOM killer takes the compiler. -# USE_LIBRICE=ON, and media.stack builds librice just before webkitgtk. -# -# webkitgtk 2.52 can take its ICE from librice instead of libnice, and with -# the GStreamer WebRTC backend on it defaults to librice: -# Could NOT find Rice (missing: Io Proto) -# librice-{io,proto} is needed for USE_LIBRICE. -# The BLFS page never hits this -- it does not enable USE_GSTREAMER_WEBRTC, -# so librice is not in the book at all (checked: the 13.1 book does not -# mention it). libnice IS installed here and is what the rest of the -# WebRTC chain already uses, so this says "keep using it". -# -# To go the other way instead -- libnice, which the rest of the chain uses -# and which is already installed -- set this to OFF and nothing needs -# building. -# POPPLER WITHOUT THE SIGNATURE STACK. -# -# poppler's cmake calls these "soft mandatory": absent, they stop configure -# and name the switch themselves -- -# Could not find the 3.98 version of NSS3. If you're not interested in -# the features it provides set the cmake ENABLE_NSS3 option to OFF -# nss, boost and gpgmepp are all in avoid.stack AS POPPLER'S CHILDREN (they -# were avoided because poppler was), so each would fail the same way in -# turn. They serve PDF signature verification and the splash backend; -# zathura's poppler-glib backend needs none of it. -# -# OBSERVED: ENABLE_NSS3 -- the failure above. -# ANTICIPATED: ENABLE_GPGME and ENABLE_BOOST, from the same avoid list and -# the same "soft mandatory" mechanism. If a later poppler stops on one of -# these with a different spelling, its message will name the real one. -[poppler] -# ENABLE_GLIB=ON: the whole reason poppler is here is zathura's -# poppler-glib backend -- -# Run-time dependency poppler-glib found: NO -# -- and cmake autodetects that frontend. The thing a chain exists for -# should be asked for, not inferred (the gst-plugins-ugly gpl rule again). -ENABLE_GLIB = ON -ENABLE_NSS3 = OFF -ENABLE_GPGME = OFF -ENABLE_BOOST = OFF -ENABLE_QT6 = OFF -# -# FOUR OF THEM, FOUND ONE CONFIGURE AT A TIME. NSS3, then GPGME and BOOST -# (guessed from the avoid list), then QT6 -- each a full cmake run apart. -# The list is not a guessing game: poppler's own CMakeLists names every one -# of them, and on this machine that file is right there -- -# -# grep -n find_soft_mandatory_package \ -# /usr/src/pkgusr/p_poppler/src/poppler-*/CMakeLists.txt -# -# Read that once when poppler moves version, instead of discovering them -# one failed configure at a time. The book's page passes ENABLE_QT5=OFF -# and nothing else, because a BLFS system is expected to have qt6. - -[webkitgtk] -# ENABLE_JOURNALD_LOG=OFF, or the build stops thousands of files in: -# wtf/Assertions.h:64: fatal error: systemd/sd-journal.h: No such file -# webkit autodetects journald and there is no systemd here. This was in the -# machine's own conf and NOT in the section shipped at 1.14.203 -- my -# omission, and it cost a nine-thousand-file build. -ENABLE_JOURNALD_LOG = OFF -# and the sandbox: bubblewrap is installed, but it also wants xdg-dbus-proxy -# and a working /proc setup; OFF is what this machine had and what it built -# with before. Turn it ON deliberately, not by inheriting a default. -ENABLE_BUBBLEWRAP_SANDBOX = OFF -USE_LIBRICE = ON -USE_WOFF2 = ON -ENABLE_WEB_RTC = ON -ENABLE_MEDIA_STREAM = ON -USE_GSTREAMER_WEBRTC = ON -jobs = 8 - -[wlroots] -werror = false diff --git a/stacks/network.stack b/stacks/network.stack index 1c327e4..7666a55 100644 --- a/stacks/network.stack +++ b/stacks/network.stack @@ -1,11 +1,4 @@ # network.stack -- wired and wireless networking, for a SysV system. -# stack-version: 1.14.292 -# -# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack -# here: the exclusions are a property of the machine, not of one stack. -# Without it a dependency plan walks into systemd, qt6, cups and texlive. -include avoid - # # The sway stack builds a desktop; nothing in it brings up a network. This is # what `wifi` (wpa_cli / wpa_supplicant / ifup) needs, plus a firewall. @@ -15,15 +8,6 @@ include avoid # # Every entry here is a BLFS package, checked against the book. -# THE SYSTEM ACCOUNTS THIS MACHINE NEEDS, DECLARED ONCE. -# -# A package user cannot create an account, so every book "Configuring" -# section that says `useradd` is refused and recorded -- correctly, and then -# somebody types it from the failure box hours later, from memory, with the -# uid it happened to suggest. accounts.conf is where they are written down -# instead, and this step creates them BEFORE the package that wants them. -cfg accounts - book libnl # netlink, what wpa_supplicant talks to the kernel with book wpa_supplicant # wpa_supplicant, wpa_cli, wpa_passphrase book wireless_tools # iwconfig/iwlist, if you want them @@ -37,43 +21,10 @@ book libmnl # minimal netlink library; libnftnl needs it # upstream moves -- `--check-refs` asks git remotes and cannot see a tar url. tar libnftnl url=https://www.netfilter.org/pub/libnftnl/libnftnl-1.3.2.tar.xz pkg=libnftnl-1.3.2.tar.xz have=/usr/lib/libnftnl.so build="./configure --prefix=/usr --disable-static; make" install="make install" -# THE BOOK DISABLES NFTABLES; THIS MACHINE'S KERNEL HAS NOTHING ELSE. -# -# BLFS configures iptables with --disable-nftables, so you get the legacy -# binaries only -- and a kernel built without CONFIG_IP_NF_IPTABLES answers -# modprobe: FATAL: Module ip_tables not found -# iptables: can't initialize iptables table `filter' -# for every rule. `[iptables] configure_args = --enable-nftables` in -# machine.conf fixes it (the merge drops the book's --disable-nftables when -# it adds the opposite), and the probe makes sure: with nftables support -# `iptables --version` says (nf_tables), without it (legacy). -book iptables expect="test -x /usr/sbin/xtables-nft-multi" - -# ...AND THEN SWITCH TO IT. `--enable-nftables` builds both backends and -# upstream's install leaves the plain names on the legacy one, so -# `iptables --version` still says (legacy) after a build whose log says -# "nftables support: yes". This step points iptables/ip6tables and their -# save/restore at xtables-nft-multi, and the probe checks the result rather -# than the build. -cfg iptables-nft # nft backend, via machine.conf (see below) +book iptables # nft backend, via machine.conf (see below) book net-tools # ifconfig, route, netstat -# THE ADDRESS HAS TO COME FROM SOMEWHERE. `lfs config network dhcp` writes -# no static interface files on purpose and says a DHCP client comes from BLFS -# afterwards -- and this stack did not have one, so a joined wireless link sat -# there with no address and `ifup` had no service to hand off to. -book dhcpcd # the DHCP client; SERVICE=dhcpcd in ifconfig.* - -# openssh lives HERE, not in services.stack and not anywhere near sway: it -# is networking. Its bootscript comes from blfs-bootscripts -# (install-sshd), which the services stack installs afterwards -- so sshd is -# startable once both stacks have run, and `expect` below checks only what -# this entry can be responsible for. -book openssh expect="test -x /usr/sbin/sshd" - -check net-wifi # what is still missing to bring a link up - # --------------------------------------------------------------------------- # iptables-nft # diff --git a/stacks/services.stack b/stacks/services.stack index e153e92..c2ba73c 100644 --- a/stacks/services.stack +++ b/stacks/services.stack @@ -1,11 +1,4 @@ # services.stack -- the boot-time services a SysV desktop needs. -# stack-version: 1.14.292 -# -# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack -# here: the exclusions are a property of the machine, not of one stack. -# Without it a dependency plan walks into systemd, qt6, cups and texlive. -include avoid - # # /etc/init.d after an LFS build holds only LFS's own scripts; nothing starts # dbus, alsa, bluetooth or a cron daemon. Those scripts live in @@ -41,85 +34,12 @@ git bash-completion url=https://github.com/scop/bash-completion ref=2.18.0 \ build="autoreconf -i && ./configure --prefix=/usr --sysconfdir=/etc" \ install="make install" -# coloured ls, the BLFS /etc/bashrc, `su <tab>` over the package users, and -# $LANG. The Bash Shell Startup Files page is CONFIG, not a package, so no -# package stack ever ran it. -# -# expect= is what this step keeps true, re-checked even when the entry is -# cached (see shadow-pam). Without it a machine that ran this step before -# the locale was part of it stays at `done` for ever -- and a terminal keeps -# saying `'C' is not a UTF-8 locale`, with nothing to say why. -cfg shell-env \ - expect='test -f /etc/bashrc && test -f /etc/locale.conf && { test ! -d /etc/profile.d || test -f /etc/profile.d/i18n.sh; }' +# coloured ls, the BLFS /etc/bashrc, and `su <tab>` over the package users. +# The Bash Shell Startup Files page is CONFIG, not a package, so no package +# stack ever ran it. +cfg shell-env book acpid # power/lid/button events -- what a laptop needs - -# ONE GROUP CARRIES THE PRIVILEGE. halt/poweroff/reboot must become root, so -# they are root's with a group deciding who may run them, and the kernel's -# power interface is re-grouped on every boot (/sys does not survive one). -# Set POWER_GROUP to override the <collector_prefix>_power default. -# linux-pam, BEFORE the accounts that need it. -# -# `su - u_claude` without a password works through pam_exec.so reading -# /etc/pam.d/su_u_user -- which needs Linux-PAM, and LFS installs neither pam -# nor a pam-aware shadow. Without it the accounts, groups, shared -# directories and launchers all still work; only PASSWORDLESS su does not. -# -# This entry installs the LIBRARY, which on its own changes nothing: until -# shadow is rebuilt against it, nothing reads /etc/pam.d at all. That -# rebuild is deliberately NOT here: -# -# packagemanager update shadow --reinstall --regenerate --run --yes -# -# It is the one step in this whole system that can leave you unable to `su` -# or log in, so it wants a root shell open, the book's Shadow page read, and -# a moment when nothing else is running. `cfg app-users` prints the same -# thing when it finds no /etc/pam.d. -# -# It was in avoid.stack until 1.14.235 -- as systemd's dependency, which is -# not why it is here now. With `install`/`update` honouring the avoid list -# (also 1.14.234), asking for it no longer drags systemd in behind it. -book linux-pam - -# ...AND THE REBUILD THAT MAKES IT MEAN ANYTHING, immediately after it. -# -# Linux-PAM above installs a library and changes nothing: until shadow is -# rebuilt against it, nothing reads /etc/pam.d, so `su - u_claude` keeps -# asking for a password however right su_u_user is. The book's Linux-PAM -# page ends with "You should now reinstall the Shadow package" -- so this -# step does, and then puts back the setuid bits that the rebuild strips -# (the BLFS Shadow page never sets them; the LFS one does). -# -# It is idempotent and checks its own effect: pam linkage with ldd, the -# modes by reading them back. With no Linux-PAM installed it says so and -# does nothing. -# -# expect= is what this step keeps true, and it is re-checked on every run even -# when the entry is cached: shadow's configure phase or a rebuild can undo it -# later, and a cached "done" then hides it (1.14.275). -cfg shadow-pam \ - expect='test -u /usr/bin/su && test "$(stat -c %U /usr/bin/su)" = root && { test ! -d /etc/pam.d || { test -f /etc/pam.d/login && test -f /etc/pam.d/passwd && test -f /etc/pam.d/system-auth; }; }' - -# THE APPLICATION ACCOUNTS, from a file rather than a command line. -# -# Each line of app-users.conf becomes a `packagemanager user setup` call, and -# this step also does the two things no per-account command can: merge -# /etc/pam.d/su_u_user (one file listing every account that may be entered -# without a password) and write /etc/profile.d/shared-dirs.sh (one file -# exporting SHARED_<NAME> for each ~/Shared/<Name>). -# -# Needs a main user: `packagemanager config --main-user <your-login>`. -# Nothing of yours is edited -- no .bashrc, no .bash_profile -- and the PAM -# file is backed up before it is touched. -# -# Re-checked on every run (see shadow-pam): the accounts IT MANAGES (the -# names in app-users.conf -- not every u_*: a hand-made u_test is not its -# business) are not directly in /home, su calls su_u_user, root owns it, and -# it checks the caller, not the target. -cfg app-users \ - expect='test -x /usr/bin/xdg-runtime-share && ! awk -F"|" "\$1 !~ /^[[:space:]]*(#|name[[:space:]]|\$)/ { gsub(/[[:space:]]/, \"\", \$1); print \"u_\" \$1 }" /etc/pkgusr/stacks/app-users.conf | while read -r u; do getent passwd "$u" | cut -d: -f6; done | grep -qE "^/home/[^/]+\$" && { test ! -f /etc/pam.d/su || grep -qE "^[[:space:]]*auth[[:space:]]+sufficient[[:space:]]+pam_exec[.]so.*su_u_user" /etc/pam.d/su; } && { test ! -e /etc/pam.d/su_u_user || { test "$(stat -c %U:%a /etc/pam.d/su_u_user)" = root:755 && ! grep -qF -e "=~ \"\$PAM_USER\"" /etc/pam.d/su_u_user; }; }' - -cfg power book fcron # the cron daemon; make-ca's weekly update needs one # blfs-bootscripts: 20250225 is the FINAL release (BLFS has stopped shipping @@ -135,30 +55,9 @@ book fcron # the cron daemon; make-ca's weekly update needs # iptables restores the firewall ruleset at boot # random seeds /dev/urandom from the saved pool # service-wpa the `wpa` helper `ifup` uses for a wireless interface -# install-service-dhcpcd WAS MISSING. The dhcpcd PACKAGE installs no service -# script; /lib/services/dhcpcd comes from here, and `SERVICE=dhcpcd` (or a -# wpa hand-off to it) silently has nothing to run without it. Adding a -# package to the network stack was only half the job. tar blfs-bootscripts \ url=https://anduin.linuxfromscratch.org/BLFS/blfs-bootscripts/blfs-bootscripts-20250225.tar.xz \ pkg=blfs-bootscripts-20250225.tar.xz \ have=/etc/rc.d/init.d/dbus \ build="true" \ - install="make install-dbus install-alsa install-bluetooth install-acpid install-fcron install-iptables install-random install-service-wpa install-service-dhcpcd install-sshd" - -# ...AND THE SAME LIST AS A MAPPING, so it cannot go stale again. -# -# The install= line above is a list written on one day. Add a package to a -# stack afterwards and its bootscript is simply not installed: the daemon is -# there, nothing starts it, and nothing says so. `cfg bootscripts` reads -# bootscripts.conf -- package | proof | targets | creates -- and installs the -# targets whose package IS here and whose script is NOT, then checks the disk -# rather than make's exit status. It also writes the one thing a bootscript -# cannot supply: /etc/sysconfig/ifconfig.<iface> for a WIRED interface, which -# is what actually makes dhcpcd run at boot (there is no init.d/dhcpcd in -# BLFS at all). Wireless is left to `check net-wifi`: that file needs an SSID. -# -# expect= is re-checked even when the entry is cached, so a package installed -# later is noticed on the next run. -cfg bootscripts \ - expect='! command -v dhcpcd >/dev/null 2>&1 || { { test -e /lib/services/dhcpcd || test -e /usr/lib/services/dhcpcd; } && ls /etc/sysconfig/ifconfig.* >/dev/null 2>&1; }' + install="make install-dbus install-alsa install-bluetooth install-acpid install-fcron install-iptables install-random install-service-wpa" diff --git a/stacks/shell-env.sh b/stacks/shell-env.sh index 1d736be..c254b33 100755 --- a/stacks/shell-env.sh +++ b/stacks/shell-env.sh @@ -1,6 +1,6 @@ #!/bin/bash -# shell-env.sh -- interactive shell environment: the system locale, coloured -# ls, bash completion, and username completion for su (root step, idempotent). +# shell-env.sh -- interactive shell environment: coloured ls, bash +# completion, and username completion for su (root step, idempotent). # # BLFS's "The Bash Shell Startup Files" page writes /etc/profile and # /etc/bashrc, and /etc/bashrc is what makes ls colourful -- it evals @@ -11,23 +11,10 @@ # is a `tar` entry in the stack beside this script. set -u -# ROOT WROTE IT, SO ROOT OWNS IT. -# -# On a package-user system a file's owner records which package installed it, -# and anything written while a build window is open is attributed to that -# package user -- /etc/profile.d/pkgusr-system.sh owned by p_gstreamer10, -# 00-path-functions.sh by p_gettext, /etc/profile.d itself by p_libnice. -# For a file a PACKAGE installed that is correct and useful. For a config -# file root writes it is wrong twice over: it says a package owns something -# no package installed, and every login shell sources it, so whoever can -# write it can run code as everyone who logs in. Re-asserted on every run. -own_root() { [ -e "$1" ] && chown root:root "$1" 2>/dev/null; return 0; } - # 1. /etc/dircolors -- the database ls colours come from if [ ! -f /etc/dircolors ]; then if command -v dircolors >/dev/null 2>&1; then dircolors -p > /etc/dircolors - own_root /etc/dircolors echo "# wrote /etc/dircolors (from dircolors -p)" fi else @@ -82,7 +69,6 @@ complete -A user su sudo passwd chage groups id # End /etc/bashrc BASHRC chmod 644 "$_dest" -own_root "$_dest" echo "# wrote $_dest" # 3. make sure /etc/profile actually sources it, and profile.d @@ -99,141 +85,4 @@ if [ -f /etc/pkgusr/bashrc ] && ! grep -q '/etc/bashrc' /etc/pkgusr/bashrc; then echo "# package users now source /etc/bashrc too" fi - -# 5. THE SYSTEM LOCALE -- $LANG, and why a terminal says the locale is "C" -# -# warning: foot: 'C' is not a UTF-8 locale, falling back to 'C.UTF-8' -# -# Nothing is compiled wrong when that appears: LANG is simply not in the -# environment. LFS's own /etc/profile reads /etc/locale.conf, but BLFS's -# "The Bash Shell Startup Files" page REPLACES /etc/profile with one that -# reads /etc/profile.d/*.sh instead -- and its i18n.sh, the piece that -# exports LANG, is a separate block on that page that is easy to walk past. -# Take either route: this writes both files, and neither profile minds the -# other's. -# -# Without it every program falls back to the C locale: sorting, dates, -# translated messages, and anything that has to decide whether a byte is a -# character. -LOCALE_CONF="${SHELL_ENV_LOCALE_CONF:-/etc/locale.conf}" # overridable, for tests -I18N_SH="${PROFILE_D:-/etc/profile.d}/i18n.sh" -LFS_CONFIG_JSON="${LFS_STORE:-/usr/share/lfs}/config.json" -_norm() { printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | sed 's/utf-8/utf8/; s/-//g'; } -_have_locale() { - [ -n "$1" ] || return 1 - locale -a 2>/dev/null | while read -r l; do - [ "$(_norm "$l")" = "$(_norm "$1")" ] && echo yes - done | grep -q yes -} - -# WHAT THIS MACHINE ALREADY SAYS IT WANTS, in order. Nothing is guessed -# from the keyboard or the timezone: a wrong locale is worse than none. -conf_lang="$(sed -n 's/^[[:space:]]*LANG=//p' "$LOCALE_CONF" 2>/dev/null \ - | tr -d '"'"'"' ' | head -n1)" -asked="${LFS_LOCALE:-$conf_lang}" -[ -n "$asked" ] || asked="$(sed -n 's/.*"locale"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' \ - "$LFS_CONFIG_JSON" 2>/dev/null | head -n1)" -# A NAME IS NOT A LOCALE UNTIL glibc HAS BUILT IT. locale.conf may well name -# one nobody ever generated, and then LANG is set to something that does not -# exist -- which looks exactly like not setting it at all. -want="" -for _c in "$asked" en_US.UTF-8 C.UTF-8; do - if _have_locale "$_c"; then want="$_c"; break; fi -done -if [ -n "$asked" ] && [ "$(_norm "$asked")" != "$(_norm "${want:-}")" ]; then - echo "!! $asked is asked for but not installed (see: locale -a)" - echo "# make it as glibc's package user: localedef -i ${asked%%.*} -f UTF-8 $asked" - [ -n "$want" ] && echo "# using $want until then" -fi - -if [ -z "$want" ]; then - echo "!! no UTF-8 locale is installed -- \$LANG cannot be set to one" - echo "# glibc builds them; the archive is glibc's, so make one as its" - echo "# package user (p_glibc here) rather than as root:" - echo "# localedef -i en_US -f UTF-8 en_US.UTF-8" - echo "# then run this step again." -else - case "$(_norm "$want")" in - *utf8) ;; - *) echo "!! $want is not a UTF-8 locale -- a terminal will still complain" ;; - esac - if [ -f "$LOCALE_CONF" ] && [ "$(_norm "$conf_lang")" = "$(_norm "$want")" ]; then - echo "# $LOCALE_CONF already says LANG=$conf_lang" - elif [ -f "$LOCALE_CONF" ]; then - echo "# $LOCALE_CONF says LANG=${conf_lang:-nothing} -- left alone (it is yours)" - echo "# $want is what works here; change it there if you want it" - else - printf 'LANG=%s\n' "$want" > "$LOCALE_CONF" - chmod 644 "$LOCALE_CONF"; own_root "$LOCALE_CONF" - echo "# wrote $LOCALE_CONF (LANG=$want)" - fi - # i18n.sh: JUDGED BY WHAT IT PRODUCES, NOT BY WHO WROTE IT. - # - # BLFS's page ships an i18n.sh of its own, and it is usually already - # there -- `. /etc/locale.conf` and export what came out. "Exists and is - # not ours, left alone" would have been the end of it, and $LANG would - # still be empty: the file was never the problem, the missing - # /etc/locale.conf above was. So source it in an empty environment and - # look at the LANG that comes out. TERM is set to something that is not - # `linux`, because both the LFS and BLFS versions special-case the - # console with a hard-coded C.UTF-8. - _i18n_gives() { - env -i HOME=/root TERM=xterm bash --noprofile --norc -c \ - ". \"\$1\" >/dev/null 2>&1; printf '%s' \"\${LANG:-}\"" _ "$1" 2>/dev/null - } - # WHOSE FILE IS IT. Three writers put an i18n.sh here and all three can - # leave LANG unset, so all three are ours to replace: - # this script -- its own marker - # the book -- `Begin /etc/profile.d/i18n.sh`, placeholder - # LANG that was never filled in - # lfs-helper -- its marker, and before 1.14.298 an i18n.sh - # with the export COMMENTED OUT: - # : # export LANG=en_US.UTF-8 - # which is what is on a machine built with it - _ours_i18n() { - grep -q '^# written by stacks/shell-env.sh' "$1" \ - || grep -q '^# written by lfs-helper' "$1" \ - || grep -q 'Begin /etc/profile.d/i18n.sh' "$1" \ - || grep -q '^[[:space:]]*:[[:space:]]*#[[:space:]]*export LANG=' "$1" - } - got="" - [ -f "$I18N_SH" ] && got="$(_i18n_gives "$I18N_SH")" - if [ -f "$I18N_SH" ] && [ "$(_norm "$got")" = "$(_norm "$want")" ]; then - echo "# $I18N_SH already gives LANG=$got -- left alone" - elif [ -f "$I18N_SH" ] && ! _ours_i18n "$I18N_SH"; then - # not ours, and not the book's either: somebody wrote this on purpose - echo "# $I18N_SH is hand-written and gives LANG=${got:-nothing} -- left alone" - echo "# $want is what this machine has; set it there, or delete the file" - else - if [ -f "$I18N_SH" ]; then - bak="$I18N_SH.bak-$(date +%Y%m%d-%H%M%S)" - cp -p "$I18N_SH" "$bak" \ - && echo "# $I18N_SH gave LANG=${got:-nothing} -- replaced (backup: $(basename "$bak"))" - fi - cat > "$I18N_SH" <<'I18N' -# written by stacks/shell-env.sh -- safe to delete, regenerated -# Begin /etc/profile.d/i18n.sh -# -# $LANG for every login shell, from /etc/locale.conf. BLFS's /etc/profile -# reads this directory; LFS's reads /etc/locale.conf itself. Either way the -# locale comes from one file. -if [ -r /etc/locale.conf ]; then - . /etc/locale.conf - export LANG - if [ -n "${LC_ALL:-}" ]; then export LC_ALL; fi -fi -# less(1) needs telling separately -- it reads this, not the locale. -export LESSCHARSET="${LESSCHARSET:-utf-8}" -# End /etc/profile.d/i18n.sh -: -I18N - chmod 644 "$I18N_SH"; own_root "$I18N_SH" - echo "# wrote $I18N_SH (exports LANG from $LOCALE_CONF)" - fi - if [ "$(_norm "${LANG:-C}")" != "$(_norm "$want")" ]; then - echo "# this shell still has LANG=${LANG:-unset}: /etc/profile.d is read at" - echo "# LOGIN, so log in again -- or now: . $I18N_SH" - fi -fi - echo "# done -- open a new shell, or: . /etc/bashrc" diff --git a/stacks/sway-session.sh b/stacks/sway-session.sh index e54478d..189420e 100755 --- a/stacks/sway-session.sh +++ b/stacks/sway-session.sh @@ -11,7 +11,7 @@ HOME_USER="${LFS_DESKTOP_USER:-${LFS_HOME_USER:-}}" [ -n "$HOME_USER" ] && printf '%s\n' "$HOME_USER" > /etc/pkgusr/desktop-user HOME_DIR="/home/${HOME_USER:-nobody-set}" -# 1. one launcher: the checks sway needs, then sway. +# 1. one launcher: a dbus session around sway. # # The seat comes from seatd, not elogind: elogind only registers a session # through pam_elogind, and Linux-PAM is not on this system, so a tty login @@ -19,8 +19,7 @@ HOME_DIR="/home/${HOME_USER:-nobody-set}" # means libseat does not try logind first and fail with a puzzling message. cat > /usr/bin/sway-session <<'LAUNCH' #!/bin/bash -# start sway from a plain tty login. The session bus and XDG_RUNTIME_DIR -# are NOT set up here -- that is yours to arrange. +# start sway inside a dbus session; run from a plain tty login export XDG_CURRENT_DESKTOP=sway export LIBSEAT_BACKEND=seatd @@ -65,10 +64,11 @@ if [ -n "${XDG_VTNR:-}" ] && [ -r /sys/class/tty/tty0/active ]; then fi fi -# THE SESSION BUS AND XDG_RUNTIME_DIR ARE DELIBERATELY NOT HANDLED HERE. -# They are yours; this launcher only checks the seat, udev and the VT, and -# then starts the compositor with whatever environment you gave it. -exec sway "$@" +if [ -z "${XDG_RUNTIME_DIR:-}" ]; then + export XDG_RUNTIME_DIR="/run/user/$(id -u)" + mkdir -p "$XDG_RUNTIME_DIR" 2>/dev/null && chmod 700 "$XDG_RUNTIME_DIR" +fi +exec dbus-run-session -- sway "$@" LAUNCH chmod 755 /usr/bin/sway-session @@ -78,7 +78,7 @@ cat > /usr/bin/pipewire-start <<'PW' # start pipewire + wireplumber for this session, if not already up pgrep -x pipewire >/dev/null || { pipewire & disown; } pgrep -x wireplumber >/dev/null || { wireplumber & disown; } -echo "# pipewire is up (its sockets go in \${XDG_RUNTIME_DIR:-<unset>})" +echo "# pipewire is up (pipewire-pulse socket in $XDG_RUNTIME_DIR)" PW chmod 755 /usr/bin/pipewire-start diff --git a/stacks/sway.stack b/stacks/sway.stack index 9338cb9..2e4a652 100644 --- a/stacks/sway.stack +++ b/stacks/sway.stack @@ -1,12 +1,5 @@ # sway.stack -- a working Sway desktop with a minimal WebKit browser. -# stack-version: 1.14.292 -# -# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack -# here: the exclusions are a property of the machine, not of one stack. -# Without it, `stack media` planned 274 packages -- systemd, polkit, qt6, -# cups -- because those exclusions lived in sway.stack alone. -include avoid - +# stack-version: 1.13.67 # Replay with: packagemanager stack sway --run # Entries run top to bottom; book deps are resolved from the BLFS book, # installed entries are skipped, the first failure stops the run. @@ -20,9 +13,33 @@ include avoid # libxml2) must not be dropped by a blanket rule. These are the branches # this desktop does not want, named one at a time, and everything they alone # pull in goes with them. +avoid libreoffice +avoid asymptote +avoid biber +avoid qt6 +avoid okular +avoid phonon +avoid phonon-backend-vlc +avoid vlc +avoid cups +avoid openjdk +avoid java +avoid apache-ant +avoid postgresql +avoid openldap +avoid unixodbc +avoid graphviz +avoid gegl # the KDE leaves okular pulls in: named individually, because a package can # have several parents and pruning by one of them drops shared dependencies # (graphite2 was lost that way) +avoid plasma-activities +avoid libkexiv2 +avoid kwindowsystem +avoid libfm-qt +avoid xdg-desktop-portal-lxqt +avoid xdg-desktop-portal-gnome +avoid gnome-desktop # The desktop-portal tail. The book makes xdg-desktop-portal REQUIRE all # four backends -- gtk, wlr, gnome AND lxqt -- so one recommendation from @@ -30,21 +47,79 @@ include avoid # Sway needs none of it to run; screen sharing wants xdg-desktop-portal-wlr, # which can be built on its own later: # packagemanager install xdg-desktop-portal-wlr --run +avoid xdg-desktop-portal +avoid xdg-desktop-portal-gtk +avoid xdg-desktop-portal-wlr +avoid xdg-desktop-portal-gnome +avoid xdg-desktop-portal-lxqt +avoid xfce4-panel +avoid xfce4-notifyd +avoid xfce4-dev-tools +avoid xfconf +avoid libxfce4util +avoid libxfce4ui +avoid libxfce4windowing +avoid exo +avoid garcon +avoid liblxqt +avoid lxqt-build-tools +avoid lxqt-notificationd +avoid lxqt-kwindowsystem +avoid libqtxdg +avoid polkit-qt +avoid libfm-qt +avoid menu-cache +avoid libwnck +avoid notification-daemon # Not this desktop either: a systemd package on a SysV system, phone modem # stacks, a mail/ssh server, spell-checking dictionaries. -# openssh is AVOIDED here and INSTALLED by network.stack, where it belongs: -# it is networking, not desktop. Letting a dependency plan pull it in -# mid-sway would put its system-account question in the middle of a graphics -# build. +avoid systemd +avoid ModemManager +avoid libmbim +avoid libqmi +avoid geoclue2 +avoid openssh +avoid ruby +avoid aspell +avoid enchant +avoid gnome-keyring +avoid gcr +avoid startup-notification # Tails of things already avoided. Each of these is in the plan ONLY to # serve a package this stack does not build, and the tools cannot yet work # that out for themselves (see 1.12.83: pruning by the single parent that # `blfs order` reports dropped shared dependencies, and broke harfbuzz). # Checked one by one against the plan's parent column: +avoid libcupsfilters # cups (avoided) +avoid libppd # cups +avoid cups-filters # cups +avoid libkexiv2 # okular (avoided) +avoid exiv2 # libkexiv2 +avoid clucene # libreoffice (avoided) +avoid redland # libreoffice +avoid rasqal # redland +avoid raptor # rasqal +avoid jasper # qt6 (avoided) +avoid libmng # qt6 +avoid double-conversion # qt6 +avoid poppler # libcupsfilters -- its ONLY parent, avoided +avoid mupdf # libcupsfilters +avoid gs # libcupsfilters +avoid qpdf # libcupsfilters +avoid nss # poppler +avoid boost # poppler +avoid pdfio # libcupsfilters +avoid gpgmepp # poppler # Checked against EVERY parent, not just the one the plan prints: these have # no parent left in the plan at all. +avoid colord # cups (avoided) -- no live parent +avoid linux-pam # systemd (avoided) -- no live parent +avoid xdg-utils # cups -- no live parent +avoid freeglut # mupdf (avoided) -- no live parent +avoid polkit # colord + systemd (both avoided) -- no live parent +avoid libgusb # colord # NOT avoided, though the plan labels them with an avoided parent: the label # shows only the FIRST parent. gtk3 is really wanted by adwaita-icon-theme, # ffmpeg by mpv, polkit by colord... and pulseaudio by pipewire/sdl3/gst. @@ -58,23 +133,6 @@ check kernel-sway book dbus book elogind url=https://github.com/elogind/elogind ref=v255.17 prog=loginctl build="meson setup build --prefix=/usr --buildtype=release -Dcgroup-controller=elogind -Ddbuspolicydir=/etc/dbus-1/system.d; ninja -C build" install="ninja -C build install" -# 1b. THE AUDIO FLOOR, BEFORE ANYTHING THAT LOOKS FOR IT. -# -# alsa-lib used to sit at entry 7, after mpv -- and ffmpeg is built inside -# mesa's plan (entry 2) and SDL3 inside gstreamer's, both of which autodetect -# ALSA. They configured cleanly, installed cleanly, and had no ALSA support, -# because the library did not exist yet. Nothing failed, so nothing said so. -# -# This is the same rule the harfbuzz comment below states for libass: what an -# earlier entry installed is simply there. An optional dependency of -# something built inside a plan can only be placed by entry order. -book alsa-lib - -# ffmpeg and SDL3 are named here, before mesa and gstreamer, so they are -# built in THIS order rather than wherever a dependency plan reaches them -- -# and expect= proves the feature afterwards instead of trusting the order. -book ffmpeg expect="ffmpeg -hide_banner -devices 2>/dev/null | grep -qw alsa" - # 2. graphics -- llvm feeds mesa; mesa options come from the machine # (machine.conf wiring pending; book defaults until then) # cmake (pulled in by llvm and others) has NO required dependencies and four @@ -103,40 +161,7 @@ book harfbuzz # text shaping: freetype recommends it, book graphite2 # and webkit/pango need both book fontconfig -# LIBGLVND, BEFORE MESA -- because machine.conf builds Mesa with -# glvnd=enabled, and Mesa's configure needs it there. -# -# glvnd is the GL dispatcher: libGL, libEGL and libGLES are its, and -# Mesa installs only its vendor libraries behind it (libGLX_mesa, -# libEGL_mesa + 50_mesa.json). It is what provides pkg-config `opengl`, -# and what lets another GL driver (NVIDIA's) sit beside Mesa. -# -# NOT a BLFS page -- the 13.1 book only links to GLFS for it. The canonical -# repo is gitlab.freedesktop.org/glvnd/libglvnd; the NVIDIA GitHub copy is -# the one reachable from where this was written. CHECKED there: newest tag -# v1.7.0, and the option names (all left at their defaults). -# -# SWITCHING AN EXISTING MACHINE: libglvnd and mesa must be done in ONE go. -# Between the two, libGL.so.1 is glvnd's with no vendor behind it and every -# GL program fails: -# packagemanager stack sway --only libglvnd,mesa --run -# Programs already built keep working: the sonames (libGL.so.1, libEGL.so.1, -# libGLESv2.so.2) do not change, only who provides them. -git libglvnd \ - url=https://gitlab.freedesktop.org/glvnd/libglvnd.git \ - ref=v1.7.0 \ - needs=xorg7-lib \ - have=/usr/lib/pkgconfig/opengl.pc \ - build="rm -rf build; meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ - install="ninja -C build install" \ - expect="pkg-config --exists opengl && pkg-config --exists glx && test -e /usr/lib/libGLdispatch.so.0" - -# THE SOFTWARE RENDERER IS A SEPARATE DRIVER, and a hardware-only mesa -# fails with "egl: failed to create dri2 screen" and no fallback. The probe -# wants both: radeonsi for the GPU, swrast_dri.so for llvmpipe. (llvmpipe -# installs as swrast_dri.so -- the file is named after the interface, not -# the rasteriser, which is why looking for llvmpipe_dri.so finds nothing.) -book mesa expect="test -e /usr/lib/dri/swrast_dri.so -o -e /usr/lib/dri/kms_swrast_dri.so && test -e /usr/share/glvnd/egl_vendor.d/50_mesa.json && test -e /usr/lib/libGLX_mesa.so.0" +book mesa book xwayland book libinput book libxkbcommon @@ -164,33 +189,55 @@ book adwaita-icon-theme # cursors and the default icon set book gtk4 book glib-networking # TLS for libsoup, or the browser has no https book libsoup3 -# The CODECS, before the plugins that look for them. A WebRTC call -# negotiates Opus for audio and VP8 for video; both plugins are autodetected -# (opus in -base, vpx in -good), so a missing library means a call that -# connects and carries nothing. -# 6b. the media chain, in its own stack so it can be rebuilt on its own: -# codecs, gstreamer, the WebRTC pieces, woff2, webkitgtk, mpv. +book gstreamer10 +book gst10-plugins-base +book gst10-plugins-good +# --------------------------------------------------------------------- +# WebRTC and camera streaming. # -# ONE LIST, TWO ENTRY POINTS. `packagemanager stack media --run` builds -# just that part; this include means the full desktop build is still one -# command and the entries are written down once. It sits HERE because -# webkitgtk needs gtk4, pango and libsoup3 above, and mesa and ffmpeg from -# further up. -include media - -# 7. audio: alsa-lib is entry 1b (it has to be); the tools can come later +# BLFS 13.0 carries NONE of these, so they are git entries. They must be +# built BEFORE gst10-plugins-bad and webkitgtk, because webrtcbin -- the +# element a browser uses for WebRTC -- is compiled into gst-plugins-bad only +# if libnice is present, and webkitgtk links the result. Adding them later +# means recompiling both (hours). +# +# What each is for: +# libnice ICE: the NAT traversal WebRTC cannot work without +# libsrtp the encrypted media transport +# usrsctp WebRTC data channels +# webrtc-audio-processing echo cancellation for a microphone (the log's +# "webrtc-audio-processing-2 found: NO") +# +# The camera side is already in this stack: v4l-utils for the device and +# pipewire for routing it. +# +# UNTESTED -- the build lines follow each project's README, and none of these +# has been built here. Expect to adjust them; that is what the stack file is +# for. +git libnice url=https://gitlab.freedesktop.org/libnice/libnice.git ref=0.1.22 have=/usr/lib/pkgconfig/nice.pc build="meson setup build --prefix=/usr --buildtype=release -Dgtk_doc=disabled; ninja -C build" install="ninja -C build install" +git libsrtp url=https://github.com/cisco/libsrtp.git ref=v2.6.0 have=/usr/lib/pkgconfig/libsrtp2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +# ref: 1.0.0 does not exist upstream (checked: the tags are 0.9.3.0, 0.9.4.0, +# 0.9.5.0). 0.9.5.0 has meson.build, so the build line below still applies. +git usrsctp url=https://github.com/sctplab/usrsctp.git ref=0.9.5.0 have=/usr/lib/pkgconfig/usrsctp.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +git webrtc-audio-processing url=https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing.git ref=v2.1 have=/usr/lib/pkgconfig/webrtc-audio-processing-2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" + +book gst10-plugins-bad +# No exclusions. The book's dependency graph is built as the book states +# it, recommendations included: a recommendation is there for a reason, and +# skipping one is how texlive came to build before FreeType. This costs +# build time (texlive, LibreOffice and their tails are hours) and buys a +# system that matches the book. +book webkitgtk +book mpv + +# 7. audio: ALSA is the floor; pipewire installed, started on demand +book alsa-lib book alsa-utils book pipewire book wireplumber -# 8. SDL3 for the Sw* tools. -# -# SDL3 dlopens libasound rather than linking it, so ldd says nothing; the -# name it will dlopen is in the binary, and that is what to look for. -# the glob matters: SDL3 installs libSDL3.so.0.<n> and a bare libSDL3.so -# symlink only when the -devel side is there, so naming one path makes the -# probe fail for the wrong reason -book sdl3 expect="strings /usr/lib/libSDL3.so* 2>/dev/null | grep -q libasound" +# 8. SDL3 for the Sw* tools +book sdl3 # 9. the nimbin2 family: make all && make PREFIX=/usr install @@ -207,6 +254,7 @@ book sdl3 expect="strings /usr/lib/libSDL3.so* 2>/dev/null | grep -q libasound # and asymptote/biber are avoided. # # To build it later, on its own: packagemanager install texlive --run +avoid texlive # --------------------------------------------------------------------- @@ -240,200 +288,7 @@ git sway url=https://github.com/swaywm/sway ref=1.11 prog=sway needs=json- git swaybg url=https://github.com/swaywm/swaybg ref=v1.2.1 prog=swaybg build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" git tllist url=https://codeberg.org/dnkl/tllist ref=1.1.0 have=/usr/include/tllist.h build="meson setup build --prefix=/usr" install="ninja -C build install" git fcft url=https://codeberg.org/dnkl/fcft ref=3.3.2 have=/usr/include/fcft build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" -# light: backlight and keyboard brightness. Not in the book. -# -# THE UPSTREAM MOVED. haikarainen/light on GitHub is gone -- checked, and -# `git ls-remote` on it fails while three other GitHub URLs in this stack -# answer, so it was the path and not the network. GitHub returns 401 for a -# repository it does not have, which is why the clone asked for a username. -# -# gitlab.com/dpeukert/light is where it continued. Checked on the machine -# this runs on: the URL answers and `git ls-remote --tags` prints NOTHING -- -# the fork carries no tags at all, only `main`. So `ref=main` is not -# laziness here, it is the only ref there is, and it means this entry -# follows upstream: the commit that built is the one worth writing down if -# a build ever breaks. -# -# git ls-remote --heads https://gitlab.com/dpeukert/light.git -# -# CFLAGS='-fcommon' IS NOT A WORKAROUND FOR A BROKEN PACKAGE. light's -# helpers.h declares `light_loglevel` without `extern`, which every C -# compiler accepted until GCC 10 made -fno-common the default; each object -# then defines it and the link fails with -# multiple definition of `light_loglevel' -# -fcommon restores the pre-GCC-10 behaviour, which is what this code was -# written against. The alternative is patching the header, and that is -# upstream's call, not this stack's. -# -# IT HAS TO BE IN THE ENVIRONMENT, NOT ON THE CONFIGURE LINE. autogen.sh -# here is not just autoreconf: it ends by running ./config.status --recheck -# and then `make`, so the compile that fails happens BEFORE the ./configure -# on this line is ever reached. Passing CFLAGS to that configure fixed -# nothing and the link failed exactly as before. Exported, it reaches -# autogen.sh's own configure run, which is the one that writes the Makefile. -# -# And `rm -f config.status` first: autogen.sh re-runs whatever config.status -# is lying around, so a tree configured by hand earlier keeps its old flags -# no matter what this line says. -# -# If that fails too, one of these: -# - the new URL, if light moved: just fix url= below. -# - the source you built from last time, as a tar entry: -# tar light url=file:///path/to/light-1.2.2.tar.gz pkg=light-1.2.2.tar.gz \ -# have=/usr/bin/light build="./autogen.sh; ./configure --prefix=/usr \ -# --with-udev; make" install="make install" -# - brightnessctl, which does the same job and DOES resolve (tag 0.5.1 -# verified from here). It uses the same udev/video-group approach: -# git brightnessctl url=https://github.com/Hummer12007/brightnessctl.git \ -# ref=0.5.1 prog=brightnessctl build="make" \ -# install="make install" expect="brightnessctl --version >/dev/null 2>&1" -# -# It writes to /sys/class/backlight/*/brightness, which root owns, so the -# binary needs either setuid root or a udev rule putting those files in a -# group you are in. --with-udev installs 90-backlight.rules, which chgrps -# them to `video`; that is the one to prefer, and it means the desktop user -# must be in `video` (cfg power puts them in the power group, not this one). -git light url=https://gitlab.com/dpeukert/light.git ref=main prog=light \ - build="export CFLAGS='-O2 -fcommon'; rm -f config.status; ./autogen.sh; ./configure --prefix=/usr --with-udev; make" \ - install="make install" \ - expect="light -V >/dev/null 2>&1" - -# jq. Not in the book, so this entry is ours -- and it was BUILT to write -# it, not guessed: -# -# - jq-1.8.2 is the newest tag (jq-1.8.2rc1 exists too; the release -# tarball is the non-rc one), md5 e288a84cf2957ea2cfc643aa9e59b473 -# - the release tarball ships `configure`, so no autogen.sh step -# - oniguruma is NOT on the system and does not need to be: configure says -# "Oniguruma was not found. Will use the packaged oniguruma", builds the -# copy in vendor/, and `jq -n '"abc"|test("b")'` answers true. That is -# the probe below -- `jq --version` would pass on a jq with no regex, -# which is the half of jq most scripts use. -tar jq url=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-1.8.2.tar.gz \ - pkg=jq-1.8.2.tar.gz prog=jq \ - build="./configure --prefix=/usr; make" \ - install="make install" \ - expect="jq -n '\"abc\" | test(\"b\")' >/dev/null 2>&1" - -# wf-recorder: screen recording for a wlroots compositor. Not in the book. -# -# CHECKED against the repo at v0.6.0 (newest tag; master and two feature -# branches also exist): -# -# * audio is CONDITIONAL. meson_options.txt has `pulse` and `pipewire` -# as auto features and config.h.in has `#mesondefine HAVE_AUDIO`, which -# gates src/audio.cpp -- so with neither library found at configure -# time you get a recorder that cannot record sound. -# * and `--help` lists `-a, --audio` EITHER WAY: that text is not inside -# the #ifdef. So the probe checks what the binary LINKS, which is the -# only honest answer to "was audio built in". -# * `meson build` (as in the by-hand script) still works, but `meson -# setup build` is the spelling meson documents now. -# -# Its codecs come from ffmpeg, which is entry 1b, and the default is -# libx264 -- so gst10-plugins-ugly's x264 is not what it uses; ffmpeg's own -# libx264 support is. -git wf-recorder url=https://github.com/ammen99/wf-recorder.git \ - ref=v0.6.0 prog=wf-recorder \ - build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ - install="ninja -C build install" \ - expect="ldd /usr/bin/wf-recorder 2>/dev/null | grep -qE 'libpulse|libpipewire'" - -# VIEWERS: a document one and an image one. Neither is in the BLFS book. -# -# CHECKED with git ls-remote, all four at once: girara, zathura and -# zathura-pdf-poppler share the release tag **2026.07.18** (they are one -# project, released together -- mixing tags is the usual way to get a -# zathura that cannot open a PDF), and imv's newest is **v4.3.0**. -# imv's repo answers on both github.com/exec64/imv and .../eXeC64/imv; the -# lowercase spelling is used here. -# -# zathura is THREE packages: girara is its widget library, the core opens no -# file formats at all, and the backend is a separate plugin. A zathura with -# no plugin starts, shows a grey window and says nothing -- which is what -# the expect= below is for: it looks for the PLUGIN, not the binary. -# -# poppler (the backend's dependency) is a book page. It was in avoid.stack -# as libcupsfilters' child; it is wanted now. mupdf stays avoided -- it is -# the other backend and one is enough. -book poppler - -# girara installs **girara.pc**, not girara-gtk3.pc: its meson.build says -# `pkg.generate(name: 'girara', ...)`. The old guess made the stack refuse -# to record a girara that had installed perfectly -- "finished, but -# /usr/lib/pkgconfig/girara-gtk3.pc is still missing", which is the have= -# check doing its job on a wrong have=. -git girara url=https://github.com/pwmt/girara.git ref=2026.07.18 \ - have=/usr/lib/pkgconfig/girara.pc \ - build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ - install="ninja -C build install" - -# xxHash, BEFORE zathura. Mentioned eight times in the BLFS book and given -# no page of its own, so this entry is ours. -# -# zathura 2026.07.18 takes it as a hard dependency: -# Run-time dependency libxxhash found: NO (tried pkgconfig and cmake) -# meson.build:57:9: ERROR: Dependency "libxxhash" not found -# -# CHECKED in the checkout: newest tag v0.8.3; it ships libxxhash.pc.in, so -# the pkg-config module really is `libxxhash` -- the name meson asks for -- -# and `make install` runs install_libxxhash.pc among its targets. A plain -# Makefile, no cmake or meson: PREFIX and LIBDIR are what it takes. -git xxhash url=https://github.com/Cyan4973/xxHash.git ref=v0.8.3 \ - have=/usr/lib/pkgconfig/libxxhash.pc \ - build="make PREFIX=/usr LIBDIR=/usr/lib" \ - install="make PREFIX=/usr LIBDIR=/usr/lib install" \ - expect="pkg-config --exists libxxhash" - -# NO -Dsqlite: zathura has no such option. Its meson.build takes sqlite as -# an unconditional dependency -- `dependency('sqlite3', version: '>=3.35.0')` -# -- and its only options are synctex, seccomp, landlock, manpages, -# tests-x11, tests-wayland and convert-icon. `-Dsqlite=enabled` stopped -# meson dead with `Unknown option: "sqlite"`, which is what guessing an -# option name earns. sqlite3 is already installed here (p_sqlite). -git zathura url=https://github.com/pwmt/zathura.git ref=2026.07.18 \ - prog=zathura \ - build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ - install="ninja -C build install" - -git zathura-pdf-poppler url=https://github.com/pwmt/zathura-pdf-poppler.git \ - ref=2026.07.18 \ - have=/usr/lib/zathura/libpdf-poppler.so \ - build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ - install="ninja -C build install" \ - expect="test -e /usr/lib/zathura/libpdf-poppler.so" - -# imv: wayland image viewer. The binary is `imv`, which dispatches to -# imv-wayland or imv-x11 -- so the probe wants the wayland one, or on this -# machine it would pass on an imv that can only open windows under X. -# -licuuc: imv links libicuio and CALLS libicuuc (ubrk_close), and its -# link line carries -Wl,--no-undefined, so the indirect dependency is an -# error rather than a warning: -# undefined reference to symbol 'ubrk_close_78' -# /lib/libicuuc.so.78: error adding symbols: DSO missing from command line -# icu splits its API across libicuuc (common), libicui18n and libicuio; -# imv's meson names only the one it uses directly. Naming the other on the -# link line is the whole fix -- nothing is patched. -git imv url=https://github.com/exec64/imv.git ref=v4.3.0 \ - prog=imv \ - build="meson setup build --prefix=/usr --buildtype=release -D c_link_args=-licuuc; ninja -C build" \ - install="ninja -C build install" \ - expect="command -v imv-wayland >/dev/null 2>&1 || ldd /usr/bin/imv 2>/dev/null | grep -q wayland" - -# wl-clipboard: wl-copy and wl-paste. Not in the book either, and sway is -# unusable without it once anything wants a clipboard from a script. -git wl-clipboard url=https://github.com/bugaevc/wl-clipboard ref=v2.2.1 \ - prog=wl-copy \ - build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ - install="ninja -C build install" \ - expect="wl-paste --version >/dev/null 2>&1" - git foot url=https://codeberg.org/dnkl/foot ref=1.28.0 prog=foot build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" -# the rest of the fonts: icons, a patched terminal font, and Noto for -# everything DejaVu does not cover. One step, because they are all "unpack -# an archive into /usr/share/fonts and run fc-cache". -cfg fonts - tar dejavu-fonts url=https://downloads.sourceforge.net/dejavu/dejavu-fonts-ttf-2.37.tar.bz2 pkg=dejavu-fonts-ttf-2.37.tar.bz2 have=/usr/share/fonts/dejavu build="true" install="install -v -d -m755 /usr/share/fonts/dejavu; install -v -m644 ttf/*.ttf /usr/share/fonts/dejavu; fc-cache /usr/share/fonts/dejavu" git sdl3-image url=https://github.com/libsdl-org/SDL_image ref=release-3.2.4 have=/usr/lib/libSDL3_image.so build="cmake -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release; cmake --build build" install="cmake --install build" git sdl3-ttf url=https://github.com/libsdl-org/SDL_ttf ref=release-3.2.2 have=/usr/lib/libSDL3_ttf.so build="cmake -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release; cmake --build build" install="cmake --install build" diff --git a/test_lfs_crosschain.sh b/test_lfs_crosschain.sh index d973042..2a61229 100644 --- a/test_lfs_crosschain.sh +++ b/test_lfs_crosschain.sh @@ -6,148 +6,9 @@ # Then checks the failure paths -- because "it printed done" is not evidence # that anything worked. # -# Usage: bash test_lfs_crosschain.sh [path-to-lfs-script] [path-to-book] -# -# RUNNING PART OF IT. The suite outgrew one execution slot, so it can run in -# pieces. Blocks are the `# ---- <title>` sections, numbered in file order: -# -# --list number and name every block -# --from N [--to M] run that range (M defaults to the last block) -# --only N,M,... run just those blocks -# --only <text> run every block whose title contains <text> -# -# A selection re-executes a generated subset: the preamble, the chosen -# blocks, and the summary. The preamble is what every block needs; a block -# that ALSO needs something an earlier block computed cannot run alone, and -# says so with bash's own unbound-variable message naming the variable -- -# rather than passing because its assertions never ran. +# Usage: bash test_lfs_crosschain.sh [path-to-lfs-script] set -u -_SELF="${BASH_SOURCE[0]}" -_SEL_FROM=""; _SEL_TO=""; _SEL_ONLY=""; _SEL_LIST=0 -_args=() -while [ $# -gt 0 ]; do - case "$1" in - --list) _SEL_LIST=1; shift ;; - --from) _SEL_FROM="${2:-}"; shift 2 ;; - --to) _SEL_TO="${2:-}"; shift 2 ;; - --only) _SEL_ONLY="${2:-}"; shift 2 ;; - --from=*) _SEL_FROM="${1#*=}"; shift ;; - --to=*) _SEL_TO="${1#*=}"; shift ;; - --only=*) _SEL_ONLY="${1#*=}"; shift ;; - -h|--help) - sed -n '3,22p' "$_SELF" | sed 's/^# \{0,1\}//'; exit 0 ;; - --*) echo "!! unknown option: $1" >&2 - echo " try: --list --from N --to M --only N,M|<text>" >&2 - exit 2 ;; - *) _args+=("$1"); shift ;; - esac -done -set -- ${_args[@]+"${_args[@]}"} - -# The splitter is ONE piece of code with two jobs (list, extract), so a -# numbering the list prints cannot disagree with the numbering a range uses. -_blocks() { # _blocks list | _blocks emit <from> <to> <only> - python3 - "$_SELF" "$@" <<'PYSPLIT' -import re, sys -path, mode = sys.argv[1], sys.argv[2] -lines = open(path, encoding="utf-8", errors="replace").read().split("\n") - -# HEREDOC BODIES ARE NOT CODE. Every python test in this suite arrives as a -# quoted heredoc, and their bodies contain lines starting with `# ---- ` and -# with `name=`. Counting those as block markers renumbered the blocks and -# hoisted python assignments into bash. -heredoc = [False] * len(lines) -tag = None -for i, ln in enumerate(lines): - if tag is None: - m = re.search(r"<<-?\s*[\"']?([A-Za-z_][A-Za-z0-9_]*)[\"']?\s*$", ln) - if m: - tag = m.group(1) - continue - heredoc[i] = True - if ln.strip() == tag: - tag = None - -marks = [i for i, ln in enumerate(lines) - if ln.startswith("# ---- ") and not heredoc[i]] -summary = next((i for i, ln in enumerate(lines) - if ln.startswith("#@@ SUMMARY @@")), len(lines)) -bounds = [] -for n, start in enumerate(marks): - end = marks[n + 1] if n + 1 < len(marks) else summary - bounds.append((start, min(end, summary))) - -def title(i): - return re.sub(r"^# ---+ | *-* *#$| -+ #$", "", - lines[i]).strip(" -#").strip() - -if mode == "list": - for n, (start, _end) in enumerate(bounds, 1): - print("%4d %s" % (n, title(start))) - print("\n%d blocks. Run some: --from N --to M | --only N,M |" - " --only <text>" % len(bounds)) - sys.exit(0) - -frm, to, only = sys.argv[3], sys.argv[4], sys.argv[5] -sel = [] -if only: - if re.fullmatch(r"[0-9,\s]+", only): - sel = [int(x) for x in re.split(r"[,\s]+", only.strip()) if x] - else: - sel = [n for n, (s, _e) in enumerate(bounds, 1) - if only.lower() in title(s).lower()] - if not sel: - sys.stderr.write("!! no block title contains %r\n" % only) - sys.exit(2) -else: - a = int(frm) if frm else 1 - b = int(to) if to else len(bounds) - sel = list(range(a, b + 1)) -sel = [n for n in sorted(set(sel)) if 1 <= n <= len(bounds)] -if not sel: - sys.stderr.write("!! that selection is empty (1-%d)\n" % len(bounds)) - sys.exit(2) - -out = lines[:marks[0]] -# What a later block may still need from an earlier one: the tool paths and -# the empty accumulators, which are plain assignments and run nothing. A -# value produced by a command substitution other than `dirname` is NOT -# hoisted -- copying it would silently re-run work. -safe = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=(\"\"|''|)$" - r"|^[A-Za-z_][A-Za-z0-9_]*=\"\$\(dirname [^\n]*\)[^$`]*\"$") -first = bounds[sel[0] - 1][0] -for i in range(marks[0], first): - if not heredoc[i] and safe.match(lines[i].strip()): - out.append(lines[i]) -for n in sel: - s, e = bounds[n - 1] - out.append("") - out.append("#== block %d/%d ==" % (n, len(bounds))) - out.extend(lines[s:e]) -out.extend(lines[summary:]) -sys.stderr.write("running %d of %d blocks: %s\n" - % (len(sel), len(bounds), - ",".join(str(n) for n in sel) if len(sel) < 12 - else "%d-%d" % (sel[0], sel[-1]))) -print("\n".join(out)) -PYSPLIT -} - -if [ "$_SEL_LIST" = 1 ]; then - _blocks list - exit 0 -fi -if [ -n "$_SEL_FROM$_SEL_TO$_SEL_ONLY" ]; then - _sub="$(mktemp /tmp/lfstest-subset.XXXXXX.sh)" - _blocks emit "$_SEL_FROM" "$_SEL_TO" "$_SEL_ONLY" > "$_sub" || exit $? - # exec, so the subset's exit status is the suite's. The temp file is the - # thing that actually ran: keep it, and say where, because a subset that - # dies on an unbound variable is only diagnosable by reading it. - echo " (subset script: $_sub)" >&2 - exec bash "$_sub" ${1+"$@"} -fi - LFS_TOOL="${1:-./lfs}" # A bare name is resolved off $PATH; anything unreadable stops NOW with one # message. `bash test_lfs_crosschain.sh lfs` used to produce hundreds of @@ -197,15 +58,9 @@ _COUNTDIR="${TMPDIR:-/tmp}/lfstest-count.$$" mkdir -p "$_COUNTDIR" : > "$_COUNTDIR/pass" : > "$_COUNTDIR/fail" -: > "$_COUNTDIR/skip" ok() { echo " PASS $1"; echo x >> "$_COUNTDIR/pass"; } bad() { echo " FAIL $1"; echo x >> "$_COUNTDIR/fail"; } -# A check whose PRECONDITION is absent is neither pass nor fail. The rc.site -# check needs a SysV book on disk and reported FAIL without one, so a clean -# run looked like a broken one and the real failures had to be counted by -# hand every time. -skip() { echo " SKIP $1"; echo x >> "$_COUNTDIR/skip"; } # Lift ONE function out of a bash source file, to exercise it in isolation. # @@ -232,36 +87,14 @@ _slice_fn() { # _slice_fn <file> <fn> ' "$1" } -# _setid_table <lfs-helper>: what _setid_known PRINTS, from the repo's -# setid.conf. Since 1.14.263 the table is that file, not text in the -# function, so the function's body no longer names any path. -_setid_table() { - ( warn() { :; }; PKGUSR_ETC=/nonexistent - eval "$(_slice_fn "$1" _setid_conf)" - eval "$(_slice_fn "$1" _setid_known)" - SETID_CONF="$(dirname "$LFS_TOOL")/stacks/setid.conf" _setid_known ) -} - -_slice_def() { # _slice_def <file> <python def> - # _slice_fn matches SHELL functions. A python def needs its own slice, - # and a slice that returns nothing makes every assertion on it pass - # vacuously -- which is worse than any of them failing. - awk -v fn="$2" 'BEGIN{s="^def " fn "\\("} - !i && $0 ~ s { i=1; print; next } - i && /^[a-zA-Z@]/ { i=0 } i { print }' "$1" -} - # the python-heredoc blocks report their own verdict; fold them in the same way -# (77 = the block printed SKIP: its precondition is not on this machine) _count_rc() { if [ "$1" -eq 0 ]; then echo x >> "$_COUNTDIR/pass" - elif [ "$1" -eq 77 ]; then echo x >> "$_COUNTDIR/skip" else echo x >> "$_COUNTDIR/fail"; fi } _pass_total() { wc -l < "$_COUNTDIR/pass" | tr -d ' '; } _fail_total() { wc -l < "$_COUNTDIR/fail" | tr -d ' '; } -_skip_total() { wc -l < "$_COUNTDIR/skip" | tr -d ' '; } cleanup() { rm -rf "$T" "$_COUNTDIR"; } trap cleanup EXIT @@ -3489,1943 +3322,6 @@ print(" PASS entering the chroot refreshes the tools first") PY52 _count_rc $? -# ---- a test block the book words differently (1.14.211) -------------------- # -# The generator moves a test block to the `test` phase by matching the -# sentence that introduces it, and the pattern was anchored: `^to test`, -# `^to check`, `^the test suite`. numpy's book page says -# "The installation can be tested with the following commands:" -# so a venv, `pip3 install hypothesis attrs` and `numpy.test()` were -# generated into INSTALL. numpy installed -- 1320 files -- and then the -# package failed on `No module named 'pytest'`: a test dependency, in an -# install. -_p="" -_bl="$(dirname "$LFS_TOOL")/blfs" -if [ -f "$_bl" ]; then - grep -q 'can be tested' "$_bl" \ - || _p="$_p;the book's 'can be tested' wording is still read as an install" - grep -q '_TEST_BLOCK_RE.search' "$_bl" \ - || _p="$_p;the test intro is matched only at the start of the sentence" - # THE COMMANDS SAY WHAT THEY ARE. Widening the SENTENCE match was not - # enough: the regenerated script still had numpy's venv and - # `numpy.test()` in install, because the book on that machine (13.0) - # words or nests it differently again. Chasing prose is a losing game - # (1.14.212). - grep -q '_TEST_CONTENT_RE' "$_bl" \ - || _p="$_p;a test block is classified only by the prose around it" - grep -q '_by_content = _test_content_reason' "$_bl" \ - || _p="$_p;the content check is defined and not used first" - _tc="$(python3 - "$_bl" <<'PYTC' -import re, sys -src = open(sys.argv[1]).read() -m = re.search(r'_TEST_CONTENT_RE = re\.compile\(\s*r"""(.*?)"""\s*\)', src, re.S) -rx = re.compile(m.group(1), re.M | re.X) -cases = [ - ('python3 -m venv --system-site-packages testenv &&\n' - 'python3 -c "import numpy, sys; sys.exit(numpy.test() is False)"', True), - ('ninja test', True), ('make check', True), ('make -k check', True), - ('meson test --print-errorlogs', True), ('ctest', True), - ('pytest', True), ('python3 -m pytest', True), - ('pip3 wheel -w dist --no-build-isolation --no-deps --no-cache-dir $PWD', False), - ('pip3 install --no-index --find-links dist --no-user numpy', False), - ('ninja install', False), ('make install', False), - ('install -v -m644 README.md /usr/share/doc/x', False), - ('cmake -D CMAKE_INSTALL_PREFIX=/usr ..', False)] -bad = [t.splitlines()[0] for t, w in cases if bool(rx.search(t)) != w] -print("BAD: " + "; ".join(bad) if bad else "OK") -PYTC -)" - [ "$_tc" = OK ] || _p="$_p;test-by-content is wrong for: $_tc" - # END TO END, THROUGH THE REAL GENERATOR. Two releases in a row shipped - # a regex that passed its own unit check while the GENERATED SCRIPT was - # unchanged -- 1.14.211 widened a sentence match and the script still - # had numpy's venv in install. A pattern test cannot see that; only - # generating a script can. This fixture is numpy's page shape, cut down - # to what the parser requires (a sect2 with div.package, an a.ulink - # download in an itemizedlist, and pre/kbd class="command" blocks). - _fx2="$T/fixbook"; rm -rf "$_fx2"; mkdir -p "$_fx2" - cat > "$_fx2/book.html" <<'FIXBOOK' -<html><body> -<div class="sect2" lang="en"> -<h3 class="sect2"><a id="zzdemo" name="zzdemo"></a>zzdemo-1.2.3</h3> -<div class="package" lang="en"> -<div class="titlepage"><div><div><h4 class="title">Introduction to zzdemo</h4></div></div></div> -<div class="itemizedlist"><ul><li><p>Download (HTTP): <a class="ulink" - href="https://files.pythonhosted.org/packages/source/z/zzdemo/zzdemo-1.2.3.tar.gz" - >zzdemo-1.2.3.tar.gz</a></p></li></ul></div> -<p>Download MD5 sum: 00000000000000000000000000000000</p> -</div> -<div class="installation" lang="en"> -<div class="titlepage"><div><div><h4 class="title">Installation of zzdemo</h4></div></div></div> -<p>Build the module:</p> -<pre class="userinput"><kbd class="command">pip3 wheel -w dist --no-build-isolation --no-deps --no-cache-dir $PWD</kbd></pre> -<p>Now, as the <code class="systemitem">root</code> user:</p> -<pre class="root"><kbd class="command">pip3 install --no-index --find-links dist --no-user zzdemo</kbd></pre> -<p>The installation can be tested with the following commands:</p> -<pre class="userinput"><kbd class="command">mkdir -p test && -cd test && -python3 -m venv --system-site-packages testenv && -source testenv/bin/activate && -pip3 install hypothesis attrs && -python3 -c "import zzdemo, sys; sys.exit(zzdemo.test() is False)" -deactivate</kbd></pre> -</div></div></body></html> -FIXBOOK - if PKGUSR_BASE="$_fx2" PKGUSR_CONFIG=/dev/null \ - python3 "$_bl" --book-file "$_fx2/book.html" script zzdemo -o "$_fx2" \ - >"$_fx2/out" 2>&1; then - _ip="$(sed -n '/^install_pkg()/,/^}/p' "$_fx2/install_zzdemo-1.2.3")" - printf '%s' "$_ip" | grep -q 'pip3 install .*--find-links dist' \ - || _p="$_p;the generated install phase lost the actual install command" - printf '%s' "$_ip" | grep -q '^## SKIPPED' \ - || _p="$_p;the generated install phase does not skip the test block" - printf '%s' "$_ip" | grep -qE '^[^#]*numpy\.test|^[^#]*zzdemo\.test' \ - && _p="$_p;the test call is still live in the install phase" - printf '%s' "$_ip" | grep -qE '^[^#]*python3 -m venv' \ - && _p="$_p;the test venv is still built during install" - else - _p="$_p;the generator could not read a page shaped like numpy's: $(tail -2 "$_fx2/out" | tr '\n' ' ')" - fi - # and a skipped block must not claim the book said something it did not - grep -q 'OPTIONAL, skipped -- the book says: " + _g' "$_bl" \ - && _p="$_p;a content-detected test is attributed to the book" - # the real sentences, from the book - _tb="$(python3 - "$_bl" <<'PYTB' -import re, sys -src = open(sys.argv[1]).read() -m = re.search(r"_TEST_BLOCK_RE = re\.compile\((.*?), re\.I\)\n", src, re.S) -rx = re.compile(eval("(" + m.group(1) + ")"), re.I) -cases = [("The installation can be tested with the following commands:", True), - ("To test the results, issue: ninja test.", True), - ("To run the tests, issue: make check.", True), - ("The test suite requires a graphical session.", True), - ("This package does not come with a test suite.", False), - ("Now, as the root user:", False), - ("Build the module:", False), - ("Install WOFF2 by running the following commands:", False)] -bad = [t for t, want in cases if bool(rx.search(t)) != want] -print("BAD: " + "; ".join(bad) if bad else "OK") -PYTB -)" - [ "$_tb" = OK ] || _p="$_p;test-intro matching is wrong for: $_tb" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a test block is recognised however the book introduces it" -fi - -# ---- configure commands one sub-section down (1.14.262) -------------------- # -# Shadow's configure phase was prose and a bare `:` -- no /etc/pam.d/login, -# passwd, chage or the ten admin tools, so `passwd` said "pam_start() failed, -# error 26" and login "PAM Failure, aborting". The blocks are on the page one -# level down (div.configuration > div.sect3 > div.sect4 > pre.root), and the -# walk read only sect3's children. The fixture is the RENDERED book's shape, -# cut down. BLFS_STORE is fresh: the index cache is what hid three earlier -# extractor fixes, so this must not read one. -_p="" -if [ -f "$_bl" ]; then - _fx3="$T/fixconf"; rm -rf "$_fx3"; mkdir -p "$_fx3/store" - cat > "$_fx3/book.html" <<'FIXCONF' -<html><body> -<div class="titlepage"><div><div><h2 class="title"><a id="zzshadow"></a>zzshadow-4.2.0</h2></div></div></div><div class="sect1" lang="en"> -<div class="package" lang="en"><h3 class="sect2">Introduction to zzshadow</h3> -<div class="itemizedlist"><ul class="compact"><li class="listitem"><p>Download (HTTP): <a class="ulink" - href="https://example.org/zzshadow-4.2.0.tar.xz">https://example.org/zzshadow-4.2.0.tar.xz</a></p></li> -<li class="listitem"><p>Download MD5 sum: 00000000000000000000000000000000</p></li></ul></div></div> -<div class="installation" lang="en"><h3 class="sect2">Installation of zzshadow</h3> -<pre class="userinput"><kbd class="command">./configure --prefix=/usr && make</kbd></pre> -<p>Now, as the root user:</p> -<pre class="root"><kbd class="command">make install</kbd></pre></div> -<div class="configuration" lang="en"><h3 class="sect2">Configuring zzshadow</h3> -<div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title">Configuration Information</h4></div></div></div> -<p>Top level, as the root user:</p> -<pre class="root"><kbd class="command">touch /etc/zz-top-level</kbd></pre> -<div class="sect4"><div class="titlepage"><div><div><h5 class="title">'login'</h5></div></div></div><pre class="root"><kbd class="command">cat > /etc/pam.d/login << "EOF"</kbd> -<code class="literal">auth include system-auth</code> -EOF</pre></div> -<div class="sect4"><div class="titlepage"><div><div><h5 class="title">'chage'</h5></div></div></div><pre class="root"><kbd class="command">cat > /etc/pam.d/chage << "EOF"</kbd> -<code class="literal">auth sufficient pam_rootok.so</code> -EOF</pre></div> -<div class="sect4"><div class="titlepage"><div><div><h5 class="title">Other utilities</h5></div></div></div><pre class="root"><kbd class="command">for PROGRAM in chfn \ - useradd -do - install -v -m644 /etc/pam.d/chage /etc/pam.d/${PROGRAM} -done</kbd></pre></div> -</div></div></div></body></html> -FIXCONF - if BLFS_STORE="$_fx3/store" PKGUSR_BASE="$_fx3" PKGUSR_CONFIG=/dev/null \ - python3 "$_bl" --book-file "$_fx3/book.html" script zzshadow -o "$_fx3" \ - >"$_fx3/out" 2>&1; then - _cp="$(sed -n '/^configure_pkg()/,/^}/p' "$_fx3"/install_zzshadow-*)" - printf '%s' "$_cp" | grep -q '^touch /etc/zz-top-level' \ - || _p="$_p;a top-level configure block was lost" - printf '%s' "$_cp" | grep -q '^cat > /etc/pam.d/login' \ - || _p="$_p;a block inside a sub-section of the configuration is dropped" - printf '%s' "$_cp" | grep -q '^for PROGRAM in chfn' \ - || _p="$_p;the loop in the last sub-section is dropped" - printf '%s' "$_cp" | grep -q "^## 'login'" \ - || _p="$_p;a sub-section's title is not kept as a comment" - # the top level is unchanged: its own title was never emitted - printf '%s' "$_cp" | grep -q 'Configuration Information' \ - && _p="$_p;the top-level titlepage is now emitted (the top level changed)" - # THE SET, DERIVED. shadow-pam checks what the configure phase - # writes, read from this same function -- so run it on this output. - _spf="$(dirname "$LFS_TOOL")/stacks/shadow-pam.sh" - if [ -f "$_spf" ]; then - _set="$( (HELPER="bash $(dirname "$LFS_TOOL")/lfs-helper" - eval "$(sed -n '/^_book_pam_set() {/,/^}/p' "$_spf")" - _book_pam_set "$(ls "$_fx3"/install_zzshadow-*)") | tr '\n' ' ')" - [ "$_set" = "chage chfn login useradd " ] \ - || _p="$_p;the pam.d set read from configure_pkg is wrong: '$_set'" - fi - else - _p="$_p;the generator could not read a shadow-shaped page: $(tail -2 "$_fx3/out" | tr '\n' ' ')" - fi - # THE CACHE KEY INCLUDES THE CODE. It was book size+mtime+a hand-bumped - # number, so an edited extractor rendered from the old index and three - # fixes were reverted as "did not fire" (1.14.262). - grep -q '_build_id()}"$' "$_bl" \ - || _p="$_p;the index cache is not keyed on blfs's own build id" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "configure blocks in sub-sections are generated, and the cache cannot hide it" -fi - -# ---- the lease: hand over before, restore on failure, settle on success ---- # -# (1.14.263) Shadow's configure died on `/etc/pam.d/su: Permission denied`: -# a bash redirect no log parser knew, so the repair granted the DIRECTORY and -# stopped. The lease hands such files to the installer BEFORE the phase, -# puts them back if it fails, and applies setid.conf if it succeeds. -# Exercised with real chown on real accounts, in a sticky directory, without -# su: the phase is simulated by writing between lease and settle. -_p="" -_hlL="$(dirname "$LFS_TOOL")/lfs-helper" -if [ "$(id -u)" != 0 ]; then - skip "the lease (needs root for chown)" -elif [ -f "$_hlL" ]; then - _lz="$( - # NOT UNDER /tmp: script-targets ignores /tmp, /dev, /proc, /sys and - # /run on purpose, and $T lives in /tmp -- every configure target in - # this fixture was silently dropped until it moved. - R="$(mktemp -d /var/tmp/lfslease.XXXXXX)"; trap 'rm -rf "$R"' EXIT - mkdir -p "$R/sbin" "$R/etc" "$R/state/manifests" - # _setid_conf reads $PKGUSR_ETC; under `set -u` an unset one killed - # the lookup, no policy was found, and 4711 came out 711 - PKGUSR_ETC="$R/no-etc" - chmod 1777 "$R/sbin" "$R/etc" - say() { :; }; note() { :; }; warn() { :; }; detail() { :; }; ok() { :; } - unprefix_pkg_user() { echo "$1"; } - user_exists() { id "$1" >/dev/null 2>&1; } - # two real non-root accounts stand in for package users - _a="$(awk -F: '$3>0 && $3<1000 {print $1}' /etc/passwd | sed -n 1p)" - _b="$(awk -F: '$3>0 && $3<1000 {print $1}' /etc/passwd | sed -n 2p)" - pkg_users_list() { printf '%s\n%s\n' "$_a" "$_b"; } - STATE="$R/state"; MANIFESTS="$R/state/manifests"; LEASES="$R/state/leases" - for _f in script_write_targets _setid_conf _setid_policy_for _lease_dir \ - _lease_candidates lease_files lease_restore lease_settle \ - _setid_state _setid_apply; do - eval "$(_slice_fn "$_hlL" "$_f")" - done - printf 'old\n' > "$R/sbin/tool"; chown "root:$_a" "$R/sbin/tool"; chmod 4711 "$R/sbin/tool" - printf 'other\n' > "$R/etc/shared"; chown "$_b:$_b" "$R/etc/shared"; chmod 644 "$R/etc/shared" - printf 'secret\n' > "$R/etc/rootonly"; chmod 600 "$R/etc/rootonly" - printf '%s\n' "$R/sbin/tool" > "$MANIFESTS/$_a.files" - printf '%s | 4711 | root | -\n' "$R/sbin/tool" > "$R/setid.conf" - export SETID_CONF="$R/setid.conf" - printf 'configure_pkg() {\ncat > %s << "X"\nnew\nX\necho x >> %s\n}\n' \ - "$R/etc/shared" "$R/etc/rootonly" > "$R/script" - out="" - # 1. a failing phase: both leased files rewritten, then restored - lease_files "$_a" "$R/script" all - [ "$(stat -c %U "$R/sbin/tool")" = "$_a" ] || out="$out;the manifest's root file was not handed over" - [ "$(stat -c %U "$R/etc/shared")" = "$_a" ] || out="$out;another package's configure target was not handed over" - [ "$(stat -c %U "$R/etc/rootonly")" = root ] || out="$out;a root file no manifest claims was handed over" - printf 'broken\n' > "$R/sbin/tool"; printf 'broken\n' > "$R/etc/shared" - lease_restore "$_a" - [ "$(cat "$R/sbin/tool")" = old ] && [ "$(stat -c '%a %U:%G' "$R/sbin/tool")" = "4711 root:$_a" ] \ - || out="$out;a failed phase did not restore the root file ($(stat -c '%a %U:%G' "$R/sbin/tool"))" - [ "$(cat "$R/etc/shared")" = other ] && [ "$(stat -c %U "$R/etc/shared")" = "$_b" ] \ - || out="$out;a failed phase did not restore the other package's file" - [ -e "$LEASES/$_a" ] && out="$out;a restored lease left its ledger behind" - # 2. a succeeding phase: setid.conf applies, a rewrite belongs to the installer - lease_files "$_a" "$R/script" all - printf 'new\n' > "$R/sbin/tool"; printf 'new\n' > "$R/etc/shared" - lease_settle "$_a" - [ "$(stat -c '%a %U:%G' "$R/sbin/tool")" = "4711 root:$_a" ] \ - || out="$out;setid.conf was not applied after success ($(stat -c '%a %U:%G' "$R/sbin/tool"))" - [ "$(stat -c %U "$R/etc/shared")" = "$_a" ] || out="$out;a rewritten file did not stay with its installer" - # 3. a leased file the phase did not change goes back to its owner - chown "$_b:$_b" "$R/etc/shared" - lease_files "$_a" "$R/script" configure - lease_settle "$_a" - [ "$(stat -c %U "$R/etc/shared")" = "$_b" ] || out="$out;an unchanged file was not given back" - # 4. an interrupted lease is restored before the next one - lease_files "$_a" "$R/script" configure - printf 'crashed\n' > "$R/etc/shared" - lease_files "$_a" "$R/script" configure - [ "$(cat "$R/etc/shared")" = new ] || out="$out;an interrupted lease was not restored first" - lease_restore "$_a" - # 5. its OWN file, taken by another package, is reclaimed on install - printf 'mine\n' > "$R/etc/own"; chown "$_b:$_b" "$R/etc/own" - printf '%s\n%s\n' "$R/sbin/tool" "$R/etc/own" > "$MANIFESTS/$_a.files" - lease_files "$_a" "$R/script" install - [ "$(stat -c %U "$R/etc/own")" = "$_a" ] \ - || out="$out;a manifest file another package holds was not leased on install" - printf 'reinstalled\n' > "$R/etc/own" - lease_settle "$_a" - [ "$(stat -c %U "$R/etc/own")" = "$_a" ] \ - || out="$out;a package could not reclaim its own file" - # 7. a root file a manifest claims only by time window is NOT leased - # (1.14.278: LLVM's manifest held /etc/pam.d/su_u_user) - printf 'x\n' > "$R/etc/rootonly"; chown root:root "$R/etc/rootonly"; chmod 755 "$R/etc/rootonly" - printf '%s\n' "$R/etc/rootonly" > "$MANIFESTS/$_a.files" - lease_files "$_a" "$R/script" install - [ "$(stat -c %U "$R/etc/rootonly")" = root ] \ - || out="$out;a root file that is neither setid nor configured was leased" - lease_restore "$_a" 2>/dev/null - # 6. the earlier-stages claim must not strip a setuid bit (1.14.265) - eval "$(_slice_fn "$_hlL" _claim_candidates)" - printf 'x\n' > "$R/sbin/suid"; chmod 4711 "$R/sbin/suid" - printf 'x\n' > "$R/sbin/plain"; chmod 755 "$R/sbin/plain" - printf '%s\n%s\n' "$R/sbin/suid" "$R/sbin/plain" > "$R/claim.files" - _cc="$(LFS_HOST_MOUNT="" SNAP_ROOT="" _claim_candidates "$_a" "$R/claim.files")" - printf '%s\n' "$_cc" | grep -qxF "$R/sbin/suid" \ - && out="$out;the earlier-stages claim takes a setuid root file (chown strips the bit)" - printf '%s\n' "$_cc" | grep -qxF "$R/sbin/plain" \ - || out="$out;the earlier-stages claim no longer takes a plain root file" - rm -rf "$R" - echo "${out:-OK}" )" - [ "$_lz" = OK ] || _p="$_p${_lz#OK}" - # it is wired into the ONE runner, and settles after the log check - _rpa="$(_slice_fn "$_hlL" run_phase_as)" - printf '%s' "$_rpa" | grep -q 'lease_files "$owner" "$script" "$phase"' \ - || _p="$_p;run_phase_as does not lease" - _i_sw="$(printf '%s\n' "$_rpa" | grep -n '_swallowed=' | head -1 | cut -d: -f1)" - _i_st="$(printf '%s\n' "$_rpa" | grep -n 'lease_settle "$owner"' | head -1 | cut -d: -f1)" - [ -n "$_i_sw" ] && [ -n "$_i_st" ] && [ "$_i_sw" -lt "$_i_st" ] \ - || _p="$_p;the lease settles before the log is checked for a swallowed failure" - grep -q ' script-targets) shift; cmd_script_targets' "$_hlL" \ - || _p="$_p;script-targets is not dispatched" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -elif [ "$(id -u)" = 0 ]; then - ok "a leased file is restored on failure and settled on success" -fi - -# ---- one package, another book (1.14.237) ---------------------------------- # -# The systemd book's linux-pam page lists systemd as REQUIRED -- which is -# how `install linux-pam --recursive` came to plan Systemd-261.2 on a SysV -# machine. The 12.4 SysV book's page has NO Required section at all (only -# Shadow, as a recommendation): verified by generating it, `required= -# ('Shadow-4.18.0')`. So a package can be pointed at another book. -_p="" -if [ -f "$pm_src" ]; then - _bf="$(_slice_def "$pm_src" _blfs_book_flags)" - printf '%s' "$_bf" | grep -q '_book_override' \ - || _p="$_p;a package cannot be read from a different book" - _bov="$(_slice_def "$pm_src" _book_override)" - printf '%s' "$_bov" | grep -q '_machine_opts_for' \ - || _p="$_p;the override does not come from machine.conf" - # AND THE CASE IT IS ACTUALLY FOR: shadow 4.20 auto-detects logind, the - # book's page (written for a systemd host) passes no switch, and - # configure stops with "libsystemd not found ... use --disable-logind". - # One switch beats four versions back (1.14.239). - _mc6="$(dirname "$LFS_TOOL")/stacks/machine.conf" - grep -A2 '^\[shadow\]' "$_mc6" 2>/dev/null | grep -q -- '--disable-logind' \ - || _p="$_p;shadow will stop at configure for want of libsystemd" - grep -B10 '^\[shadow\]' "$_mc6" 2>/dev/null | grep -q 'libsystemd not found' \ - || _p="$_p;nothing records the error that switch answers" - # A KEY THAT ADDRESSES THE TOOL MUST NOT REACH THE BUILD LINE. - # `[linux-pam] book = ...` went onto the meson line as -Dbook=... and - # meson stopped with `ERROR: Unknown option: "book"` (1.14.240). - # _warn_unknown_meson_opts cannot catch it: at script-generation time - # the source is not unpacked and there is no meson.options to check. - _am2="$(_slice_def "$pm_src" apply_machine_opts)" - printf '%s' "$_am2" | grep -q 'for _reserved in' \ - || _p="$_p;a tool-level machine.conf key can still reach the build line" - _bko="$(python3 - "$pm_src" "$T" 2>&1 <<'PYBK' -import sys, types, os, io, contextlib -d = os.path.join(sys.argv[2], "bk"); os.makedirs(d, exist_ok=True) -open(os.path.join(d, "machine.conf"), "w").write( - "[zzpam]\nbook = /somewhere/BOOK.html\njobs = 4\n") -os.environ.update(PM_MACHINE_CONF=os.path.join(d, "machine.conf"), - PM_PACKAGE_FIXES=os.path.join(d, "none.conf"), - PKGUSR_BASE=d, PKGUSR_CONFIG="/dev/null") -sp = os.path.join(d, "install_ZzPam-1.0") -open(sp, "w").write("build_pkg() {\nmeson setup .. --prefix=/usr &&\nninja\n}\n") -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -buf = io.StringIO() -with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): - mod.apply_machine_opts("zzpam", sp) -body = open(sp).read() -bad = [] -if "-Dbook" in body or "book=" in body: - bad.append("book reached the build line") -if "ninja -j4" not in body: - bad.append("jobs no longer applied") -print("BAD: " + "; ".join(bad) if bad else "OK") -PYBK -)" - _bko="$(printf '%s\n' "$_bko" | tail -1)" - [ "$_bko" = OK ] || _p="$_p;reserved machine.conf keys: $_bko" - - # NOT A MERGE, and the reason has to be written down - grep -q 'NOT a merge' "$pm_src" \ - || _p="$_p;nothing says why two pages are not merged" - grep -q 'NOT a merge' "$(dirname "$LFS_TOOL")/stacks/machine.conf" \ - || _p="$_p;machine.conf does not explain the whole-page rule" - _boo="$(PM_MACHINE_CONF=$T/bo.conf PM_PACKAGE_FIXES=$T/bo-none.conf \ - PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYBO' -import sys, types, os -open(os.environ["PM_MACHINE_CONF"], "w").write( - "[linux-pam]\nbook = /usr/share/blfs/books/zz-sysv.html\n") -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -one = mod._blfs_book_flags("linux-pam") -other = mod._blfs_book_flags("webkitgtk") -print("OK" if one and "zz-sysv.html" in one[-1] and not other - else "BAD %r %r" % (one, other)) -PYBO -)" - _boo="$(printf '%s\n' "$_boo" | tail -1)" - [ "$_boo" = OK ] || _p="$_p;the per-package book override is wrong: $_boo" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "one package can be read from another book, whole pages only" -fi - -# ---- cmake will not overwrite a symlink (1.14.242) ------------------------- # -# tiff's round 1 handed /usr/lib/pkgconfig/libtiff-4.pc from p_libtiff to -# p_tiff and the retry then died on -# file INSTALL cannot duplicate symlink ... at /usr/lib/libtiff.so.6 -# because: File exists -# Ownership was not the problem: cmake refuses to REPLACE a symlink at all, -# so handing it over changes nothing -- it has to go. And the paths in that -# message are NOT quoted, unlike the copy-file one, which is why the -# extractor saw nothing. -_p="" -_hl12="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl12" ]; then - _tk="$(_slice_fn "$_hl12" _take_cmake_dest)" - printf '%s' "$_tk" | grep -q 'duplicate symlink' \ - || _p="$_p;a symlink cmake refuses to replace is not recognised" - printf '%s' "$_tk" | grep -q 'rm -f "$dest"' \ - || _p="$_p;it hands the symlink over instead of removing it" - printf '%s' "$_tk" | grep -q 'root|""' \ - || _p="$_p;root's symlink could be removed" - # ROOT'S FILES ARE NEVER TAKEN HERE (1.14.263). 1.14.261 handed a - # manifest-claimed root file back AFTER the install failed on it; the - # lease now does that BEFORE the phase, and restores it on failure. - printf '%s' "$_tk" | grep -q "Operation not permitted: '" \ - || _p="$_p;the gate does not know meson's wording" - printf '%s' "$_tk" | sed -n '/root|"")/,/;;/p' | grep -q 'chown' \ - && _p="$_p;the repair loop still takes root's files" - _tbo="$( - R="$T/tb"; rm -rf "$R"; mkdir -p "$R/sbin" "$R/man" - say() { :; }; note() { :; }; warn() { :; } - unprefix_pkg_user() { echo "${1#p_}"; } - MANIFESTS="$R/man" - : > "$R/sbin/ours" - _asuser="$(awk -F: '$3>0 && $3<1000 {print $1; exit}' /etc/passwd)" - [ -n "$_asuser" ] || _asuser=nobody - printf '%s\n' "$R/sbin/ours" > "$MANIFESTS/$_asuser.files" - printf "Operation not permitted: '%s'\n" "$R/sbin/ours" > "$R/log" - eval "$(_slice_fn "$_hl12" _take_cmake_dest)" - _take_cmake_dest "$R/log" "$_asuser" >/dev/null 2>&1 - stat -c '%U' "$R/sbin/ours" )" - [ "$_tbo" = root ] || _p="$_p;the repair loop took a root file ($_tbo)" - # the unquoted form must extract - printf '%s\n' 'CMake Error at x/cmake_install.cmake:57 (file):' \ - ' file INSTALL cannot duplicate symlink' '' ' /src/b/libtiff.so.6' \ - '' ' at' '' ' /usr/lib/libtiff.so.6' '' ' because: File exists' \ - > "$T/sym.log" - _got="$(tr '\n' ' ' < "$T/sym.log" \ - | grep -aoE 'cannot duplicate symlink[[:space:]]+[^[:space:]]+[[:space:]]+at[[:space:]]+[^[:space:]]+' \ - | awk '{print $NF}')" - [ "$_got" = /usr/lib/libtiff.so.6 ] \ - || _p="$_p;the unquoted destination parses to '$_got'" -fi - # AN ENTRY IS ONLY DONE IF IT IS THERE AFTERWARDS -- EVERY KIND. The - # check covered git and tar only, so poppler (a `book` entry) came out - # `done (stack cache)` while pkg-config found no poppler at all, and - # every later run skipped it -- leaving zathura-pdf-poppler failing on a - # dependency the plan believed it had (1.14.246). - _cs="$(_slice_def "$pm_src" cmd_stack)" - printf '%s' "$_cs" | grep -q 'elif e\["kind"\] == "book"' \ - || _p="$_p;a book entry is recorded done without being checked" - printf '%s' "$_cs" | grep -q '_installed_now' \ - || _p="$_p;the done-check is still per-kind" - # and the reason zathura's backend needs poppler at all must be asked for - grep -A14 '^\[poppler\]' "$(dirname "$LFS_TOOL")/stacks/machine.conf" \ - 2>/dev/null | grep -q 'ENABLE_GLIB *= *ON' \ - || _p="$_p;poppler may build without the glib frontend the chain needs" - - # A PACKAGE PROMOTED OUT OF avoid.stack BRINGS ITS AVOIDED CHILDREN - # BACK INTO QUESTION. poppler became wanted for zathura's backend, and - # nss, boost and gpgmepp are all in avoid.stack AS POPPLER'S CHILDREN -- - # so each stops configure in turn, naming its own switch (1.14.243). - _mc7="$(dirname "$LFS_TOOL")/stacks/machine.conf" - _av7="$(dirname "$LFS_TOOL")/stacks/avoid.stack" - # FOUR soft-mandatory switches, found one configure at a time: NSS3, - # GPGME, BOOST, then QT6 (1.14.247). qt6 is avoided and poppler's page - # passes only ENABLE_QT5=OFF, because a BLFS system is expected to have - # qt6. The comment names the grep that lists them all at once. - for _sw in ENABLE_NSS3 ENABLE_GPGME ENABLE_BOOST ENABLE_QT6; do - grep -A14 '^\[poppler\]' "$_mc7" 2>/dev/null | grep -q "$_sw *= *OFF" \ - || _p="$_p;poppler will stop at configure for $_sw" - done - grep -B14 '^\[poppler\]' "$_mc7" 2>/dev/null | grep -q 'OBSERVED' \ - || _p="$_p;nothing separates the observed failure from the anticipated ones" - grep -A26 '^\[poppler\]' "$_mc7" 2>/dev/null \ - | grep -q 'find_soft_mandatory_package' \ - || _p="$_p;nothing says how to list poppler's soft-mandatory switches at once" - for _c in nss boost gpgmepp; do - grep -qE "^avoid +$_c" "$_av7" \ - || _p="$_p;$_c is no longer excluded, so poppler would build it" - done - -# AND THE SAME LIBRARY UNDER TWO NAMES. p_libtiff and p_tiff are the same -# upstream package: 12.4 titles the page libtiff, 13.1 titles it tiff. Two -# accounts own two copies of one library, and whichever installed last wins -# every file. Reported, not fixed: `packagemanager which-package -# /usr/lib/libtiff.so.6` says who holds it today. -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a symlink another package owns is removed, not handed over" -fi - -# ---- the version is what FINISHED (1.14.236) ------------------------------- # -# install_last is staged into the package home BEFORE the build runs, and -# the installed version was read out of it -- so webkitgtk 2.52.5 died at -# configure and `update webkitgtk` then answered "already up to date -# (WebKitGTK-2.52.5)" while /usr/lib/libwebkitgtk-6.0.so.4.13.8 on disk was -# still 2.50.5's. A failed build had advanced the version. -_p="" -_hlv="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$pm_src" ] && [ -f "$_hlv" ]; then - grep -q '.pkgusr-installed' "$pm_src" \ - || _p="$_p;the version still comes from the staged script" - grep -q '.pkgusr-installed' "$_hlv" \ - || _p="$_p;nothing records which version finished installing" - # ...AND THE VARIABLE MUST EXIST IN THAT SCOPE. `$home` is a local of - # the install function, not of the one that prints "done (all phases - # complete)", so with set -u the block died with - # lfs-helper: line 7345: home: unbound variable - # and the marker -- the entire point of it -- was never written - # (1.14.250). - grep -A14 'WHAT IS INSTALLED IS WHAT FINISHED' "$_hlv" \ - | grep -q '"\$home/install_last"' \ - && _p="$_p;the marker still reads a variable that is not in scope" - grep -A20 'WHAT IS INSTALLED IS WHAT FINISHED' "$_hlv" \ - | grep -q 'pkgusr_home_for "\$name"' \ - || _p="$_p;the marker does not resolve the home the way other callers do" - # the whole tool must survive set -u: an unbound variable anywhere in a - # success path is a silent no-op at best - bash -n "$_hlv" || _p="$_p;lfs-helper does not parse" - # written ONLY after every phase succeeded - # the marker line and the success line must be in the same few lines: - # a marker written anywhere else is a marker written on failure too - grep -A34 'WHAT IS INSTALLED IS WHAT FINISHED' "$_hlv" \ - | grep -q 'all phases complete' \ - || _p="$_p;the marker is not written at the success point" - _mk="$T/mk"; rm -rf "$_mk"; mkdir -p "$_mk/pkgusr/p_zzver" - printf "name='zzver'\nname_version='ZzVer-2.0'\n" \ - > "$_mk/pkgusr/p_zzver/install_last" - echo x > "$_mk/pkgusr/p_zzver/pkg.lst" - _vo="$(PKGUSR_BASE=$_mk PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYVER' -import sys, types, os -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -home = os.path.join(os.environ["PKGUSR_BASE"], "pkgusr", "p_zzver") -a = mod.gather_user("zzver").version # no marker: staged script -open(os.path.join(home, ".pkgusr-installed"), "w").write("ZzVer-1.0") -b = mod.gather_user("zzver").version # marker wins -print("OK" if a == "ZzVer-2.0" and b == "ZzVer-1.0" else "BAD %r %r" % (a, b)) -PYVER -)" - _vo="$(printf '%s\n' "$_vo" | tail -1)" - [ "$_vo" = OK ] || _p="$_p;the installed version is read wrongly: $_vo" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "the recorded version is the one that finished, not the one staged" -fi - -# ---- one scalar, two paths (1.14.230) -------------------------------------- # -# `info opus` said "unvalidated: missing: dir /usr/include/opus -# /usr/share/doc/opus-1.6.1" with /usr/include/opus sitting right there. -# The book writes "Installed Directories: A and B", both landed in ONE -# `installed_directory=`, and the check looked for a directory literally -# called "A B". And the doc half was never going to exist: it needs Doxygen -# and texlive, which this system deliberately does not have. -_p="" -if [ -f "$pm_src" ]; then - _it="$(_slice_def "$pm_src" install_targets)" - printf '%s' "$_it" | grep -q 'def _paths' \ - || _p="$_p;a field with two paths in it is treated as one path" - _ito="$(python3 - "$pm_src" "$T" <<'PYIT' -import sys, types, os -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -bad = [] -_, _, dirs = mod.install_targets( - {"name": "opus", - "installed_directory": "/usr/include/opus /usr/share/doc/opus-1.6.1"}) -if dirs != ["/usr/include/opus", "/usr/share/doc/opus-1.6.1"]: - bad.append("two directories parse as %r" % (dirs,)) -progs, _, _ = mod.install_targets( - {"name": "x", "installed_programs": ["gst-launch-1.0 gst-inspect-1.0"]}) -if progs != ["gst-launch-1.0", "gst-inspect-1.0"]: - bad.append("two programs parse as %r" % (progs,)) -# a missing DOC dir is documentation, not a fault; a missing real one is -base = os.path.join(sys.argv[2], "op") -for name, extra, want in (("docsgone", "/nope-doc/share/doc/x-1.0", "installed"), - ("realgone", "/nope-real", "unvalidated")): - home = os.path.join(base, "pkgusr", "p_" + name) - os.makedirs(os.path.join(base, "inc"), exist_ok=True) - os.makedirs(home, exist_ok=True) - open(os.path.join(home, "install_last"), "w").write( - "name='%s'\nname_version='%s-1.0'\ninstalled_directory='%s %s'\n" - % (name, name, os.path.join(base, "inc"), extra)) - open(os.path.join(home, "pkg.lst"), "w").write(os.path.join(base, "inc") + "\n") -os.environ["PKGUSR_BASE"] = base -os.environ["PKGUSR_CONFIG"] = "/dev/null" -mod2 = types.ModuleType("pm2"); mod2.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod2.__dict__) -for name, want in (("docsgone", "installed"), ("realgone", "unvalidated")): - u = mod2.gather_user(name) - if not u.has_user: - continue # no account here; the parse tests stand - st, why = mod2.assess(u, None, run_cmd=False) - if st != want: - bad.append("%s -> %s (%s), wanted %s" % (name, st, why, want)) -print("BAD: " + "; ".join(bad) if bad else "OK") -PYIT -)" - [ "$_ito" = OK ] || _p="$_p;install target parsing: $_ito" - printf '%s' "$(_slice_def "$pm_src" assess)" | grep -q 'share/doc/' \ - || _p="$_p;a doc directory that was never built counts as missing" - # `why` EXPLAINS A VERDICT THAT NEEDS EXPLAINING. For an installed - # package it printed the routine sentence every time, above the very - # fields it describes, so the one line that ever matters was - # indistinguishable from noise (1.14.231). - grep -q '_routine = (' "$pm_src" \ - || _p="$_p;a routine verdict is still printed as a reason" - # _slice_def stops at the first line starting with a letter, and - # print_user_details has one before this point -- grep the file - grep -q '_label = "note" if state == "installed"' "$pm_src" \ - || _p="$_p;an installed package's note is still labelled why" - grep -q 'docs not built (no doxygen/texlive here)' "$pm_src" \ - || _p="$_p;it does not say WHY the doc directory is absent" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "two paths in one field parse as two, and absent docs are not a fault" -fi - -# ---- one quote character, three symptoms (1.14.220) ------------------------ # -# `_parse_bash_vars` read only key="value". The BOOK-generated scripts -# write name_version='Rustc-1.93.1' in single quotes, so name_version was -# never parsed for any package built from the book -- and that is where the -# installed version comes from. Hence: `info webkitgtk` showed uid, home, -# 1727 files and every dependency but not which webkitgtk; `update` filed it -# under "cannot judge"; late-deps reported 28 packages with "no recorded -# version". -_p="" -if [ -f "$pm_src" ]; then - _pv="$(python3 - "$pm_src" <<'PYPV' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -cases = [("name_version='Rustc-1.93.1'\n", "Rustc-1.93.1"), - ('name_version="openh264-2.6.0"\n', "openh264-2.6.0"), - ("name_version='WebKitGTK-2.50.2' # from the book\n", "WebKitGTK-2.50.2"), - ('name_version=""\n', "")] -bad = [t.strip() for t, w in cases - if mod._parse_bash_vars(t, ("name_version",), ()).get("name_version") != w] -print("BAD: " + "; ".join(bad) if bad else "OK") -PYPV -)" - [ "$_pv" = OK ] || _p="$_p;a scalar is misparsed: $_pv" - # THE BOOK PUTS cmake AFTER `&&`, and the detector anchored to the start - # of a line: webkitgtk's page writes - # mkdir -vp build && cd build && cmake -D CMAKE_BUILD_TYPE=Release \\ - # so all seven [webkitgtk] options were reported as "NOT applied" on the - # one package where they matter most (1.14.222). - _hc="$(_slice_def "$pm_src" _has_command)" - [ -n "$_hc" ] || _p="$_p;whether a build command exists is still a bare regex" - grep -q '_has_command(body, "cmake")' "$pm_src" \ - || _p="$_p;the cmake detector does not use it" - grep -q 'r"\^\\s\*cmake' "$pm_src" \ - && _p="$_p;the old line-anchored cmake test is still there" - _hco="$(python3 - "$pm_src" <<'PYHC' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -cases = [("mkdir -vp build && cd build && cmake -D X=1 ..\n", "cmake", True), - ("mkdir -vp build &&\ncd build &&\ncmake -D X=1 ..\n", "cmake", True), - ("## NOTE: add it to the cmake command\n:\n", "cmake", False), - ("mkdir build && cd build && meson setup --prefix=/usr ..\n", - "meson setup", True), - ("./configure --prefix=/usr\nmake\n", "cmake", False)] -bad = [c[0].splitlines()[0] for c in cases - if mod._has_command(c[0], c[1]) != c[2]] -b = "## NOTE: add it to the cmake command\nmkdir -p build && cd build && cmake -D P=G ..\n" -if "-DUSE_WOFF2=ON" not in mod._insert_on_command(b, "cmake", "-DUSE_WOFF2=ON").splitlines()[1]: - bad.append("insertion missed the real command line") -print("BAD: " + "; ".join(bad) if bad else "OK") -PYHC -)" - [ "$_hco" = OK ] || _p="$_p;command detection is wrong for: $_hco" - # AND THE VERSION MAP IS CACHED BETWEEN RUNS. `blfs versions` re-reads - # the whole book -- eighteen seconds -- and `info` paid it per command. - _bva="$(_slice_def "$pm_src" book_versions_all)" - printf '%s' "$_bva" | grep -q 'book-versions.json\|_versions_cache_path' \ - || _p="$_p;the book version map is rebuilt on every command" - printf '%s' "$_bva" | grep -q 'st_mtime' \ - || _p="$_p;the cache is not keyed to the book, so it can outlive it" - printf '%s' "$_bva" | grep -q 'did not read back' \ - || _p="$_p;a cache that fails to load would be silent" - - # THE PREFIX IS NOT PART OF THE BOOK'S NAME. `info webkitgtk` asked the - # version map for `p_webkitgtk`, missed, and fell through to a `blfs - # debug` SUBPROCESS that reads the book cache to answer one lookup and - # then fails -- the pause after "looking up p_webkitgtk in the book", - # and the reason no book version was shown (1.14.221). - _bvf="$(_slice_def "$pm_src" book_version)" - printf '%s' "$_bvf" | grep -q 'pkgusr_prefix()' \ - || _p="$_p;the account prefix is not stripped before the book lookup" - printf '%s' "$_bvf" | grep -q 'run_blfs(\["debug", _tries\[-1\]\])' \ - || _p="$_p;the fallback still asks with the prefixed name" - # no subprocess for a plain info, and the book version is shown - if [ -d "$T/iw/pkgusr/p_zzwk" ] || mkdir -p "$T/iw/pkgusr/p_zzwk"; then - printf "name='zzwk'\nname_version='ZzWk-1.2.3'\n" \ - > "$T/iw/pkgusr/p_zzwk/install_last" - echo x > "$T/iw/pkgusr/p_zzwk/pkg.lst" - _io2="$(PKGUSR_BASE=$T/iw PKGUSR_CONFIG=/dev/null \ - timeout 120 python3 "$pm_src" info zzwk 2>&1)" - printf '%s' "$_io2" | grep -q 'looking up p_' \ - && _p="$_p;info still spawns a book lookup for a prefixed name" - printf '%s' "$_io2" | grep -q 'version : 1.2.3' \ - || _p="$_p;info does not show the bare version number" - fi - # the label column: {colors.normal:<9} padded the ESCAPE SEQUENCE - printf '%s' "$_pd" | grep -q "colors.normal:<9" \ - && _p="$_p;the why label pads a colour code instead of the word" - # and info must PRINT the version, both the installed one and the book's - _pd="$(_slice_def "$pm_src" print_user_details)" - printf '%s' "$_pd" | grep -q '("version"' \ - || _p="$_p;info does not print the installed version as a field" - printf '%s' "$_pd" | grep -q '("book"' \ - || _p="$_p;info does not say what the book carries" - printf '%s' "$_pd" | grep -q 'name_version' \ - || _p="$_p;info has no answer when the version was not recorded" - # ...and update's "cannot judge" must name the book version beside each - grep -q 'no_record.append("%s (book: %s)"' "$pm_src" \ - || _p="$_p;'no recorded version' still does not say what the book has" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a version is parsed in either quote style, and both versions are shown" -fi - -# ---- application users from a file (1.14.217) ------------------------------ # -# The accounts a browser runs as were a command line per account, and the -# two files that list them all -- /etc/pam.d/su_u_user and the SHARED_* -# variables -- were hand-kept. Both are now derived from app-users.conf, -# and neither may clobber a hand-edited file. -_p="" -_au="$(dirname "$LFS_TOOL")/stacks/app-users.sh" -_ac2="$(dirname "$LFS_TOOL")/stacks/app-users.conf" -if [ -f "$_au" ] && [ -f "$_ac2" ]; then - bash -n "$_au" || _p="$_p;app-users.sh does not parse" - grep -qE '^cfg +app-users' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ - || _p="$_p;nothing runs the app-users step" - # ONE NAMESPACE FOR THIS SYSTEM'S FACTS, AND IT MUST BE INERT. - # PKGUSR_BASE, PKGUSR_CONFIG and PKGUSR_LIB are READ by the tools as - # overrides, so exporting that namespace system-wide would reconfigure - # every tool in every login shell. PKGUSR_SYS_* is consumed by nothing - # (1.14.219). - _se3="$(_slice_def "$pm_src" _system_env_values)" - [ -n "$_se3" ] || _p="$_p;there is no place that states this system's facts" - printf '%s' "$_se3" | grep -q 'PKGUSR_SYS_MAIN_USER' \ - || _p="$_p;the main user is not among the exported facts" - for _bad in PKGUSR_BASE PKGUSR_CONFIG PKGUSR_LIB; do - printf '%s' "$_se3" | grep -q "\"$_bad\"" \ - && _p="$_p;$_bad is exported, which reconfigures the tools" - done - _ces="$(_slice_def "$pm_src" cmd_env_system)" - printf '%s' "$_ces" | grep -q 'SYSTEM_ENV_MARKER not in had' \ - || _p="$_p;a profile fragment that is not ours would be overwritten" - printf '%s' "$_ces" | grep -q 'getattr(args, "run", False)' \ - || _p="$_p;env --system writes without being asked to" - # the real command prints the facts and names what it does not know - _seo="$(timeout 60 python3 "$pm_src" env --system 2>&1)" - printf '%s' "$_seo" | grep -q 'export PKGUSR_SYS_USER_PREFIX=' \ - || _p="$_p;env --system does not print the prefixes it knows" - printf '%s' "$_seo" | grep -q 'dry run' \ - || _p="$_p;env --system is not a dry run by default" - # and the app-users step asks for them rather than guessing - grep -q 'PKGUSR_SYS_MAIN_USER' "$_au" \ - || _p="$_p;app-users.sh still guesses the main user first" - - # THE PREFIX IS CONFIGURABLE and must not be hard-coded - grep -q 'user-prefix' "$_au" \ - || _p="$_p;the account prefix is assumed rather than read" - # the quoted example in the comment is not code: skip comment lines - grep -vE '^\s*#' "$_au" | grep -qE '"u_[a-z]' \ - && _p="$_p;a u_ prefix is hard-coded in app-users.sh" - # IT IS A SCRIPT, NOT A CONFIG FILE. /etc/pam.d/su_u_user starts with - # `#!/bin/bash` and is RUN by pam_exec.so; written 644 it cannot be - # executed, and `su - u_claude` just asks for a password with nothing in - # the logs to say why (1.14.232). - [ "$(grep -c 'chmod 755 "$PAM_FILE"' "$_au")" -ge 2 ] \ - || _p="$_p;the PAM script is not executable after both write and merge" - grep -q 'pam_exec.so quiet' "$_au" \ - || _p="$_p;nothing says how pam is told to run the script" - # AND THE LINE IS ADDED, with a backup, above the other auth lines -- - # `sufficient` succeeds the stack immediately, so after them it would - # never run for the accounts it exists for. On a machine with no - # Linux-PAM (avoid.stack avoids it) the step says so instead of writing - # into a file pam never reads (1.14.233). - grep -q 'cp -p "$PAM_SU" "$bak"' "$_au" \ - || _p="$_p;/etc/pam.d/su is edited without a backup" - grep -q 'Linux-PAM is not installed here' "$_au" \ - || _p="$_p;a machine with no pam gets a line it will never read" - grep -q 'restoring the backup' "$_au" \ - || _p="$_p;a failed edit of /etc/pam.d/su is not undone" - _psu="$( - R="$T/au8"; rm -rf "$R"; mkdir -p "$R/etc/pam.d" "$R/etc/profile.d" - { echo '#!/bin/sh'; echo 'case "$1 $2" in' - echo ' "config --show") case "$3" in main-user) echo t;; user-prefix) echo u_;; esac; exit 0;;' - echo ' "user setup"|"user create"|"user --help"|"env --system") exit 0;;' - echo 'esac'; echo 'exit 2'; } > "$R/pm"; chmod 755 "$R/pm" - printf 'name|app|share|dir|desktop|launcher|groups\nzz1|/bin/true|no|no|no|no|-\n' > "$R/conf" - printf '%s\n' '# Begin /etc/pam.d/su' 'auth sufficient pam_rootok.so' \ - 'auth required pam_unix.so' 'account required pam_unix.so' \ - > "$R/etc/pam.d/su" - APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ - APP_USERS_PAM_SU="$R/etc/pam.d/su" APP_USERS_PROFILE_D="$R/etc/profile.d" \ - APP_USERS_PM="$R/pm" bash "$_au" >/dev/null 2>&1 - head -2 "$R/etc/pam.d/su" | tail -1 | grep -c pam_exec )" - [ "${_psu:-0}" = 1 ] \ - || _p="$_p;the pam_exec line is not the first auth line in /etc/pam.d/su" - # (the step EDITS /etc/pam.d/su now, with a backup -- the old assertion - # asked for the refusal message it no longer prints) - grep -q 'if su is broken, put the backup back' "$_au" \ - || _p="$_p;it does not say how to undo the /etc/pam.d/su change" - _pmo="$( - R="$T/au7"; rm -rf "$R"; mkdir -p "$R/etc/pam.d" "$R/etc/profile.d" - { echo '#!/bin/sh'; echo 'case "$1 $2" in' - echo ' "config --show") case "$3" in main-user) echo t;; user-prefix) echo u_;; esac; exit 0;;' - echo ' "user setup"|"user create"|"user --help"|"env --system") exit 0;;' - echo 'esac'; echo 'exit 2'; } > "$R/pm"; chmod 755 "$R/pm" - printf 'name|app|share|dir|desktop|launcher|groups\nzz1|/bin/true|no|no|no|no|-\n' > "$R/conf" - APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ - APP_USERS_PROFILE_D="$R/etc/profile.d" APP_USERS_PM="$R/pm" \ - bash "$_au" >/dev/null 2>&1 - _m1="$(stat -c %a "$R/etc/pam.d/su_u_user" 2>/dev/null)" - chmod 644 "$R/etc/pam.d/su_u_user" 2>/dev/null - printf 'name|app|share|dir|desktop|launcher|groups\nzz1|/bin/true|no|no|no|no|-\nzz2|/bin/true|no|no|no|no|-\n' > "$R/conf" - APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ - APP_USERS_PROFILE_D="$R/etc/profile.d" APP_USERS_PM="$R/pm" \ - bash "$_au" >/dev/null 2>&1 - echo "$_m1 $(stat -c %a "$R/etc/pam.d/su_u_user" 2>/dev/null)" )" - [ "$_pmo" = "755 755" ] \ - || _p="$_p;PAM script modes are '$_pmo', wanted '755 755'" - - # A HAND-EDITED FILE IS BACKED UP AND MERGED, NEVER OVERWRITTEN - grep -q 'cp -p "$PAM_FILE" "$bak"' "$_au" \ - || _p="$_p;the PAM file is changed without a backup" - grep -q 'pam_has' "$_au" \ - || _p="$_p;accounts already in the PAM file would be duplicated" - grep -q 'left alone' "$_au" \ - || _p="$_p;a profile fragment that is not ours would be overwritten" - # `user setup` NEEDS THE ACCOUNT TO EXIST. All three lines of - # app-users.conf answered "u_browser-mini: no such user. Create it - # first" -- which makes a file that DECLARES accounts useless for the - # one job it has. `user create` does both (1.14.231). - grep -q 'set -- user create' "$_au" \ - || _p="$_p;an account that does not exist yet can never be created" - grep -q 'id "$PREFIX$name" >/dev/null' "$_au" \ - || _p="$_p;it does not check whether the account exists first" - _cro="$( - R="$T/au4"; rm -rf "$R"; mkdir -p "$R/etc/pam.d" "$R/etc/profile.d" - { - echo '#!/bin/sh' - echo 'case "$1 $2" in' - echo ' "config --show") case "$3" in main-user) echo tester;; user-prefix) echo u_;; esac; exit 0;;' - echo ' "user setup") echo SETUP; exit 0;;' - echo ' "user create") echo CREATE; exit 0;;' - echo ' "user --help"|"env --system") exit 0;;' - echo 'esac' - echo 'exit 2' - } > "$R/pm"; chmod 755 "$R/pm" - printf 'name|app|share|dir|desktop|launcher|groups\nzz-no-such-acct|/bin/true|no|no|no|no|-\n' > "$R/conf" - APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ - APP_USERS_PROFILE_D="$R/etc/profile.d" APP_USERS_PM="$R/pm" \ - bash "$_au" 2>&1 | grep -cE '^CREATE$' )" - [ "${_cro:-0}" -ge 1 ] \ - || _p="$_p;a declared account that does not exist is not created" - - # THE COMMAND IS `packagemanager USER setup`. The first version called - # `packagemanager setup`, which is an ALIAS OF BOOTSTRAP -- so it did not - # fail as an unknown command, it reached bootstrap's parser and said - # "unrecognized arguments", three times. The stub in the first test said - # yes to anything, which is why the test passed (1.14.218). - grep -q 'set -- user setup' "$_au" \ - || _p="$_p;the setup command is missing its 'user' subcommand" - grep -q '$PM user setup --help' "$_au" \ - || _p="$_p;it does not check the command exists before the loop" - # ...and the stub below REFUSES anything that is not a real subcommand - # path, so a wrong command fails the test instead of passing it - # and the merge, for real: an existing file with one of the accounts - _pr="$T/au"; rm -rf "$_pr"; mkdir -p "$_pr/etc/pam.d" "$_pr/etc/profile.d" - { - echo '#!/bin/sh' - echo 'case "$1 $2" in' - echo ' "config --show") case "$3" in main-user) echo tester;; user-prefix) echo u_;; esac; exit 0;;' - echo ' "user setup") exit 0;;' - echo ' "user create") exit 0;;' - echo ' "user --help") exit 0;;' - echo ' "env --system") exit 0;;' - echo 'esac' - echo 'echo "STUB refused: $*" >&2; exit 2' - } > "$_pr/pm"; chmod 755 "$_pr/pm" - printf '%s\n' '#!/bin/bash' 'case "$PAM_USER" in' \ - ' u_claude) USERS="u_claude tester"; ;;' \ - ' u_keepme) USERS="u_keepme tester"; ;;' \ - ' *) exit 1 ; ;;' 'esac' > "$_pr/etc/pam.d/su_u_user" - printf 'name|app|share|dir|desktop|launcher|groups\nclaude|/bin/true|slim|no|no|no|-\nzznew|/bin/true|full|no|no|no|-\n' \ - > "$_pr/conf.conf" - APP_USERS_CONF="$_pr/conf.conf" APP_USERS_PAM="$_pr/etc/pam.d/su_u_user" \ - APP_USERS_PROFILE_D="$_pr/etc/profile.d" APP_USERS_PM="$_pr/pm" \ - bash "$_au" >"$_pr/out" 2>&1 - [ "$(grep -c 'u_claude)' "$_pr/etc/pam.d/su_u_user")" = 1 ] \ - || _p="$_p;an account already in the PAM file was duplicated" - grep -q 'u_keepme)' "$_pr/etc/pam.d/su_u_user" \ - || _p="$_p;a hand-written PAM entry was lost in the merge" - grep -q 'u_zznew)' "$_pr/etc/pam.d/su_u_user" \ - || _p="$_p;a new account was not added to the PAM file" - grep -q 'STUB refused' "$_pr/out" \ - && _p="$_p;it called a subcommand that does not exist: $(grep -m1 'STUB refused' "$_pr/out")" - ls "$_pr/etc/pam.d/" | grep -q 'bak-' \ - || _p="$_p;the PAM file was edited with no backup beside it" - # the env fragment enumerates rather than lists - grep -q 'HOME"/Shared/\*' "$_pr/etc/profile.d/shared-dirs.sh" 2>/dev/null \ - || _p="$_p;the SHARED_* variables are a fixed list, not what is there" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "application users come from a file, and neither list-file is clobbered" -fi - -# ---- the handover selects by NUMBER (1.14.216) ----------------------------- # -# 7.2's `chown -R root:root` is refused on a chapter-8 tree (it would strip -# every package user), and the precise version selects files by the HOST's -# lfs uid instead -- the only thing that means the same on both sides of a -# chroot. Book 4.3 pins no uid for `lfs`, and package users here are -# created in the 10000-20000 range: if those numbers ever meet, a package -# user's files are handed to root and it loses all of them silently. -_p="" -if [ -f "$LFS_TOOL" ]; then - _hb="$(_slice_def "$LFS_TOOL" _handover_build_user_only)" - [ -n "$_hb" ] || _p="$_p;the precise handover cannot be read" - printf '%s' "$_hb" | grep -q 'etc", "passwd"' \ - || _p="$_p;it does not check the tree's passwd for that uid" - printf '%s' "$_hb" | grep -q 'REFUSING to hand files back' \ - || _p="$_p;a uid collision is not refused" - printf '%s' "$_hb" | grep -q 'chown -h root {}' \ - || _p="$_p;the handover follows symlinks or takes the group too" - printf '%s' "$_hb" | grep -q -- '-xdev' \ - || _p="$_p;the handover can cross into another filesystem" - # and it must refuse for real, with a tree whose passwd collides - rm -rf "$T/ho"; mkdir -p "$T/ho/etc" "$T/ho/usr" - printf 'root:x:0:0::/root:/bin/bash\np_light:x:10753:10753::/x:/bin/bash\n' \ - > "$T/ho/etc/passwd" - cat > "$T/ho/probe.py" <<'PYHO' -import sys, types, io, contextlib -mod = types.ModuleType("lfsmod"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -mod._build_user_uid = lambda: 10753 -buf = io.StringIO() -with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): - n = mod._handover_build_user_only(sys.argv[2], run=True) -print("touched=%d" % n) -print("refused" if "REFUSING" in buf.getvalue() else "NOT-REFUSED") -PYHO - _ho="$(python3 "$T/ho/probe.py" "$LFS_TOOL" "$T/ho" 2>&1)" - printf '%s\n' "$_ho" | grep -qx 'refused' \ - || _p="$_p;a colliding uid is not refused in practice: $(printf '%s' "$_ho" | tr '\n' ' ')" - printf '%s\n' "$_ho" | grep -qx 'touched=0' \ - || _p="$_p;it changed files despite the collision" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "the handover refuses when the build uid is also a package user's" -fi - -# ---- a missing python backend is a package (1.14.210) ---------------------- # -# numpy died on `BackendUnavailable: Cannot import 'mesonpy'` after 60 lines -# of pip internals, the gate did not recognise it, and the page that -# provides `mesonpy` is called **meson_python** -- not guessable from the -# module name. numpy's Required in the book is exactly -# `cython-3.2.9, meson_python-0.20.0`, so a --recursive install would have -# built it. -# (its own variable: borrowing $_hl9 from the block below made this block -# die with "unbound variable" when run alone -- the third time a borrowed -# variable has done that) -_p="" -_hl10="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl10" ]; then - grep -qF "BackendUnavailable: Cannot import" "$_hl10" \ - || _p="$_p;a missing python build backend is not recognised as a failure" - grep -q 'mesonpy) echo meson_python' "$_hl10" \ - || _p="$_p;the page that provides mesonpy is not named" - # AUTO-INSTALL IS OPT-IN, AND REFUSES TEST-ONLY MODULES. Installing - # what a failure names would have installed pytest for numpy, run a - # multi-minute test suite inside an install, and hidden the generator - # bug that put the test there (1.14.214). - _am="$(_slice_fn "$_hl10" _install_missing_module)" - [ -n "$_am" ] || _p="$_p;there is no way to install a missing build backend" - printf '%s' "$_am" | grep -q '_PY_TEST_ONLY' \ - || _p="$_p;a test-only module would be installed to satisfy an install" - printf '%s' "$_am" | grep -q 'NOT installing it' \ - || _p="$_p;it does not refuse a test dependency out loud" - grep -q 'auto_modules=1' "$_hl10" \ - || _p="$_p;there is no flag for it" - grep -q 'auto_modules="${PM_AUTO_MODULES:-0}"' "$_hl10" \ - || _p="$_p;auto-install is on by default, which changes the system unasked" - # reported on the SUCCESS path as well, which is when nobody looks - grep -q 'installed on the way (--auto-modules), not asked for' "$_hl10" \ - || _p="$_p;packages installed on the way are not reported" - [ "$(grep -c 'installed on the way (--auto-modules)' "$_hl10")" -ge 2 ] \ - || _p="$_p;it is reported only when the build fails" - # and the three outcomes, for real. The slices go into a FILE and the - # file is run: an eval nested inside a command substitution inside a - # double-quoted assignment is four levels of quoting, and getting one - # wrong made every case come out "refused" -- a test that fails for its - # own reasons teaches nothing. - { - echo 'say(){ :; }; note(){ :; }; warn(){ :; }' - echo 'unprefix_pkg_user(){ echo "${1#p_}"; }' - echo 'name=p_numpy' - echo 'packagemanager(){ return 0; }' - _slice_fn "$_hl10" _missing_py_module - awk '/^_PY_TEST_ONLY=/,/flaky"/' "$_hl10" - _slice_fn "$_hl10" _py_module_provider - _slice_fn "$_hl10" _install_missing_module - echo '_AUTO_MODULES_LOG="$1"; : > "$_AUTO_MODULES_LOG"' - echo 'printf "No module named %s\n" "'"'"'pytest'"'"'" > "$2/l1"' - echo '_install_missing_module "$2/l1" && echo pytest=INSTALLED || echo pytest=refused' - echo 'printf "Cannot import %s\n" "'"'"'mesonpy'"'"'" > "$2/l2"' - echo '_install_missing_module "$2/l2" && echo mesonpy=installed || echo mesonpy=REFUSED' - echo 'printf "No module named %s\n" "'"'"'zzunknown'"'"'" > "$2/l3"' - echo '_install_missing_module "$2/l3" && echo unknown=INSTALLED || echo unknown=refused' - echo 'grep -q meson_python "$1" && echo recorded=yes || echo recorded=NO' - } > "$T/am_probe.sh" - _amo="$(bash "$T/am_probe.sh" "$T/am.log" "$T" 2>&1)" - for _want in "pytest=refused" "mesonpy=installed" "unknown=refused" "recorded=yes"; do - printf '%s\n' "$_amo" | grep -qx "$_want" \ - || _p="$_p;auto-modules: expected $_want, got: $(printf '%s' "$_amo" | tr '\n' ' ')" - done - grep -q 'a python module is missing, not a permission' "$_hl10" \ - || _p="$_p;the failure box does not say what kind of failure it is" - # the extraction, on both wordings - for _pair in "Cannot import 'mesonpy'|mesonpy" \ - "ModuleNotFoundError: No module named 'numpy'|numpy"; do - printf '%s\n' "${_pair%|*}" > "$T/pylog" - _got="$(grep -aoE "Cannot import '[a-zA-Z0-9_.]+'|No module named '[a-zA-Z0-9_.]+'" \ - "$T/pylog" | head -n1 | cut -d"'" -f2)" - [ "$_got" = "${_pair#*|}" ] \ - || _p="$_p;'${_pair%|*}' parses to '$_got'" - done -fi -if [ -f "$pm_src" ]; then - # AND THE DEPENDENCY CHECK MUST NOT SWALLOW ITS OWN FAILURE: silence - # there is indistinguishable from "nothing missing" - # the SHAPE, not the phrase: the comment explaining this bug quotes the - # old code, and matching the quote matched the explanation (the fourth - # time prose has failed a check about code) - grep -vE '^\s*#' "$pm_src" | grep -q 'except (SystemExit, Exception):' \ - && _p="$_p;a failed dependency lookup is still silent" - grep -q 'could not work out what' "$pm_src" \ - || _p="$_p;a failed dependency lookup does not say so" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a missing python backend names its package, and a failed dep lookup is not silence" -fi - -# ---- the second retry must not re-clone (1.14.209) ------------------------- # -# opencv needed two directory grants (licenses, then cmake). Round 1 -# retried `install` only -- correct, the build was at 100%. Round 2 then -# ran `phase all` and re-cloned and rebuilt the whole tree, because -# LAST_FAILED_PHASE is cleared at the top of run_phase_as and was only SET -# in its all|update branch: a single-phase run left it empty, and cmd_build -# fell back to $phase. -_p="" -_hl9="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl9" ]; then - _rpa="$(_slice_fn "$_hl9" run_phase_as)" - _hb10="$(grep -A14 'IS A COMPOUND PHASE TOO' "$_hl9")" - printf '%s' "$_rpa" | grep -q '\*) LAST_FAILED_PHASE="$phase"' \ - || _p="$_p;a single-phase run does not record which phase failed" - # `update` IS A COMPOUND PHASE TOO. The mapping covered `all` only, so - # a failure inside `update` retried `update`: poppler's retry, after a - # grant for /usr/share/gir-1.0, re-unpacked the tarball and re-ran cmake - # with 145 files already installed (1.14.248). - printf '%s' "$_hb10" | grep -q 'all|update)' \ - || _p="$_p;a failure inside update re-runs the whole update" - _upd="$( phase=update; LAST_FAILED_PHASE=install - _r="$phase" - case "$phase" in all|update) - case "${LAST_FAILED_PHASE:-}" in - install|configure) _r="$LAST_FAILED_PHASE" ;; esac ;; esac - echo "$_r" )" - [ "$_upd" = install ] \ - || _p="$_p;update+failed install retries '$_upd', not install" - # and the decision it feeds: install stays install across rounds - _dec="$( phase=install; rc=1; LAST_FAILED_PHASE="" - case "$phase" in all|update) ;; *) LAST_FAILED_PHASE="$phase" ;; esac - _r="$phase" - [ "$phase" = all ] && case "${LAST_FAILED_PHASE:-}" in - install|configure) _r="$LAST_FAILED_PHASE" ;; esac - echo "$_r" )" - [ "$_dec" = install ] \ - || _p="$_p;a failed install retry resolves to '$_dec', not install" - # the comment that explains why must name the cost, not just the rule - grep -q 'reached 100%' "$_hl9" \ - || _p="$_p;nothing records what the full rebuild cost" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a second repair round retries the failed phase, it does not re-clone" -fi - -# ---- one list, two entry points (1.14.223) --------------------------------- # -# The media chain is the part that gets rebuilt -- a codec appears, a plugin -# set was configured without a feature, webkitgtk moves a version -- and it -# lived inside a 158-entry desktop stack. Copying it into a second file -# would give two lists of the same packages, and two lists drift: -# gst10-libav sat in one of them for three releases before anyone noticed. -# So `include <stack>` splices one stack into another. -_p="" -_ms="$(dirname "$LFS_TOOL")/stacks/media.stack" -_sw3="$(dirname "$LFS_TOOL")/stacks/sway.stack" -if [ -f "$_ms" ] && [ -f "$_sw3" ]; then - grep -qE '^include +media' "$_sw3" \ - || _p="$_p;sway.stack does not include the media stack" - # the ENTRIES must exist once, in media.stack, and not be copied back - for _e in webkitgtk gst10-plugins-bad gst10-libav opus; do - [ "$(grep -cE "^(book|git|tar) +$_e( |$)" "$_sw3")" = 0 ] \ - || _p="$_p;$_e is written in sway.stack as well as media.stack" - grep -qE "^(book|git|tar) +$_e( |$)" "$_ms" \ - || _p="$_p;$_e is in neither stack" - done - # A STATUS IS NOT A PLAN. On a system built from an older book, half - # the tree reads "(update available)" and the summary said "building - # next: ..." -- so `stack media --run` looked like it was about to - # rebuild glib2 and everything under it. A run builds what is MISSING - # and never touches an installed package (1.14.225). - grep -q 'to build (missing)' "$pm_src" \ - || _p="$_p;the status still presents missing packages as a build plan" - grep -q 'Installed packages are NOT' "$pm_src" \ - || _p="$_p;nothing says a stack run leaves installed packages alone" - # the SHAPE: the phrase in a printed string, not in the comment that - # explains why it was dropped (fifth time prose has failed a code check) - grep -vE '^\s*#' "$pm_src" | grep -q 'installed {u.version} (update available)' \ - && _p="$_p;'update available' still implies the stack will update it" - # ...and a single update names the requirements that differ, which is the - # real question behind "can I update webkitgtk without glib2" - _sr2="$(_slice_def "$pm_src" _report_stale_requires)" - [ -n "$_sr2" ] || _p="$_p;a single update says nothing about stale requirements" - printf '%s' "$_sr2" | grep -q 'not a refusal' \ - || _p="$_p;it refuses instead of reporting, which breaks the one workflow" - printf '%s' "$_sr2" | grep -q '_same_version' \ - || _p="$_p;it compares version strings without the book's spelling rules" - rm -rf "$T/sr"; mkdir -p "$T/sr/pkgusr/p_zzwk" "$T/sr/pkgusr/p_zzlib" - printf "name='zzwk'\nname_version='ZzWk-1.0'\nrequired=('ZzLib-9.9')\n" \ - > "$T/sr/pkgusr/p_zzwk/install_last" - echo x > "$T/sr/pkgusr/p_zzwk/pkg.lst" - printf "name='zzlib'\nname_version='ZzLib-1.0'\n" \ - > "$T/sr/pkgusr/p_zzlib/install_last" - echo x > "$T/sr/pkgusr/p_zzlib/pkg.lst" - _sro="$(PKGUSR_BASE=$T/sr PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYSR' -import sys, types, io, contextlib -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -buf = io.StringIO() -with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): - mod._report_stale_requires(["zzwk"]) -out = buf.getvalue() -print("NAMED" if "zzlib" in out and "ZzLib-1.0" in out else "MISSED:%s" % out[:120]) -PYSR -)" - printf '%s\n' "$_sro" | grep -q NAMED \ - || _p="$_p;a requirement older than the book is not named: $_sro" - # THE MAJOR VERSION IS PART OF THE NAME. gtk3 and gtk4 both record - # name_version 'GTK-<v>', so the base 'GTK' matched whichever was - # indexed first and `GTK-4.20.3` resolved to **gtk3** -- webkitgtk - # listed gtk3 twice and gtk4 not at all. Same collision hid libsoup-3.x - # behind libsoup, sdl3 behind sdl, qt6 behind qt (1.14.226). - _rdo="$(python3 - "$pm_src" "$T/rd" 2>&1 <<'PYRD' -import sys, types, os -base = os.path.join(sys.argv[2], "pkgusr") -for name, nv in (("gtk3", "GTK-3.24.51"), ("gtk4", "GTK-4.20.3"), - ("libsoup3", "libsoup-3.6.6"), ("opus", "Opus-1.6.1"), - ("sdl3", "SDL3-3.4.0"), ("lcms2", "Little-CMS-2.18")): - d = os.path.join(base, "p_" + name) - os.makedirs(d, exist_ok=True) - open(os.path.join(d, "install_last"), "w").write( - "name='%s'\nname_version='%s'\n" % (name, nv)) -os.environ["PKGUSR_BASE"] = sys.argv[2] -os.environ["PKGUSR_CONFIG"] = "/dev/null" -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -inst = mod._pkg_dirs() -bad = [] -for dep, want in (("GTK-3.24.51", "gtk3"), ("GTK-4.20.3", "gtk4"), - ("libsoup-3.6.6", "libsoup3"), ("Opus-1.6.1", "opus"), - ("SDL3-3.4.0", "sdl3"), ("Little-CMS-2.18", "lcms2")): - got = mod._resolve_dep(dep, inst) - if got != want: - bad.append("%s -> %s (want %s)" % (dep, got, want)) -print("BAD: " + "; ".join(bad) if bad else "OK") -PYRD -)" - [ "$_rdo" = OK ] || _p="$_p;dependency resolution: $_rdo" - printf '%s' "$_sr2" | grep -q '_row not in rows' \ - || _p="$_p;the same package can be listed twice in the report" - # A TRUNCATED LIST MUST SAY HOW TO SEE THE REST, and there must be a way - # to see every package the book has moved past, not just the one asked - # about (1.14.227). - # (the literal is wrapped across two source lines, so match the part - # that is on one of them -- a pattern spanning a line break matches - # nothing and the assertion fails on correct code) - printf '%s' "$_sr2" | grep -q 'all of them: packagemanager' \ - || _p="$_p;'... and N more' is a dead end" - printf '%s' "$_sr2" | grep -q '_verbose' \ - || _p="$_p;-v does not show the full requirement list" - grep -q 'everything the book has moved past' "$pm_src" \ - || _p="$_p;nothing points at the whole-system update list" - # THE HINT MUST BE A COMMAND THAT WORKS. It printed - # `packagemanager update webkitgtk -v` and -v was a TOP-LEVEL flag only, - # so the tool answered "unrecognized arguments: -v" (1.14.228). - timeout 60 python3 "$pm_src" update --help 2>&1 | grep -q -- '\[-v\]' \ - || _p="$_p;-v is not accepted where the hint says to type it" - # A MIDDLE GROUND BETWEEN ONE PACKAGE AND EVERYTHING: a stack's own - # packages. `stack media --run` builds what is missing and updates - # nothing; `update webkitgtk` is one package; between them there was - # only a list of names to type. - timeout 60 python3 "$pm_src" update --help 2>&1 | grep -q -- '--stack NAME' \ - || _p="$_p;there is no way to update just one stack's packages" - _st2="$(_slice_def "$pm_src" _stack_targets)" - printf '%s' "$_st2" | grep -q '"book", "git", "tar"' \ - || _p="$_p;--stack would try to update check/cfg entries" - # 2>/dev/null and tail -1: _stack_targets PRINTS its refusal, so the - # message arrived before the verdict and the comparison read it as the - # answer - _sto="$(python3 - "$pm_src" "$(dirname "$LFS_TOOL")/stacks/media.stack" 2>/dev/null <<'PYST' -import sys, types, os -os.environ["PKGUSR_CONFIG"] = "/dev/null" -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -t = mod._stack_targets(sys.argv[2]) or [] -bad = [] -if "webkitgtk" not in t: bad.append("webkitgtk missing") -if "webrtc" in t: bad.append("the check entry is treated as a package") -if mod._stack_targets(sys.argv[2] + ".nope") is not None: - bad.append("a missing stack is not refused") -print("BAD: " + "; ".join(bad) if bad else "OK") -PYST -)" - _sto="$(printf '%s\n' "$_sto" | tail -1)" - [ "$_sto" = OK ] || _p="$_p;--stack targets: $_sto" - # and the plan must say what it will change, in its own line - grep -q 'this plan updates %d package' "$pm_src" \ - || _p="$_p;the plan does not state which packages it updates" - grep -q 'a comparison, not a list of work' "$pm_src" \ - || _p="$_p;the requirement report can still be read as a plan" - # A DRY RUN ASKS NOTHING. `update --stack media` prompted four times, - # once per already-current package, before printing a plan it was never - # going to carry out (1.14.229). - grep -q 'A DRY RUN ASKS NOTHING' "$pm_src" \ - || _p="$_p;a preview still prompts for already-current packages" - _cu="$(_slice_def "$pm_src" cmd_update)" - # -B8: the guard is the `continue` branch above the prompt, and three - # lines of it are the comment explaining why - printf '%s' "$_cu" | grep -B8 'Re-apply the update anyway' \ - | grep -q 'getattr(args, "run", False)' \ - || _p="$_p;the up-to-date prompt is not gated on --run" - # A GIT-TRACKED PACKAGE HAS NO VERSION TO COMPARE. gst-plugins-rs is a - # git entry following upstream; its name_version is - # `gst-plugins-rs-git`, which against the book's 1.28.6 read as "older" - # -- so the plan offered to replace a cargo-built checkout with the - # book's page, silently. - printf '%s' "$_cu" | grep -q 'endswith("-git")' \ - || _p="$_p;a git-tracked package is compared against a book version" - printf '%s' "$_cu" | grep -q 'git_tracked.append' \ - || _p="$_p;a git-tracked package is not reported as left alone" - grep -q 'follow GIT, not the book -- left alone' "$pm_src" \ - || _p="$_p;nothing says why those packages are not in the plan" - grep -q 'stack <name> --only <pkg> --run --force' "$pm_src" \ - || _p="$_p;it does not say how to move a git-tracked package" - # ...and that command really does default to everything - timeout 60 python3 "$pm_src" update --help 2>&1 \ - | grep -q 'default: all installed' \ - || _p="$_p;bare \`update\` does not list every installed package" - # THE SKIP MARKER IS ONE CONSTANT. _enable_one_alternative matched - # "## OPTIONAL, skipped" and 1.14.212 renamed the marker, so webkitgtk's - # script came out with BOTH build blocks commented: a build phase of `:` - # and an install phase running ninja against no build.ninja. - _bl2="$(dirname "$LFS_TOOL")/blfs" - if [ -f "$_bl2" ]; then - grep -q '^SKIP_MARK = ' "$_bl2" \ - || _p="$_p;the skip marker is a literal in more than one place" - grep -q '_is_skip_line' "$_bl2" \ - || _p="$_p;the alternative picker matches the marker by hand" - _alt="$(python3 - "$_bl2" <<'PYALT' -import sys, types -mod = types.ModuleType("b"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -# both blocks skipped, under the CURRENT marker: one must come back -cmds = ("## SKIPPED -- the book says: If you want the GTK-3 version:\n" - "# mkdir -vp build &&\n# cd build &&\n# cmake -D PORT=GTK ..\n# ninja\n" - "## SKIPPED -- the book says: If you want the GTK-4 version:\n" - "# rm -rf * &&\n# cmake -D USE_GTK4=ON ..\n# ninja\n") -out = mod._enable_one_alternative(cmds) -live = [l for l in out.split("\n") - if l.strip() and not l.strip().startswith("#") and l.strip() != ":"] -print("OK" if any("cmake" in l for l in live) else "NONE-ENABLED") -PYALT -)" - [ "$_alt" = OK ] \ - || _p="$_p;a page of two alternatives still builds neither: $_alt" - fi - - # THE EXCLUSIONS ARE A PROPERTY OF THE MACHINE, not of one stack. When - # the media chain moved out, it lost all 87 `avoid` entries -- so - # `stack media --status` planned 274 packages including systemd, polkit - # and qt6, on a SysV machine that must never have systemd. avoid.stack - # is included by both (1.14.224). - _av="$(dirname "$LFS_TOOL")/stacks/avoid.stack" - [ -f "$_av" ] || _p="$_p;there is no shared avoid list" - # EVERY stack, not the two that were split. network, services and - # vision never got the include, so `stack services --status` planned - # systemd just as media had -- the same bug, three files over - # (1.14.235). - for _f in "$(dirname "$LFS_TOOL")"/stacks/*.stack; do - case "$(basename "$_f")" in avoid.stack) continue ;; esac - grep -qE '^include +avoid' "$_f" \ - || _p="$_p;$(basename "$_f") does not include the avoid list" - [ "$(grep -c '^avoid ' "$_f")" = 0 ] \ - || _p="$_p;$(basename "$_f") still carries its own avoid lines" - done - # A PACKAGE MUST NOT BE BOTH BUILT AND AVOIDED. `avoid openssh` sat in - # avoid.stack while network.stack built it with the install-sshd - # bootscript -- two files contradicting each other, and whichever a - # plan saw first won. Checked for every stack now (1.14.241). - _clash="$(python3 - "$pm_src" "$(dirname "$LFS_TOOL")"/stacks/*.stack <<'PYCL' -import sys, types, os -os.environ["PKGUSR_CONFIG"] = "/dev/null" -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -bad = [] -for f in sys.argv[2:]: - if os.path.basename(f) == "avoid.stack": - continue - rows = mod._parse_stack_file(f) - build = {r["target"] for r in rows if r["kind"] in ("book", "git", "tar")} - avoid = {r["target"] for r in rows if r["kind"] == "avoid"} - both = sorted(build & avoid) - if both: - bad.append("%s builds and avoids %s" % (os.path.basename(f), ", ".join(both))) -print("BAD: " + "; ".join(bad) if bad else "OK") -PYCL -)" - _clash="$(printf '%s\n' "$_clash" | tail -1)" - [ "$_clash" = OK ] || _p="$_p;$_clash" - - # INSTALLING PAM IS NOT ENOUGH, AND WARNING ABOUT MODES IS NOT ENOUGH. - # A rebuild of shadow strips /usr/bin/su's setuid bit (the BLFS page - # never sets it; the LFS one does), so su was broken after the very - # rebuild that made PAM usable. `cfg shadow-pam` does both halves and - # FIXES the modes rather than reporting them (1.14.252). - _sp="$(dirname "$LFS_TOOL")/stacks/shadow-pam.sh" - _sc="$(dirname "$LFS_TOOL")/stacks/setid.conf" - # set HERE: only an earlier block set it, so a --only selection that - # did not pull that line in died on `set -u` - _hlv="$(dirname "$LFS_TOOL")/lfs-helper" - if [ -f "$_sp" ] && [ -f "$_sc" ]; then - bash -n "$_sp" || _p="$_p;shadow-pam.sh does not parse" - grep -qE '^cfg +shadow-pam' \ - "$(dirname "$LFS_TOOL")/stacks/services.stack" \ - || _p="$_p;nothing rebuilds shadow after linux-pam" - grep -q 'update shadow --reinstall --regenerate' "$_sp" \ - || _p="$_p;it does not rebuild shadow against pam" - grep -q 'libpam\\.so' "$_sp" \ - || _p="$_p;it does not check whether su actually links pam" - # A PAM-LINKED su WITH NO /etc/pam.d/su IS WORSE THAN A NON-PAM ONE: - # su: pam_start: error 26 (PAM_ABORT -- no service file) - # `update` installs and does NOT configure, and those files come - # from the CONFIGURE phase of linux-pam and shadow (1.14.253). - grep -q 'script configure' "$_sp" \ - || _p="$_p;a pam-linked su is left with no /etc/pam.d/su to read" - # CONFIG BEFORE REBUILD. Between a pam-linked su and its config - # file, NOTHING can be built: every build runs as its package user - # through `su - p_<pkg>`, including the configure phases that would - # write the missing file. Root survives it only because pam_rootok - # is sufficient (1.14.254). - _i_cfg="$(grep -n 'script configure' "$_sp" | head -1 | cut -d: -f1)" - _i_reb="$(grep -n 'update shadow --reinstall' "$_sp" | head -1 | cut -d: -f1)" - [ -n "$_i_cfg" ] && [ -n "$_i_reb" ] && [ "$_i_cfg" -lt "$_i_reb" ] \ - || _p="$_p;shadow is rebuilt before its pam config is written" - grep -q 'NOTHING can be built' "$_sp" \ - || _p="$_p;nothing records why that order matters" - # the helpers must be defined before the first step that calls them - _i_def="$(grep -n '^_pam_installed()' "$_sp" | head -1 | cut -d: -f1)" - _i_use="$(grep -n 'if _pam_installed; then' "$_sp" | head -1 | cut -d: -f1)" - [ -n "$_i_def" ] && [ -n "$_i_use" ] && [ "$_i_def" -lt "$_i_use" ] \ - || _p="$_p;_pam_installed is called before it is defined" - # THE SET, NOT A SAMPLE: su and system-auth existed while fourteen - # other files did not, and it said "in place" (1.14.262). - grep -q '^_book_pam_set() {' "$_sp" \ - || _p="$_p;shadow-pam checks a sample of the pam.d files, not the set" - grep -q 'script fetch "$1"' "$_sp" \ - || _p="$_p;shadow-pam does not read the package's own script" - grep -q '_fetch_script "$_cfgpkg" --regenerate' "$_sp" \ - || _p="$_p;a stale script that names no pam file is run again as it is" - # power.sh read packagemanager.conf (key=value) with a JSON pattern - # and stopped the stack on the booted machine (1.14.265). The - # prefix comes from the resolver, exported to cfg scripts. - _pws="$(dirname "$LFS_TOOL")/stacks/power.sh" - grep -q '"collector_prefix"' "$_pws" \ - && _p="$_p;power.sh parses the conf with the lfs config's JSON syntax" - grep -q 'LFS_COLLECTOR_PREFIX' "$_pws" \ - || _p="$_p;power.sh does not read the exported collector prefix" - grep -q '_cenv\["LFS_COLLECTOR_PREFIX"\]\|("collector_prefix", "LFS_COLLECTOR_PREFIX")' \ - "$(dirname "$LFS_TOOL")/packagemanager" \ - || _p="$_p;cfg scripts are not given the resolved collector prefix" - # ID RANGES (1.14.267): collector groups at 90000+ with no account, - # application users at 30000+, power's group not a system gid, and - # a symlinked binary judged by its target. - _pmf="$(dirname "$LFS_TOOL")/packagemanager" - sed -n '/^def cmd_nimgnu_create/,/^def /p' "$_pmf" | grep -q 'add_package_user\|find_add_package_user' \ - && _p="$_p;sysgroup create makes an account in the package-user range" - sed -n '/^def cmd_user_create/,/^def /p' "$_pmf" | grep -q 'APP_UID_MIN' \ - || _p="$_p;application users are created in the package-user range" - grep -q 'groupadd -r' "$_pws" \ - && _p="$_p;power.sh makes its collector group with a system gid" - grep -q 'readlink -f' "$_pws" \ - || _p="$_p;power.sh judges a symlink instead of its target" - grep -q '/etc/rc.d/rcS.d' "$_pws" \ - || _p="$_p;power.sh does not know LFS's rcS.d" - # GLVND (1.14.270): machine.conf enables it, so libglvnd must come - # before mesa, and --clean must never delete a file that moved to - # another package (libGL.so.1 goes from mesa to libglvnd). - _swy="$(dirname "$LFS_TOOL")/stacks/sway.stack" - _mcf="$(dirname "$LFS_TOOL")/stacks/machine.conf" - _l_g="$(grep -nE '^git +libglvnd' "$_swy" | head -1 | cut -d: -f1)" - _l_m="$(grep -nE '^book +mesa' "$_swy" | head -1 | cut -d: -f1)" - if grep -qE '^glvnd *= *enabled' "$_mcf"; then - [ -n "$_l_g" ] && [ -n "$_l_m" ] && [ "$_l_g" -lt "$_l_m" ] \ - || _p="$_p;mesa is built with glvnd but libglvnd does not come before it" - fi - sed -n '/^def _orphan_sweep/,/^def /p' "$_pmf" | grep -q 'st.st_uid != own_uid' \ - || _p="$_p;--clean can delete a file another package now owns" - # A MOVE OF MANY FILES IS ONE ROUND (1.14.271): libglvnd taking - # Mesa's headers went one file per round and was called a bundled - # subproject. meson's install plan hands the rest over at once, and - # only a subprojects/ source counts as a bundled copy. - _lhf="$(dirname "$LFS_TOOL")/lfs-helper" - sed -n '/^meson_takeover_from_log() {/,/^}/p' "$_lhf" | grep -q 'intro-installed.json' \ - || _p="$_p;a file move between packages still takes one meson round per file" - sed -n '/^meson_takeover_from_log() {/,/^}/p' "$_lhf" | grep -q '/subprojects/' \ - || _p="$_p;any one-file handover streak is called a bundled subproject" - # AN EXTRA DOWNLOAD UNDER ITS OWN NAME (1.14.272): the patch was - # fetched with -O $pkg, wget said 416, and patch found no file. - grep -qF '_pp_fetch "$li" "$f"' "$(dirname "$LFS_TOOL")/lfs-phases" \ - || _p="$_p;additional downloads are saved under the tarball's name" - # A FAILED BUILD MUST NOT ADVANCE THE VERSION (1.14.273): a package - # with no .pkgusr-installed had its version read from install_last, - # which is replaced before the build. - grep -q 'RECORD WHAT IS INSTALLED BEFORE OVERWRITING THE RECORD' "$_lhf" \ - || _p="$_p;a legacy package's failed build still advances its version" - # SCANS PRUNE; CMAKE'S WRAPPED ERROR IS READ (1.14.274) - sed -n '/^snapshot() {/,/^}/p' "$_lhf" | grep -q -- '-prune' \ - || _p="$_p;the tree scan walks /usr/src and filters afterwards" - grep -q '_cmakedirs=' "$_lhf" \ - || _p="$_p;a permission error CMake wraps over two lines is not read" - # su_u_user CHECKS THE CALLER, root OWNS IT, AND A CACHED cfg IS - # RE-CHECKED (1.14.275) - _aus="$(dirname "$LFS_TOOL")/stacks/app-users.sh" - grep -qF -- "echo '[[ \" \$USERS \" == *\" \$PAM_RUSER \"* ]]'" "$_aus" \ - || _p="$_p;su_u_user compares the target, not the caller (anyone gets in)" - grep -q 'chown root:root "$PAM_FILE"' "$_aus" \ - || _p="$_p;su_u_user, which root runs, is not forced to root ownership" - grep -q 'cached as done, but its check fails' "$_pmf" \ - || _p="$_p;a cached cfg entry is never re-checked" - grep -qE '^cfg +app-users +\\$' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ - || _p="$_p;cfg app-users has no expect= to re-check" - # REGROUP A DIRECTORY (1.14.276): a command, dispatched, that asks - # and shows a plan before it changes anything - grep -q '^ regroup-dir) shift; cmd_regroup_dir' "$_lhf" \ - || _p="$_p;lfs-helper has no regroup-dir" - sed -n '/^cmd_regroup_dir() {/,/^}/p' "$_lhf" | grep -q "apply? \[y/N\]" \ - || _p="$_p;regroup-dir changes a directory without confirmation" - # KERNEL OPTIONS THIS MACHINE WANTS SURVIVE A CONFIG REFRESH (1.14.277) - grep -q '^CONFIG_SET=.*LEGACY_TIOCSTI=n' "$(dirname "$LFS_TOOL")/kernel.conf" \ - || _p="$_p;kernel.conf does not disable legacy TIOCSTI (su loses job control)" - grep -q 'CONFIG_SET did not take' "$(dirname "$LFS_TOOL")/lfs-kernel" \ - || _p="$_p;lfs-kernel applies CONFIG_SET without reading it back" - # FIRMWARE PATTERNS AT ANY DEPTH, FLAT NAMES FROM WHENCE (1.14.279) - grep -q '^_fw_whence_links() {' "$(dirname "$LFS_TOOL")/lfs-kernel" \ - || _p="$_p;flat firmware names (WHENCE Link:) are never created" - grep -q -- '-name "$d" -print' "$(dirname "$LFS_TOOL")/lfs-kernel" \ - || _p="$_p;a FIRMWARE_ONLY pattern only matches the top level" - # AFTER ITS FIRST INSTALL A PACKAGE CLAIMS ONLY -tmp/-pass STAGES - # (1.14.280: "gave 'p_llvm' 13 path(s)" was su_u_user and the tools) - sed -n '/^_claim_earlier_stages() {/,/^}/p' "$_lhf" | grep -q '_installed_before' \ - || _p="$_p;an installed package still claims root files from its own manifest" - # THE INSTALLED VERSION IS RECORDED ON EVERY PATH THAT FINISHES - # (1.14.281): `--phase all` never wrote .pkgusr-installed, and - # VERSION could not parse name_version='...'. - [ "$(_slice_fn "$_lhf" cmd_build | grep -c '_record_installed "$name" "$owner" "$s"')" -ge 2 ] \ - || _p="$_p;a completed --phase all install does not record its version" - _ri_out="$( _rt="$(mktemp -d)"; mkdir -p "$_rt/home" - printf "name_version='Zz-2.0'\n" > "$_rt/script" - eval "$(_slice_fn "$_lhf" _record_installed)" - pkgusr_home_for() { echo "$_rt/home"; }; real_chown() { :; } - _record_installed zz p_zz "$_rt/script" - cat "$_rt/home/.pkgusr-installed"; head -1 "$_rt/home/VERSION"; rm -rf "$_rt" )" - [ "$_ri_out" = "Zz-2.0 -Zz-2.0" ] || _p="$_p;_record_installed does not write both records from name_version='...' (got: $_ri_out)" - # EXISTING APPLICATION ACCOUNTS MOVE ON `user setup` TOO (1.14.282) - sed -n '/^def cmd_user_setup/,/^def /p' "$_pmf" | grep -q '_relocate_app_home(u)' \ - || _p="$_p;an existing application account is never moved into its subdirectory" - # `packagemanager stack` WITH NO NAME LISTS THE STACKS (1.14.284) - _sl="$(cd "$(dirname "$LFS_TOOL")" && PKGUSR_CONFIG=/dev/null python3 packagemanager stack 2>/dev/null)" - printf '%s\n' "$_sl" | grep -qE '^ sway +[0-9]+/[0-9]+ done' \ - || _p="$_p;packagemanager stack without a name does not list the stacks" - # NO UNDEFINED NAMES IN cmd_install (1.14.287: `ignore` was - # build_install_plan's local, used in cmd_install) - _pyf="$(python3 -m pyflakes --version >/dev/null 2>&1 && echo y)" - if [ -n "$_pyf" ]; then - # every undefined name in the Python tools (STACK_DIR is read - # through a globals() guard on purpose) - _un="$(cd "$(dirname "$LFS_TOOL")" && python3 -m pyflakes packagemanager lfs blfs 2>/dev/null \ - | grep "undefined name" | grep -v "'STACK_DIR'")" - [ -z "$_un" ] || _p="$_p;undefined names in the Python tools: $(printf '%s' "$_un" | tr '\n' ' ')" - else - sed -n '/^def cmd_install(/,/^def /p' "$_pmf" | grep -q 'ignore=ignore)' \ - && _p="$_p;cmd_install passes a name it never defines (ignore)" - fi - # THE SCRIPT LAST INSTALLED, NOT THE FIRST IN THE ALPHABET (1.14.288) - _vs="$( _vd="$(mktemp -d)"; touch "$_vd/install_Mesa-26.1.7"; sleep 0.01; touch "$_vd/install_Mesa-25.3.5" - printf "name_version='Mesa-26.1.7'\n" > "$_vd/install_last" - cd "$(dirname "$LFS_TOOL")" && PKGUSR_CONFIG=/dev/null python3 -c " -import types,sys -m=types.ModuleType('pm'); exec(compile(open('packagemanager').read(),'pm','exec'), m.__dict__) -print(m._versioned_script_in(sys.argv[1]))" "$_vd" 2>/dev/null | tail -1; rm -rf "$_vd" )" - case "$_vs" in */install_Mesa-26.1.7) ;; *) _p="$_p;script runs the alphabetically first version, not the installed one ($_vs)" ;; esac - # THE su TRAP: shadow's configure writes su WITHOUT the pam_exec - # line, so app-users must run after shadow-pam to put it back. - _svs="$(dirname "$LFS_TOOL")/stacks/services.stack" - _i_sp="$(grep -nE '^cfg +shadow-pam' "$_svs" | head -1 | cut -d: -f1)" - _i_au="$(grep -nE '^cfg +app-users' "$_svs" | head -1 | cut -d: -f1)" - [ -n "$_i_sp" ] && [ -n "$_i_au" ] && [ "$_i_sp" -lt "$_i_au" ] \ - || _p="$_p;app-users does not run after shadow-pam, so shadow's su loses the pam_exec line" - grep -q 'grep -qF "$(basename "$PAM_FILE")" "$PAM_SU"' \ - "$(dirname "$LFS_TOOL")/stacks/app-users.sh" \ - || _p="$_p;app-users would not notice su lost its pam_exec line" - grep -q 'pam.d/system-auth' "$_sp" \ - || _p="$_p;system-auth is not checked, and su needs it too" - grep -q 'pam_start' "$_sp" \ - || _p="$_p;nothing records the error that missing file produces" - # EVERY BUILD IS A `su - p_<pkg>`, so a pam_unix lookup failure is - # not cosmetic: add_package_user writes /etc/passwd and leaves no - # /etc/shadow entry, and pam_unix says "cannot retrieve - # authentication info" on every package build for ever (1.14.255). - grep -q 'pwconv' "$_sp" \ - || _p="$_p;package users have no shadow entry and pam says so on every build" - grep -q 'cp -p /etc/shadow' "$_sp" \ - || _p="$_p;/etc/shadow is rewritten without a backup" - # NO /etc/shadow AT ALL was the case the guard missed -- and it is - # the system that needs this most: pam_unix has nowhere to look for - # ANY account (1.14.257). pwconv CREATES the file from - # /etc/passwd, which is a change to how the machine authenticates, - # so passwd is what gets backed up and root's line is checked. - grep -q 'elif \[ ! -e /etc/shadow \]' "$_sp" \ - || _p="$_p;a system with no /etc/shadow is skipped in silence" - grep -q 'cp -p /etc/passwd "$_pwbak"' "$_sp" \ - || _p="$_p;/etc/passwd is changed by pwconv without a backup" - grep -q "grep -q '\^root:' /etc/shadow" "$_sp" \ - || _p="$_p;it does not check root survived the conversion" - grep -q 'cp -p "$_pwbak" /etc/passwd' "$_sp" \ - || _p="$_p;a failed conversion is not rolled back" - grep -q 'if login is broken' "$_sp" \ - || _p="$_p;it does not say how to undo it by hand" - grep -q 'still without a shadow entry' "$_sp" \ - || _p="$_p;it does not read back whether the entries appeared" - grep -q 'UNVERIFIED FROM WHERE THIS WAS WRITTEN' "$_sp" \ - || _p="$_p;the unverified part is not marked unverified" - # ORDER: chown clears the setuid bit, so chmod must come last - _ci="$(grep -n 'chown' "$_sp" | head -1 | cut -d: -f1)" - _cm="$(grep -n 'chmod "$mode"' "$_sp" | head -1 | cut -d: -f1)" - [ -n "$_ci" ] && [ -n "$_cm" ] && [ "$_ci" -lt "$_cm" ] \ - || _p="$_p;chmod runs before chown, which clears the setuid bit" - grep -q 'su.*| *4711 * | *root' "$_sc" \ - || _p="$_p;setid.conf does not declare su as setuid root" - # shadow's install asks for NINE bits and the first list declared - # five. These four came off that request list (1.14.258). - for _f in chfn chsh newgidmap newuidmap; do - grep -q "/$_f *|" "$_sc" \ - || _p="$_p;$_f is missing from setid.conf, so its bit stays lost" - done - # APPLY, DO NOT RECITE: report_setid_requests runs as root right - # after the install, which is the one moment the missing thing can - # simply be done. It printed nine chmod lines instead. - _rsr="$(_slice_fn "$_hlv" report_setid_requests)" - printf '%s' "$_rsr" | grep -q 'cmd_setid --run' \ - || _p="$_p;the setid requests are printed and not applied" - printf '%s' "$_rsr" | grep -q 'id -u.*= 0' \ - || _p="$_p;it would try to apply them as a non-root caller" - printf '%s' "$_rsr" | grep -q 'SOME COULD NOT BE APPLIED' \ - || _p="$_p;a failed apply is not reported" - # A BACKTICK INSIDE DOUBLE QUOTES RUNS THE COMMAND. The success - # line said `lfs-helper setid` and therefore EXECUTED it, printing a - # second full report in the middle of a sentence (1.14.259). - printf '%s' "$_rsr" | grep -q '`lfs-helper setid`' \ - && _p="$_p;a backtick in the success line runs lfs-helper again" - # KEEP THE GROUP: only the USER must be root for a setuid bit to - # mean root, and the group is how this system records which package - # owns a file. It forced root:root. - _cs2="$(_slice_fn "$_hlv" cmd_setid)" - printf '%s' "$_cs2" | grep -q '_setid_apply "$mode" root "$f"' \ - && _p="$_p;the setid apply throws away the package group" - printf '%s' "$_cs2" | grep -q '_grp_now' \ - || _p="$_p;it does not preserve the group the file already has" - # TWO TABLES, ONE ANSWER. The "what upstream installs" table - # hard-coded `root`, so it ran right after the request table and - # undid it: `su is now 4755 root:p_shadow` followed by `su is now - # 4755 root:root`, 17 changes to reach the wrong one (1.14.260). - _kn="$(_setid_table "$_hlv")" - printf '%s' "$_kn" | grep -q '^/usr/bin/su|[0-7]*|root|' \ - && _p="$_p;the upstream table still forces root:root on su" - printf '%s' "$_kn" | grep -q '^/usr/bin/su|[0-7]*|-|' \ - || _p="$_p;su's group is not marked as keep-what-is-there" - # ONE TABLE (1.14.263): lfs-helper said 4755 and setid.conf 4711, so - # shadow-pam and `setid --run` corrected each other. The function - # reads the file; no path may be written into it again. - _slice_fn "$_hlv" _setid_known | grep -q '^/usr/bin/' \ - && _p="$_p;_setid_known carries its own list again, beside setid.conf" - _slice_fn "$_hlv" cmd_setid | grep -q 'setid.conf decides this one' \ - || _p="$_p;a requested bit and setid.conf are both applied to one file" - # a setGID entry must keep its NAMED group -- there the group is the point - printf '%s' "$_kn" | grep -q 'dbus-daemon-launch-helper|4750|messagebus' \ - || _p="$_p;a setgid entry lost the group that is its purpose" - printf '%s' "$_cs2" | grep -q 'if \[ "$grp" = "-" \]' \ - || _p="$_p;the keep marker is not honoured when applying" - # the fixture file must carry a NON-root group, or "keep the group" - # and "force root" are indistinguishable (a root-created file is - # root:root either way -- the first version of this check compared - # nothing) - _sgo="$( - R="$T/sg"; rm -rf "$R"; mkdir -p "$R"; : > "$R/su"; chmod 755 "$R/su" - _othergrp="$(awk -F: '$1!="root" && $3>0 {print $1; exit}' /etc/group)" - chgrp "$_othergrp" "$R/su" 2>/dev/null || _othergrp=root - warn() { :; }; ok() { :; } - _setid_state() { stat -c '%a %U:%G' "$1"; } - eval "$(_slice_fn "$_hlv" _setid_apply)" - _setid_apply 4755 "$(stat -c '%G' "$R/su")" "$R/su" >/dev/null 2>&1 - stat -c '%a %U:%G' "$R/su" )" - case "$_sgo" in - "4755 root:root") _p="$_p;the group became root instead of staying put" ;; - "4755 root:"*) ;; # kept whatever it was - *) _p="$_p;setid apply produced '$_sgo'" ;; - esac - for _f in passwd chage gpasswd newgrp unix_chkpwd; do - grep -q "/$_f *|" "$_sc" \ - || _p="$_p;$_f is not in setid.conf, so its bit stays lost" - done - # and it must FIX, in a run, not just report - _spo="$( - R="$T/sp"; rm -rf "$R"; mkdir -p "$R/bin" - : > "$R/bin/su"; chmod 755 "$R/bin/su" - printf '%s | 4711 | root | -\n' "$R/bin/su" > "$R/setid.conf" - printf '#!/bin/sh\nexit 0\n' > "$R/pm"; chmod 755 "$R/pm" - SETID_CONF="$R/setid.conf" SHADOW_PAM_PM="$R/pm" \ - SHADOW_PAM_SU="$R/bin/su" bash "$_sp" >/dev/null 2>&1 - stat -c '%a' "$R/bin/su" 2>/dev/null )" - [ "$_spo" = 4711 ] \ - || _p="$_p;the setuid bit is reported but not applied (mode $_spo)" - else - _p="$_p;there is no step that makes su use pam and keeps its bits" - fi - - # ...and linux-pam is now a WANTED package, not an avoided one: su - # without a password needs pam_exec, which needs pam - grep -qE '^book +linux-pam' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ - || _p="$_p;nothing installs linux-pam, so passwordless su cannot work" - grep -qE '^avoid +linux-pam' "$_av" \ - && _p="$_p;linux-pam is both built and avoided" - _lpo="$(python3 - "$pm_src" "$(dirname "$LFS_TOOL")/stacks/services.stack" <<'PYLP' -import sys, types, os -os.environ["PKGUSR_CONFIG"] = "/dev/null" -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] -print("OK" if o.index("linux-pam") < o.index("app-users") - else "linux-pam is built after the accounts that need it") -PYLP -)" - [ "$_lpo" = OK ] || _p="$_p;$_lpo" - # the shadow rebuild is the dangerous half and must stay manual - grep -B14 -E '^book +linux-pam' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ - | grep -q 'update shadow --reinstall' \ - || _p="$_p;nothing says shadow must be rebuilt for pam to be read" - grep -qE '^(book|git|tar) +shadow' "$(dirname "$LFS_TOOL")"/stacks/*.stack \ - && _p="$_p;shadow is rebuilt by a stack, which can lock you out of su" - _ac3="$(python3 - "$pm_src" "$_ms" "$_sw3" <<'PYAV' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -bad = [] -for f in sys.argv[2:]: - rows = mod._parse_stack_file(f) - av = [r["target"] for r in rows if r["kind"] == "avoid"] - for must in ("systemd", "qt6", "cups", "texlive", "openjdk"): - if must not in av: - bad.append("%s does not avoid %s" % (f.split("/")[-1], must)) - # INCLUDE ONCE: owning the "already spliced" set per include LINE meant - # avoid was spliced twice over, 332 entries where there are 158 - tg = [r["target"] for r in rows] - for t in ("qt6", "systemd"): - if tg.count(t) > 1: - bad.append("%s lists %s %d times" % (f.split("/")[-1], t, tg.count(t))) -print("BAD: " + "; ".join(bad) if bad else "OK") -PYAV -)" - [ "$_ac3" = OK ] || _p="$_p;$_ac3" - _io3="$(python3 - "$pm_src" "$_sw3" "$_ms" <<'PYINC' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -sw = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] -me = [r["target"] for r in mod._parse_stack_file(sys.argv[3])] -problems = [] -# every media entry appears in the composed sway plan, exactly once -for t in me: - if sw.count(t) != 1: - problems.append("%s appears %d times in sway" % (t, sw.count(t))) -# and in the same relative order -if [t for t in sw if t in me] != me: - problems.append("the included entries are reordered") -# the include lands after what webkitgtk needs and before the compositor -for need in ("gtk4", "libsoup3", "mesa"): - if need in sw and sw.index(need) > sw.index("webkitgtk"): - problems.append("%s comes after webkitgtk" % need) -if "sway" in sw and sw.index("sway") < sw.index("webkitgtk"): - problems.append("the compositor is built before the media chain") -print("BAD: " + "; ".join(problems) if problems else "OK") -PYINC -)" - [ "$_io3" = OK ] || _p="$_p;include composition is wrong: $_io3" - # a cycle must be refused, not recursed - printf '# t\n# stack-version: 0\ninclude zzb\n' > "$T/zza.stack" - printf '# t\n# stack-version: 0\ninclude zza\n' > "$T/zzb.stack" - _cyc2="$(python3 - "$pm_src" "$T/zza.stack" 2>&1 <<'PYCY' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -try: - mod._parse_stack_file(sys.argv[2]) - print("ACCEPTED") -except ValueError as e: - print("circular" if "circular" in str(e) else "refused: %s" % e) -except RecursionError: - print("RECURSED") -PYCY -)" - case "$_cyc2" in - circular) ;; - *) _p="$_p;a circular include is not refused cleanly: $_cyc2" ;; - esac -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "the media chain is one list, included by sway, and a cycle is refused" -fi - -# ---- a new stack keeps the conventions (1.14.207) -------------------------- # -# vision.stack is the first stack added since the rules grew, and a stack -# file that skips them fails silently: no stack-version means the -# one-release-one-number check has nothing to compare, and an out-of-book -# entry with no probe installs and says nothing about whether it works. -# -# Checked, not assumed: `git ls-remote` lists opencv branch 5.x AND tag -# 5.0.0, so the tag is pinned rather than the branch. -_p="" -for _sf in "$(dirname "$LFS_TOOL")"/stacks/*.stack; do - [ -f "$_sf" ] || continue - _n="$(basename "$_sf")" - grep -q '^# stack-version:' "$_sf" \ - || _p="$_p;$_n has no stack-version line" - # every git/tar entry (an out-of-book one) needs a have= or the stack - # cannot tell installed from not, and an expect= or nothing proves it - _miss="$(python3 - "$pm_src" "$_sf" <<'PYCONV' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -for r in mod._parse_stack_file(sys.argv[2]): - if r["kind"] not in ("git", "tar"): - continue - o = r["opts"] - if not (o.get("have") or o.get("prog")): - print("NOSTATE %s" % r["target"]) - if not o.get("expect"): - print("NOPROBE %s" % r["target"]) -PYCONV -)" - case "$_miss" in - *NOSTATE*) _p="$_p;$_n: an out-of-book entry has no have= or prog=: ${_miss}" ;; - esac -done -# and the opencv entry specifically: pinned, and building where the package -# user owns the tree -_vs="$(dirname "$LFS_TOOL")/stacks/vision.stack" -if [ -f "$_vs" ]; then - grep -qE '^git +opencv' "$_vs" || _p="$_p;opencv is not in vision.stack" - grep -A4 -E '^git +opencv' "$_vs" | grep -q 'ref=5\.' \ - || _p="$_p;opencv follows a branch instead of a pinned tag" - grep -A8 -E '^git +opencv' "$_vs" | grep -q 'cmake -S . -B build' \ - || _p="$_p;opencv builds outside the tree the package user owns" - grep -A8 -E '^git +opencv' "$_vs" | grep -q 'OPENCV_GENERATE_PKGCONFIG=ON' \ - || _p="$_p;the .pc file upstream stopped generating is not asked for" - grep -A8 -E '^git +opencv' "$_vs" | grep -q 'cmake --build build' \ - || _p="$_p;the build ignores CMAKE_BUILD_PARALLEL_LEVEL" - grep -B32 -E '^git +opencv' "$_vs" | grep -q 'ls-remote' \ - || _p="$_p;nothing records how the ref was checked" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "every stack file keeps the conventions, and opencv is pinned and probed" -fi - -# ---- info printed everything, then crashed (1.14.206) ---------------------- # -# `packagemanager info x264 openh264 gst10-plugins-ugly` printed all three -# packages and then: -# NameError: name '_states' is not defined -# Three bugs in two lines: `_states` was never initialised in cmd_info (it -# belongs to verify's loop, where the lines were copied from), `name` does -# not exist in that scope -- `anchor` does -- and the line was there twice. -# py_compile cannot see any of it: a NameError needs the line to RUN. -_p="" -if [ -f "$pm_src" ]; then - _ci="$(_slice_def "$pm_src" cmd_info)" - [ -n "$_ci" ] || _p="$_p;cmd_info cannot be read" - printf '%s' "$_ci" | grep -q '_states\[name\]' \ - && _p="$_p;cmd_info still assigns _states[name], which does not exist" - printf '%s' "$_ci" | grep -q 'counts, _states = {}, {}' \ - || _p="$_p;_states is used in cmd_info without being initialised" - # and it RUNS: a name error only shows up when the line executes - rm -rf "$T/ib"; mkdir -p "$T/ib/pkgusr/p_zzdemo" - printf "name='zzdemo'\nname_version='zzdemo-1'\n" \ - > "$T/ib/pkgusr/p_zzdemo/install_last" - _io="$(PKGUSR_BASE=$T/ib PKGUSR_CONFIG=/dev/null \ - python3 "$pm_src" info zzdemo 2>&1)" - case "$_io" in - *Traceback*|*NameError*) _p="$_p;info still crashes: $(printf '%s' "$_io" | tail -1)" ;; - esac - printf '%s' "$_io" | grep -q 'Total:' \ - || _p="$_p;info does not reach its own summary line" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "info runs to the end, and names the packages that need attention" -fi - -# ---- one release, one number (1.14.194) ------------------------------------ # -# The stack files carry `# stack-version:` and it DRIFTED: network and -# services said 1.14.167 and sway said 1.14.188 while the tools were at -# 1.14.193, because each release bumped them with a sed matching the exact -# previous number -- so any release that did not touch a stack broke the -# chain for every release after it. -# -# They ship as one artifact, so they carry one number, and this asserts it -# rather than trusting a sed. -_p="" -_ver="$(python3 "$LFS_TOOL" --version 2>/dev/null \ - | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)" -if [ -n "$_ver" ]; then - for _sf in "$(dirname "$LFS_TOOL")"/stacks/*.stack; do - [ -f "$_sf" ] || continue - _sv="$(sed -n 's/^# stack-version: *//p' "$_sf" | head -1)" - if [ -z "$_sv" ]; then - _p="$_p;$(basename "$_sf") carries no stack-version" - elif [ "$_sv" != "$_ver" ]; then - _p="$_p;$(basename "$_sf") says $_sv, the tools say $_ver" - fi - done -else - _p="$_p;the tool version cannot be read, so nothing can be compared to it" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "every stack file carries the same version as the tools" -fi - # ---- every tool reports its version ----------------------------------------- # # Without one there is no way to tell whether the copy inside the chroot is the # one that was just fixed -- and several rounds of diagnosis were spent on @@ -8886,25 +6782,22 @@ if [ -f "$pmt" ]; then && ok "a shared user joins the session group" \ || bad "a shared user cannot reach the display or audio" jx="$(sed -n '/^def _join_xdg_runtime_group/,/^def _setup_application_user/p' "$pmt")" - # THE GROUP IS NAMED (1.14.286, the user's decision): <collector - # prefix>_xdg-runtime, created in the collector range, joined by the main - # user and every shared account. Reading it off the directory found the - # person's own group, refused, and nothing ever set one up. - grep -q '^def xdg_runtime_group' "$pmt" \ - && ok "the session group has one name, from the collector prefix" \ - || bad "the session group is not named" + # the group must be READ from the directory, never assumed by name + case "$jx" in + *"getgrgid(os.stat(rt).st_gid)"*) + ok "the group is read from the runtime directory" ;; + *) bad "the session group name is hardcoded" ;; + esac case "$jx" in - *"_create_collector_group(g)"*) ok "the session group is created in the collector range" ;; - *) bad "the session group is never created" ;; + *'u_xdg_runtime"'*) bad "a specific group name is hardcoded" ;; + *) ok "no group name is baked in" ;; esac + # a personal group is not a shared group case "$jx" in - *'for who in (main, u)'*) ok "the main user joins it too (chgrp needs membership)" ;; - *) bad "the main user is not in the session group" ;; + *"which is not a shared "*) + ok "it refuses to add the user to a personal group" ;; + *) bad "an application user could join the person's own group" ;; esac - _aus2="$(dirname "$LFS_TOOL")/stacks/app-users.sh" - grep -q 'cat > /usr/bin/xdg-runtime-share' "$_aus2" \ - && ok "the session applies the group (xdg-runtime-share)" \ - || bad "nothing applies the group to a runtime directory made at boot" # both /run/user/<uid> and /tmp/xdg-<user> layouts xd="$(sed -n '/^def _xdg_runtime_dir_of/,/^def _join_xdg/p' "$pmt")" for p in "/run/user" "/tmp/xdg-"; do @@ -8962,41 +6855,10 @@ if [ -f "$pmt" ]; then ok "nothing is written when the program is not installed" ;; *) bad "a launcher could point at a program that does not exist" ;; esac - # a hand-written file is kept; one of ours that went stale is rewritten - # (1.14.289: ~/bin/browser-mini kept running a renamed-away u_browser) + # and neither file is clobbered if it is already there case "$de" in - *'"Comment=Runs as the application user " not in cur'*) ok "a hand-written menu entry is kept" ;; - *) bad "the desktop entry would overwrite a hand-written one" ;; - esac - wl="$(sed -n '/^def _write_launcher/,/^def cmd_user_launcher/p' "$pmt")" - # (1.14.292) the launcher builds the argument list itself: {} is what - # you type, a search term inside a URL, nothing without {} - _lt="$( _lh="$(mktemp -d)"; PKGUSR_CONFIG=/dev/null python3 - "$pmt" "$_lh" <<'PY' 2>/dev/null -import types, sys, os -m = types.ModuleType("pm"); exec(compile(open(sys.argv[1]).read(), "pm", "exec"), m.__dict__) -h = sys.argv[2]; m._run = lambda c: True; m._home_of = lambda u: h; m._app_home = lambda u: "/home/x/" + u -m._write_launcher("me", "u_a", "browser-mini https://s.example/r?q={}", "srch") -m._write_launcher("me", "u_a", "browser-mini https://fixed.example/", "fixed") -print(h) -PY -)" - _lt="$(printf '%s\n' "$_lt" | tail -1)" - if [ -f "$_lt/bin/srch" ]; then - for _l in srch fixed; do - sed 's/^exec su - [^ ]* -c .exec "\$0" "\$@". -- /printf "<%s>" /' "$_lt/bin/$_l" > "$_lt/$_l.t" - done - [ "$(bash "$_lt/srch.t" a b)" = "<browser-mini><--app-id><srch><https://s.example/r?q=a+b>" ] \ - && [ "$(bash "$_lt/srch.t")" = "<browser-mini><--app-id><srch><https://s.example/r>" ] \ - && [ "$(bash "$_lt/fixed.t" extra)" = "<browser-mini><--app-id><fixed><https://fixed.example/>" ] \ - && ok "the launcher passes {} as arguments or a search term, and nothing without {}" \ - || bad "the launcher's argument handling is wrong: $(bash "$_lt/srch.t" a b) / $(bash "$_lt/srch.t") / $(bash "$_lt/fixed.t" extra)" - rm -rf "$_lt" - else - bad "the launcher could not be generated" - fi - case "$wl" in - *"was not written by these tools -- left alone"*) ok "a hand-written launcher is kept, a stale one of ours rewritten" ;; - *) bad "a stale launcher is never rewritten" ;; + *"already exists -- left alone"*) ok "an existing menu entry is kept" ;; + *) bad "the desktop entry would be overwritten" ;; esac fi @@ -10413,7 +8275,7 @@ if [ -f "$pmt" ]; then import importlib.machinery as m, os pm = m.SourceFileLoader('pm', '$pmt').load_module() print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" - [ "$got" = "/usr/src /home/shared_user" ] \ + [ "$got" = "/usr/src /usr/src" ] \ && ok "an unconfigured system keeps the existing layout" \ || bad "the layout changed without being asked (got '$got')" printf 'collector_prefix=n\npkgusr_home=/usr/src/package_users\nappuser_home=/home/shared_users\n' \ @@ -10995,19 +8857,7 @@ if not hasattr(lfs, "_auto_file_block"): print(" FAIL a section that ships a file instead of commands is ignored") sys.exit(1) -try: - raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() -except OSError as e: - print(" SKIP rc.site needs a book on disk (%s)" % e.strerror) - sys.exit(77) -# A CHECK'S PRECONDITION IS NOT A FAILURE. rc.site is a SysV section: the -# systemd book has no ch-config-site at all, so this reported FAIL on a -# machine where nothing was wrong, and every clean run had to be read with -# "except that one" in mind. Say what is missing instead (1.14.147). -if "ch-config-site" not in raw: - print(" SKIP this book has no rc.site section (SysV only) -- " - "build one with lfs-sysvbook, or point the suite at a SysV book") - sys.exit(77) +raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() blk = lfs._auto_file_block(raw, "ch-config-site", "/etc/sysconfig/rc.site") if not blk: problems.append("rc.site is not extracted from the book") @@ -12056,9 +9906,8 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then _cb="$(_slice_fn "$helper_src" cmd_build)" _p="" - # (1.14.274: pruned rather than filtered -- either form excludes it) case "$_cb" in - *'-not -path "$_bld/*"'*|*'-path "$_bld" \) -prune'*) ;; + *'-not -path "$_bld/*"'*) ;; *) _p="$_p;the build scratch is still scanned into every manifest" ;; esac case "$_cb" in @@ -20942,11 +18791,8 @@ printf '%s' "$_rs" | grep -q 'setuid/setgid' \ || _p="$_p;the report does not say what kind of request it was" _slice_fn "$_hlpU" cmd_build | grep -q 'report_setid_requests "$owner"' \ || _p="$_p;the build never reports them" -# it prints each line -- and KEEPS the ledger. Clearing it here made this a -# notice you had to be watching for: `su` lost its setuid bit at install time -# and six hours later nothing on the system could say so (1.14.147). The -# file is now cleared by `setid --run`, once the bit has been applied and -# read back. +# it prints each line and then clears the file, so a later build does not +# repeat requests that have been dealt with mkdir -p "$T/sidrep" printf 'chmod 2755 /usr/bin/fcrontab\n' > "$T/sidrep/.pkgusr-setid.requests" _o="$( warn() { echo "$*" >&2; } @@ -20957,12 +18803,8 @@ case "$_o" in *"chmod 2755 /usr/bin/fcrontab"*) ;; *) _p="$_p;the refused command is not shown" ;; esac -case "$_o" in - *"lfs-helper setid"*) ;; - *) _p="$_p;the report does not say how to apply them later" ;; -esac [ -s "$T/sidrep/.pkgusr-setid.requests" ] \ - || _p="$_p;the ledger is emptied by the report, so nothing can ask later" + && _p="$_p;the requests are repeated on every later build" if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" @@ -20971,2130 +18813,6 @@ else ok "a refused setuid/setgid bit is written down and reported to root" fi -# ---- the setid ledger, and su (1.14.147) ----------------------------------- # -# `su -` said -# setgid: Operation not permitted -# on a system where everything else worked. shadow installs su with -# install -m 4755 ... /usr/bin/su -# and the install WRAPPER downgraded that to 755 -- correctly, a package user -# may not set it -- but, unlike the chmod wrapper beside it, wrote nothing -# down. Same refusal, one of two places implementing the rule. (And a -# package user CAN setuid its own file: the result runs as the package user, -# which is worse than no bit at all.) -_p="" -_hlpS="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hlpS" ]; then - # the wrapper is written by make-wrappers as a heredoc; extract and run it - mkdir -p "$T/instw/home" "$T/instw/dest" - python3 - "$_hlpS" "$T/instw/install" <<'PYIN' -import sys, re -src = open(sys.argv[1], encoding="utf-8", errors="replace").read() -m = re.search(r'cat > "\$WRAPPERS/install" <<\'EOF\'\n(.*?)\nEOF\n', src, re.S) -open(sys.argv[2], "w").write(m.group(1) if m else "") -PYIN - if [ ! -s "$T/instw/install" ]; then - _p="$_p;the install wrapper could not be extracted" - else - chmod 755 "$T/instw/install" - bash -n "$T/instw/install" || _p="$_p;the install wrapper does not parse" - grep -q 'pkgusr-setid.requests' "$T/instw/install" \ - || _p="$_p;install -m 4755 is downgraded without recording it" - # a setid install records the file that ACTUALLY landed - echo x > "$T/instw/su" - ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 4755 su "$T/instw/dest/su" ) >/dev/null 2>&1 - [ -f "$T/instw/dest/su" ] || _p="$_p;the file was not installed at all" - grep -q "chmod 4755 $T/instw/dest/su" "$T/instw/home/.pkgusr-setid.requests" 2>/dev/null \ - || _p="$_p;the refused setuid bit was not written down" - # into a DIRECTORY: the request must name the installed path, not the source - ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 4755 su "$T/instw/dest/" ) >/dev/null 2>&1 - grep -q "chmod 4755 $T/instw/dest/su" "$T/instw/home/.pkgusr-setid.requests" 2>/dev/null \ - || _p="$_p;installing into a directory records the wrong path" - # an ordinary mode is not recorded - : > "$T/instw/home/.pkgusr-setid.requests" - ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 644 su "$T/instw/dest/su2" ) >/dev/null 2>&1 - [ -s "$T/instw/home/.pkgusr-setid.requests" ] \ - && _p="$_p;an ordinary install is recorded as a set-id request" - # a FAILED install must not leave a request behind: root would chmod - # a file that is not there - : > "$T/instw/home/.pkgusr-setid.requests" - ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 4755 nosuchfile "$T/instw/dest/nope" ) >/dev/null 2>&1 - [ -s "$T/instw/home/.pkgusr-setid.requests" ] \ - && _p="$_p;a failed install still records a request" - fi - # A RECORDER MUST NOT BE THE REASON A BUILD FAILS. - # - # The ledger was left root:root by a root-side truncation, the install - # wrapper (running as the package user) could not append, and the SHELL - # printed "Permission denied" for the failed redirection -- which the - # swallowed-failure check then treated as a failed install. openssh had - # finished installing (1.14.192). - _sr="$(_slice_fn "$_hlpS" _sid_record)" - [ -n "$_sr" ] || _p="$_p;the recorder still writes without checking it can" - printf '%s' "$_sr" | grep -q '\[ -w "$f" \]' \ - || _p="$_p;it redirects first and lets the shell report the failure" - printf '%s' "$_sr" | grep -q 'could not record' \ - || _p="$_p;a refusal it cannot record is silent" - # `printf >> f 2>/dev/null || echo` does NOT work: prove the shape - _rt="$T/ro"; rm -rf "$_rt"; mkdir -p "$_rt"; : > "$_rt/led"; chmod 400 "$_rt/led" - if [ "$(id -u)" = 0 ]; then - : # root can write a 400 file; the check below would prove nothing - else - _bad="$( { printf 'x\n' >> "$_rt/led" 2>/dev/null || echo NOTE; } 2>&1 )" - case "$_bad" in - *"Permission denied"*) ;; # exactly the trap, still true - *) _p="$_p;the redirection trap this guards against no longer exists" ;; - esac - fi - # and every root write hands the ledger back to its user - grep -q 'chown "$owner:$owner" "$_sidf"' "$_hlpS" \ - || _p="$_p;a root truncation leaves a ledger the package user cannot write" - # (the whole file: $_sid is a _slice_fn slice and cmd_setid contains a - # line starting with }, so the slice ends before this) - grep -q 'gave $u back' "$_hlpS" \ - || _p="$_p;setid does not repair a ledger an older release left root-owned" - - # the command that applies them - _sid="$(_slice_fn "$_hlpS" cmd_setid)" - [ -n "$_sid" ] || _p="$_p;there is no way to apply the refused bits" - printf '%s' "$_sid" | grep -q '_setid_apply' \ - || _p="$_p;setid does not use the one applier" - _ap="$(_slice_fn "$_hlpS" _setid_apply)" - # A SETUID FILE OWNED BY A PACKAGE USER RUNS AS THAT PACKAGE USER, which - # is how `su` came to be setuid something that is not root. The mode - # without the chown is not a fix. - printf '%s' "$_ap" | grep -q 'chown "root:' \ - || _p="$_p;the mode is set without giving the file to root" - printf '%s' "$_ap" | grep -q '_setid_state' \ - || _p="$_p;setid does not read back what it changed" - printf '%s' "$_ap" | grep -q 'nosuid' \ - || _p="$_p;a nosuid mount is not mentioned when the bit does not stick" - # su is checked even when no request was recorded (an older build) - _kn="$(_setid_table "$_hlpS")" - for _f in /usr/bin/su /usr/bin/passwd /usr/bin/newgrp; do - printf '%s' "$_kn" | grep -q "^$_f|" \ - || _p="$_p;$_f is not among the programs checked on disk" - done - # every flag the report suggests must be one the command accepts - _rs2="$(_slice_fn "$_hlpS" report_setid_requests)" - case "$_rs2" in - *"setid --run"*) - printf '%s' "$_sid" | grep -q -- '--run' \ - || _p="$_p;the suggested --run is not a flag setid accepts" ;; - esac - # it is dispatched, and it is in the help - grep -q ' setid)' "$_hlpS" || _p="$_p;setid is not dispatched" - grep -q 'lfs-helper setid' "$_hlpS" || _p="$_p;setid is not in the help" - # and verify says so instead of waiting to be asked - _slice_fn "$_hlpS" cmd_verify | grep -q 'setid' \ - || _p="$_p;verify is silent about refused setid bits" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a refused setuid bit is recorded by BOTH wrappers and applied by setid" -fi - -# ---- lfs-backup: back up, restore without replacing (1.14.291) ------------ # -# lfs-config (a git-repo manager) became lfs-backup: one config of paths, -# an archive with modes and owner NAMES, symlinks stored as their files, and a -# restore that creates what is missing, leaves identical files alone and puts -# everything else beside the live file as .new -- account databases always. -_p="" -_bk="$(dirname "$LFS_TOOL")/lfs-backup" -if [ -f "$_bk" ] && [ "$(id -u)" = 0 ]; then - bash -n "$_bk" || _p="$_p;lfs-backup does not parse" - grep -qE '^TOOLS[[:space:]]*:?=.*lfs-backup' "$(dirname "$LFS_TOOL")/Makefile" \ - || _p="$_p;lfs-backup is not installed with the other tools" - grep -qE '^TOOLS[[:space:]]*:?=.*lfs-config' "$(dirname "$LFS_TOOL")/Makefile" \ - && _p="$_p;the retired lfs-config is still installed" - _B="$T/bk"; rm -rf "$_B"; mkdir -p "$_B/old/etc/profile.d" "$_B/new/etc/profile.d" \ - "$_B/old/home/u1/.config/a" "$_B/old/home/u1/d" "$_B/new/home/u2" "$_B/old/root" "$_B/new/root" - printf 'root:x:0:0::/root:/bin/bash\nu1:x:1500:1500::/home/u1:/bin/bash\n' > "$_B/old/etc/passwd" - printf 'root:x:0:\nu1:x:1500:\n' > "$_B/old/etc/group" - printf "export PKGUSR_SYS_MAIN_USER='u1'\n" > "$_B/old/etc/profile.d/pkgusr-system.sh" - printf 'root:x:0:0::/root:/bin/bash\nu2:x:1600:1600::/home/u2:/bin/bash\n' > "$_B/new/etc/passwd" - printf 'root:x:0:\nu2:x:1600:\n' > "$_B/new/etc/group" - printf "export PKGUSR_SYS_MAIN_USER='u2'\n" > "$_B/new/etc/profile.d/pkgusr-system.sh" - echo "real" > "$_B/old/home/u1/d/a.conf"; ln -s ../../d/a.conf "$_B/old/home/u1/.config/a/conf" - echo "old fstab" > "$_B/old/etc/fstab"; echo "new fstab" > "$_B/new/etc/fstab" - printf 'root:secret::\n' > "$_B/old/etc/shadow"; chmod 600 "$_B/old/etc/shadow" - chown -R 1500:1500 "$_B/old/home/u1"; chown 1600:1600 "$_B/new/home/u2" - printf '[user]\n~/.config/a/conf\n[system]\n/etc/fstab\n/etc/shadow\n' > "$_B/conf" - export LFS_BACKUP_CONF="$_B/conf" - LFS_BACKUP_ROOT="$_B/old" bash "$_bk" create --run "$_B/a.tgz" >/dev/null 2>&1 \ - || _p="$_p;create failed" - [ "$(stat -c %a "$_B/a.tgz" 2>/dev/null)" = 600 ] || _p="$_p;the archive (it holds secrets) is not mode 600" - tar -xzOf "$_B/a.tgz" files/user/.config/a/conf 2>/dev/null | grep -qx real \ - || _p="$_p;a symlink was not stored as the file it points to" - _out="$(LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/a.tgz" --run 2>&1)" - grep -qx real "$_B/new/home/u2/.config/a/conf" 2>/dev/null \ - || _p="$_p;a missing file was not created in the new main user's home" - [ "$(stat -c %u "$_B/new/home/u2/.config/a/conf" 2>/dev/null)" = 1600 ] \ - || _p="$_p;the old main user's file does not belong to the new main user" - grep -qx "new fstab" "$_B/new/etc/fstab" || _p="$_p;an existing file was REPLACED" - grep -qx "old fstab" "$_B/new/etc/fstab.new" 2>/dev/null || _p="$_p;a differing file got no .new" - [ ! -e "$_B/new/etc/shadow" ] && [ -f "$_B/new/etc/shadow.new" ] \ - || _p="$_p;a missing /etc/shadow was dropped in instead of written as .new" - printf '%s' "$_out" | grep -q 'TO MERGE BY HAND' || _p="$_p;the .new files are not listed at the end" - # SECTIONS (1.14.292): --skip leaves a named section out of a restore; - # clean removes this archive's untouched .new files and keeps an edited one - printf '[system:acct]\n/etc/shadow\n[system:other]\n/etc/fstab\n' > "$_B/conf2" - export LFS_BACKUP_CONF="$_B/conf2" - rm -f "$_B/new/etc/shadow.new" "$_B/new/etc/fstab.new" - LFS_BACKUP_ROOT="$_B/old" bash "$_bk" create --run "$_B/b.tgz" >/dev/null 2>&1 - LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/b.tgz" --skip acct --run >/dev/null 2>&1 - [ ! -e "$_B/new/etc/shadow.new" ] || _p="$_p;--skip did not leave the named section out" - [ -f "$_B/new/etc/fstab.new" ] || _p="$_p;--skip left out a section it was not given" - LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/b.tgz" --only acct --run >/dev/null 2>&1 - echo "merged" >> "$_B/new/etc/shadow.new" - LFS_BACKUP_ROOT="$_B/new" bash "$_bk" clean "$_B/b.tgz" --run >/dev/null 2>&1 - [ ! -e "$_B/new/etc/fstab.new" ] || _p="$_p;clean left an untouched .new" - [ -f "$_B/new/etc/shadow.new" ] || _p="$_p;clean removed a .new that had been edited" - # PLAN (lfs-backup 1.2.0): a plan line set to skip is not restored, a - # changed target is used, and a [root] file is root's whatever it recorded - rm -f "$_B/new/etc/shadow.new" "$_B/new/etc/fstab.new" "$_B/plan" - LFS_BACKUP_ROOT="$_B/new" bash "$_bk" plan "$_B/b.tgz" "$_B/plan" >/dev/null 2>&1 - sed -i -e 's/^restore | [a-z]* *| system:acct /skip | x | system:acct /' \ - -e 's#| /etc/fstab | /etc/fstab$#| /etc/fstab | /etc/fstab.moved#' "$_B/plan" - LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/b.tgz" --plan "$_B/plan" --run >/dev/null 2>&1 - [ ! -e "$_B/new/etc/shadow.new" ] || _p="$_p;a plan line set to skip was restored" - grep -qx "old fstab" "$_B/new/etc/fstab.moved" 2>/dev/null || _p="$_p;a changed plan target was not used" - # SECRETS OWNER-ONLY (1.2.1): a world-readable recorded mode is not copied - [ "$(stat -c %a "$_B/new/etc/shadow.new" 2>/dev/null)" != "" ] || true - grep -q 'SECRET_RE=' "$_bk" || _p="$_p;a secret file is restored with whatever mode it had" - # A NAME IN BOTH [user] AND [root] (1.2.2): the plan key includes the kind - grep -q -- '-v n="$scope:$name"' "$_bk" || _p="$_p;a plan line for ~/.bashrc applies to both the user's and root's" - # MERGE (1.3.0): k removes the .new, n takes its content and keeps the - # live file's mode; the account databases cannot be taken whole - printf '[system:m]\n/etc/fstab\n' > "$_B/conf3"; export LFS_BACKUP_CONF="$_B/conf3" - echo "old fstab" > "$_B/old/etc/fstab"; echo "live fstab" > "$_B/new/etc/fstab"; chmod 640 "$_B/new/etc/fstab" - rm -f "$_B/new/etc/fstab.new" - LFS_BACKUP_ROOT="$_B/old" bash "$_bk" create --run "$_B/c.tgz" >/dev/null 2>&1 - LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/c.tgz" --run >/dev/null 2>&1 - printf 'n\n' | LFS_BACKUP_ROOT="$_B/new" setsid bash "$_bk" merge "$_B/c.tgz" >/dev/null 2>&1 - grep -qx "old fstab" "$_B/new/etc/fstab" || _p="$_p;merge n did not take the backup's content" - [ "$(stat -c %a "$_B/new/etc/fstab")" = 640 ] || _p="$_p;merge n changed the live file's mode" - [ ! -e "$_B/new/etc/fstab.new" ] || _p="$_p;merge n left the .new behind" - grep -q 'account database: no whole take' "$_bk" || _p="$_p;merge would take an old passwd whole" - unset LFS_BACKUP_CONF -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "lfs-backup stores files by name and mode, and restores without replacing anything" -fi - -# ---- a zero exit is not proof, in FOUR dialects (1.14.180) ----------------- # -# llvm's configure died -- -# CMake Error at CMakeLists.txt:6 (set): -# -- Configuring incomplete, errors occurred! -# -- and the run printed "llvm: done (all phases complete)". The swallowed- -# failure check knew make's wording only; cmake fails before make runs. And -# its `ninja` alternative required `***`, which ninja never prints, so the -# one build system it was supposed to cover was not covered either. -_p="" -_hl8="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl8" ]; then - _pat="$(grep -m1 -o '"\^(make.*could not compile"' "$_hl8" | sed 's/^"//; s/"$//')" - [ -n "$_pat" ] || _p="$_p;the swallowed-failure pattern cannot be read" - # every dialect this system builds in must be caught, and nothing else - _chk() { # _chk <name> <line> <want: 1 catch / 0 ignore> - if printf '%s\n' "$2" | grep -m1 -E "$_pat" >/dev/null 2>&1; then - [ "$3" = 1 ] || _p="$_p;'$1' is treated as a failure and is not one" - else - [ "$3" = 0 ] || _p="$_p;$1 is not recognised as a failure" - fi - } - _chk "cmake configure" "CMake Error at CMakeLists.txt:6 (set):" 1 - _chk "cmake summary" "-- Configuring incomplete, errors occurred!" 1 - _chk "autotools" "configure: error: cannot find libfoo" 1 - _chk "meson" "meson.build:12:0: ERROR: Dependency not found" 1 - _chk "meson install" "ERROR: Unhandled python OSError." 1 - _chk "cargo" "error: could not compile \`gst-plugin-rtp\`" 1 - _chk "make" "make: *** [Makefile:408: all] Error 2" 1 - _chk "ninja" "ninja: build stopped: subcommand failed." 1 - _chk "configuring done" "-- Configuring done" 0 - _chk "cmake warning" "CMake Warning at CMakeLists.txt:6" 0 -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a failure is recognised in make, ninja, cmake, meson, autotools and cargo" -fi - -# ---- how many jobs, once (1.14.179) ---------------------------------------- # -# make reads MAKEFLAGS, cmake reads CMAKE_BUILD_PARALLEL_LEVEL, cargo reads -# CARGO_BUILD_JOBS, and ninja reads NOTHING (nproc+2). Setting MAKEFLAGS -# limited one of the four, so "use at most 13" was not sayable: a cmake or -# rust package still ran wide. -_p="" -_hl7="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl7" ]; then - for _v in CMAKE_BUILD_PARALLEL_LEVEL CARGO_BUILD_JOBS; do - grep -q "$_v" "$_hl7" || _p="$_p;--jobs does not limit $_v" - done - # READ THE VALUE THAT WILL BE PASSED, not the ambient one. build.env's - # MAKEFLAGS=-j13 is in $envpass by then; ${MAKEFLAGS:-} in a chroot root - # shell is empty, so CARGO_BUILD_JOBS was never set and rust built on - # every core (1.14.181). - grep -qF 'grep -oE "MAKEFLAGS=' "$_hl7" \ - || _p="$_p;the job count in build.env is not seen at all" - _jn="$( envpass="KF6_PREFIX=/usr MAKEFLAGS=-j13 NINJAJOBS=13" - printf '%s' "$envpass" | grep -oE "MAKEFLAGS='?-j ?[0-9]+" \ - | grep -oE '[0-9]+' | head -n1 )" - [ "$_jn" = 13 ] || _p="$_p;a build.env-style MAKEFLAGS parses to '$_jn'" - # derived from MAKEFLAGS when --jobs was not given, or build.env limits - # only make - grep -qF 'grep -oE '"'"'\-j ?[0-9]+'"'"'' "$_hl7" \ - || _p="$_p;a job count in MAKEFLAGS is not carried to cmake and cargo" -fi -if [ -f "$pm_src" ]; then - # NINJA IS NOT THE ONLY ONE THAT IGNORES THE ENVIRONMENT. rust's own - # page says the build is limited by "a COMBINATION of --jobs <N> on each - # invocation of ./x.py and CARGO_BUILD_JOBS"; with only the variable set - # it used every core. - _jp="$(grep -A22 '_jobs = opts.pop' "$pm_src")" - printf '%s' "$_jp" | grep -q 'x\.py' \ - || _p="$_p;./x.py is never given a job limit" - printf '%s' "$_jp" | grep -q 'cargo' \ - || _p="$_p;cargo build lines are never given a job limit" - # an explicit -j in the script or stack entry must win - printf '%s' "$_jp" | grep -q -- '\\s--jobs' \ - || _p="$_p;a line that already names --jobs would be given a second one" - _out="$(python3 - <<'PYJ' -import re -body = ("cmake .. && ninja\n./x.py build\n./x.py install\n" - "cargo cbuild --release\ncargo cinstall -j 4 --release\nninja -j2 check\n") -n = "13" -for pat, rep in ( - (r"(?m)(^|&&[ \t]*|;[ \t]*)(ninja\b(?![^\n]*(\s-j|\s--jobs)))", r"\1\2 -j" + n), - (r"(?m)^(\s*\.?/?x\.py\s+\w+(?![^\n]*(\s-j|\s--jobs)))", r"\1 --jobs " + n), - (r"(?m)^(\s*cargo\s+c?(?:build|install)\b(?![^\n]*(\s-j|\s--jobs)))", r"\1 -j " + n)): - body = re.sub(pat, rep, body) -print("OK" if ("x.py build --jobs 13" in body - and "x.py install --jobs 13" in body - and "cargo cbuild -j 13" in body - and "cargo cinstall -j 4 --release" in body - and "&& ninja -j13" in body - and "ninja -j2 check" in body) else "WRONG:\n" + body) -PYJ -)" - case "$_out" in - OK) ;; - *) _p="$_p;the job patch does not land where it must ($_out)" ;; - esac - # A MESSAGE THAT NAMES A THING MUST BE CHECKED AGAINST THE THING. - # With only [all] jobs = 13 set, the run said - # machine.conf [rust]: x.py --jobs 13 - # machine.conf has [rust] but the script has no meson/cmake line - # -- options NOT applied - # for a file with no [rust] section, about options that HAD been applied - # and with nothing left to apply (1.14.183). - _ms2="$(_slice_def "$pm_src" apply_machine_opts)" - _os2="$(_slice_def "$pm_src" _opt_sections)" - [ -n "$_os2" ] || _p="$_p;the section a value came from is never worked out" - _out2="$(PM_MACHINE_CONF=$T/mo.conf PM_PACKAGE_FIXES=$T/nof.conf python3 - "$pm_src" <<'PYMO' -import sys, types, os, io, contextlib -open(os.environ["PM_MACHINE_CONF"], "w").write( - "[all]\njobs = 13\n\n[webkitgtk]\njobs = 8\n\n[odd]\nweird = yes\n") -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -d = os.path.dirname(os.environ["PM_MACHINE_CONF"]) -for name, body in (("rust", "install_pkg() {\n./x.py build\n}\n"), - ("webkitgtk", "build_pkg() {\ncmake ..\nninja\n}\n"), - ("odd", "install_pkg() {\nmake install\n}\n")): - sp = os.path.join(d, "install_" + name) - open(sp, "w").write(body) - buf = io.StringIO() - with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): - mod.apply_machine_opts(name, sp) - print("%s|%s" % (name, (buf.getvalue().strip().splitlines() or [""])[0])) -PYMO -)" - case "$_out2" in - *"rust|machine.conf [all]: x.py --jobs 13"*) ;; - *) _p="$_p;a jobs value from [all] is credited to a section that does not exist" ;; - esac - case "$_out2" in - *"webkitgtk|machine.conf [webkitgtk]: ninja -j8"*) ;; - *) _p="$_p;the overridden [all] value is still credited" ;; - esac - # per LINE: a glob over the whole capture matched "rust|" here and - # "NOT applied" three lines later, and called that a bug - printf '%s\n' "$_out2" | grep '^rust|' | grep -q 'NOT applied' \ - && _p="$_p;options that were applied are reported as not applied" - case "$_out2" in - *"odd|machine.conf [odd] set weird"*) ;; - *) _p="$_p;an option that really cannot be applied is not reported" ;; - esac - # [all] in machine.conf, with a named section still winning - _mo="$(_slice_def "$pm_src" _machine_opts_for)" - printf '%s' "$_mo" | grep -q '"all"' \ - || _p="$_p;there is no way to set an option for every package" - _got="$(PM_MACHINE_CONF=$T/mc.conf PM_PACKAGE_FIXES=$T/none.conf python3 - "$pm_src" <<'PYALL' -import sys, types, os -open(os.environ["PM_MACHINE_CONF"], "w").write( - "[all]\njobs = 13\n\n[webkitgtk]\njobs = 8\n") -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -print(mod._machine_opts_for("mesa").get("jobs"), - mod._machine_opts_for("webkitgtk").get("jobs")) -PYALL -)" - [ "$_got" = "13 8" ] \ - || _p="$_p;[all] and a named section do not resolve to '13 8' but '$_got'" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "one job count reaches make, cmake, cargo and ninja" -fi - -# ---- a book command that works once (1.14.176) ----------------------------- # -# libtiff's page moves its html docs into a versioned directory. The first -# install creates /usr/share/doc/libtiff-4.7.1/tiff; the second cannot: -# mv: cannot overwrite '.../tiff': Directory not empty -# Nothing was wrong with the system -- the destination was the package's own -# output from last time, and a reinstall means replacing it -- but the repair -# gate did not recognise it, so a 38-package rebuild pass stopped on a -# directory the package itself had put there. -_p="" -_hl6="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl6" ]; then - _cs="$(_slice_fn "$_hl6" _clear_stale_dest)" - [ -n "$_cs" ] || _p="$_p;a destination that already exists is still unrecognised" - printf '%s' "$_cs" | grep -q 'cannot overwrite' \ - || _p="$_p;the message mv actually prints is not matched" - # ONLY THE PACKAGE'S OWN OUTPUT. Another package's directory in the way - # is a conflict, and deleting it would be the worst possible repair. - printf '%s' "$_cs" | grep -q 'stat -c %U' \ - || _p="$_p;it would delete a directory without checking who owns it" - printf '%s' "$_cs" | grep -q 'that is a conflict' \ - || _p="$_p;another package's directory is not distinguished from stale output" - # THE GATE DECIDES; THE LOOP ACTS. Called from the gate, the repair - # removed the directory -- correctly, and said so -- and then the retry - # loop ran ZERO rounds, because the loop only counts what it fixed - # itself. The obstacle was gone and the package failed anyway (1.14.178). - # (whole file, not the cmd_build slice: _slice_fn stops at the first line - # starting with }, and cmd_build is 1700 lines with several of those -- - # the slice ended long before these lines, so both assertions were - # testing an empty string) - # A RULE ON DISK IS NOT A RULE IN FORCE. light installs - # 90-backlight.rules, and until udevd is told, the brightness files keep - # their old group and light still cannot write them -- on a system where - # every file is exactly where it should be. A reboot hides it - # (1.14.194). - _ur="$(_slice_fn "$_hl6" _udev_reload_if_rules_changed)" - [ -n "$_ur" ] || _p="$_p;a new udev rule is never loaded" - grep -q '_udev_reload_if_rules_changed "$man"' "$_hl6" \ - || _p="$_p;the reload is defined and never called" - printf '%s' "$_ur" | grep -q 'control --reload' \ - || _p="$_p;udevd is not told the rules changed" - # ONLY the subsystems the rules mention: a bare trigger re-adds every - # device on the machine, which on a desktop can drop a session - printf '%s' "$_ur" | grep -q -- '--subsystem-match=' \ - || _p="$_p;it re-triggers every device instead of the ones concerned" - # THE ACTION MATTERS AS MUCH AS THE SUBSYSTEM. `udevadm trigger` sends - # **change** by default and light's rule is ACTION=="add", so the - # trigger fired, the rule did not match, and brightness stayed - # root:root -- while the output said it had re-triggered the subsystem - # (1.14.197). - printf '%s' "$_ur" | grep -q -- '--action=' \ - || _p="$_p;the trigger sends change, so an ACTION==add rule never runs" - printf '%s' "$_ur" | grep -q 'ACTION==' \ - || _p="$_p;the action is not read out of the rule" - _acts="$(printf '%s\n' \ - 'ACTION=="add", SUBSYSTEM=="backlight", RUN+="/bin/chgrp video x"' \ - | sed -n 's/.*ACTION=="\([a-z]*\)".*/\1/p' | sort -u | tr '\n' ' ')" - [ "$_acts" = "add " ] \ - || _p="$_p;the action parses to '$_acts', not the one in the rule" - # a bare `udevadm trigger` with no match option (the comment mentioning - # one does not count) - # a line that triggers WITHOUT a match option. (`trigger([^-]|$)` - # matched the space in `trigger --subsystem-match`, so the first version - # of this failed on correct code.) - printf '%s' "$_ur" | grep -v '^ *#' | grep 'udevadm trigger' \ - | grep -qv -- '--subsystem-match' \ - && _p="$_p;there is a bare udevadm trigger in there" - printf '%s' "$_ur" | grep -q 'this is a chroot' \ - || _p="$_p;in a chroot it tries to talk to a udevd that is not there" - # prove the subsystem is read out of the rule - mkdir -p "$T/udv/usr/lib/udev/rules.d" - printf 'ACTION=="add", SUBSYSTEM=="backlight", RUN+="/bin/true"\n' \ - > "$T/udv/usr/lib/udev/rules.d/90-backlight.rules" - _sub="$(sed -n 's/.*SUBSYSTEM=="\([a-z0-9_-]*\)".*/\1/p' \ - "$T/udv/usr/lib/udev/rules.d/90-backlight.rules")" - [ "$_sub" = backlight ] \ - || _p="$_p;the subsystem parses to '$_sub', not the one in the rule" - - # AN ABSOLUTE PATH IN THE MESSAGE IS THE ANSWER. - # - # install: cannot create regular file - # '/usr/lib/udev/rules.d/90-backlight.rules': Permission denied - # - # The file does not exist (nothing to hand over) and the directory is - # another package's (a grant is what is needed), but the extractor only - # fired on a `cd` plus a RELATIVE path -- so nothing was extracted and - # light's udev rule stopped the install (1.14.193). - grep -q '_instdirs' "$_hl6" \ - || _p="$_p;an absolute path in a permission message is not used" - _got2="$(printf '%s\n' \ - "install: cannot create regular file '/usr/lib/udev/rules.d/90-backlight.rules': Permission denied" \ - "mkdir: cannot create directory '/usr/share/vim/vimfiles': Permission denied" \ - | grep -aoE "cannot create (regular file|directory) '/[^']+'" \ - | sed "s/.*'\(\/[^']*\)'.*/\1/" | sed 's|/[^/]*$||' | sort -u | tr '\n' ' ')" - case "$_got2" in - "/usr/lib/udev/rules.d /usr/share/vim "*) ;; - *) _p="$_p;the parent directory parses to '$_got2'" ;; - esac - grep -q '_clear_stale_dest "$log" "${owner:-}" && _fixed_something=1' "$_hl6" \ - || _p="$_p;clearing a stale destination does not trigger a retry" - grep -q '_clear_stale_dest .* && _perm_ish=1' "$_hl6" \ - && _p="$_p;the repair acts in the gate, so nothing re-runs the phase" - # -F: the thing being looked for IS a regex, so matching it as one - # matches something else entirely - grep -qF "cannot overwrite '[^']+'" "$_hl6" \ - || _p="$_p;the gate cannot recognise the failure it now repairs" - # CMAKE NAMES THE DESTINATION ACROSS THREE LINES, and /usr/bin is sticky - # (drwxrwxr-t root:install), so a group grant cannot help: only the - # file's owner may replace it, and /usr/bin/llvm-ar belonged to p_rust. - # The meson handover beside it needs "Unhandled python OSError" and - # "Installing SRC to DIR" lines, so it saw nothing -- zero rounds again - # (1.14.182). - _tc="$(_slice_fn "$_hl6" _take_cmake_dest)" - [ -n "$_tc" ] || _p="$_p;a cmake destination owned by another package is not handled" - printf '%s' "$_tc" | grep -qF 'file INSTALL cannot copy file' \ - || _p="$_p;cmake's own wording is not matched" - printf '%s' "$_tc" | grep -q "joined into one line" \ - || _p="$_p;a message spanning three lines cannot be read line by line" - printf '%s' "$_tc" | grep -q 'root|""' \ - || _p="$_p;it would take a file owned by root" - grep -q '_take_cmake_dest "$log" "${owner:-}" && _fixed_something=1' "$_hl6" \ - || _p="$_p;the handover does not trigger a retry" - # prove it on the real three-line text - mkdir -p "$T/cm/bin" - printf 'x\n' > "$T/cm/bin/llvm-ar" - printf 'CMake Error at tools/llvm-ar/cmake_install.cmake:52 (file):\n file INSTALL cannot copy file\n "/src/build/bin/llvm-ar" to\n "%s": Permission denied.\n' \ - "$T/cm/bin/llvm-ar" > "$T/cm/log" - _dest="$(tr '\n' ' ' < "$T/cm/log" \ - | grep -aoE 'file INSTALL cannot copy file[[:space:]]+"[^"]+"[[:space:]]+to[[:space:]]+"[^"]+"' \ - | sed 's/.*[[:space:]]to[[:space:]]*"//; s/"$//')" - [ "$_dest" = "$T/cm/bin/llvm-ar" ] \ - || _p="$_p;the destination parses to '$_dest', not the file cmake named" - - # and a second round must find nothing, or the loop spins - printf '%s' "$_cs" | grep -q '\[ -e "$path" \] || continue' \ - || _p="$_p;a path already removed is reported as fixed again" - # A REPAIR MUST NOT BE ABLE TO FAIL THE THING IT REPAIRS. The gate - # called it with "$user"; cmd_build calls the package user `owner`, so - # `set -u` aborted a git build whose install had just finished -- with a - # line number and nothing else (1.14.177). - _cb="$(_slice_fn "$_hl6" cmd_build)" - printf '%s' "$_cb" | grep -qE '"\$user"' \ - && _p="$_p;cmd_build reads \$user, which it never sets" - printf '%s' "$_cb" | grep -q '_clear_stale_dest "$log" "${owner:-}"' \ - || _p="$_p;the repair is called with a variable that may be unset" - printf '%s' "$_cs" | grep -q 'user="${2:-}"' \ - || _p="$_p;the repair itself dies when called with nothing" - # every variable the gate passes must be one the function assigns first - printf '%s' "$_cb" | grep -q 'owner=' \ - || _p="$_p;owner is read in cmd_build and never assigned there" - # prove all three outcomes on real text - mkdir -p "$T/stale/mine/tiff" "$T/stale/theirs/x" - echo x > "$T/stale/mine/tiff/i.html" - printf "mv: cannot overwrite '%s': Directory not empty\n" "$T/stale/mine/tiff" > "$T/stale/log1" - printf "make: *** Error 1\n" > "$T/stale/log2" - _out="$( warn() { echo "!! $*"; }; note() { echo "$*"; } - eval "$_cs" - _clear_stale_dest "$T/stale/log1" "$(id -un)"; echo "rc1=$?" - _clear_stale_dest "$T/stale/log2" "$(id -un)"; echo "rc2=$?" )" - case "$_out" in - *rc1=0*) ;; - *) _p="$_p;a stale destination the package owns is not cleared" ;; - esac - case "$_out" in - *rc2=0*) _p="$_p;an unrelated failure is treated as a stale destination" ;; - esac - [ -e "$T/stale/mine/tiff" ] && _p="$_p;it reported clearing the directory and did not" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a destination the package itself left behind is cleared, anyone else's is not" -fi - -# ---- the help is a list, and lists go stale (1.14.175) --------------------- # -# The grouped `packagemanager --help` -- the one a reader actually uses -- -# had no `stack`, `probe`, `late-deps`, `reclaim`, `bootstrap` or `env`, and -# stack is the primary command. The module docstring was worse: a "Roadmap" -# marking check, search, dependencies, install, remove and reload-pkg-list as -# [TODO] years after they were implemented. -# -# argparse knows the real list, and says it when given a bad command. Ask it. -_p="" -if [ -f "$pm_src" ]; then - _choices="$(python3 "$pm_src" __no_such_command__ 2>&1 \ - | sed -n "s/.*choose from //p" | tr -d "')(" | tr ',' '\n' \ - | sed "s/^ *//;s/'//g" | grep -v '^$')" - [ -n "$_choices" ] || _p="$_p;argparse does not list its own commands" - _grouped="$(python3 "$pm_src" --help 2>&1 \ - | sed -n '/commands, by what you are doing/,$p')" - # aliases do not need their own line - for _c in $_choices; do - case "$_c" in - deps|setup|which-pkg|regen-script|add-dir-to-nimgnu|nimgnu) continue ;; - esac - printf '%s' "$_grouped" | grep -q -- "$_c" \ - || _p="$_p;$_c is a command and is not in the grouped help" - done - # and no second list anywhere in the file - # the SHAPE of a roadmap entry, not the word: the comment explaining this - # bug quotes [TODO], and matching that matched the explanation - grep -qE '^ +[a-z_-]+ +\[(TODO|IMPLEMENTED)\]' "$pm_src" \ - && _p="$_p;the module docstring still keeps a roadmap of commands" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "every command appears in the grouped help, and there is one list" -fi - -# ---- a package owns what it installed, not where (1.14.158) --------------- # -# /usr/include/gstreamer-1.0/gst/mse belonged to p_linux-headers, so -# gst-plugins-bad could not install into its own directory. The repair handed -# it one file per round; after three rounds the bundled-copy heuristic -# concluded meson was vendoring linux-headers. Every step reasoned correctly -# from a wrong fact, and the wrong fact was one line: -# chown -R "$_hacct:$_hacct" /usr/include -# which gave the headers package every header on the system. -_p="" -_k5="$(dirname "$LFS_TOOL")/lfs-kernel" -if [ -f "$_k5" ]; then - # PROSE IS NOT CODE: the comment that explains this bug quotes the old - # line, and the first version of this assertion matched the comment. - grep -vE '^[[:space:]]*#' "$_k5" \ - | grep -qE 'chown -R [^ ]*_hacct[^ ]* /usr/include' \ - && _p="$_p;the headers package still takes ownership of all of /usr/include" - # it must chown the paths it copied, which means knowing what they were - grep -q "find . -mindepth 1 -printf" "$_k5" \ - || _p="$_p;nothing records which headers were installed" - grep -q 'xargs -0 -r chown -h' "$_k5" \ - || _p="$_p;the chown is not limited to the recorded paths" -fi -# AND THE REPAIR FOR A MACHINE THAT ALREADY HAS IT. One tree-wide chown is -# not a per-package problem: /usr/include/glib-2.0 belonged to -# p_linux-headers, so glib2 could not install its own headers either -# (1.14.166). -if [ -f "$pm_src" ]; then - python3 "$pm_src" reclaim --help 2>/dev/null | grep -q -- '--all' \ - || _p="$_p;the ownership repair has to be run one package at a time" - _rcl="$(_slice_def "$pm_src" cmd_reclaim)" - printf '%s' "$_rcl" | grep -q '_pkg_dirs()' \ - || _p="$_p;--all does not mean every installed package" - printf '%s' "$_rcl" | grep -q 'getattr(args, "all", False)' \ - || _p="$_p;a package with no manifest is an error under --all" -fi - -# and the audit that finds this class after the fact -if [ -f "$pm_src" ]; then - python3 "$pm_src" late-deps --help >/dev/null 2>&1 \ - || _p="$_p;there is no way to ask which packages were built too early" - _ld="$(_slice_def "$pm_src" cmd_late_deps)" - [ -n "$_ld" ] || _p="$_p;cmd_late_deps could not be read out of the tool" - printf '%s' "$_ld" | grep -q '3600' \ - || _p="$_p;two packages built in one run would be reported as a problem" - # SLACK THAT HIDES THE CASE IS TOO MUCH SLACK. A day of it filed - # `ffmpeg ... alsa-lib +0 day(s)` as fine, and alsa-lib a few HOURS after - # ffmpeg is the original bug (1.14.168). - printf '%s' "$_ld" | grep -q '> 86400' \ - && _p="$_p;a gap of hours is still treated as no gap" - printf '%s' "$_ld" | grep -q 'hour(s)" % (gap // 3600)' \ - || _p="$_p;a sub-day gap is reported as zero days" - printf '%s' "$_ld" | grep -q 'only a probe can settle it' \ - || _p="$_p;a same-session gap is presented as if timestamps settled it" - # ...and the probe exists, reads the stacks, and names the rebuild - _pr="$(_slice_def "$pm_src" cmd_probe)" - [ -n "$_pr" ] || _p="$_p;there is no way to run the expect= probes after the fact" - printf '%s' "$_pr" | grep -q 'expect' \ - || _p="$_p;the probe runner does not read the stacks' probes" - printf '%s' "$_pr" | grep -q -- '--only' \ - || _p="$_p;a failed probe does not say how to rebuild" - python3 "$pm_src" probe --help >/dev/null 2>&1 \ - || _p="$_p;probe is not a command" - # AN EDITED STACK IS NEVER OVERWRITTEN, so the tool can carry probes that - # no installed stack has -- and "no expect= probes in the stacks" then - # reads as "nothing to check" (1.14.169). - printf '%s' "$_pr" | grep -q 'stack.new' \ - || _p="$_p;no probes does not say the newer stacks are waiting" - printf '%s' "$_pr" | grep -q -- '--take-new' \ - || _p="$_p;it does not say how to take the newer stack" - grep -q -- '--take-new' "$pm_src" \ - || _p="$_p;the suggested --take-new is not a flag that exists" - # A STACK ENTRY DOES NOT REBUILD AN INSTALLED PACKAGE. The expect= - # failure suggested `stack sway.stack --only sdl3 --run` -- with the - # .stack suffix, which is not a stack name, and running `install`, which - # answers "everything is installed and up to date". Re-running it could - # never fix the feature (1.14.171). - _ex2="$(grep -n 'feature check failed' -A25 "$pm_src")" - printf '%s' "$_ex2" | grep -q '_rebuild_cmd' \ - || _p="$_p;a failed feature check still points at a command that no-ops" - # AND THE COMMAND DEPENDS ON THE KIND OF ENTRY. `update --reinstall - # --regenerate` regenerates from the BOOK, and gst-plugins-rs is a git - # entry with no book page -- that advice could not work at all - # (1.14.202). - printf '%s' "$_ex2" | grep -q 'e\["kind"\] == "book"' \ - || _p="$_p;a git entry is told to regenerate from a book page it does not have" - printf '%s' "$_ex2" | grep -q -- '--run --force' \ - || _p="$_p;nothing says --force is what rebuilds a git entry" - # ...and have= proving less than it looks is worth saying out loud: a - # part-installed package satisfies it - printf '%s' "$_ex2" | grep -q 'one file is not the whole package' \ - || _p="$_p;a part-installed package that satisfies have= is not explained" - printf '%s' "$_ex2" | grep -q 'will NOT rebuild it' \ - || _p="$_p;it does not say that re-running the entry cannot help" - printf '%s' "$_ex2" | grep -q 'stack %s --only %s --run' \ - && _p="$_p;the suggested stack name still carries the .stack suffix" - # the drift notice must show WHAT differs: two equal entry counts argued - # there was nothing to take, for files differing in every entry - grep -q 'the entry count is the same' "$pm_src" \ - || _p="$_p;the newer-stack notice offers a count as its only evidence" - grep -q 'line(s) in the file; the plan below' "$pm_src" \ - || _p="$_p;take-new and the header report two different entry counts" - - # A PROBE THAT FAILS BECAUSE THE PACKAGE IS NOT THERE IS A DIFFERENT - # FINDING. gst10-libav had never been built -- it was missing from the - # stack -- and the report said "the feature is NOT in the installed - # package", sending the reader to look for a build option (1.14.170). - printf '%s' "$_pr" | grep -q 'NOT INSTALLED' \ - || _p="$_p;a probe cannot tell an absent package from a missing feature" - printf '%s' "$_pr" | grep -q 'pkg.lst' \ - || _p="$_p;installed-ness is assumed rather than checked" - # ...and a probe can be wrong about a path as easily as a package can be - # wrong about a feature, so show what it said - printf '%s' "$_pr" | grep -q 'the check said' \ - || _p="$_p;a failing probe's own output is thrown away" - # the sdl3 probe must not name one library file: SDL3 installs - # libSDL3.so.0.<n> and the bare symlink is not always there - # $_sw belongs to the next block; name the file, do not borrow a variable - grep -q 'libSDL3.so\*' "$(dirname "$LFS_TOOL")/stacks/sway.stack" \ - || _p="$_p;the SDL3 probe names a path that may not exist" - # AN UNMATCHED DEPENDENCY IS NOT A CLEAN RESULT. A silent miss would - # under-report exactly the thing being looked for. - printf '%s' "$_ld" | grep -q 'match no INSTALLED package' \ - || _p="$_p;dependencies it could not resolve are not reported" - # ...and say that an unmatched name is usually just a package that is not - # installed, which is nothing to fix: 103 of them read as 103 problems - printf '%s' "$_ld" | grep -q 'cannot have been built after anything' \ - || _p="$_p;unmatched names read as unfinished work" - printf '%s' "$(_slice_def "$pm_src" _built_when)" | grep -q 'pkg.lst' \ - || _p="$_p;the install time does not come from the manifest" - # THREE DIFFERENT ANSWERS. On a real system the report was 40 packages - # in one list: some fixable by order, some CIRCULAR (cairo requires - # glib2, mesa requires libxcb -- no order exists, only a second pass), - # and a third of them documentation tools (graphviz, docutils). One - # list means the real findings are read as noise (1.14.159). - printf '%s' "$_ld" | grep -q 'no order exists' \ - || _p="$_p;a circular dependency is reported as an ordering mistake" - printf '%s' "$_ld" | grep -q 'in the stack: put' \ - || _p="$_p;an orderable case does not say what to change" - printf '%s' "$(_slice_def "$pm_src" _depends_on)" | grep -q 'seen' \ - || _p="$_p;the dependency walk can loop for ever" - grep -q '_DOC_DEPS' "$pm_src" \ - || _p="$_p;documentation tools are mixed in with real features" - # the rebuild order is not the report order - printf '%s' "$_ld" | grep -q 'rebuild, in this order' \ - || _p="$_p;there is no rebuild plan" - printf '%s' "$_ld" | grep -q 'g_order.get(n, set()) & nodes' \ - || _p="$_p;the rebuild plan is not ordered by dependency" - printf '%s' "$_ld" | grep -q 'add --run to do it' \ - || _p="$_p;--rebuild writes without being asked" - # INSTALL DOES NOT REBUILD. The first plan emitted `install --run - # --regenerate`, which answered "Nothing to do -- everything is installed - # and up to date" 28 times and looked like a successful pass. - _rc="$(_slice_def "$pm_src" _rebuild_cmd)" - printf '%s' "$_rc" | grep -q '"update"' \ - || _p="$_p;the rebuild plan uses a verb that does not rebuild" - printf '%s' "$_rc" | grep -q -- '--reinstall' \ - || _p="$_p;the rebuild plan does not force a reinstall" - printf '%s' "$_ld" | grep -q 'install %s --run --regenerate' \ - && _p="$_p;a rebuild line still says install" - # RECOMMENDED IS A DEPENDENCY TOO when asking whether an order exists: - # BLFS lists mesa->libxcb and cairo->glib2 as recommended, and with - # `required` alone the report advised putting mesa before libxcb. - printf '%s' "$_ld" | grep -q 'g_order' \ - || _p="$_p;only required edges are used to detect a cycle" - printf '%s' "$_ld" | grep -q '"required", "recommended"' \ - || _p="$_p;a recommended cycle is reported as an ordering mistake" - # A CYCLE BROKEN ALPHABETICALLY IS NOT AN ORDER. base needs mesa, mesa - # needs libxcb, bad needs base -- one cluster with no entry point, and - # `sorted(left)[0]` then emitted gst10-plugins-bad before - # gst10-plugins-base, cairo and fontconfig before freetype2. That is the - # exact mistake the plan exists to prevent (1.14.163). - printf '%s' "$_ld" | grep -v '^[[:space:]]*#' | grep -q 'ready = \[sorted(left)' \ - && _p="$_p;a dependency cycle is still broken alphabetically" - printf '%s' "$_ld" | grep -q 'sum(1 for m in cs' \ - || _p="$_p;the cycle pick ignores how many packages wait for it" - # ORDER THE CLUSTERS, NOT THE NODES. Picking one node out of a cycle and - # returning to the ready scan stranded the rest at the end: freetype2 and - # harfbuzz came out LAST, after fontconfig and cairo, which require - # freetype2. Tarjan's components form a DAG, so they sort (1.14.164). - printf '%s' "$_ld" | grep -q '_strong' \ - || _p="$_p;a dependency cycle is still broken one node at a time" - printf '%s' "$_ld" | grep -q 'iterative' \ - || _p="$_p;the component walk recurses, and this graph has 400 nodes" - # prove it on the shape that failed - _scc="$(PKGUSR_BASE=$T/scc PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYSCC' -import sys, types, os, time, io, contextlib -base = os.environ["PKGUSR_BASE"] + "/pkgusr" -def mk(name, nv, req, rec, opt, old=True): - d = os.path.join(base, "p_" + name); os.makedirs(d, exist_ok=True) - f = os.path.join(d, "install_last") - open(f, "w").write("name='%s'\nname_version='%s'\nrequired=(%s)\n" - "recommended=(%s)\noptional=(%s)\n" - % (name, nv, req, rec, opt)) - w = time.time() - (400 * 86400 if old else 86400) - os.utime(f, (w, w)) - if not old: - p = os.path.join(d, "pkg.lst"); open(p, "w").write("x\n"); os.utime(p, (w, w)) -mk("freetype2", "freetype-2.14.1", "", "'harfbuzz-12.1.0'", "'brotli-1.1.0'") -mk("harfbuzz", "harfbuzz-12.1.0", "'freetype-2.14.1'", "", "'brotli-1.1.0'") -mk("fontconfig", "fontconfig-2.17.1", "'freetype-2.14.1'", "", "'brotli-1.1.0'") -mk("cairo", "cairo-1.18.6", "'freetype-2.14.1' 'fontconfig-2.17.1'", "", "'brotli-1.1.0'") -mk("brotli", "brotli-1.1.0", "", "", "", old=False) -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -class A: - kind = "recommended,optional"; package = []; verbose = False; why = False - include_docs = False; rebuild = True; run = False -buf = io.StringIO() -with contextlib.redirect_stdout(buf): - mod.cmd_late_deps(A()) -order = [ln.split()[2] for ln in buf.getvalue().splitlines() - if ln.strip().startswith("packagemanager update")] -ok = (order.index("freetype2") < order.index("fontconfig") - and order.index("harfbuzz") < order.index("cairo")) -print("CLUSTER_FIRST" if ok else "WRONG %s" % order) -PYSCC -)" - case "$_scc" in - *CLUSTER_FIRST*) ;; - *) _p="$_p;a cycle is still rebuilt after the packages that need it ($_scc)" ;; - esac - # AND THE OPTIONAL EDGE DECIDES THE ORDER TOO. The graph was - # required+recommended, so mpg123 was rebuilt before pulseaudio -- and - # mpg123's missing feature IS pulseaudio (1.14.165). - printf '%s' "$_ld" | grep -q '"required", "recommended", "optional"' \ - || _p="$_p;the rebuild order ignores the optional edges it exists for" - _opt="$(PKGUSR_BASE=$T/optord PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYOPT' -import sys, types, os, time, io, contextlib -base = os.environ["PKGUSR_BASE"] + "/pkgusr" -def mk(name, nv, req, rec, opt, old=True): - d = os.path.join(base, "p_" + name); os.makedirs(d, exist_ok=True) - f = os.path.join(d, "install_last") - open(f, "w").write("name='%s'\nname_version='%s'\nrequired=(%s)\n" - "recommended=(%s)\noptional=(%s)\n" - % (name, nv, req, rec, opt)) - w = time.time() - (400 * 86400 if old else 86400) - os.utime(f, (w, w)) - if not old: - p = os.path.join(d, "pkg.lst"); open(p, "w").write("x\n"); os.utime(p, (w, w)) -mk("mpg123", "mpg123-1.33.3", "", "'alsa-lib-1.2.14'", "'PulseAudio-17.0'") -mk("pulseaudio", "pulseaudio-17.0", "", "'alsa-lib-1.2.14'", "") -mk("alsa-lib", "alsa-lib-1.2.14", "", "", "", old=False) -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -class A: - kind = "recommended,optional"; package = []; verbose = False; why = False - include_docs = False; rebuild = True; run = False -buf = io.StringIO() -with contextlib.redirect_stdout(buf): - mod.cmd_late_deps(A()) -o = [ln.split()[2] for ln in buf.getvalue().splitlines() - if ln.strip().startswith("packagemanager update")] -print("PULSE_FIRST" if o.index("pulseaudio") < o.index("mpg123") else "WRONG %s" % o) -PYOPT -)" - case "$_opt" in - *PULSE_FIRST*) ;; - *) _p="$_p;an optional dependency is still rebuilt after its user ($_opt)" ;; - esac - - # and a suspicious position must be explainable - python3 "$pm_src" late-deps --why --help >/dev/null 2>&1 \ - || _p="$_p;there is no way to see which edges were used" - printf '%s' "$_ld" | grep -q 'in dependency cycles' \ - || _p="$_p;a cycle is broken silently, so a second pass is a surprise" - # ...and prove it on the shape that failed: bad needs base, base needs - # mesa, mesa needs base (a cycle) -- base must come out before bad - _cyc="$(PKGUSR_BASE=$T/cyc PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYCYC' -import sys, types, os, time -base = os.environ["PKGUSR_BASE"] + "/pkgusr" -def mk(name, nv, req, rec, opt, old=True): - d = os.path.join(base, "p_" + name); os.makedirs(d, exist_ok=True) - f = os.path.join(d, "install_last") - open(f, "w").write("name='%s'\nname_version='%s'\nrequired=(%s)\n" - "recommended=(%s)\noptional=(%s)\n" - % (name, nv, req, rec, opt)) - when = time.time() - (400 * 86400 if old else 1 * 86400) - os.utime(f, (when, when)) - if not old: - p = os.path.join(d, "pkg.lst"); open(p, "w").write("x\n") - os.utime(p, (when, when)) -mk("gst10-plugins-base", "gst-plugins-base-1.28.1", - "'mesa-25.2.3'", "", "'alsa-lib-1.2.14'") -mk("gst10-plugins-bad", "gst-plugins-bad-1.28.1", - "'gst-plugins-base-1.28.1'", "", "'libnice-0.1.22'") -mk("mesa", "mesa-25.2.3", "'gst-plugins-base-1.28.1'", "", "'libnice-0.1.22'") -mk("alsa-lib", "alsa-lib-1.2.14", "", "", "", old=False) -mk("libnice", "libnice-0.1.22", "", "", "", old=False) -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -class A: - kind = "recommended,optional"; package = []; verbose = False - include_docs = False; rebuild = True; run = False -import io, contextlib -buf = io.StringIO() -with contextlib.redirect_stdout(buf): - mod.cmd_late_deps(A()) -out = buf.getvalue() -order = [ln.split()[2] for ln in out.splitlines() - if ln.strip().startswith("packagemanager update")] -print("BASE_BEFORE_BAD" if order.index("gst10-plugins-base") - < order.index("gst10-plugins-bad") else "WRONG %s" % order) -PYCYC -)" - case "$_cyc" in - *BASE_BEFORE_BAD*) ;; - *) _p="$_p;in a cycle the package others wait for is still rebuilt last ($_cyc)" ;; - esac - - # AN EDGE THAT DOES NOT RESOLVE IS AN EDGE THAT DOES NOT EXIST. - # 119 dependency names did not match a package, and the rebuild order was - # computed from what was left: gst10-plugins-bad sorted BEFORE - # gst10-plugins-base (which it requires), fontconfig before freetype2. - # The mapping is in the data -- each script records name_version, the - # page title the deps are written with (1.14.162). - _di="$(_slice_def "$pm_src" _dep_index)" - [ -n "$_di" ] || _p="$_p;dependency titles are matched by guesswork" - printf '%s' "$_di" | grep -q "name_version='" \ - || _p="$_p;the title-to-package mapping ignores what each script records" - # resolve the two cases that broke the order, for real - _res="$(PKGUSR_BASE=$T/depidx PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYRES' -import sys, types, os -base = os.environ["PKGUSR_BASE"] + "/pkgusr" -for name, nv in (("gst10-plugins-base", "gst-plugins-base-1.28.1"), - ("gst10-plugins-bad", "gst-plugins-bad-1.28.1"), - ("freetype2", "freetype-2.14.1"), - ("fontconfig", "fontconfig-2.17.1")): - d = os.path.join(base, "p_" + name) - os.makedirs(d, exist_ok=True) - open(os.path.join(d, "install_last"), "w").write( - "name='%s'\nname_version='%s'\n" % (name, nv)) -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -inst = mod._pkg_dirs() -print(mod._resolve_dep("gst-plugins-base-1.28.1", inst), - mod._resolve_dep("FreeType-2.14.1", inst)) -PYRES -)" - case "$_res" in - *"gst10-plugins-base freetype2"*) ;; - *) _p="$_p;a dependency written as its page title still does not resolve ($_res)" ;; - esac - # --reinstall IS NOT A QUESTION ABOUT VERSIONS. With no recorded version - # `update curl --reinstall` said "nothing to update, 1 it cannot judge" - # and did nothing -- 28 times. - python3 - "$pm_src" <<'PYRI' || _p="$_p;--reinstall does nothing for a package with no recorded version" -import sys -src = open(sys.argv[1]).read() -i = src.index("no_record.append(name)") -blk = src[max(0, i - 1200):i] -sys.exit(0 if 'reinstall' in blk and '"(reinstall)", bv' in blk else 1) -PYRI - - # NO HOST, NO STALENESS. On a booted system the stamp records what was - # copied in during the build, so every later `make install` made all - # seven tools "differ" and packagemanager printed, before every command, - # an instruction to run sync-tools on a host that no longer exists - # (1.14.161). - _st="$(_slice_def "$pm_src" warn_if_tools_are_stale)" - printf '%s' "$_st" | grep -q '_in_chroot' \ - || _p="$_p;the host-staleness warning fires on a booted system" - printf '%s' "$(_slice_def "$pm_src" _in_chroot)" | grep -q '/proc/1/root' \ - || _p="$_p;the chroot test does not ask pid 1 where its root is" - # ...and whoever installs the tools records what they are - grep -q 'tool-stamps.json' "$(dirname "$LFS_TOOL")/Makefile" \ - || _p="$_p;make install leaves the stamp describing older copies" - # every flag the message suggests must exist - printf '%s' "$_st" | grep -q 'sync-tools --run' \ - && { python3 "$(dirname "$LFS_TOOL")/lfs" build-system sync-tools --help \ - >/dev/null 2>&1 || _p="$_p;the suggested sync-tools command does not exist"; } - - # and a stack run must raise the question itself - grep -q 'packagemanager late-deps --rebuild --run' "$pm_src" \ - || _p="$_p;a finished stack never mentions the second pass" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "the headers package owns only its headers, and late builds are findable" -fi - -# ---- the desktop's non-book pieces (1.14.184) ------------------------------ # -# light (backlight), wl-clipboard, and the icon/patched/Noto fonts are in no -# book, so the stack carries them -- and everything about them is ours to -# keep true: refs, URLs, and the archive layouts. -_p="" -_sw2="$(dirname "$LFS_TOOL")/stacks/sway.stack" -_fs="$(dirname "$LFS_TOOL")/stacks/fonts.sh" -if [ -f "$_sw2" ]; then - # jq is not in the book either, and its probe has to test the half of jq - # that can be missing: `jq --version` passes on a jq built without - # oniguruma, and `test`/`match`/`capture` are what most scripts use - # (1.14.198). - grep -qE '^tar +jq ' "$_sw2" || _p="$_p;jq is in no stack" - grep -A6 -E '^tar +jq ' "$_sw2" | grep -q 'test(' \ - || _p="$_p;the jq probe would pass on a jq with no regex" - grep -B10 -E '^tar +jq ' "$_sw2" | grep -q 'packaged oniguruma' \ - || _p="$_p;nothing records whether jq needs a system oniguruma" - # the probe string must survive the parser and still be runnable - _jqp="$(python3 - "$pm_src" "$_sw2" <<'PYJQ' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -for r in mod._parse_stack_file(sys.argv[2]): - if r["target"] == "jq": - print(r["opts"].get("expect", "")) - break -PYJQ -)" - case "$_jqp" in - *'test('*) ;; - *) _p="$_p;the jq probe does not survive the stack parser: '$_jqp'" ;; - esac - # it must be valid shell, whatever jq itself says - ( eval "jq() { :; }; $_jqp" ) >/dev/null 2>&1 \ - || _p="$_p;the jq probe is not runnable shell" - # wf-recorder's AUDIO IS CONDITIONAL and its --help says `-a, --audio` - # either way -- that text is outside the #ifdef. So the probe has to - # check what the binary LINKS, not what it prints, and the entry has to - # come after pipewire so the library is there at configure time - # (1.14.208). - grep -qE '^git +wf-recorder' "$_sw2" || _p="$_p;wf-recorder is in no stack" - grep -A6 -E '^git +wf-recorder' "$_sw2" | grep -q 'ldd /usr/bin/wf-recorder' \ - || _p="$_p;the wf-recorder probe does not check what it linked" - grep -A6 -E '^git +wf-recorder' "$_sw2" | grep -q -- '--help' \ - && _p="$_p;the probe reads help text, which says --audio either way" - _wo="$(python3 - "$pm_src" "$_sw2" <<'PYWF' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] -bad = [n for n in ("ffmpeg", "pipewire") - if n in o and o.index(n) > o.index("wf-recorder")] -print("LATE " + " ".join(bad) if bad else "OK") -PYWF -)" - [ "$_wo" = OK ] \ - || _p="$_p;wf-recorder configures before what it needs: $_wo" - # ZATHURA IS THREE PACKAGES and the core opens no formats at all: girara - # is its widget library, the backend is a separate plugin, and a zathura - # with no plugin starts, shows a grey window and says nothing. So the - # probe looks for the PLUGIN. All three share one release tag -- mixing - # tags is the usual way to get a zathura that cannot open a PDF - # (1.14.241). - for _e in girara zathura zathura-pdf-poppler imv; do - grep -qE "^git +$_e " "$_sw2" || _p="$_p;$_e is in no stack" - done - # NO INVENTED OPTIONS, NO INVENTED FILENAMES. `-Dsqlite=enabled` - # stopped meson with `Unknown option: "sqlite"` (zathura takes sqlite3 - # unconditionally and has no such option), and have=girara-gtk3.pc made - # the stack refuse to record a girara that had installed perfectly -- - # its meson.build says `pkg.generate(name: 'girara')`, so the file is - # girara.pc (1.14.244). - # zathura 2026.07.18 ALSO needs libxxhash, which the book mentions eight - # times and gives no page (1.14.245). Its checkout ships - # libxxhash.pc.in, so `libxxhash` really is the module meson asks for. - grep -qE '^git +xxhash ' "$_sw2" \ - || _p="$_p;zathura will stop at configure for libxxhash" - _xo="$(python3 - "$pm_src" "$_sw2" <<'PYXX' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] -bad = [n for n in ("xxhash", "girara") if n in o and o.index(n) > o.index("zathura")] -print("BAD: " + ", ".join(bad) if bad else "OK") -PYXX -)" - [ "$_xo" = OK ] || _p="$_p;built after zathura, which needs it: $_xo" - grep -A6 -E '^git +xxhash ' "$_sw2" | grep -q 'libxxhash.pc' \ - || _p="$_p;nothing proves xxhash installed the module meson wants" - grep -A6 -E '^git +zathura ' "$_sw2" | grep -q 'Dsqlite' \ - && _p="$_p;zathura is built with an option it does not have" - grep -A6 -E '^git +girara ' "$_sw2" | grep -q 'girara-gtk3.pc' \ - && _p="$_p;girara's have= names a file it does not install" - grep -A6 -E '^git +girara ' "$_sw2" | grep -q 'pkgconfig/girara.pc' \ - || _p="$_p;girara has no have= that its build actually produces" - _zt="$(grep -oE 'ref=2026\.[0-9.]+' "$_sw2" | sort -u | wc -l)" - [ "$_zt" = 1 ] \ - || _p="$_p;the zathura family does not share one release tag ($_zt seen)" - grep -A6 -E '^git +zathura-pdf-poppler' "$_sw2" | grep -q 'libpdf-poppler.so' \ - || _p="$_p;nothing proves zathura got a PDF backend" - grep -qE '^book +poppler' "$_sw2" \ - || _p="$_p;the pdf backend has no poppler to build against" - grep -qE '^avoid +poppler' "$_av" \ - && _p="$_p;poppler is both built and avoided" - grep -qE '^avoid +mupdf' "$_av" \ - || _p="$_p;the second pdf backend is no longer excluded" - # imv links libicuio and CALLS libicuuc, with -Wl,--no-undefined on the - # link line, so the indirect dependency is an ERROR: "undefined - # reference to symbol 'ubrk_close_78' ... DSO missing from command - # line" (1.14.251). - grep -A6 -E '^git +imv ' "$_sw2" | grep -q 'c_link_args=-licuuc' \ - || _p="$_p;imv will not link: libicuuc is not on its link line" - grep -B10 -E '^git +imv ' "$_sw2" | grep -q 'DSO missing' \ - || _p="$_p;nothing records why that flag is there" - # ...AND NOR IS ITS SHARED DIRECTORY. `appuser_home()` returned - # BASE_DIR, so `cfg app-users` ran `mkdir -p /usr/src/u_claude/Shared` - # and pointed ~/Shared/Claude into the package tree -- even after the - # homes had been moved to /home with usermod (1.14.256). - _ah="$(_slice_def "$pm_src" appuser_home)" - # (1.14.282: the default is ONE subdirectory, APPUSER_HOME_DEFAULT) - printf '%s' "$_ah" | grep -q 'or APPUSER_HOME_DEFAULT' \ - || _p="$_p;application users default to the package base, not their subdirectory" - grep -q '^APPUSER_HOME_DEFAULT = "/home/shared_user"$' "$pm_src" \ - || _p="$_p;application users do not default to /home/shared_user" - sed -n '/^def _relocate_app_home/,/^def /p' "$pm_src" | grep -q 'want = os.path.join(root, u)' \ - || _p="$_p;an application home is still moved to a hard-coded /home/<user>" - _aph="$(_slice_def "$pm_src" _app_home)" - [ -n "$_aph" ] || _p="$_p;nothing resolves an application user's real home" - printf '%s' "$_aph" | grep -q '_home_of(u) or' \ - || _p="$_p;a home moved with usermod is ignored" - grep -q 'share = os.path.join(_app_home(u), "Shared")' "$pm_src" \ - || _p="$_p;the shared directory is built from the wrong home" - _apo="$(PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYAH' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -mod._home_of = lambda u: "/home/u_moved" if u == "u_moved" else None -bad = [] -if mod.appuser_home() != "/home/shared_user": - bad.append("appuser_home() = %s" % mod.appuser_home()) -if mod._app_home("u_moved") != "/home/u_moved": - bad.append("a moved home is not used") -if not mod._app_home("u_fresh").startswith("/home/shared_user/"): - bad.append("a new account does not land in /home/shared_user") -print("BAD: " + "; ".join(bad) if bad else "OK") -PYAH -)" - _apo="$(printf '%s\n' "$_apo" | tail -1)" - [ "$_apo" = OK ] || _p="$_p;application home resolution: $_apo" - - # AN APPLICATION USER'S HOME IS NOT A BUILD TREE. add_package_user - # creates the account and puts the home where PACKAGE homes go, so - # u_claude landed in /usr/src. It holds a browser profile, not a build. - _rah="$(_slice_def "$pm_src" _relocate_app_home)" - [ -n "$_rah" ] || _p="$_p;an application user's home is left in the package tree" - printf '%s' "$_rah" | grep -q 'usermod", "-d", want, "-m"' \ - || _p="$_p;the home is moved without taking its contents" - printf '%s' "$_rah" | grep -q 'os.path.exists(want)' \ - || _p="$_p;it would move a home onto an existing directory" - printf '%s' "$_rah" | grep -q 'startswith(base' \ - || _p="$_p;it would move a home that is already somewhere deliberate" - grep -q '_relocate_app_home(u)' "$pm_src" \ - || _p="$_p;the relocation is defined and never called" - # a probe with an unexpanded \$( in it tests nothing - grep -A6 -E '^git +imv ' "$_sw2" | grep -q 'expect=' \ - || _p="$_p;imv is installed without proving it can use wayland" - grep -A6 -E '^git +imv ' "$_sw2" | grep -q 'ldd /usr/bin/imv' \ - || _p="$_p;the imv probe relies on a substitution the parser eats" - for _e in light wl-clipboard; do - grep -qE "^git +$_e " "$_sw2" || _p="$_p;$_e is not in the sway stack" - done - # light writes to /sys/class/backlight, which root owns: the udev rule is - # what makes it usable without setuid - grep -A6 '^git light ' "$_sw2" | grep -q -- '--with-udev' \ - || _p="$_p;light is built without the udev rule that grants access" - grep -B8 '^git light ' "$_sw2" | grep -q 'video' \ - || _p="$_p;nothing says which group the backlight rule grants to" - grep -qE '^cfg +fonts' "$_sw2" || _p="$_p;the fonts step is not in the stack" - # -fno-common HAS BEEN GCC'S DEFAULT SINCE GCC 10, and light's helpers.h - # declares light_loglevel without extern: every object defines it and - # the link fails. -fcommon is not a workaround for a broken package, it - # is the behaviour the code was written against -- and the entry has to - # say which, or the next reader deletes it (1.14.189). - grep -A8 '^git light ' "$_sw2" | grep -q -- '-fcommon' \ - || _p="$_p;light will not link on any compiler from GCC 10 onwards" - # ...AND IN THE ENVIRONMENT, NOT ON THE CONFIGURE LINE. light's - # autogen.sh ends by running ./config.status --recheck and then `make`, - # so the compile happens BEFORE the ./configure in the build line is - # reached: CFLAGS passed there fixed nothing and the link failed exactly - # as before (1.14.191). - grep -A8 '^git light ' "$_sw2" | grep -q "export CFLAGS=" \ - || _p="$_p;CFLAGS reaches only the configure that runs too late" - grep -A8 '^git light ' "$_sw2" | grep -q 'rm -f config.status' \ - || _p="$_p;autogen.sh will re-run a stale config.status with the old flags" - # and the flag does what the comment says it does - if command -v gcc >/dev/null 2>&1; then - printf 'int lvl;\n' > "$T/h.h" - printf '#include "h.h"\nint main(void){return lvl;}\n' > "$T/a.c" - printf '#include "h.h"\nint f(void){return lvl;}\n' > "$T/b.c" - ( cd "$T" && gcc -c a.c -o a.o && gcc -c b.c -o b.o \ - && gcc a.o b.o -o demo ) >/dev/null 2>&1 \ - && _p="$_p;this compiler links a tentative definition anyway, so the -fcommon claim is wrong here" - ( cd "$T" && gcc -fcommon -c a.c -o a.o && gcc -fcommon -c b.c -o b.o \ - && gcc a.o b.o -o demo ) >/dev/null 2>&1 \ - || _p="$_p;-fcommon does not fix the multiple definition it is there for" - fi - grep -B64 '^git light ' "$_sw2" | grep -q 'multiple definition' \ - || _p="$_p;the reason for -fcommon is not written down" - # OPENSSH IS NETWORKING. It sat in services.stack when the stack that - # installs wpa_supplicant and dhcpcd is where it belongs (1.14.190). - _ns2="$(dirname "$LFS_TOOL")/stacks/network.stack" - _sv2="$(dirname "$LFS_TOOL")/stacks/services.stack" - grep -qE '^book +openssh' "$_ns2" 2>/dev/null \ - || _p="$_p;openssh is not in the network stack" - grep -qE '^book +openssh' "$_sv2" 2>/dev/null \ - && _p="$_p;openssh is in two stacks" - grep -q 'install-sshd' "$_sv2" 2>/dev/null \ - || _p="$_p;openssh is installed with no bootscript to start it" - # A LEGACY iptables ON AN NFT-ONLY KERNEL FAILS EVERY RULE. BLFS - # configures --disable-nftables, the kernel had no ip_tables, and the - # symptom was "Module ip_tables not found" once per rule at boot - # (1.14.195). - # BUILDING NFTABLES SUPPORT DOES NOT SWITCH TO IT. The build log said - # "nftables support: yes", xtables-nft-multi was installed, and - # `iptables --version` still said (legacy) because the plain names stay - # on xtables-legacy-multi. So the package entry proves the BACKEND - # exists and a separate step makes it the one in use (1.14.196). - grep -A2 -E '^book +iptables' "$_ns2" 2>/dev/null | grep -q 'xtables-nft-multi' \ - || _p="$_p;nothing checks that the nft backend was built" - grep -qE '^cfg +iptables-nft' "$_ns2" 2>/dev/null \ - || _p="$_p;the nft backend is built and never selected" - _in="$(dirname "$LFS_TOOL")/stacks/iptables-nft.sh" - if [ -f "$_in" ]; then - bash -n "$_in" || _p="$_p;iptables-nft.sh does not parse" - grep -q 'iptables --version' "$_in" \ - || _p="$_p;it does not check what the switch achieved" - grep -q 'real file, not a symlink' "$_in" \ - || _p="$_p;it would replace a real file it did not install" - grep -q 'xtables-legacy-multi /usr/sbin/iptables' "$_in" \ - || _p="$_p;there is no way back to the legacy backend" - grep -q 'already uses the nft backend' "$_in" \ - || _p="$_p;it relinks on every replay instead of noticing" - # every generic name, not just iptables: a legacy iptables-restore - # reading an nft ruleset is the same failure one command later - for _n in iptables-save iptables-restore ip6tables ip6tables-save; do - grep -q "$_n" "$_in" || _p="$_p;$_n is left on the legacy backend" - done - else - _p="$_p;there is no step that selects the nft backend" - fi - grep -B14 -E '^book +iptables' "$_ns2" 2>/dev/null | grep -q 'disable-nftables' \ - || _p="$_p;the book's --disable-nftables is not mentioned where it bites" - _nw2="$(dirname "$LFS_TOOL")/stacks/net-wifi.sh" - if [ -f "$_nw2" ]; then - grep -q 'ip_tables(legacy)' "$_nw2" \ - || _p="$_p;the report does not say what the kernel offers" - grep -q 'every rule will fail' "$_nw2" \ - || _p="$_p;a userspace/kernel mismatch is not called out" - # both directions, not just the one that happened - [ "$(grep -c 'every rule will fail' "$_nw2")" -ge 2 ] \ - || _p="$_p;only one of the two mismatch directions is reported" - # and the variant parses out of the real strings - for _pair in "iptables v1.8.12 (legacy)|legacy" "iptables v1.8.12 (nf_tables)|nft"; do - _in="${_pair%|*}"; _want="${_pair#*|}" - case "$_in" in - *nf_tables*) _got=nft ;; - *legacy*) _got=legacy ;; - *) _got=unknown ;; - esac - [ "$_got" = "$_want" ] \ - || _p="$_p;'$_in' reads as $_got, not $_want" - done - fi - - # AND ITS ACCOUNT IS DECLARED, not typed out of a failure box hours later - _ac="$(dirname "$LFS_TOOL")/stacks/accounts.conf" - _as="$(dirname "$LFS_TOOL")/stacks/accounts.sh" - grep -qE '^cfg +accounts' "$_ns2" 2>/dev/null \ - || _p="$_p;nothing creates the declared system accounts" - grep -q '^sshd:' "$_ac" 2>/dev/null \ - || _p="$_p;the sshd account is not declared anywhere" - if [ -f "$_as" ]; then - bash -n "$_as" || _p="$_p;accounts.sh does not parse" - _ord="$(python3 - "$pm_src" "$_ns2" <<'PYORD2' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -# BUILD entries only. Since every stack includes avoid.stack, an `avoid -# openssh` line sat at index 50 while the real `book openssh` was at 73 -- -# so this compared an exclusion against a build step and called the order -# wrong (1.14.241). -o = [r["target"] for r in mod._parse_stack_file(sys.argv[2]) - if r["kind"] != "avoid"] -print("OK" if "accounts" in o and "openssh" in o - and o.index("accounts") < o.index("openssh") else "LATE") -PYORD2 -)" - [ "$_ord" = OK ] \ - || _p="$_p;the accounts step runs after openssh, so the build still refuses" - grep -q -- '"$@" -r' "$_as" \ - || _p="$_p;an account with no declared number gets an ordinary uid" - grep -q 'left alone: it may own files' "$_as" \ - || _p="$_p;it would renumber an account that already owns files" - grep -q 'is already group' "$_as" \ - || _p="$_p;a gid collision does not say what holds the number" - printf 'zz-test-acct:-:-:-:-:suite\n' > "$T/acc.conf" - ACCOUNTS_CONF="$T/acc.conf" bash "$_as" >/dev/null 2>&1 - if id zz-test-acct >/dev/null 2>&1; then - [ "$(id -u zz-test-acct)" -lt 1000 ] \ - || _p="$_p;a declared system account got uid $(id -u zz-test-acct)" - ACCOUNTS_CONF="$T/acc.conf" bash "$_as" 2>&1 \ - | grep -q '0 created, 1 already there' \ - || _p="$_p;a second run does not leave the account alone" - userdel zz-test-acct 2>/dev/null; groupdel zz-test-acct 2>/dev/null - else - _p="$_p;accounts.sh did not create the account it was given" - fi - fi - # A URL THAT WAS CHECKED SAYS SO. haikarainen/light does not resolve - # while three other github URLs in this stack do, and the clone failed - # with "could not read Username" -- GitHub's answer for a repository it - # does not have. An unverified URL that looks verified costs someone a - # build (1.14.186). - # -B32: the note is as long as the finding needed, and -B14 cut it off - # the finding moved from "this URL is dead" to "upstream moved, and the - # new host could not be reached from here": both are provenance, and the - # assertion is that the entry carries it - grep -B64 '^git light ' "$_sw2" \ - | grep -qiE 'did not resolve|could NOT be verified|upstream moved' \ - || _p="$_p;the light URL carries no note about how it was checked" - grep -B64 '^git light ' "$_sw2" | grep -q 'git ls-remote' \ - || _p="$_p;it does not say how to check the URL" -fi -if [ -f "$_fs" ]; then - bash -n "$_fs" || _p="$_p;fonts.sh does not parse" - # THREE UNPACKERS: unzip is a separate BLFS page and nothing requires it, - # so depending on one is how this fails on a system that could unpack - grep -q 'command -v bsdtar' "$_fs" || _p="$_p;fonts.sh cannot use libarchive" - grep -q 'zipfile' "$_fs" || _p="$_p;fonts.sh has no unpacker that is always present" - # a font present but uncached is a font nothing can find - grep -q 'fc-cache -f' "$_fs" || _p="$_p;the font cache is never rebuilt" - grep -q 'fc-list' "$_fs" || _p="$_p;it does not check that fontconfig sees them" - # already installed is done: this runs on every replay - grep -q 'already installed' "$_fs" || _p="$_p;it re-downloads fonts on every run" - # one bad URL must not stop the rest - grep -q 'continue' "$_fs" || _p="$_p;one moved URL stops every other font" - # A GLOB DOES NOT CROSS A SLASH. `*otfs/*.otf` could never match - # fontawesome-free-6.7.2-desktop/otfs/fa-solid-900.otf -- the layout - # every release actually has (1.14.187). - grep -q 'find . -type f -path' "$_fs" \ - || _p="$_p;the archive is searched with a glob that cannot cross a directory" - _fx="$T/fx"; rm -rf "$_fx"; mkdir -p "$_fx/pkg-6.7.2-desktop/otfs" - : > "$_fx/pkg-6.7.2-desktop/otfs/fa-solid-900.otf" - _hit="$(cd "$_fx" && find . -type f -path "*/*.otf" -o -type f -path "./*.otf" | head -1)" - [ -n "$_hit" ] || _p="$_p;find cannot see a font one directory down either" - # and the Noto lines had to be checked, not guessed - grep -q 'WHAT WAS CHECKED, AND HOW' "$_fs" \ - || _p="$_p;the URLs do not say whether they were verified" - grep -q 'notofonts.github.io' "$_fs" \ - || _p="$_p;the release-asset URLs that did not exist are not recorded as such" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "backlight, clipboard and fonts are in the stack, and the fonts step survives a moved URL" -fi - -# ---- a feature is not proved by a file existing (1.14.156) ----------------- # -# ffmpeg and SDL3 came out with no ALSA support on a machine where alsa-lib -# was built and verified. Both autodetect ALSA; both were built BEFORE it, -# because `book alsa-lib` sat at entry 7 while ffmpeg is built inside mesa's -# plan (entry 2) and SDL3 inside gstreamer's. Nothing failed: they -# configured cleanly, installed cleanly, passed have=, and had no sound. -# -# Two things were missing. An order (the stack already states this rule for -# harfbuzz/libass: what an earlier entry installed is simply there), and a -# way to check what was BUILT rather than whether a build happened. -_p="" -# THE STACK IS NOW TWO FILES. The media chain moved into media.stack -# and sway.stack pulls it in with `include media`, so a grep for an -# ENTRY has to read both -- eight assertions here failed the moment -# the split landed, which is the split being noticed rather than a -# fault in it (1.14.223). -_sw="$T/stacks-all.stack" -cat "$(dirname "$LFS_TOOL")/stacks/sway.stack" \ - "$(dirname "$LFS_TOOL")/stacks/media.stack" > "$_sw" 2>/dev/null \ - || _sw="$(dirname "$LFS_TOOL")/stacks/sway.stack" -if [ -f "$_sw" ] && [ -f "$pm_src" ]; then - # the order, read out of the stack itself - _ord="$(python3 - "$pm_src" "$_sw" <<'PYORD' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -rows = [r for r in mod._parse_stack_file(sys.argv[2])] -pos = {} -for i, r in enumerate(rows): - pos.setdefault(r["target"], i) -# a library that something autodetects must come before every consumer of it -for lib, users in (("alsa-lib", ("ffmpeg", "sdl3", "mpv", "gstreamer10", - "pipewire", "mesa", "wireplumber")),): - if lib not in pos: - print("MISSING %s" % lib); continue - for u in users: - if u in pos and pos[u] < pos[lib]: - print("LATE %s after %s" % (lib, u)) -print("OK") -PYORD -)" - case "$_ord" in - *"LATE alsa-lib"*) _p="$_p;alsa-lib is built after something that autodetects it: ${_ord%%$'\n'*}" ;; - *"MISSING alsa-lib"*) _p="$_p;alsa-lib is not in the sway stack at all" ;; - esac - # the probe: expect= must be parsed, kept for the runner, and enforced - _ex="$(python3 - "$pm_src" "$_sw" <<'PYEX' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -n = 0 -for r in mod._parse_stack_file(sys.argv[2]): - if r["opts"].get("expect"): - n += 1 - # a probe must run something and test its result -- a bare command - # name with no pipe, grep or redirect is not a check. (First version - # only allowed pipes and grep, so `gst-inspect-1.0 webrtcbin - # >/dev/null 2>&1` -- a perfectly good probe -- was called thin.) - _e = r["opts"]["expect"] - if any(t in _e for t in ("|", "grep", ">/dev/null", "test ", "[ ")): - continue - print("THIN %s" % r["target"]) -print("COUNT %d" % n) -PYEX -)" - case "$_ex" in - *"COUNT 0"*) _p="$_p;nothing in the stack checks what it was built with" ;; - esac - case "$_ex" in *THIN*) _p="$_p;an expect= probe does not actually test anything" ;; esac - grep -q 'expect' "$pm_src" || _p="$_p;the runner does not know about expect=" - # A FAILED PROBE MUST NOT BE RECORDED AS DONE, or the entry disappears - # from every later run and the missing feature becomes permanent. - python3 - "$pm_src" <<'PYRC' || _p="$_p;a failed feature check is still recorded as done" -import sys, re -src = open(sys.argv[1]).read() -i = src.index('_expect = e["opts"].get("expect")') -j = src.index("_stack_mark_done(path, e[\"_key\"])", i) -blk = src[i:j] -sys.exit(0 if "_expect_ok" in blk and "not _expect_ok" in blk else 1) -PYRC - # and the key must not reach a build as an option - grep -q 'opts.pop("expect"' "$pm_src" \ - || _p="$_p;expect= can be passed to meson as -Dexpect=" -fi -# the probes themselves have to work on the output they claim to read -_o="$(printf 'Devices:\n D alsa ALSA audio output\n')" -printf '%s' "$_o" | grep -qw alsa || _p="$_p;the ffmpeg probe does not match ffmpeg -devices output" -printf 'Devices:\n D oss\n' | grep -qw alsa && _p="$_p;the ffmpeg probe matches output with no alsa" -printf 'libasound.so.2\0' > "$T/fakelib.so" -strings "$T/fakelib.so" 2>/dev/null | grep -q libasound \ - || _p="$_p;the SDL3 probe cannot see a dlopen name in a library" -# A LIBRARY IS NOT A PLUGIN. gst-plugins-bad always builds -# libgstwebrtc-1.0.so, so the file list and `verify` are clean while the -# webrtcbin ELEMENT -- the thing a browser loads -- is absent, because -# libnice was not installed when it configured. The browser said so; the -# system did not (1.14.157). -_gst="$(python3 - "$pm_src" "$_sw" <<'PYGST' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -pos, exp = {}, {} -for i, r in enumerate(mod._parse_stack_file(sys.argv[2])): - pos.setdefault(r["target"], i) - if r["opts"].get("expect"): - exp[r["target"]] = r["opts"]["expect"] -# each codec and transport library before the plugin package that looks for it -# gst10-libav needs ffmpeg AND gst-plugins-base, and it was not in the stack -# at all -- so H.264 decoded by nothing and the symptom was a black frame -# (1.14.167). -# absent entirely is the case that started this: name it as missing rather -# than as an unproven probe -if "gst10-libav" not in pos: - print("MISSING gst10-libav") -for lib, pkg in (("ffmpeg", "gst10-libav"), ("gst10-plugins-base", "gst10-libav"), - ("opus", "gst10-plugins-base"), ("libvpx", "gst10-plugins-good"), - ("libnice", "gst10-plugins-bad"), ("libsrtp", "gst10-plugins-bad"), - ("usrsctp", "gst10-plugins-bad")): - if lib not in pos: - print("MISSING %s" % lib) - elif pkg in pos and pos[pkg] < pos[lib]: - print("LATE %s after %s" % (lib, pkg)) -# and the element probe, not a file probe -b = exp.get("gst10-plugins-bad", "") -if "webrtcbin" not in b: - print("NOPROBE gst10-plugins-bad") -if "gst-inspect" not in exp.get("gst10-plugins-good", ""): - print("NOPROBE gst10-plugins-good") -if "avdec" not in exp.get("gst10-libav", ""): - print("NOPROBE gst10-libav") -# DECODING IS NOT ENCODING. avdec_h264 was present while x264enc, -# openh264enc and openh264dec were all missing, so the browser could play -# H.264 and not produce it -- which is what a camera site needs to publish. -# Checked against BLFS 13.1: gst10-plugins-ugly IS a book page, openh264 is -# not (1.14.199). -if "gst10-plugins-ugly" not in pos: - print("MISSING gst10-plugins-ugly") -elif "x264enc" not in exp.get("gst10-plugins-ugly", ""): - print("NOPROBE gst10-plugins-ugly") -if "openh264" not in pos: - print("MISSING openh264") -elif "gst10-plugins-bad" in pos and pos["openh264"] > pos["gst10-plugins-bad"]: - print("LATE openh264 after gst10-plugins-bad") -print("OK") -PYGST -)" -case "$_gst" in - *"LATE "*) _p="$_p;a WebRTC library is built after the plugins that look for it: ${_gst%%$'\n'*}" ;; - *"MISSING "*) _p="$_p;the WebRTC chain is incomplete: ${_gst%%$'\n'*}" ;; -esac -case "$_gst" in - *"NOPROBE "*) _p="$_p;a gst plugin package does not prove its elements registered" ;; -esac -# AN OUT-OF-BOOK ENTRY CARRIES ITS OWN TRUTH. gst-plugins-rs has no BLFS -# page, so the ref is not a pin unless we say so, the crates come from the -# network, and the plugin list is a choice -- all three belong next to the -# entry (1.14.172). -if [ -f "$_sw" ]; then - grep -A26 '^git gst-plugins-rs' "$_sw" | grep -q 'cargo cinstall' \ - || _p="$_p;the rust plugins are built without cargo-c" - # A PLUGIN IS A SHARED OBJECT. Without --library-type cdylib cargo-c - # also installs a static libgstrs*.a into /usr/lib/gstreamer-1.0, which - # nothing loads and which is what the first real run died on. - grep -A26 '^git gst-plugins-rs' "$_sw" | grep -q -- '--library-type cdylib' \ - || _p="$_p;cargo-c installs a static library nothing loads" - # audiornnoise IS NOT ITS OWN PLUGIN. It lives in gst-plugin-audiofx, so - # a -p naming the element would fail before anything built (1.14.174). - _rsb="$(grep -A26 '^git gst-plugins-rs' "$_sw")" - printf '%s' "$_rsb" | grep -q 'gst-plugin-audiofx' \ - || _p="$_p;audiornnoise is asked for without the plugin that carries it" - printf '%s' "$_rsb" | grep -q -- '-p gst-plugin-audiornnoise' \ - && _p="$_p;audiornnoise is named as a plugin, which it is not" - # every element in the probe must be asked for by some -p - for _el in webrtcsink audiornnoise; do - printf '%s' "$_rsb" | grep -q "$_el" \ - || _p="$_p;the probe checks $_el and nothing builds it" - done - # and the build and install lines must ask for the SAME set: a plugin - # built and not installed is the most confusing outcome available - _b="$(printf '%s' "$_rsb" | grep -o 'build="[^"]*"' | grep -o -- '-p [a-z0-9-]*' | sort)" - _i="$(printf '%s' "$_rsb" | grep -o 'install="[^"]*"' | grep -o -- '-p [a-z0-9-]*' | sort)" - [ "$_b" = "$_i" ] \ - || _p="$_p;the build and install lines ask for different plugins" - grep -B20 '^git gst-plugins-rs' "$_sw" | grep -q 'not a pin' \ - || _p="$_p;an unpinned ref is not called out as unpinned" - grep -B20 '^git gst-plugins-rs' "$_sw" | grep -q 'downloads crates' \ - || _p="$_p;nothing says the rust build needs the network" - grep -q 'gst-plugins-rs is not built (optional)' \ - "$(dirname "$LFS_TOOL")/stacks/webrtc.sh" \ - || _p="$_p;a missing optional plugin set fails the chain report" -fi - -# CARGO NAMES THE DESTINATION IN BACKTICKS, on the line above the reason: -# failed to copy `...a` to `/usr/lib/gstreamer-1.0/...a` -# Caused by: -# Permission denied (os error 13) -# No quotes, and no "Permission denied" on the same line -- so the repair -# extracted nothing, ran zero rounds, and still printed "grant access and -# retry" (1.14.173). -_hl5="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_hl5" ]; then - grep -q 'os error 13' "$_hl5" \ - || _p="$_p;cargo's permission failure is invisible to the repair" - printf '%s\n' \ - 'Error: CliError { error: Some(failed to copy `/src/libgstrsrtp.a` to `/usr/lib/gstreamer-1.0/libgstrsrtp.a`' \ - 'Caused by:' \ - ' Permission denied (os error 13)), exit_code: 101 }' > "$T/cargo.log" - _got="$(grep -aoE 'to `/[^`]+`' "$T/cargo.log" \ - | sed 's/^to `//; s/`$//' | sed 's|/[^/]*$||' | sort -u)" - [ "$_got" = /usr/lib/gstreamer-1.0 ] \ - || _p="$_p;the cargo destination parses to '$_got', not the directory" -fi - -# the chain report: builds nothing, and names the rebuild command -_wr="$(dirname "$LFS_TOOL")/stacks/webrtc.sh" -if [ -f "$_wr" ]; then - bash -n "$_wr" || _p="$_p;webrtc.sh does not parse" - grep -qE '^check +webrtc' "$_sw" || _p="$_p;the chain report is not in the stack" - grep -q 'gst-inspect-1.0' "$_wr" \ - || _p="$_p;it checks for files instead of registered elements" - grep -qE '^[[:space:]]*(install|cp|mv|rm|mkdir|chmod|chown|tee|ln|meson|ninja)[[:space:]]' "$_wr" \ - && _p="$_p;the chain report builds or writes something" - grep -q 'only if' "$_wr" \ - || _p="$_p;a missing element does not say which library it needed" -else - _p="$_p;there is no report of the WebRTC chain" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "a library goes in before its consumers, expect= proves the feature, and the chain is reportable" -fi - -# ---- privilege is a group, not a person (1.14.154) ------------------------- # -# The reference machine delegated halt/poweroff/reboot/pm-suspend to a group -# by hand: 4550 root:<prefix>_power, plus a boot script re-applying the modes -# on /sys/power/state because /sys does not survive a reboot. That is the -# same shape as seatd-<user>, and it was nowhere in the stacks. -# -# Two pieces of it could never have worked, and both looked like working -# configuration: -# SUBSYSTEM=="power", KERNEL=="state", MODE="4550" -- /sys/power/state has -# no uevent, so udev is never asked; and 550 has no write bit, so a group -# member could not write "mem" to it even if it had applied. -# ACTION=="add", KERNEL=="bootlog", OWNER="..." -- /run/bootlog is a -# file the bootscripts write, not a device. No account is needed. -_p="" -_pw="$(dirname "$LFS_TOOL")/stacks/power.sh" -_sv="$(dirname "$LFS_TOOL")/stacks/services.stack" -if [ -f "$_pw" ]; then - bash -n "$_pw" || _p="$_p;power.sh does not parse" - grep -qE '^cfg +power' "$_sv" 2>/dev/null \ - || _p="$_p;the power step is not in the services stack" - grep -q '4550' "$_pw" || _p="$_p;the power binaries are not setuid root" - grep -q 'chown "root:\$grp"' "$_pw" \ - || _p="$_p;a setuid binary is left owned by something other than root" - # THE GROUP NAME IS READ, NOT DEFAULTED. Reporting a built-in default as - # the configured value is a bug this project has already paid for. - grep -q 'collector_prefix' "$_pw" \ - || _p="$_p;the group name ignores the collector prefix" - grep -q 'POWER_GROUP' "$_pw" || _p="$_p;the group name cannot be overridden" - grep -q 'no collector_prefix in' "$_pw" \ - || _p="$_p;a missing prefix silently becomes a default group name" - # 0660 on the sysfs attribute: writing to it IS the suspend - grep -q 'chmod 0660' "$_pw" \ - || _p="$_p;the power interface is not writable by the group" - grep -qE 'chmod 0?550 .*power/state' "$_pw" \ - && _p="$_p;the power interface is set read-only" - # ONE file, symlinked -- the machine had a second copy under rc3.d, which - # is a second place to edit - grep -q 'ln -sf ../init.d/power-perms' "$_pw" \ - || _p="$_p;the boot script is copied into the runlevel instead of linked" - [ "$(grep -c 'cat > "\$init"' "$_pw")" = 1 ] \ - || _p="$_p;the boot script is written in more than one place" - # usermod has returned 0 and changed nothing on this project, three times - grep -q 'id -nG' "$_pw" || _p="$_p;the group membership is not read back" - # and the two rules that cannot fire are named - grep -q 'KERNEL==\"state\"' "$_pw" \ - || _p="$_p;a udev rule that cannot fire is left unexplained" - grep -q 'KERNEL==\"bootlog\"' "$_pw" \ - || _p="$_p;the bootlog rule is left looking like something that works" - grep -q 'rc.site' "$_pw" \ - || _p="$_p;it does not say where the boot log IS configured" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "power is delegated to a group, by the step, with the dead rules named" -fi - -# ---- something has to fetch an address (1.14.153) -------------------------- # -# The wireless link came up and `wlp3s0` would not go anywhere: the network -# stack had wpa_supplicant, libnl, iptables and net-tools, and no DHCP -# client at all -- while `lfs config network dhcp` writes no static -# interface files precisely because a client from BLFS is supposed to follow. -# Nothing followed. -_p="" -_ns="$(dirname "$LFS_TOOL")/stacks/network.stack" -if [ -f "$_ns" ]; then - grep -qE '^(book|tar|git) +dhcpcd\b' "$_ns" \ - || _p="$_p;the network stack still has no DHCP client" - # A PACKAGE IS NOT A SERVICE. dhcpcd installs no /lib/services/dhcpcd; - # that script comes from blfs-bootscripts' install-service-dhcpcd, and - # without it SERVICE=dhcpcd (or a wpa hand-off to it) has nothing to run. - # Adding the package was half the job (1.14.155). - grep -q 'install-service-dhcpcd' "$_sv" 2>/dev/null \ - || _p="$_p;the dhcpcd service script is never installed" - grep -q 'install-service-wpa' "$_sv" 2>/dev/null \ - || _p="$_p;the wpa service script is never installed" - # and the interview's promise must name something that exists in it - _dh="$(grep -oE 'dhcpcd' "$(dirname "$LFS_TOOL")/lfs" | head -n1)" - [ -n "$_dh" ] && ! grep -q 'dhcpcd' "$_ns" \ - && _p="$_p;lfs promises dhcpcd from BLFS and no stack provides it" -fi -_nw="$(dirname "$LFS_TOOL")/stacks/net-wifi.sh" -if [ -f "$_nw" ]; then - bash -n "$_nw" || _p="$_p;net-wifi.sh does not parse" - grep -qE '^check +net-wifi' "$_ns" 2>/dev/null \ - || _p="$_p;the wifi check is not in the stack that installs wifi" - # THE SERVICE NAMES COME OFF THE DISK. Which variable the wpa service - # chains through differs between bootscript releases, and a remembered - # one is how you get a joined link with no address. - grep -q 'SERVICE' "$_nw" || _p="$_p;the ifconfig SERVICE= is never checked" - grep -q 'services' "$_nw" || _p="$_p;it does not look at the installed services" - # predictable interface names: a file called ifconfig.wifi0 does nothing - # for wlp3s0 - grep -q '/sys/class/net' "$_nw" \ - || _p="$_p;it assumes an interface name instead of reading the kernel's" - grep -q 'phy80211' "$_nw" \ - || _p="$_p;wireless interfaces are identified by name rather than by sysfs" - # IT MUST WRITE NOTHING: the files hold the user's SSID and key. - # (First version of this assertion matched `>/dev/null` and a `>` inside - # a quoted suggestion -- text about a command is not the command, which - # is a rule this suite has caught five filters and one injector on.) - grep -qE '^[[:space:]]*(install|cp|mv|rm|mkdir|chmod|chown|tee|ln)[[:space:]]' "$_nw" \ - && _p="$_p;a check writes to the system" - # a check must be quiet about what it cannot know and loud about files - grep -q 'chmod 600' "$_nw" \ - || _p="$_p;a world-readable psk file is not called out" - # A STEP MUST NOT FAIL FOR WORK IT WAS NEVER GOING TO DO. This check - # stopped the stack at 8 of 9 because no /etc/sysconfig/ifconfig.wlan0 - # existed -- a file that holds the user's SSID. Two levels now: !! is - # the stack's and fails, -> is yours and does not. - grep -q '^todo()' "$_nw" \ - || _p="$_p;the user's own files still fail the step" - # GIT MUST NEVER ASK A QUESTION IN A BUILD: on a terminal that hangs, - # and in a log it reads like a credentials problem on a public repo. - _ph="$(dirname "$LFS_TOOL")/lfs-phases" - if [ -f "$_ph" ]; then - grep -q 'GIT_TERMINAL_PROMPT=0' "$_ph" \ - || _p="$_p;a clone can still stop and ask for a password" - grep -q 'PATH IS WRONG' "$_ph" \ - || _p="$_p;a 401 on a public repository is not explained" - grep -q 'git ls-remote \$link' "$_ph" \ - || _p="$_p;the failure does not say how to test the URL" - # AN EMPTY LIST UNDER A HEADING IS A RIDDLE. A fork with no tags - # printed "Recent tags:" and then nothing, and the branch list - # offered `origin`, which is not a ref anyone can check out. - grep -q 'none at all' "$_ph" \ - || _p="$_p;a repository with no tags prints an empty list" - grep -q 'HEAD' "$_ph" \ - || _p="$_p;the branch list offers the remote name and HEAD as refs" - fi - grep -q 'todo "write \$f' "$_nw" \ - || _p="$_p;a missing ifconfig file is still the stack's failure" - # THE INTERFACE FILE HOLDS NO SECRET, so there is no reason to make the - # reader work out its syntax: print it, with the interface that exists - # and the hand-off variable this machine's wpa service reads (1.14.185). - grep -q 'this is the whole file' "$_nw" \ - || _p="$_p;it asks for a file without saying what goes in it" - grep -q 'IFACE=' "$_nw" || _p="$_p;the printed file names no interface" - grep -q '_chain_var' "$_nw" \ - || _p="$_p;the printed file hard-codes the hand-off variable" - # and a file with no key cannot be a leaked key: it said both, two lines - # apart, about one file - grep -q 'it holds a key and is world-readable' "$_nw" \ - || _p="$_p;the permission warning is not tied to actually holding a key" - grep -q '__\*__' "$_nw" \ - || _p="$_p;a placeholder template is reported as a broken config" -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "the network stack can actually get an address, and says what is missing" -fi - -# ---- firmware is answerable in a chroot, and fixable without a kernel build - # -# FIRMWARE_ONLY="amdgpu amd-ucode amd intel" installed intel/, and the AX200 -# still had no firmware: -# Direct firmware load for iwlwifi-cc-a0-77.ucode failed with error -2 -# no suitable firmware found! -# The driver asks for a FLAT path, so a directory named `intel` never -# satisfies it -- and there was no way to install one blob without building a -# whole kernel. The running kernel printed the exact path it wanted; the -# checkout's git index knows where that file is. Ask both. -_p="" -_k4="$(dirname "$LFS_TOOL")/lfs-kernel" -if [ -f "$_k4" ]; then - bash -n "$_k4" || _p="$_p;lfs-kernel does not parse" - grep -q '^ firmware)' "$_k4" \ - || _p="$_p;there is no firmware subcommand" - grep -q 'lfs-kernel firmware' "$_k4" \ - || _p="$_p;the firmware subcommand is not in the help" - # --help must not silently drop it: the slice used to be a line range - bash "$_k4" --help 2>/dev/null | grep -q 'firmware --missing' \ - || _p="$_p;--help does not mention firmware --missing" - # the functions must be defined BEFORE the build runs, or a firmware-only - # run would have to build a kernel to reach them - _defl="$(grep -n '^_install_firmware()' "$_k4" | cut -d: -f1)" - _bldl="$(grep -n '^echo "# kernel \$VERSION -> \$ESP' "$_k4" | cut -d: -f1)" - [ -n "$_defl" ] && [ -n "$_bldl" ] && [ "$_defl" -lt "$_bldl" ] \ - || _p="$_p;the firmware code is only reachable after a kernel build" - # it reads the failures out of dmesg, and strips the trailing punctuation - _rq="$(_slice_fn "$_k4" _fw_requests)" - [ -n "$_rq" ] || _p="$_p;nothing reads what the kernel asked for" - _got="$( eval "$_rq" - dmesg() { printf '%s\n' \ - "[ 4.051584] iwlwifi 0000:03:00.0: Direct firmware load for iwlwifi-cc-a0-77.ucode failed with error -2" \ - "[ 1.698606] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2" \ - "[ 4.073516] Bluetooth: hci0: Found device firmware: intel/ibt-20-1-3.sfi"; } - _fw_requests )" - case "$_got" in - *iwlwifi-cc-a0-77.ucode*) ;; - *) _p="$_p;the wireless firmware request is not extracted from dmesg" ;; - esac - case "$_got" in - *regulatory.db*) ;; - *) _p="$_p;the regulatory database request is not extracted" ;; - esac - # a firmware that LOADED is not a missing one - case "$_got" in - *ibt-20-1-3*) _p="$_p;a successful firmware load is reported as missing" ;; - esac - # the index, not the working tree: a sparse checkout has removed the very - # file being looked for - _ms="$(_slice_fn "$_k4" _firmware_missing)" - printf '%s' "$_ms" | grep -q '_fw_index' \ - || _p="$_p;it searches the checkout instead of asking the index" - # what it installs is the path the DRIVER asked for - _pl="$(_slice_fn "$_k4" _fw_place)" - printf '%s' "$_pl" | grep -q '/lib/firmware/\$req' \ - || _p="$_p;the blob is not placed at the path the driver asked for" - # a file in the index but not in the sparse checkout must still be - # installable -- and by ONE blob. The first version widened the sparse - # checkout and fell back to `sparse-checkout disable`, which in cone mode - # would have materialised all ten gigabytes to fix one 2 MB blob. - printf '%s' "$_pl" | grep -q "show 'HEAD:" \ - || _p="$_p;a file in the index but not on disk cannot be installed" - printf '%s' "$_pl" | grep -q 'sparse-checkout disable' \ - && _p="$_p;the fallback would check out the whole of linux-firmware" - printf '%s' "$_pl" | grep -q 'stat -c %s' \ - || _p="$_p;the copy is not confirmed by reading the file back" - # ONE `local` PER DERIVED VALUE. `local a="$2" d="/lib/firmware/$a"` - # reads $a before this local assigned it, and set -u killed the function - # at the first --fix on a real machine. - printf '%s' "$_pl" | grep -qE '^ *local [a-z_]+="\$1".*dst=' \ - && _p="$_p;dst is derived inside the local that assigns its source" - # git's own message, not a guess: "does not exist in HEAD" and a missing - # blob in a partial clone want different answers - printf '%s' "$_pl" | grep -q '_gerr' \ - || _p="$_p;git's error is discarded and replaced with a guess" - # THE REPOSITORY IS THE FIRMWARE ACCOUNT'S AND THIS RUNS AS ROOT, which - # git refuses: the empty result was reported as "linux-firmware does not - # have a file by that name at all", about a file that was right there. - _ix="$(_slice_fn "$_k4" _fw_index)" - [ -n "$_ix" ] || _p="$_p;there is no single place that asks the index" - printf '%s' "$_ix" | grep -q 'safe.directory' \ - || _p="$_p;git will refuse the account's repository when run as root" - # ls-TREE, not ls-files: with a sparse index git collapses an excluded - # directory into one entry, so every nested blob looked absent -- 56 - # files reported as "no file by that name" while they were all there. - printf '%s' "$_ix" | grep -q 'ls-tree -r' \ - || _p="$_p;a sparse index hides nested blobs from the lookup" - printf '%s' "$_ix" | grep -q '_FW_INDEX_ERR' \ - || _p="$_p;a failed index lookup cannot be told from an absent file" - printf '%s' "$_ms" | grep -q 'UNKNOWN -- not absent' \ - || _p="$_p;a failed lookup is still reported as a fact" - # the scratch file must not land in someone else's checkout - printf '%s' "$_ix" | grep -q 'mktemp' \ - || _p="$_p;the index listing is written inside the repository" - # A CHROOT'S dmesg IS THE HOST'S -- and a chroot is where this work - # happens, so refusing to answer there is no answer. The tree can be - # asked directly: its modules declare their firmware, and the hardware is - # in /sys. Every line says which source named it. - _wt="$(_slice_fn "$_k4" _fw_wanted)" - [ -n "$_wt" ] || _p="$_p;the list comes from the running boot alone" - printf '%s' "$_wt" | grep -q 'modinfo' \ - || _p="$_p;the tree's own modules are never asked what they need" - printf '%s' "$_wt" | grep -q '/sys/devices' \ - || _p="$_p;every driver's firmware is listed, not this machine's" - printf '%s' "$_wt" | grep -q 'modprobe -S' \ - || _p="$_p;aliases are resolved against the running kernel's modules" - printf '%s' "$_wt" | grep -q 'CONFIG_EXTRA_FIRMWARE' \ - || _p="$_p;firmware the config embeds is not counted" - # ONE dmesg PARSER: _fw_wanted must not re-implement _fw_requests - printf '%s' "$_wt" | grep -q '_fw_requests' \ - || _p="$_p;there are two dmesg parsers to keep in step" - printf '%s' "$_ms" | grep -q 'the HOST kernel' \ - || _p="$_p;a chroot does not say whose boot the boot lines describe" - printf '%s' "$_ms" | grep -q 'modules scanned' \ - || _p="$_p;the report does not say which kernel's modules it read" - # A DECLARATION IS NOT A FAILURE. `modinfo -F firmware amdgpu` lists - # every chip amdgpu supports, so the first modules scan printed 56 - # MISSING lines -- Vega, Aldebaran, five iwlwifi RF families this laptop - # does not have -- each with its own remedy, and concluded "56 missing". - printf '%s' "$_ms" | grep -q 'CANDIDATES, not failures' \ - || _p="$_p;a module's declarations are reported as failures" - printf '%s' "$_wt" | grep -q 'modules:\$m' \ - || _p="$_p;a declared file does not say which module declared it" - printf '%s' "$_ms" | grep -q -- '--fix --all' \ - || _p="$_p;there is no way to install the declared files anyway" - grep -q -- '--all) _FWALL=1' "$_k4" \ - || _p="$_p;the suggested --all is not a flag firmware accepts" - # WHAT IS THERE, NEXT TO WHAT IS NOT. A missing blob on a machine whose - # wifi worked under the previous kernel is either the same file under - # another path (FIRMWARE_ONLY entries keep the repository's, the loader - # wants its own) or an older firmware API the new driver refuses. Both - # change what you do next, and the report said neither. - _nb="$(_slice_fn "$_k4" _fw_neighbours)" - [ -n "$_nb" ] || _p="$_p;nothing looks for what IS installed nearby" - printf '%s' "$_nb" | grep -q 'find /lib/firmware -name' \ - || _p="$_p;the same file under another path is not noticed" - printf '%s' "$_nb" | grep -q 'older' \ - || _p="$_p;an older firmware API on disk is not reported" - # the path is returned in a variable, not on stdout: the caller's $(...) - # swallowed the whole explanation with it - printf '%s' "$_nb" | grep -q '_FW_LOCAL=' \ - || _p="$_p;the explanation is captured instead of printed" - printf '%s' "$_ms" | grep -q '_fw_neighbours "\$req"' \ - || _p="$_p;the neighbour check is never called" - # SAY WHAT IS SATISFIED. A boot line whose file is now on disk was - # skipped in silence, so a run with everything in place looked exactly - # like one that had not looked -- and in a chroot that line is the - # reassurance the reader came for. - printf '%s' "$_ms" | grep -q 'installed now' \ - || _p="$_p;a request that is now satisfied is passed over silently" - # ...and a green run is still not a promise: the driver derives the - # firmware name from ids it reads off the card at run time - printf '%s' "$_ms" | grep -q 'Only a boot of THIS kernel settles it' \ - || _p="$_p;the report reads as a verdict on a kernel that has not run" - # a chroot host can be another distribution on other hardware - printf '%s' "$_ms" | grep -q 'neither confirm nor deny' \ - || _p="$_p;the host's boot log is presented as evidence about this tree" - # WHICH KERNEL: uname -r is the host's in a chroot - _mv="$(_slice_fn "$_k4" _fw_modver)" - printf '%s' "$_mv" | grep -q 'VERSION' \ - || _p="$_p;the modules of the configured kernel are not preferred" - # regulatory.db is NOT in linux-firmware, so "update the checkout" is - # advice that can never work - _el="$(_slice_fn "$_k4" _fw_elsewhere)" - printf '%s' "$_el" | grep -q 'regulatory.db|wireless-regdb' \ - || _p="$_p;regulatory.db is looked for in the wrong project" - # FIRMWARE_ONLY is copied to the same relative path, so it can only - # satisfy a request whose path matches the repository's - printf '%s' "$_ms" | grep -q 'no FIRMWARE_ONLY entry can express this' \ - || _p="$_p;a nested blob with a flat request gets advice that cannot work" - _if="$(_slice_fn "$_k4" _install_firmware)" - printf '%s' "$_if" | grep -q 'cp -a --parents' \ - || _p="$_p;a nested FIRMWARE_ONLY entry is installed with its path dropped" - # and an unchanged FIRMWARE_ONLY is not advice - printf '%s' "$_ms" | grep -q 'if \[ -n "\$sugg" \]' \ - || _p="$_p;it prints the unchanged FIRMWARE_ONLY as though it were a fix" - # every flag the messages suggest is one the command accepts - printf '%s' "$_ms" | grep -q 'firmware --missing --fix' \ - && { grep -q -- '--fix)' "$_k4" \ - || _p="$_p;the suggested --fix is not a flag firmware accepts"; } - # THE REGULATORY DATABASE IS A DIFFERENT PROJECT. Naming it in - # FIRMWARE_ONLY failed the entire firmware install, because a pattern - # matching nothing in the repository is an error -- so the shipped config - # must not name it, and an entry someone else already copied must be - # skipped with a note rather than taken as a failure. - _rg="$(_slice_fn "$_k4" _install_regdb)" - [ -n "$_rg" ] || _p="$_p;nothing installs the regulatory database" - printf '%s' "$_rg" | grep -q 'sort -V | tail -n1' \ - || _p="$_p;the wireless-regdb version is hard-coded instead of asked for" - printf '%s' "$_rg" | grep -q 'stat -c %s /lib/firmware/regulatory.db' \ - || _p="$_p;the regulatory database install is not confirmed" - printf '%s' "$_rg" | grep -q 'REGDB:-yes' \ - || _p="$_p;the regulatory database cannot be turned off" - printf '%s' "$_if" | grep -q '_install_regdb' \ - || _p="$_p;a normal firmware run does not install the database" - printf '%s' "$_if" | grep -q 'is not in linux-firmware' \ - || _p="$_p;a FIRMWARE_ONLY entry for it still fails the whole run" - # in a chroot the boot list is NOT the reliable one, and the advice used - # to say it was, two lines under a note saying the opposite - printf '%s' "$_ms" | grep -q 'boot THIS kernel and run this' \ - || _p="$_p;in a chroot it still points at the host's boot list as evidence" - # AN OPTION IN THE WRONG SECTION FAILS THE WHOLE BUILD. `alsa` reads - # perfectly well under [gst10-plugins-bad] and is a gst-plugins-BASE - # option; meson answers "Unknown option: alsa" and stops (1.14.200). - _wu="$(_slice_def "$pm_src" _warn_unknown_meson_opts)" - [ -n "$_wu" ] || _p="$_p;a machine.conf key is passed without checking it exists" - printf '%s' "$_wu" | grep -q 'NOT passed' \ - || _p="$_p;an unknown option is passed to meson anyway" - printf '%s' "$_wu" | grep -q 'it does have' \ - || _p="$_p;it does not offer the near misses" - printf '%s' "$(_slice_def "$pm_src" _meson_option_names)" | grep -q 'meson.options' \ - || _p="$_p;the option names are not read from the package's own source" - # builtins are not in meson.options and must still be allowed - printf '%s' "$_wu" | grep -q 'wrap_mode' \ - || _p="$_p;a meson built-in like wrap_mode would be dropped" - # and the shipped machine.conf must put alsa where it belongs - grep -A2 '^\[gst10-plugins-base\]' "$_mc5" 2>/dev/null | grep -q 'alsa' \ - || _p="$_p;alsa is not asked for where the option exists" - grep -A4 '^\[gst10-plugins-bad\]' "$_mc5" 2>/dev/null | grep -q '^alsa' \ - && _p="$_p;alsa is still under the section that has no such option" - - # A HARDWARE-ONLY MESA HAS NOTHING TO FALL BACK TO. `gallium-drivers = - # radeonsi` alone gave "libEGL warning: egl: failed to create dri2 - # screen" and then nothing. The book, twice: add softpipe or llvmpipe - # as a backup gallium driver, and swrast as a backup vulkan one - # (1.14.215). - grep -A3 '^\[mesa\]' "$_mc5" 2>/dev/null | grep -q 'gallium-drivers.*llvmpipe\|gallium-drivers.*softpipe' \ - || _p="$_p;mesa is built with no software gallium driver" - grep -A4 '^\[mesa\]' "$_mc5" 2>/dev/null | grep -q 'vulkan-drivers.*swrast' \ - || _p="$_p;mesa is built with no software vulkan driver" - grep -A2 -E '^book +mesa' "$_sw2" | grep -q 'swrast_dri.so' \ - || _p="$_p;nothing proves the software renderer was built" - # the probe must pass when EITHER file is there -- `ls a b` needs both - grep -A2 -E '^book +mesa' "$_sw2" | grep -q -- '-o -e' \ - || _p="$_p;the mesa probe demands both driver files, not either" - # llvmpipe installs as swrast_dri.so, and the comment has to say so or - # the next reader looks for llvmpipe_dri.so and finds nothing - grep -B6 -E '^book +mesa' "$_sw2" | grep -q 'named after the interface' \ - || _p="$_p;nothing explains why the file is not called llvmpipe" - # and the kernel option the book ties to it - _ks2="$(dirname "$LFS_TOOL")/stacks/kernel-sway.sh" - if [ -f "$_ks2" ]; then - bash -n "$_ks2" || _p="$_p;kernel-sway.sh does not parse" - grep -q 'CONFIG_DRM_VGEM' "$_ks2" \ - || _p="$_p;the kernel option llvmpipe needs is not checked" - # reported, not fatal: a machine that never falls back does not need it - grep -q 'need CONFIG_DRM_VGEM' "$_ks2" \ - && _p="$_p;a missing VGEM fails the stack, which is too strong" - fi - - # USE_GSTREAMER_WEBRTC=ON MAKES webkit WANT librice. webkitgtk 2.52 - # takes its ICE from librice rather than libnice when the GStreamer - # WebRTC backend is on, and the BLFS page never hits this because it - # does not enable that backend -- librice is not in the 13.1 book at - # all. libnice IS installed and is what the rest of the chain uses - # (1.14.232). - # USE_GSTREAMER_WEBRTC=ON MAKES webkit WANT librice, which is not in the - # book at all. The option and the package must agree: the option - # without the package fails configure, the package without the option - # wastes the build (1.14.233). - _wk_rice="$(grep -A8 '^\[webkitgtk\]' "$_mc5" 2>/dev/null \ - | grep -oE 'USE_LIBRICE *= *(ON|OFF)' | grep -oE '(ON|OFF)')" - case "$_wk_rice" in - ON) - grep -qE '^git +librice' "$_ms" \ - || _p="$_p;USE_LIBRICE=ON and nothing builds librice" - _ro="$(python3 - "$pm_src" "$_ms" <<'PYRICE' -import sys, types -mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] -exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) -o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] -print("OK" if "librice" in o and o.index("librice") < o.index("webkitgtk") - else "librice is missing or comes after webkitgtk") -PYRICE -)" - [ "$_ro" = OK ] || _p="$_p;$_ro" - grep -A8 -E '^git +librice' "$_ms" | grep -q 'rice-proto' \ - || _p="$_p;nothing proves librice installed its pkg-config files" - # AND THE URL AND CRATE NAMES WERE CHECKED, after a first - # attempt that guessed the host, the ref and both package - # names and got all four wrong. - grep -A6 -E '^git +librice' "$_ms" | grep -q 'ystreet/librice' \ - || _p="$_p;librice points at a host that answers 401" - grep -A8 -E '^git +librice' "$_ms" | grep -q -- '-p rice-proto -p rice-io' \ - || _p="$_p;the librice crate names are the guessed librice-* ones" - grep -B24 -E '^git +librice' "$_ms" | grep -q 'CHECKED against the repository' \ - || _p="$_p;nothing records that the librice entry was verified" - ;; - OFF) - grep -qE '^git +librice' "$_ms" \ - && _p="$_p;librice is built and USE_LIBRICE is OFF" - ;; - *) _p="$_p;USE_LIBRICE is not set either way for webkitgtk" ;; - esac - - # x264enc EXISTS ONLY WITH -D gpl=enabled, in the book's own words, so - # the shipped machine.conf has to ask for it - _mc5="$(dirname "$LFS_TOOL")/stacks/machine.conf" - grep -A2 '^\[gst10-plugins-ugly\]' "$_mc5" 2>/dev/null | grep -q 'gpl *= *enabled' \ - || _p="$_p;x264enc will not be built: gpl is not enabled for -ugly" - grep -B4 '^\[gst10-plugins-ugly\]' "$_mc5" 2>/dev/null | grep -q 'A)GPL' \ - || _p="$_p;nothing says why that switch matters" - # the chain report must name the encoders, not just the decoder - for _el in x264enc openh264enc; do - grep -q "$_el" "$(dirname "$LFS_TOOL")/stacks/webrtc.sh" \ - || _p="$_p;$_el is not in the chain report" - done - # A GITHUB ARCHIVE URL ENDS IN THE TAG, NOT THE PROJECT. wget saved - # v2.6.0.tar.gz, the unpack looked for the declared - # pkg=openh264-2.6.0.tar.gz, and the run said "nowhere to fetch it from" - # one line under "'v2.6.0.tar.gz' saved [60302243]" (1.14.201). - _ph2="$(dirname "$LFS_TOOL")/lfs-phases" - if [ -f "$_ph2" ]; then - _fe="$(_slice_fn "$_ph2" _pp_fetch)" - printf '%s' "$_fe" | grep -q -- '-O "$out"' \ - || _p="$_p;a download does not land under the name pkg= declares" - # -O leaves an empty file on a 404 and -c would resume it for ever - printf '%s' "$_fe" | grep -q 'rm -f "$out"' \ - || _p="$_p;a failed download leaves an empty file that -c resumes" - grep -q 'what IS here, newest first' "$_ph2" \ - || _p="$_p;a name mismatch still reads as a failed download" - # prove it lands under pkg= - _fx="$T/fx"; rm -rf "$_fx"; mkdir -p "$_fx" - ( cd "$_fx" - _pp_say() { :; } - _pp_mirrors_for() { printf '%s\n' "$1"; } - wget() { _o=""; while [ $# -gt 0 ]; do case "$1" in - -O) _o="$2"; shift 2 ;; -4|-c) shift ;; *) shift ;; esac; done - echo body > "${_o:-wrong-name.tar.gz}"; } - eval "$_fe" - pkg="openh264-2.6.0.tar.gz" - _pp_fetch "https://github.com/cisco/openh264/archive/refs/tags/v2.6.0.tar.gz" "$pkg" - [ -f "$pkg" ] || exit 1 ) \ - || _p="$_p;the fetch still writes the URL's basename, not pkg=" - fi - - # THE BOOK PASSES -D USE_WOFF2=OFF, so a .woff2 icon font renders as a - # blank square with no error anywhere. woff2 IS a book page, so it goes - # in the stack before webkitgtk, and the switch goes in machine.conf -- - # either alone changes nothing (1.14.203). - _ord3="$(python3 - "$pm_src" "$_sw2" <<'PYW' -import sys, types -mod = types.ModuleType("pm") -exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) -o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] -print("OK" if "woff2" in o and "webkitgtk" in o - and o.index("woff2") < o.index("webkitgtk") else "BAD") -PYW -)" - [ "$_ord3" = OK ] \ - || _p="$_p;woff2 is missing from the stack or comes after webkitgtk" - grep -A6 '^\[webkitgtk\]' "$_mc5" 2>/dev/null | grep -q 'USE_WOFF2 *= *ON' \ - || _p="$_p;USE_WOFF2 is not turned on, so installing woff2 changes nothing" - grep -A2 -E '^book +webkitgtk' "$_sw2" | grep -q 'woff2' \ - || _p="$_p;nothing proves webkitgtk actually linked woff2" - - # and openh264 is ours: the book does not have it - grep -B8 -E '^tar +openh264' "$_sw2" | grep -q 'NOT in the BLFS book' \ - || _p="$_p;openh264 is not marked as an out-of-book entry" - grep -B8 -E '^tar +openh264' "$_sw2" | grep -qi 'meson path was NOT built' \ - || _p="$_p;the unverified part of the openh264 entry is not said to be unverified" - - # and the shipped config covers this machine's wireless - _kc4="$(dirname "$LFS_TOOL")/kernel.conf" - if [ -f "$_kc4" ]; then - grep -q 'iwlwifi-\*\.ucode' "$_kc4" \ - || _p="$_p;FIRMWARE_ONLY does not cover the flat iwlwifi blobs" - grep -qE '^FIRMWARE_ONLY=.*regulatory' "$_kc4" \ - && _p="$_p;FIRMWARE_ONLY names regulatory.db, which fails the run" - grep -q '^REGDB=' "$_kc4" \ - || _p="$_p;the config cannot ask for the regulatory database" - fi -fi -if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done -else - ok "missing firmware is derived from the tree, not only from dmesg, and installed without a kernel build" -fi - # ---- fix them together, not one round each (1.14.83) ----------------------- # # fcron names ONE file at a time. First /etc/fcron.conf must be owned by # root:fcron; fixed that, and the next run said /etc/fcron.allow: Permission @@ -25432,8 +21150,6 @@ else ok "slow steps announce themselves, and headers install as their own package" fi -#@@ SUMMARY @@ -- everything below runs whichever blocks were selected echo -echo " PASS: $(_pass_total) FAIL: $(_fail_total) SKIP: $(_skip_total) (final)" -[ "$(_skip_total)" -eq 0 ] || echo " (skipped checks need something this machine has not got)" +echo " PASS: $(_pass_total) FAIL: $(_fail_total) (final)" [ "$(_fail_total)" -eq 0 ]