Packagemanager-LFS-PackageUser-System git · main
Package-user based package manager for LFS/BLFS
Shell 63.7% Markdown 35.8%git clone https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System.gitwget https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System/archive/Packagemanager-LFS-PackageUser-System.tar.gzFixed cfg steps And stuff
HANDOFF.md | 377 +++++++++++++++++++++++----- Makefile | 31 ++- README.md | 29 ++- blfs | 2 +- lfs | 207 ++++++++++----- lfs-helper | 413 +++++++++++++++++++++++++----- packagemanager | 348 +++++++++++++++++++++++--- packagemanager_install | 57 ++++- test_lfs_crosschain.sh | 664 +++++++++++++++++++++++++++++++++++++++++++++---- 9 files changed, 1812 insertions(+), 316 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index c7307b1..63269ee 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -11,15 +11,25 @@ Attach: the five scripts (`lfs`, `lfs-helper`, `packagemanager`, ## Current state -All five tools at **1.10.1**. Build ids — check these match what is installed: +All five tools at **1.11.6**. Build ids — check these match what is installed: - lfs 8b2cb98 - lfs-helper c2f6fbc - packagemanager d051474 - blfs 36a4717 - packagemanager_install 1c068ed + lfs f1f6164 + lfs-helper 3e99aa1 + packagemanager df7bb26 + blfs c479327 + packagemanager_install 1d6c685 -569 regression tests, 568 pass. +589 regression tests, 584 pass in a bare checkout. + +**This is a build candidate.** Everything since 1.11.0 is desk work — no +change has met a real build. The remaining open items need one to make +progress, so the next move is a full run, not more code. The five failures are +environment-only: four need the book HTML beside the tool, one needs +`skel-u_xdg/.bash_profile`. + +**A full 104/104 build has completed on 1.10.1** and verified clean: 66265 +paths owned by the right package, every account prefixed, every home owned by +its user, install directories sealed, `check-toolchain` usable. **A full build on 1.9.3 verified clean: 66265 paths owned by the right package, nothing else outstanding.** The one failure is `the XDG profile is not @@ -37,7 +47,7 @@ directories, the build scratch and the state directory have all moved. lfs build-system session --reconfigure # several new questions lfs run # inside the chroot: - lfs-helper --version # MUST read c2f6fbc + lfs-helper --version # MUST read 7a7b240 lfs-helper build-all Afterwards `lfs-helper verify` is the real check — it reconciles ownership @@ -188,9 +198,9 @@ prefix and the root are both answered from it, so a name cannot be given one kind's prefix and placed in another kind's root. `lfs-helper pkgusr-home <name>` exposes the rule to shell scripts, so nothing -else re-derives it. `packagemanager_install`, `lfs-completion.bash` and -`last_build_step.sh` all go through a chokepoint now; a test fails if any of -them builds `/usr/src/$something` by hand. +else re-derives it. `packagemanager_install` and `lfs-completion.bash` go +through a chokepoint now; a test fails if either builds `/usr/src/$something` +by hand. **The chokepoint must resolve to the ACCOUNT home, not just the root.** Callers pass a mixture and always will: `build_pkg` has the step name @@ -360,6 +370,7 @@ because the next one will look like them. | Handover guarded by one inode | `_owner_is_root` checks `$LFS/usr`; the pass it guards covers eight trees | | uid range used to identify the build user | host `lfs` was 10753, inside the package range | | Adoption reached from two directions | an up-front pass and a mid-loop "late" pass with its own flag | +| Ownership scans kept their own exclusion lists | build trees moved into the package homes in 1.7.6, the snapshot scans followed and the ownership scans did not — `verify` ran for minutes walking every unpacked source tree | | Generated scripts had no version stamp | the step order lives in the tree, the tools outside it — a tree generated by an older version went on running `last-step` under a lfs-helper that had removed it | | Collector export written and read in two places | 1.9.0 sorted the state dir; the writer stayed at the top level, the reader moved into `groups/` | | `pkgusr-home` was exposed, `owner-name` was not | a script could ask where a package lives but not what its user is called, so `last_build_step.sh` assumed — `chown: invalid user: 'wget:wget'` at step 104 of 105 | @@ -509,6 +520,64 @@ finished, unbootable system without ever having been offered one. GRUB is unchanged and still withheld unless explicitly chosen: its book instructions run `grub-install /dev/sda`, which writes a disk's boot sector. +## A command that works in silence looks like one that hung (1.10.3) + +`lfs-helper verify` ran for five minutes printing nothing, on a tree it had +already verified clean, and was reported as a hang. It was not. Two faults: + +**A fork per path, twice over.** `stat -c %U` for each path, and +`is_never_claimed` running `< <(never_claim_list)` — a process substitution — +for each path too. Across 66265 manifest paths that is ~130000 processes. + +- `_never_claim_load` reads the list once into `_NEVER_CLAIM`; matching is then + pure shell. +- Owners come from **one batched `stat` per manifest** via `xargs -0`. + Measured on 8000 paths: **12.6s → 0.15s**. + +**No output until every pass had finished.** Each pass now announces itself +before it runs, and the long one prints `[n/m] <package>` live. + +Two traps worth remembering, both of which produced code that was *correct and +slow* — the hardest kind of wrong to notice: + +- **`stat` does not expand escapes.** `stat -c '%U\t%n'` prints a literal + backslash-t, so every line came back as one field, the map stayed empty, and + the loop fell through to the per-path `stat` it was written to replace. Use a + real tab (`$'\t'`). +- **I first pasted the batching into `cmd_build`**, not the verify pass — both + contain `for man in ...` over manifests. The tests caught it. + +## Scratch, and the one list that names it (1.10.2) + +`scan_prune_paths` / `scan_prune_set` name everything no ownership scan should +walk: `/dev /proc /sys /run /tmp`, `$LFS/sources`, `$BUILD_ROOT`, `$STATE`, and +**every package's unpacked source tree** (`<home>/$PKGUSR_BUILD_SUBDIR`). + +There were five such lists and they had already diverged. Build trees moved from +`/build` into each package user's home in 1.7.6; the snapshot scans followed and +the ownership scans did not. So `lfs-helper verify` began walking every unpacked +source tree in the system — minutes of silence — and a finished tree filled the +sanity report with tcl's own documentation: + + !! files with no owner (first 40): + /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm + +A tarball can carry any uid it likes. Unpacked sources are not installed, +nothing owns them, and asking who does has no answer. + +**`SCAN_PRUNE` is an ARRAY, not a string spliced through `eval`.** The patterns +contain `*`, and `eval` lets the shell expand them against the real filesystem +before `find` ever sees them — `/usr/src/pkgusr/*/src/*` becomes a handful of +literal directory names and the prune silently matches almost nothing. That is +what the first version of this fix did, and it looked like it worked. + +`lfs-sanity.sh` prunes the same paths, or it reports what `verify` does not. + +Also: a config step that writes **root-owned** files (`cfg_clock` → `/etc/adjtime`) +is not an orphaned manifest. It installs no software, so it has no account, and +there is nothing adoption could ever want. Eleven such findings on a perfect +tree. + ## The tree remembers which version generated it (1.10.1) `gen-chroot-scripts` writes `progress/generated-by` with the version, build id @@ -556,6 +625,215 @@ it. Set it empty to turn it off. back it should be a directory of drop-in scripts, not a template copied once that then drifts from the tool that generated it. +## The kernel and the bootloader are yours (1.11.6) + +Both are decisions about the whole machine rather than about a package, and +getting either wrong costs the system doing the building. So the build does +neither, and ends by **saying** so — `_say_what_is_yours_to_finish`, beside the +login and strip warnings — rather than leaving it to be discovered after a +reboot. + +`bootloader` was already `none` by default and stays that way. The prompt now +says what that means: nothing is written to any ESP, boot sector or partition +table, and the machine keeps booting exactly as it does today. rEFInd is opt-in +by name, and the closing note prints how to ask for it. + +## $LFS may never be the running system (1.11.5) + +Every path this tool writes is `$LFS/something`. Nothing checked that the tree +was not the host's own, so a config holding `/`, an `export LFS=/` in the wrong +shell, or a typo in the interview was enough for +`build-system restart --run` to delete the top-level directories of the machine +doing the building, and for `_verify_lfs_ownership` to run `chown -R lfs /usr` +on it. + +`_is_host_path` is the rule; `_refuse_host_tree` is the hard stop. It is called +from `require_lfs_for_root`, the single door every command resolves `$LFS` +through, and from `_verify_lfs_ownership`, which resolved `$LFS` itself and was +therefore the one destructive path with no check at all. The interview refuses +the same paths at the prompt, where you can just type another one. + +**Both spellings are checked.** On a merged-`/usr` system `/bin` is a symlink +to `usr/bin`, so `realpath("/bin")` is `/usr/bin` and a check on the resolved +path alone let `$LFS=/bin` through. The first version of the test caught this. + +**Inside a system tree counts.** `/usr/src/lfs` is not a build tree, it is a +directory in the system's own `/usr`. `/home` and `/srv` are deliberately not +treated that way: `/home/you/lfs` is an ordinary place to build, and a check +that refuses `$LFS/usr` refuses every build there is. + +What was already safe, and stays so: `restart` refuses while the chroot's bind +mounts are up, because deleting through `/dev` would reach the host. GRUB +scripts are not generated at all unless GRUB is the chosen bootloader, so +nothing that could run `grub-install /dev/sda` is left lying around. The rEFInd +step never formats, partitions or writes a raw device; it adds files to an +already-mounted ESP, backs the config up, adds a menu entry beside your +existing bootloader, and refuses to overwrite an existing rEFInd without +`REFIND_OVERWRITE=1`. + +## A suppressed failure is a bug waiting to be found (1.11.4) + +`soft <what> -- <cmd ...>` runs a command that may fail, and **says so** when +it does, without stopping the build. Every ownership and permission call whose +failure changes what the system looks like goes through it: the wrappers' +own root:root, the install directories' group-writable bit, the staging tree's +inherited owner and mode, `passwd`/`group` backups, the `lfs` build user, and +the mode `verify --fix` counts as fixed. + +Fifteen call sites. `2>/dev/null || true` is not banned — `mkdir -p` on a +directory that exists is fine — but an ownership call that keeps it must carry +a comment on the line immediately above saying **SUPPRESSED DELIBERATELY** or +**benign:**, and a test counts the ones that do not. + +Any comment was not enough as a rule. The surrounding prose explains what a +call does, not why its failure may be thrown away, so a reintroduced +suppression inherits whatever comment happened to be above it — the first +version of the test passed a mutation that put one straight back. + +The two deliberate ones are `mkstate` and `ensure_pkgusr_roots`: both run +before book 7.6, when the `install` group does not exist and the call cannot +succeed. Reporting them would print a failure on every invocation for the +first third of a build, and that is how a report becomes noise. + +## The build says what it did not do (1.11.4) + +`strip` is collected by the interview, carried into the chroot environment as +`LFS_STRIP`, and acted on by nothing. `_warn_if_strip_was_asked_for` is now the +build's second-to-last word, beside `_warn_if_no_login` — the place where the +things you still have to do yourself are collected. It fires only if you asked. + +Still not implemented, and the reason stands: stripping rewrites every binary, +and here a file's owner is the record of which package installed it, so it has +to run as that owner, package by package. + +## One door per decision (1.11.3) + +Item 6 does not need a rewrite. `grant_dir_to_user` already showed the shape: +**ask `lfs-helper` when it is on the system, keep the local copy as the +fallback for when it is not.** Three more decisions now go through it. + +**The wrappers.** `packagemanager_install` shipped its own 480-line copy and +wrote it to `/tmp/pkgusr-wrappers.$$` on every run. The wrappers *are* the rule +about what a package user may do — chown skipped, chgrp skipped, `install -d` +on an existing directory allowed to succeed — so that was a second answer to +the question, and `/etc/pkgusr/bash_profile` pointed at neither copy: it names +`/usr/lib/pkgusr`, which is where lfs-helper writes them. Now +`_wrapper_dir_from_lfs_helper` asks, and writes them through +`lfs-helper make-wrappers` if they are missing or incomplete — which also +repairs a tree whose wrappers predate a fix. The local copy stays, for a system +with no lfs-helper. + +**Only a temporary directory is deleted.** Both call sites ended with +`rm -rf "$_wrapdir"`. Pointed at the shared directory that removes the wrappers +from every package user on the system, so the cleanup is now conditional on +having built a temporary one. + +**The account home.** `pkgusr_home_for` re-derived the layout — root per kind, +prefix, `cfg_` special case. Correct today, wrong the moment the layout is +configured differently, which it can be. It asks `lfs-helper pkgusr-home` now. + +**site-packages.** `pip_install_module` did its own `groupadd` + `chgrp -R` + +`chmod -R g+w`: a third implementation, and the only one that never asked what +group the directory already carried. It goes through `lfs-helper grant-dir`. +That is the debt 1.11.2 left. + +Two new commands make the doors reachable: `lfs-helper wrapper-dir` and +`lfs-helper make-wrappers --run`. + +**The dispatch repeated six entries** — `verify`, `install-as`, `pkgusr-info`, +`pkgusr-home`, `owner-name`, `clean-state` — and `case` takes the first match, +so the second set was unreachable code that looked like a second decision. A +test now fails on any repeat. + +What is left of item 6 is the build loop itself: `cmd_build` and +`packagemanager_install` still each drive unpack/build/install/configure. That +is where staging (item 4) lives, and it is the part that needs a real build to +prove. + +## packagemanager setup installs, not just declares (1.11.2) + +`--run` was a stub. It now does the bootstrap half of the old final step, and +`--sources` is unchanged, so `get-sources` still asks it the same question. + +- **wget** comes from `packagemanager install wget --run`, invoked as a + command. A build plan, a script, a package user and a manifest — there is no + "just the wget part" of that which is not a second implementation of it. +- **The wheels** go through `pip_install_module`, extracted from `cmd_pip` so + both commands run the same code. `--no-deps`, in the order `_SETUP_WHEELS` + lists them, `--no-index --find-links /sources`: a fresh system has no CA + certificates, so a pip index lookup is a hang rather than an error. +- **Idempotent.** It reads `<dist>-<version>.dist-info` in site-packages — + what pip itself writes, so the check is an identity rather than a guess — + and skips what is there. `--force` redoes it. +- **It refuses** when the wheels are not on disk, naming each one and saying + they came from `get-sources` before the reboot. + +**A bug fixed on the way in.** `cmd_pip` took `_sanitise_user_name(mod)` and +used it everywhere. `create_package_user` normalises internally, so the account +it created was `p_requests` while `usermod` and `su` were handed `requests`: + + usermod: user 'requests' does not exist + +Nobody hit it because nothing called `packagemanager pip` on a real system. +Same species as `chown: invalid user: 'wget:wget'` — half the code asking for +the name, half assembling it. + +Still owed: site-packages access is granted by `chgrp -R`/`chmod -R g+w` inside +`pip_install_module` rather than by `lfs-helper grant-dir`. Correct group, +right result, wrong door — fold it in with item 6. + +## last-step is cleaned out of the tree (1.11.1) + +`last_build_step.sh` stopped existing in 1.10.0, but three things still spoke +as if it did: + +- **`make install` copied it into `/etc/lfs`** and aborted, because the file is + not in the repo. Installing was broken and nothing said so. +- **`build-all` sealed the install directories early** when the step name was + `last-step` — dead since the final step became `init-accounts`, which runs as + root and installs nothing. The seal after the loop was already doing the work. +- **Three tests looked for the file beside `lfs`** and skipped silently when it + was absent, which is every run since 1.10.0. + +The name stays in `_is_not_a_package`'s match: a tree generated before 1.10.0 +still carries it in `steporder` and in its manifests, and the rule must still +say no. It is marked as legacy there. + +`make clean` exists now, and `make install` no longer hides a failed test +install behind `2>/dev/null || true` — the first of item 10. + +## The build's last word is whether you can log in (1.11.0) + +`init-accounts` asks for a root password, but it is skippable — a scripted run +has no terminal, and an interactive one can be answered with a blank line. So +`_warn_if_no_login` is the **last thing `build-all` does**, because it is the +last moment it can still be fixed. After the reboot the chroot is gone and the +way back is booting the host and mounting the tree by hand. + +It checks the **result**, not that the step ran: `/etc/shadow` for a real root +password, or any account between uid 1000 and 9999 with one. + + !! NOBODY CAN LOG INTO THIS SYSTEM. + Fix it NOW, while you are still in here: + lfs-helper init-accounts --force + +`lfs-helper check-login` runs it on its own. + +## wget belongs to packagemanager setup (1.11.0) + +`packagemanager setup --sources` prints every URL setup needs, and +`lfs build-system get-sources` asks it. **One list, owned by the tool that +installs from it.** + +The first version of this had `bootstrap_packages` in `lfs`'s own config, +resolving wget from the BLFS book separately from the tool that installs it — +a second thing to go stale, which is the mistake this project keeps making. +That key is gone. + +`setup` itself is a stub: `--sources` works, `--run` says it is not implemented +and exits non-zero. That is deliberate — `get-sources` needs the list *now*, so +a fresh build still downloads what the finished system will need. + ## packagemanager setup: what it owes Three guarantees left the suite with `last_build_step.sh` and belong to `setup`. @@ -575,21 +853,6 @@ Also: wget needs `check_certificate = off` in `/etc/wgetrc` with the `lfs-temporary-no-verify` marker, because a fresh system has no CA certificates until BLFS `make-ca` — and you need wget to fetch make-ca. -## The final step is yours, so fixes do not reach it - -`/etc/lfs/last_build_step.sh` is created once from a template embedded in `lfs` -and then **belongs to the person** — it is meant to be edited, so it is never -overwritten. Which means a fix to the template reaches nobody who already has a -copy, and the stale copy fails at step 104 of 105, after six hours, with an -error that looks like it came from the tools: - - chown: invalid user: 'wget:wget' - -`_LAST_STEP_STALE` lists known-broken patterns and -`_warn_if_last_step_is_stale` reports them at generation time with the exact -repair. A fixed copy stays quiet, or the warning becomes noise. **Fixing the -template alone is not fixing the bug** — there are two copies by design. - ## Things that belong to nobody `never_claim_list` — files no package may own, however many write to them: @@ -721,29 +984,11 @@ Then `bash lfs-sanity.sh` and read the `!!` lines. ## Open items -1. **NEXT JOB: `packagemanager setup`.** `last_build_step.sh` does two - unrelated things: a *bootstrap* (builds `wget`, installs seven Python wheels - so `lfs` and `blfs` can parse the books at all) and *configuration* (root - password, login account). Only the second belongs in a file whose header - says "EDIT THIS FILE. It is yours." The bootstrap should become - `packagemanager setup`: idempotent, re-runnable, testable — today it runs - exactly once, at the end of a six-hour build. - - Requirements to preserve: wheels not sdists (a modern sdist needs its build - backend and no index is reachable — `BackendUnavailable: Cannot import - 'hatchling.build'`); dependency order with `--no-deps`; `PIP_USER=0 - PYTHONNOUSERSITE=1`; each module as its own package user; site-packages - reached via `lfs-helper grant-dir` joining the collector group the directory - already carries, never the `install` group; and the `/etc/wgetrc` - `lfs-temporary-no-verify` marker, which is how the tools later find and - remove that weakening once `make-ca` exists. - - **Trap:** `lfs build-system get-sources` parses the `SOURCES=()` array - *out of `last_build_step.sh`*. If the wheel list moves, `get-sources` must - still find it. There is a test on that path: *the final step's SOURCES are - parsed (never executed)*. - -2. **`strip` is asked for but does nothing.** The answer is collected and the +1. ~~`packagemanager setup`~~ — **done in 1.11.2.** What is left of it is the + grant-dir door, folded into item 6. + +2. **`strip` is asked for but does nothing** — and since 1.11.4 the build says + so at the end, so it is no longer silent. Implementing it: The answer is collected and the prompt says NOT YET IMPLEMENTED. Not a book copy-paste: stripping rewrites each binary, and here a file's owner is the record of which package installed it, so it must run as that owner or the record drifts. Book 8.85 @@ -763,11 +1008,18 @@ Then `bash lfs-sanity.sh` and read the `!!` lines. the build loop twice. 5. **Root password** — the one decision the interview cannot hold; a plaintext - config would be worse. `last_build_step.sh` still prompts, and prints - `chroot /mnt/lfs /usr/bin/passwd root` when non-interactive. - -6. **`lfs-helper` and `packagemanager_install` are two implementations of one - thing.** Nearly every bug this project has had was the same species: one + config would be worse. Handled as far as it can be: `init-accounts` prompts, + and `_warn_if_no_login` checks the *result* as build-all's last word. A + non-interactive run therefore finishes with a system nobody can log into, + loudly. Nothing further is planned. + +6. **The build loop is still two implementations.** Everything else went + through one door in 1.11.3 — wrappers, account home, collector grants — and + what remains is the loop itself: `cmd_build` and `packagemanager_install` + each drive unpack/build/install/configure. Item 4 (staging) lives inside it. + Needs a real build to prove, so it is not a desk change. + + The original framing, still true of the loop: Nearly every bug this project has had was the same species: one decision held in several places, free to disagree — `cfg_bootscripts`, `adopted.list`, the package-user prefix, the `/sources` permissions, the account's kind. `install-as` is the entry point that makes unifying them @@ -790,12 +1042,10 @@ Then `bash lfs-sanity.sh` and read the `!!` lines. has no fallback for a tree built before the move. Harmless given that no tree is migrated; worth deleting the ambiguity if migration is ever written. -10. **A sweep for `2>/dev/null || true` is still overdue.** Six suppressed - errors were made visible in an earlier session and they were the cause of - nearly every bug that session. `ensure_pkgusr_roots` adds two more - suppressed `chown`/`chmod` calls — deliberately, because the roots being - wrong is not fatal and `verify --fix` repairs it, but it is the same - pattern and should be counted in the sweep. +10. ~~The `2>/dev/null || true` sweep~~ — **done in 1.11.4** for ownership and + permission calls, with `soft` and a test on the marker. `packagemanager_install` + still has seven, all benign (chmod on a scratch directory, `disown` on a + backgrounded mandb); they have not been reviewed one by one. ## Useful commands @@ -814,7 +1064,7 @@ lfs-helper pkgusr-info # where each package came from ## Tests ```sh -bash test_lfs_crosschain.sh ./lfs # 569 tests, 568 pass +bash test_lfs_crosschain.sh ./lfs # 577 tests, 576 pass ``` Counts are kept in files, not shell variables, so subshell increments survive. @@ -840,8 +1090,9 @@ no longer calls something. The comment explaining *why* it no longer calls it names the thing, so a naive grep matches the explanation and the test can never pass. Strip comments first, or anchor on the call syntax. -Run it from a directory holding ALL five scripts plus `last_build_step.sh` — -about 100 tests silently skip if the others are not beside `lfs`. +Run it from a directory holding ALL five scripts, plus `lfs-sanity.sh` and +`skel-u_xdg/.bash_profile` — about 100 tests silently skip if the others are +not beside `lfs`. ## Working style that worked diff --git a/Makefile b/Makefile index d37b440..b395a75 100644 --- a/Makefile +++ b/Makefile @@ -3,6 +3,7 @@ # make install -> /usr # make install PREFIX=/usr/local -> /usr/local # make uninstall +# make clean -> scratch files, not installed ones # make check -> run the regression tests # # DESTDIR is honoured, so this works for staged installs and for installing @@ -22,12 +23,11 @@ SYSCONFDIR ?= $(DESTDIR)/etc # The four tools, plus the bash engine packagemanager calls. TOOLS := lfs lfs-helper packagemanager packagemanager_install blfs COMPLETION := lfs-completion.bash -LAST_STEP := last_build_step.sh TESTS := test_lfs_crosschain.sh INSTALL ?= install -.PHONY: all install uninstall check help +.PHONY: all install uninstall check clean help all: help @@ -35,6 +35,7 @@ help: @echo "make install [PREFIX=...] [DESTDIR=...] install the tools" @echo "make uninstall [PREFIX=...] remove them" @echo "make check run the tests" + @echo "make clean remove scratch files" @echo "" @echo "installs into: $(PREFIX)/bin" @@ -63,18 +64,15 @@ install: "$(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile"; \ echo " installed (XDG_RUNTIME_DIR is filled in on first use)"; \ fi - @echo "==> last build step -> $(SYSCONFDIR)/lfs" - $(INSTALL) -d $(SYSCONFDIR)/lfs - @if [ -f "$(SYSCONFDIR)/lfs/$(LAST_STEP)" ]; then \ - echo " kept your existing $(SYSCONFDIR)/lfs/$(LAST_STEP)"; \ - $(INSTALL) -m 644 $(LAST_STEP) "$(SYSCONFDIR)/lfs/$(LAST_STEP).new"; \ - echo " new version alongside it as $(LAST_STEP).new"; \ - else \ - $(INSTALL) -m 755 $(LAST_STEP) "$(SYSCONFDIR)/lfs/$(LAST_STEP)"; \ - fi @echo "==> tests -> $(SHAREDIR)" $(INSTALL) -d $(SHAREDIR) - $(INSTALL) -m 644 $(TESTS) $(SHAREDIR)/ 2>/dev/null || true +# The tests are optional -- say so when they are absent, rather than hiding a +# real failure. `2>/dev/null || true` hid both cases alike. + @if [ -f "$(TESTS)" ]; then \ + $(INSTALL) -m 644 $(TESTS) $(SHAREDIR)/ && echo " $(TESTS)"; \ + else \ + echo " (not shipped: $(TESTS))"; \ + fi @echo "" @echo "Installed. Check with:" @echo " $(PREFIX)/bin/lfs --version" @@ -88,10 +86,17 @@ uninstall: rm -rfv $(SHAREDIR) @echo "" @echo "Left alone (they are yours):" - @echo " $(SYSCONFDIR)/lfs/$(LAST_STEP)" @echo " $(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile" @echo " /usr/share/lfs books, settings, snapshots" @echo " /etc/pkgusr packagemanager settings" check: @bash $(TESTS) ./lfs + +# Nothing here is compiled, so clean removes only what running the tools here +# leaves behind: Python bytecode and the test suite's scratch directories. +# It never touches an installed tree -- that is what uninstall is for. +clean: + rm -rf __pycache__ *.pyc + rm -rf /tmp/lfstest.* /tmp/lfstest-count.* + @echo "clean. (installed files are untouched -- use 'make uninstall')" diff --git a/README.md b/README.md index 3d76163..86b60d5 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ Build and maintain an LFS/BLFS system where **every package is owned by its own user**. > **100% vibecode, but tested.** Prompted into existence rather than hand -> written. 568 regression tests. It builds my own system — read it before -> you point it at yours. +> written. Every fix carries a regression test. It builds my own system — +> read it before you point it at yours. --- @@ -48,7 +48,7 @@ make uninstall `/usr` is the default: the chroot invokes these by name, and `packagemanager` runs `packagemanager_install` off `$PATH`. -Installs the four tools, the completion script, and `last_build_step.sh`. +Installs the four tools, `packagemanager_install`, and the completion script. **Requires:** Python 3.9+, bash, coreutils, tar. Book parsing needs `beautifulsoup4` and `requests`. @@ -78,6 +78,18 @@ failure, or a cancel. ## Use the built system +A fresh system has no download tool, and `lfs` and `blfs` need `requests` and +`beautifulsoup4` to read the books. Bootstrap both, offline, from what +`get-sources` already downloaded: + +```sh +packagemanager setup # what it would do +packagemanager setup --run +``` + +Each module is installed as its own package user. Run it again after fixing a +failure; it skips what is already there. + ```sh packagemanager install <pkg> --recursive --run packagemanager update <pkg> --run @@ -199,7 +211,7 @@ let any package user delete another package's home. The tools print a fingerprint of their own contents: ```sh -lfs --version # lfs 1.7.1 (build 4f5f344) +lfs --version # name, version, build id lfs-helper --version # inside the chroot ``` @@ -214,7 +226,7 @@ enter it. bash test_lfs_crosschain.sh ./lfs ``` -568 tests. Each encodes a bug that actually happened, with a comment explaining +Each test encodes a bug that actually happened, with a comment explaining what broke. Run it from a directory holding all five scripts — about 100 tests skip @@ -224,7 +236,12 @@ without them. ## Caveats -- The kernel and the bootloader's kernel line are yours to finish. +- **The kernel and the bootloader are yours.** Nothing here configures, builds + or installs a kernel, and nothing is written to any ESP, boot sector or + partition table. The machine keeps booting exactly as it does now. The build + ends by saying so and printing what to run. +- A rEFInd entry can be added beside your existing bootloader if you ask for it + (`lfs config bootloader refind`). It only ever adds to a mounted ESP. - `lfs-helper` and `packagemanager_install` duplicate some logic. Unifying them is worthwhile but not done. - Tested on x86_64 only. diff --git a/blfs b/blfs index f851ef0..943b4ec 100644 --- a/blfs +++ b/blfs @@ -118,7 +118,7 @@ CACHE_VERSION = 4 # tool + generated-script format versions. Bump SCRIPT_VERSION when the shape # of generated install scripts changes (phases, dispatcher, ...), and # CACHE_VERSION when the parsed/cached data structure changes. -BLFS_VERSION = "1.10.1" +BLFS_VERSION = "1.11.6" def _build_id(): diff --git a/lfs b/lfs index 721b1a1..ac5c67b 100644 --- a/lfs +++ b/lfs @@ -60,7 +60,7 @@ except ImportError: " packagemanager pip install beautifulsoup4\n") sys.exit(2) -LFS_VERSION = "1.10.1" +LFS_VERSION = "1.11.6" def _build_id(): @@ -1222,6 +1222,84 @@ def _lfs_dir(): return os.environ.get("LFS") or load_config().get("lfs_mount") +# THE ONE THING THAT COULD DESTROY THE MACHINE YOU ARE BUILDING ON. +# +# Every path in this tool is `$LFS/something`. If $LFS is the running system, +# `$LFS/usr` is the HOST's /usr -- and `build-system restart --run` deletes the +# top-level directories of the tree it is given. Nothing checked that the tree +# was not the host's own: a config holding `/` as the mount point, an +# `export LFS=/` in the wrong shell, or a typo in the interview was enough. +# +# The chroot's bind mounts are already guarded (restart refuses while /dev is +# mounted, because deleting through it would reach the host's /dev). This is +# the other half: refusing when the tree IS the host. +# +# It lives in require_lfs_for_root because that is the single door every +# command resolves $LFS through -- put it anywhere else and the next command +# added is the one that misses it. +_HOST_DIRS = ("/", "/usr", "/etc", "/boot", "/bin", "/sbin", "/lib", "/lib64", + "/var", "/home", "/root", "/opt", "/srv", "/run", "/dev", + "/proc", "/sys", + # merged-/usr: /bin is a symlink to usr/bin, so realpath("/bin") + # is "/usr/bin" and the names above never match it. A check that + # only looked at the resolved path let $LFS=/bin through. + "/usr/bin", "/usr/sbin", "/usr/lib", "/usr/lib64", "/usr/share", + "/usr/include", "/usr/local", "/usr/src") + +# Trees that belong to the running system whole. Anything inside one of these +# is the host, however deep -- $LFS=/usr/src/lfs is not a build tree, it is a +# directory in the system's own /usr. /home and /srv are deliberately NOT +# here: /home/you/lfs is a perfectly ordinary place to build. +_HOST_TREES = ("/usr/", "/etc/", "/boot/", "/bin/", "/sbin/", "/lib/", + "/lib64/", "/proc/", "/sys/", "/dev/", "/run/", "/var/") + + +def _is_host_path(p): + """True if this path is part of the running system. The test behind both + the interview's refusal and the hard stop below, so they cannot disagree.""" + try: + real = os.path.realpath(p) + except OSError: + return False + # BOTH spellings. On a merged-/usr system /bin resolves to /usr/bin, so + # checking only the resolved path misses the name the person typed, and + # checking only the name misses where it actually points. + given = os.path.abspath(os.path.expanduser(str(p))).rstrip("/") or "/" + if real in _HOST_DIRS or given in _HOST_DIRS: + return True + for t in _HOST_TREES: + if real.startswith(t) or given.startswith(t): + return True + try: + if os.path.samefile(real, "/"): + return True + except OSError: + pass + for d in ("/usr", "/etc", "/boot"): + cand = os.path.join(real, d.lstrip("/")) + try: + if os.path.exists(cand) and os.path.samefile(cand, d): + return True + except OSError: + continue + return False + + +def _refuse_host_tree(lfs): + """Hard stop if $LFS is, or contains, the running system.""" + if not _is_host_path(lfs): + return lfs + real = os.path.realpath(lfs) + sys.stderr.write( + "\n!! REFUSING: %s is part of the running system.\n" + " Every path this tool writes is $LFS/something, so this would\n" + " build into, and `restart` would delete, the machine you are\n" + " sitting on.\n" + "\n Set it to the mount point of the LFS partition:\n" + " lfs build-system session\n" % real) + sys.exit(2) + + def require_lfs_for_root(): """Ensure an LFS mount is known (book rule: root needs $LFS after 2.4). Falls back to the saved session config and exports it so child shells inherit it.""" @@ -1232,6 +1310,7 @@ def require_lfs_for_root(): " Configure it once: lfs build-system session\n" " or for this shell: export LFS=/mnt/lfs\n") sys.exit(2) + _refuse_host_tree(d) os.environ["LFS"] = d return d @@ -1301,10 +1380,10 @@ _SESSION_KEYS = [ ("locale", "system locale (see 'locale -a')", "en_US.UTF-8"), ("paper_size", "paper size for groff: A4 or letter", "A4"), # Everything below is a decision the build used to discover halfway - # through, or at the very end -- `last_build_step.sh` blocked on `passwd - # root` after a six-hour build, and its own comment admits the - # non-interactive path leaves the system "unbootable-but-known". Asking - # here is what lets `lfs run` and `build-all` finish without a person. + # through, or at the very end -- the old final step blocked on `passwd + # root` after a six-hour build, and its non-interactive path left the + # system "unbootable-but-known". Asking here is what lets `lfs run` and + # `build-all` finish without a person. ("timezone", "time zone for the built system (e.g. Europe/Berlin, " "or UTC)", "UTC"), # Deleting a working cross-toolchain should never happen by default, so @@ -1320,10 +1399,11 @@ _SESSION_KEYS = [ ("strip", "strip debug symbols at the end? Saves several GB, " "but rewrites every binary -- slower, and harder to " "debug a crash afterwards. NOT YET IMPLEMENTED: the " - "answer is stored and nothing acts on it (yes/no)", + "answer is stored and the build says so at the end " + "rather than acting on it (yes/no)", "no"), - # The account you will actually log in as. last_build_step.sh reads this; - # without it, a freshly booted system has only root, with no password. + # The account you will actually log in as. The `init-accounts` step reads + # this; without it, a freshly booted system has only root, with no password. ("main_user", "login account to create on the built system " "(blank for none)", ""), ] @@ -1353,6 +1433,17 @@ def _prompt_session_config(cfg, force=False): continue val = input(f" {desc} [{cur}]: ").strip() val = val if val else cur + # Refuse a mount point that is the running system, HERE, where you can + # simply type another one. The guard in require_lfs_for_root catches + # it later too, but by then it is stored, every command refuses, and + # the fix is to work out which config key to edit. + if k == "lfs_mount" and val: + while _is_host_path(val): + sys.stderr.write( + " !! %s is part of the running system -- refusing.\n" + " The LFS mount point is an empty directory the LFS\n" + " partition mounts on, e.g. /mnt/lfs.\n" % val) + val = input(f" {desc} [{cur}]: ").strip() or cur if k == "collector_import_file" and val: # Accepting a path that is not there means the groups silently # never get imported and every sharing question comes back. @@ -1598,6 +1689,12 @@ def _verify_lfs_ownership(): lfs = _lfs_dir() if not lfs: return + # This function runs `chown -R lfs` on $LFS/usr, $LFS/lib64, $LFS/var and + # $LFS/etc. It resolves $LFS itself rather than going through + # require_lfs_for_root, so it was the one destructive path with no check + # that the tree is not the running system -- and on a host, /usr has no + # package users, so _handover_is_safe would have said yes. + _refuse_host_tree(lfs) import pwd as _pwd try: want = _pwd.getpwnam("lfs").pw_uid @@ -2838,17 +2935,15 @@ CONFIG_KEYS = [ ("editor", "editor for --edit; default: $EDITOR or vim"), ("sudo", "prefix suggested root commands with 'sudo' " "(on/off; default off)"), - ("bootstrap_packages", - "BLFS packages the built system needs before it can " - "fetch anything at all (default: wget). Their tarballs " - "are downloaded by get-sources, while there is still a " - "network; `packagemanager setup` installs them on the " - "booted system. Blank for none"), - ("bootloader", "which bootloader to set up: none (default), refind, " - "or grub. GRUB's book instructions run " - "'grub-install /dev/sda', which writes to a disk's boot " - "sector -- with 'refind' that section is skipped " - "entirely and a rEFInd script is generated instead"), + ("bootloader", "which bootloader to set up: none (DEFAULT -- nothing " + "is written to any ESP, boot sector or partition " + "table, and the machine keeps booting exactly as it " + "does now), refind, or grub. GRUB's book instructions " + "run 'grub-install /dev/sda', which writes to a disk's " + "boot sector -- with 'refind' that section is skipped " + "entirely and a rEFInd script is generated instead, " + "which only ADDS an entry beside your existing " + "bootloader"), ("esp", "path of the EFI System Partition inside the built " "system (e.g. /boot/efi). rEFInd installs there; it is " "never formatted, and an existing installation is left " @@ -3476,7 +3571,7 @@ def write_chroot_env(lfs, quiet=False): f'# languages to keep -- see `lfs-helper prune-locales`\n' f'export LFS_LOCALES="{cfg.get("locales") or "en de"}"\n' f'# answers collected by `lfs build-system session`, so the\n' - f'# chroot side never has to stop and ask. last_build_step.sh\n' + f'# chroot side never has to stop and ask. `init-accounts`\n' f'# reads LFS_MAIN_USER instead of blocking on a prompt after\n' f'# a six-hour build.\n' f'export LFS_TIMEZONE="{cfg.get("timezone") or "UTC"}"\n' @@ -4081,60 +4176,36 @@ def _gen_blfs_extra(outdir, order, unresolved): -def bootstrap_packages(): - """BLFS packages the built system needs before it can fetch anything. - - Just wget by default: it is the one thing LFS leaves out that makes a fresh - system unable to download its own next package. `packagemanager setup` - installs it; this is only about having the tarball on disk in time. - """ - v = load_config().get("bootstrap_packages") - if v is None: - return ["wget"] - return [p for p in str(v).replace(",", " ").split() if p] - - def bootstrap_sources(): - """Their download URLs, asked of the BLFS book. + """What the built system needs downloaded before it can fetch anything. - `blfs` runs on the host and keeps its own book store, so this works before - the chroot exists. Asking the book rather than hardcoding URLs means there - is one place that knows where wget comes from, and it is the same place - `packagemanager` will ask when it installs it. + ASKED OF `packagemanager setup`, which owns that decision -- wget and the + Python modules the tools need are its job to install, so the list of what + to download is its to declare. This command only has to make sure they are + on disk before the chroot is sealed, because get-sources is the last moment + there is still a network. + + Two tools, one list. A copy here would be a second thing to go stale, and + the first version of this did exactly that: `bootstrap_packages` in lfs's + own config, resolving wget separately from the tool that installs it. """ import subprocess - urls = [] - for pkg in bootstrap_packages(): - try: - out = subprocess.run(["blfs", "sources", pkg], - capture_output=True, text=True, timeout=120) - except (OSError, subprocess.SubprocessError): - warn("! could not run `blfs sources %s` -- is blfs installed?" % pkg) - continue - got = [l.strip() for l in (out.stdout or "").splitlines() - if l.strip().startswith(("http://", "https://", "ftp://"))] - if not got: - warn("! the BLFS book gave no download for '%s'." % pkg) - warn(" A fresh system will have no way to fetch anything. Either") - warn(" fetch a book (blfs fetch) or drop it:") - warn(" lfs config bootstrap_packages ''") - continue - urls += got + try: + out = subprocess.run(["packagemanager", "setup", "--sources"], + capture_output=True, text=True, timeout=180) + except (OSError, subprocess.SubprocessError): + warn("! could not run `packagemanager setup --sources`.") + warn(" The built system will have no way to fetch anything until you") + warn(" put a download tool on it by hand.") + return [] + urls = [l.strip() for l in (out.stdout or "").splitlines() + if l.strip().startswith(("http://", "https://", "ftp://"))] + if not urls: + warn("! `packagemanager setup --sources` listed nothing.") + for line in (out.stderr or "").splitlines(): + warn(" %s" % line) return urls - - - - - - - - - - - - - def _write_tool_stamps(lfs, tools): """Record the md5 of every tool we copied in. diff --git a/lfs-helper b/lfs-helper index c8cf943..6503e33 100644 --- a/lfs-helper +++ b/lfs-helper @@ -22,7 +22,7 @@ set -u # Bump on every change that alters behaviour. `lfs-helper --version` makes it # possible to tell at a glance whether the copy inside the chroot is the one # that was just fixed -- guessing at that has wasted a lot of time. -LFS_HELPER_VERSION="1.10.1" +LFS_HELPER_VERSION="1.11.6" # A fingerprint of this file's own contents. A hand-maintained version number # goes stale the moment someone forgets to bump it -- and the reason for @@ -185,6 +185,35 @@ say() { echo "$*"; } ok() { echo "${C_OK}$*${C_OFF}"; } # yellow on stderr: something needs attention, and is not part of normal flow warn() { echo "${C_WARN}$*${C_OFF}" >&2; } + +# `cmd ... 2>/dev/null || true` -- the pattern that has cost this project more +# debugging than any other. A whole tree came out with every package home +# root:root 755, and not one line of output said why; the reason it took a +# filesystem check, an su test and three rounds of guessing was that the +# failure had been suppressed at the point it happened. +# +# There are two honest reasons to write it, and they are different: +# +# soft <what> -- <cmd ...> the command MAY fail and the build carries on, +# but the failure is SAID. Use this wherever a +# failure changes what the system looks like: +# ownership, modes, group membership, backups. +# +# <cmd> 2>/dev/null || true only where failing is NORMAL and means nothing: +# mkdir -p on a directory that exists, appending +# to a log that may not be writable, sorting a +# file in place. Every remaining one carries a +# comment saying which. +soft() { + local what="$1"; shift + [ "${1:-}" = "--" ] && shift + local err rc + err="$("$@" 2>&1 >/dev/null)"; rc=$? + [ "$rc" = 0 ] && return 0 + warn "!! $what" + [ -n "$err" ] && warn " $err" + return 0 +} # yellow on stdout: needs attention, but IS part of the flow -- a retry, a # question, a directory being granted. Same colour because it means the same # thing; stdout because it belongs in sequence with the build output. @@ -214,7 +243,13 @@ mkstate() { mkdir -p "$STATE" "$SCRIPTS" "$MANIFESTS" "$LOGS" \ "$STATE_PROGRESS" "$STATE_GROUPS" "$STATE_CONF" || return 0 if [ "$(id -u)" = 0 ]; then + # SUPPRESSED DELIBERATELY. mkstate runs at the top of every command, + # including ones from before book 7.6, when the `install` group does + # not exist yet and this cannot succeed. Reporting it would print a + # failure on every invocation for the first third of a build, which is + # how a report becomes noise. seal-install-dirs sets it for real. set_install_dir_owner "$STATE" 2>/dev/null || true + # SUPPRESSED DELIBERATELY: same moment, same reason as the line above. real_chmod 1775 "$STATE" 2>/dev/null || true fi } @@ -655,7 +690,8 @@ cmd_init_pkgusr() { # o+t (sticky) is deliberately deferred to seal-install-dirs, because # it would stop package users replacing files the temporary system # installed as root. - real_chmod g+w "$d" 2>/dev/null || true + soft "$d is not group-writable -- package users cannot install into it" \ + -- real_chmod g+w "$d" n=$((n+1)) done < <(install_dirs_list) ok "# $n install directories are now root:install (group-writable)" @@ -798,11 +834,14 @@ EOF # Steps that are not packages, and must never get a package user. # -# last-step, init-* and refind are actions, not software: they own no files of -# their own and installing "as" them makes no sense. A `last-step` user -# appeared because a manifest existed under that name, and everything the step -# touched -- including wget's binary, installed as the wget user -- was then -# handed to it. +# init-* and refind are actions, not software: they own no files of their own +# and installing "as" them makes no sense. A `last-step` user appeared because +# a manifest existed under that name, and everything the step touched -- +# including wget's binary, installed as the wget user -- was then handed to it. +# +# `last-step` itself was removed in 1.10.0 and no tool generates it any more. +# It stays in the match below for trees generated before that: their steporder +# and manifests still carry the name, and the rule must still say no. # # cfg_* is NOT decidable from the name. Most chapter-9 "configuration" # sections write a file (/etc/hostname, /etc/hosts) and are root's work, but @@ -1493,7 +1532,11 @@ ensure_pkgusr_roots() { # another package's home. With it, only the owner can. Group write then # grants only the ability to CREATE, which nothing unprivileged does -- # `add-user` is root-only -- so this is consistency, not capability. + # SUPPRESSED DELIBERATELY, same reason as mkstate: the roots are + # created before the `install` group exists. `lfs-helper verify` + # checks the result afterwards, which is the honest place to find out. set_install_dir_owner "$r" 2>/dev/null || true + # SUPPRESSED DELIBERATELY: same moment, same reason as the line above. real_chmod 1775 "$r" 2>/dev/null || true done < <(pkgusr_roots) } @@ -1741,7 +1784,8 @@ grant_dir_access() { if [ -n "$curgrp" ] && [ "$curgrp" != root ] \ && case "$curgrp" in "$COLLECTOR_PREFIX"*) true ;; *) false ;; esac \ && id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx "$curgrp"; then - real_chmod g+rwx "$dir" 2>/dev/null || true + soft "$dir is group $curgrp but not group-writable -- the grant will not take" \ + -- real_chmod g+rwx "$dir" say " $dir -> already group $curgrp, and '$user' is a member" return 0 fi @@ -1836,7 +1880,8 @@ grant_dir_access() { # is already a member of the group by then, that grant is automatic (see # the fast path at the top of this function). real_chgrp "$grp" "$dir" 2>/dev/null || return 1 - real_chmod g+rwx "$dir" 2>/dev/null || true + soft "$dir is group $grp but not group-writable -- the grant will not take" \ + -- real_chmod g+rwx "$dir" say " $dir -> group $grp (owner $owner, + $user)" mkdir -p "$(dirname "$GRANTED")" grep -qxF "$grp|$user|$owner" "$GRANTED" 2>/dev/null \ @@ -2403,8 +2448,10 @@ EOF # an install directory: root:install and group-writable during the build. # Inheriting that would let any package user rewrite the wrappers -- that # is, rewrite the rule that stops it changing ownership. - chmod 0755 "$WRAPPERS" 2>/dev/null || true - chown root:root "$WRAPPERS" "$WRAPPERS"/* 2>/dev/null || true + soft "could not set $WRAPPERS to 0755 -- a package user may be able to rewrite the wrappers" \ + -- chmod 0755 "$WRAPPERS" + soft "could not give $WRAPPERS to root -- the wrappers are writable by whoever owns them now" \ + -- chown root:root "$WRAPPERS" "$WRAPPERS"/* } # Re-apply the install-directory permissions. @@ -2682,15 +2729,32 @@ NCLIST printf '%s\n' "$STATE" } -is_never_claimed() { - local f="${1%/}" p - [ -n "$f" ] || return 1 +# The list, read ONCE into an array. +# +# is_never_claimed used to run `< <(never_claim_list)` on every call, which is a +# process substitution -- a fork -- per path. `verify` calls it for every path +# in every manifest: 66265 forks on a finished system, and that was only half of +# them. The command ran for minutes in silence. +_TAB=$'\t' +_NEVER_CLAIM=() +_never_claim_load() { + [ "${#_NEVER_CLAIM[@]}" -gt 0 ] && return 0 + local p while IFS= read -r p; do [ -n "$p" ] || continue p="${SNAP_ROOT%/}${p}" - [ "$f" = "${p%/}" ] && return 0 - case "$f" in "${p%/}"/*) return 0 ;; esac + _NEVER_CLAIM+=( "${p%/}" ) done < <(never_claim_list) +} + +is_never_claimed() { + local f="${1%/}" p + [ -n "$f" ] || return 1 + _never_claim_load + for p in "${_NEVER_CLAIM[@]}"; do + [ "$f" = "$p" ] && return 0 + case "$f" in "$p"/*) return 0 ;; esac + done return 1 } @@ -2845,8 +2909,12 @@ _install_staged_tree() { continue fi mkdir -p "$dest" || { rc=1; break; } - real_chown --reference="$d" "$dest" 2>/dev/null || true - real_chmod --reference="$d" "$dest" 2>/dev/null || true + # A staged directory that does not inherit the live one's owner and + # mode is how a package quietly takes /usr/bin. + soft "$dest did not inherit the owner of $d" \ + -- real_chown --reference="$d" "$dest" + soft "$dest did not inherit the mode of $d" \ + -- real_chmod --reference="$d" "$dest" done < <(find "$stage" -mindepth 1 -type d 2>/dev/null | sort) [ "$rc" = 0 ] || return 1 @@ -3348,9 +3416,11 @@ cmd_build() { real_chown "$owner:$owner" "$_stage" 2>/dev/null \ || warn "# could not give $_stage to '$owner'" fi - real_chmod 755 "$_stage" 2>/dev/null || true + soft "$_stage is not 755 -- the package user may not be able to stage into it" \ + -- real_chmod 755 "$_stage" # the parent too, or the package user cannot traverse into it - real_chmod 755 "$(dirname "$_stage")" 2>/dev/null || true + soft "$(dirname "$_stage") is not 755 -- the package user cannot traverse into the staging tree" \ + -- real_chmod 755 "$(dirname "$_stage")" else _stage="" fi @@ -3383,7 +3453,8 @@ cmd_build() { # already owned correctly and nothing has to be chowned afterwards user_exists "$owner" && real_chown "$owner:$owner" "$_bldroot" 2>/dev/null fi - real_chmod 755 "$_bldroot" 2>/dev/null || true + soft "$_bldroot is not 755 -- the build may not be able to enter its own tree" \ + -- real_chmod 755 "$_bldroot" local envpass="LFS_CC_PHASE='$phase' BUILD_ROOT='$_bldroot'" [ -n "$_stage" ] && envpass="$envpass DESTDIR='$_stage'" @@ -3851,6 +3922,7 @@ cmd_build() { echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper" [ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script" } > "$_home/VERSION" + # benign: a note in the package's own home, rewritten every install real_chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true fi @@ -4028,6 +4100,33 @@ _needs_adoption() { # Give one user access to one directory, using the normal rules. +# The wrappers, on demand. +# +# `packagemanager_install` shipped its own 480-line copy and wrote it to a +# fresh /tmp directory on every run -- a second set of wrappers, with its own +# idea of what `chown` should do, for the same job. This is the door it asks +# through instead. Idempotent: writing them again is how a fixed wrapper +# reaches a system that already has the old one. +cmd_make_wrappers() { + need_root + local run=0 + case "${1:-}" in --run) run=1 ;; esac + if [ "$run" != 1 ]; then + say "# would write the package-user wrappers to $WRAPPERS" + say "# $(cd "$WRAPPERS" 2>/dev/null && echo "currently: $(ls | tr '\n' ' ')")" + say "# apply with: lfs-helper make-wrappers --run" + return 0 + fi + make_wrappers + ok "wrappers written to $WRAPPERS" + printf '%s\n' "$WRAPPERS" +} + +# Where they are, for a caller that only needs the path. +cmd_wrapper_dir() { + printf '%s\n' "$WRAPPERS" +} + cmd_grant_dir() { need_root local dir="${1:?usage: lfs-helper grant-dir <dir> <user> [--run]}" @@ -4109,10 +4208,8 @@ _has_orphaned_files() { local r="${SNAP_ROOT%/}" hit # see _vfy_orphans: before 7.6 nothing resolves, so everything looks orphaned _user_db_ready || return 1 - hit="$(find "${r:-/}" -xdev -nouser \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - -not -path "$r/sources/*" \ + scan_prune_set + hit="$(find "${r:-/}" -xdev -nouser "${SCAN_PRUNE[@]}" \ -print -quit 2>/dev/null)" [ -n "$hit" ] } @@ -4472,7 +4569,7 @@ cmd_sort_users() { continue fi if [ "$run" = 1 ]; then - cp -p "$f" "$f.bak" 2>/dev/null || true + soft "no backup of $f -- editing it without one" -- cp -p "$f" "$f.bak" cat "$tmp" > "$f" rm -f "$tmp" ok " $f sorted by id (previous kept as $f.bak)" @@ -4607,7 +4704,10 @@ _vfy_manifest_ownership() { # gave man-pages (the first chapter-8 package) uid 10000 and binutils -- # built first, in chapter 5 -- a much higher one, which makes the uids # useless for reading the build order off the passwd file. - for man in $(_manifests_in_build_order) $(_manifests_in_build_order dirs); do + local _mans _nman=0 _ntot + _mans="$(_manifests_in_build_order) $(_manifests_in_build_order dirs)" + _ntot="$(printf '%s\n' $_mans | grep -c .)" + for man in $_mans; do [ -e "$man" ] || continue [ -s "$man" ] || continue case "$man" in *.dirs) kind=dir ;; *) kind=file ;; esac @@ -4642,6 +4742,30 @@ _vfy_manifest_ownership() { _vfy_n_fixed=$((_vfy_n_fixed + 1)) fi + # Which package, live. This pass walks every path in every manifest, + # and it used to print nothing until it had finished. A command that + # works for minutes in silence is indistinguishable from one that has + # hung -- and was reported as one. + _nman=$((_nman + 1)) + [ -t 1 ] && printf '\r [%d/%d] %-24s\033[K' "$_nman" "$_ntot" "$name" >&2 + + # Every owner in ONE stat call, not one call per path. + # + # `stat -c %U` per path is a fork per path, and `verify` walks every + # path in every manifest -- 66265 of them on a finished system. With + # is_never_claimed forking too that was ~130000 processes, which is + # where the minutes went. xargs batches them into a handful. + # A REAL tab in the format string. `stat -c '%U\t%n'` prints a + # literal backslash-t -- stat does not expand escapes -- so every line + # came back as one field, the map stayed empty, and the loop below fell + # through to the per-path stat it was written to avoid. It would have + # been correct and slow, which is the hardest kind of wrong to notice. + declare -A _own_of=() + while IFS="$_TAB" read -r _o _f; do + [ -n "$_f" ] && _own_of["$_f"]="$_o" + done < <(grep -v '^[[:space:]]*$' "$man" \ + | tr '\n' '\0' | xargs -0 -r stat -c "%U${_TAB}%n" 2>/dev/null) + while IFS= read -r p; do [ -n "$p" ] || continue p="$(strip_host_prefix "$p")" @@ -4653,7 +4777,8 @@ _vfy_manifest_ownership() { is_never_claimed "$p" && continue case "$p" in "${BUILD_ROOT:-${SNAP_ROOT%/}/build}"/*) continue ;; esac _vfy_n_checked=$((_vfy_n_checked + 1)) - cur="$(stat -c %U "$p" 2>/dev/null)" || continue + cur="${_own_of[$p]:-}" + [ -n "$cur" ] || cur="$(stat -c %U "$p" 2>/dev/null)" || continue [ "$cur" = "$owner" ] && continue # A shared install directory belongs to root:install, never to a # package -- even when the package's manifest records creating it. @@ -4676,6 +4801,8 @@ _vfy_manifest_ownership() { fi done < "$man" done + [ -t 1 ] && printf '\r\033[K' >&2 + return 0 } # --- rule 2: shared install directories belong to root:install ------------- # @@ -4715,8 +4842,12 @@ _vfy_install_dirs() { "$(stat -c '%U:%G' "$live" 2>/dev/null) $mode" \ "root:install $want_mode" if [ "$fix" = 1 ]; then + # COUNTED AS FIXED, so it must not fail in silence: the next + # run would find the same directory and report it again with + # no explanation of why the repair never took. set_install_dir_owner "$live" || true - real_chmod "$want_mode" "$live" 2>/dev/null || true + soft "could not set $live to $want_mode -- counted as fixed but it did not take" \ + -- real_chmod "$want_mode" "$live" _vfy_n_fixed=$((_vfy_n_fixed + 1)) fi fi @@ -4724,8 +4855,53 @@ _vfy_install_dirs() { } # --- rule 3: every file has an owner that exists here ---------------------- # +# The paths no ownership scan should walk, as find arguments. +# +# ONE list, because there were five and they had already diverged: the snapshot +# scans excluded the build scratch and the orphan scan did not, so when build +# trees moved from /build into each package user's home (1.7.6) the orphan scan +# started walking every unpacked source tree in the system. `lfs-helper verify` +# went from seconds to minutes with no output, and the sanity report filled up +# with tcl's own documentation: +# +# !! files with no owner (first 40): +# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm +# +# A tarball can contain any uid it likes. Unpacked sources are scratch -- they +# are not installed, nothing owns them, and asking who does has no answer. +scan_prune_paths() { + local r="${SNAP_ROOT%/}" + printf '%s\n' \ + "$r/dev" "$r/proc" "$r/sys" "$r/run" "$r/tmp" \ + "$r/sources" "${BUILD_ROOT:-$r/build}" "$STATE" + # every package's unpacked source tree, wherever the homes are + local d + while IFS= read -r d; do + [ -n "$d" ] || continue + printf '%s\n' "$d/*/$PKGUSR_BUILD_SUBDIR" + done < <(pkgusr_roots) +} + +# The same list as a find argument ARRAY, in SCAN_PRUNE. +# +# An array, not a string spliced through `eval`: the patterns contain `*`, and +# eval lets the shell expand them against the real filesystem before find ever +# sees them. `/usr/src/pkgusr/*/src/*` then becomes a handful of literal +# directory names instead of a wildcard, and the prune silently matches almost +# nothing -- which is exactly how it looked like it was working. +SCAN_PRUNE=() +scan_prune_set() { + local p + SCAN_PRUNE=() + while IFS= read -r p; do + [ -n "$p" ] || continue + SCAN_PRUNE+=( -not -path "$p" -not -path "$p/*" ) + done < <(scan_prune_paths) +} + _vfy_orphans() { local fix="$1" r="${SNAP_ROOT%/}" p n=0 + scan_prune_set # NOT before book 7.6 has written /etc/passwd. # # `find -nouser` asks whether a uid resolves to a name. With no passwd file @@ -4751,10 +4927,7 @@ _vfy_orphans() { real_chown -h lfs:lfs "$p" 2>/dev/null \ && _vfy_n_fixed=$((_vfy_n_fixed + 1)) fi - done < <(find "${r:-/}" -xdev -nouser \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - -not -path "$r/sources/*" 2>/dev/null) + done < <(find "${r:-/}" -xdev -nouser "${SCAN_PRUNE[@]}" 2>/dev/null) [ "$n" -gt 5 ] && say " ... and $((n - 5)) more with no owner" [ -n "$VERIFY_LOG" ] && say " all $n: grep ^no-owner $VERIFY_LOG" } @@ -4774,6 +4947,7 @@ _vfy_orphans() { # which package they belong to. _vfy_build_user_leftovers() { local fix="$1" r="${SNAP_ROOT%/}" p n=0 + scan_prune_set _user_db_ready || return 0 user_exists lfs || return 0 while IFS= read -r p; do @@ -4797,13 +4971,7 @@ _vfy_build_user_leftovers() { # the chroot and should be root's. Its CONTENTS are not -- scratch has no # owner worth arguing about. The state directory is under $SRCROOT, which # is excluded wholesale. - done < <(find "${r:-/}" -xdev -user lfs \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - -not -path "$r/tmp/*" \ - -not -path "$r/sources" -not -path "$r/sources/*" \ - -not -path "${BUILD_ROOT:-$r/build}/*" \ - -not -path "$STATE/*" \ + done < <(find "${r:-/}" -xdev -user lfs "${SCAN_PRUNE[@]}" \ -not -path "$SRCROOT/*" 2>/dev/null) [ "$n" -gt 5 ] && say " ... and $((n - 5)) more still owned by the build user" [ "$n" -gt 0 ] && [ -n "$VERIFY_LOG" ] \ @@ -4813,9 +4981,17 @@ _vfy_build_user_leftovers() { _vfy_make_lfs_user() { if have_shadow_tools; then - groupadd -g "$LFS_BUILD_UID" lfs 2>/dev/null || true - useradd -M -u "$LFS_BUILD_UID" -g lfs -s /bin/bash \ - -c "build user from outside the chroot" lfs 2>/dev/null || true + # Failing here is NORMAL on a second run -- the account exists. It + # is not normal on the first, and that case used to look identical. + if ! getent group lfs >/dev/null 2>&1; then + soft "could not create the 'lfs' group" \ + -- groupadd -g "$LFS_BUILD_UID" lfs + fi + if ! id lfs >/dev/null 2>&1; then + soft "could not create the 'lfs' build user" \ + -- useradd -M -u "$LFS_BUILD_UID" -g lfs -s /bin/bash \ + -c "build user from outside the chroot" lfs + fi else printf 'lfs:x:%s:\n' "$LFS_BUILD_UID" >> "$ETC/group" printf 'lfs:x:%s:%s:build user from outside the chroot:/:/bin/bash\n' \ @@ -4855,7 +5031,7 @@ _vfy_user_order() { if ! cmp -s "$f" "$tmp"; then _vfy_report order "$f" "build order" "id order" if [ "$fix" = 1 ]; then - cp -p "$f" "$f.bak" 2>/dev/null || true + soft "no backup of $f -- editing it without one" -- cp -p "$f" "$f.bak" cat "$tmp" > "$f" _vfy_n_fixed=$((_vfy_n_fixed + 1)) fi @@ -5156,6 +5332,8 @@ _vfy_unclaimed() { [ -n "$p" ] || continue is_never_claimed "$p" || printf '%s\n' "$p" >> "$_keep" done < "$found" + # benign: $_keep is a filtered copy of $found -- if the move fails the + # unfiltered list is still correct, just longer mv -f "$_keep" "$found" 2>/dev/null || true n="$(LC_ALL=C comm -23 "$found" "$list" 2>/dev/null | wc -l | tr -d ' ')" @@ -5199,6 +5377,7 @@ cmd_verify() { local problems=0 fixed=0 + detail "# checking for accounts that share an id ..." # ---- 1. two accounts sharing one id ----------------------------------- # # Ownership is a number: a collision makes files report as the wrong name # and a package unable to write its own files. @@ -5207,6 +5386,7 @@ cmd_verify() { warn " (verify cannot repair this: which name owns what is yours to decide)" fi + detail "# checking the shared install directories ..." # ---- 2. the shared install directories -------------------------------- # # root:install, group-writable. A staged install used to stamp the package # user's ownership onto these, handing /usr and /usr/bin to one package and @@ -5217,6 +5397,7 @@ cmd_verify() { # to be a second copy that did neither. _vfy_install_dirs "$fix" || problems=$((problems+1)) + detail "# checking every package owns the files it installed ..." # ---- 3. every manifest's files belong to its package ------------------- # # # A SECOND copy of _vfy_manifest_ownership used to live here, inline, and @@ -5246,9 +5427,11 @@ cmd_verify() { ok "package files: $checked path(s) owned by the right package" fi + detail "# looking for files no manifest claims ..." # ---- 3b. files no manifest claims ------------------------------------- # _vfy_unclaimed || problems=$((problems+1)) + detail "# looking for files with no owner ..." # ---- 4. files with no owner at all ------------------------------------ # if _has_orphaned_files; then warn "some files are owned by a uid with no passwd entry" @@ -5272,6 +5455,7 @@ cmd_verify() { ok "every file has a real owner" fi + detail "# checking homes, the build user and the uid order ..." # ---- 4b. the passes the dead copy had and this one did not ------------- # # homes owned by their own user, the build user's leftovers, and the uid # order. They were written, and reachable only from the definition bash @@ -5281,6 +5465,7 @@ cmd_verify() { _vfy_build_user_leftovers "$fix" || true _vfy_user_order "$fix" || true + detail "# looking for left-over staging trees ..." # ---- 5. staging trees left behind ------------------------------------- # local st n_stage=0 for st in "$STATE"/stage/*; do @@ -5760,19 +5945,12 @@ cmd_build_all() { warn " lfs-helper build-all" return 0 fi - # Seal immediately BEFORE the final step. By now every package is - # built, so the install directories no longer need to be writable by - # everyone -- and last-step runs after the seal, which is the correct - # order: it installs new files (allowed under the sticky bit) rather - # than replacing other packages' (which is what the seal prevents). - if [ "$s" = "last-step" ] && ! _install_dirs_are_sealed; then - say "" - say "Everything before the final step is built -- sealing the" - say "install directories now, so packages can no longer overwrite" - say "each other's files:" - say "" - _vfy_seal_install_dirs - fi + # The seal happens once, after the loop -- see the end of this + # function. There used to be a second one here, firing just before + # `last-step` because that step still INSTALLED software (wget) and had + # to do it under the sticky bit. `last-step` is gone; the final step + # is now `init-accounts`, which runs as root and installs nothing, so + # there is nothing left to seal early for. say "" say "===== [$i/$total] $s =====" @@ -5833,6 +6011,116 @@ cmd_build_all() { # cosmetic, so it is safe to do without asking. say "" _vfy_user_order 1 || true + + _warn_if_strip_was_asked_for + _warn_if_no_login + _say_what_is_yours_to_finish +} + +# The two things this tool deliberately does not do. +# +# The kernel and the bootloader are decisions about the whole MACHINE, not +# about a package, and getting either wrong costs you the system you are +# building on -- so they are yours, on purpose, and the build ends by saying +# so rather than leaving you to discover it after the reboot. +# +# rEFInd is opt-in for the same reason and stays that way: the machine booted +# in order to build LFS at all, so it already has a working bootloader, and +# adding a menu entry to it is not something a build should do because nobody +# said no. +_say_what_is_yours_to_finish() { + say "" + say "# Two things are yours to finish, on purpose:" + say "#" + say "# 1. THE KERNEL. Nothing here configures, builds or installs one." + say "# Book 10.3, from inside the chroot:" + say "# cd /sources && tar -xf linux-*.tar.xz && cd linux-*" + say "# make menuconfig && make && make modules_install" + say "# cp -iv arch/x86/boot/bzImage /boot/vmlinuz-lfs" + say "#" + say "# 2. THE BOOTLOADER. Nothing was written to any ESP, boot sector" + say "# or partition table, and your current bootloader is untouched." + say "# Add an entry for this system by hand, or let the tools add a" + say "# rEFInd entry beside your existing one:" + say "# lfs config bootloader refind" + say "# lfs config esp /boot/efi" + say "# lfs-helper build refind --force" + say "# That only ever ADDS to a mounted ESP: it never formats, never" + say "# writes a raw device, and never replaces what boots you today." +} + +# You asked for stripping. Nothing did it, and nothing will say so. +# +# `lfs build-system session` stores the answer and the prompt says NOT YET +# IMPLEMENTED -- but that was six hours ago, at the top of an interview, and +# the build otherwise finishes as though every answer had been acted on. A +# setting that is collected, carried into the chroot environment, and then +# silently ignored is worse than one that was never offered. +# +# Said at the END, next to the login warning, because that is where the things +# you still have to do yourself are collected. +_warn_if_strip_was_asked_for() { + [ "${LFS_STRIP:-0}" = 1 ] || return 0 + say "" + warn "# you asked for debug symbols to be stripped, and NOTHING DID IT." + warn "#" + warn "# Stripping rewrites every binary, and in this system a file's" + warn "# owner is the record of which package installed it -- so it has" + warn "# to run as that owner, package by package. That is not written" + warn "# yet, and doing it as root would hand the whole tree to root." + warn "#" + warn "# The system is complete and correct; it is only larger than you" + warn "# asked for. Book 8.85 also removes libtool .la files." + return 1 +} + +# Can anyone actually log into this system? +# +# THE LAST THING SAID, because it is the last thing that can still be fixed. +# A fresh LFS has root with no password. Depending on the login manager that is +# either "anyone can log in as root" or "nobody can log in at all", and you find +# out after the reboot -- when the chroot is gone and there is no way back in +# short of booting the host again and mounting the tree by hand. +# +# init-accounts asks for it, but it is skippable: a scripted run has no +# terminal, and an interactive one can be answered with a blank line. So this +# checks the RESULT rather than trusting that the step ran. +_warn_if_no_login() { + local root_ok=0 user_ok=0 u + if grep -qE '^root:[^:*!]' "$ETC/shadow" 2>/dev/null; then + root_ok=1 + fi + # any non-system account with a real password + while IFS=: read -r u _ uid _; do + case "$uid" in ''|*[!0-9]*) continue ;; esac + [ "$uid" -ge 1000 ] && [ "$uid" -lt 10000 ] || continue + grep -qE "^$u:[^:*!]" "$ETC/shadow" 2>/dev/null && { user_ok=1; break; } + done < "$ETC/passwd" + + if [ "$root_ok" = 1 ] || [ "$user_ok" = 1 ]; then + say "" + ok "# you can log in after rebooting:" + [ "$root_ok" = 1 ] && ok "# root has a password" + [ "$user_ok" = 1 ] && ok "# '$u' has a password" + return 0 + fi + + say "" + warn "==============================================================" + warn "!! NOBODY CAN LOG INTO THIS SYSTEM." + warn "" + warn " root has no password, and no other account has one either." + warn " Depending on the login manager that means either anyone can" + warn " log in as root, or nobody can log in at all -- and you would" + warn " find out after the reboot, with the chroot gone." + warn "" + warn " Fix it NOW, while you are still in here:" + warn " lfs-helper init-accounts --force" + warn "" + warn " or, from outside the chroot:" + warn " chroot /mnt/lfs /usr/bin/passwd root" + warn "==============================================================" + return 1 } # Are the install directories already sticky? Checking a handful of the ones @@ -6333,6 +6621,8 @@ Look around / tidy up: lfs-helper verify [--fix] does the tree match the manifests? lfs-helper pkgusr-info [<name>] where each package came from lfs-helper pkgusr-home <name> print an account's home directory + lfs-helper wrapper-dir print where the build wrappers live + lfs-helper make-wrappers --run (re)write the build wrappers lfs-helper owner-name <package> print the account name for a package lfs-helper manifests [<name>] what each package installed lfs-helper which-package <path> which package installed this file @@ -6380,12 +6670,8 @@ case "${1:-}" in clean-state) shift; cmd_clean_state "$@" ;; which-package) shift; cmd_which_package "$@" ;; grant-dir) shift; cmd_grant_dir "$@" ;; - verify) shift; cmd_verify "$@" ;; - install-as) shift; cmd_install_as "$@" ;; - pkgusr-info) shift; cmd_pkgusr_info "$@" ;; - pkgusr-home) shift; cmd_pkgusr_home "$@" ;; - owner-name) shift; cmd_owner_name "$@" ;; - clean-state) shift; cmd_clean_state "$@" ;; + make-wrappers) shift; cmd_make_wrappers "$@" ;; + wrapper-dir) shift; cmd_wrapper_dir "$@" ;; export-groups) shift; cmd_export_groups "$@" ;; import-groups) shift; cmd_import_groups "$@" ;; prune-locales) shift; cmd_prune_locales "$@" ;; @@ -6397,6 +6683,7 @@ case "${1:-}" in # itself the moment book 7.6 has created the user database; this is for # doing it by hand, or checking whether it has happened. establish-ownership) shift; cmd_establish_ownership "$@" ;; + check-login) shift; _warn_if_no_login ;; init-accounts) shift; cmd_init_accounts "$@" ;; # The repair commands people were told to run for years. verify is the # ONE implementation; these are aliases, not second copies -- a command diff --git a/packagemanager b/packagemanager index e7f2844..1a0e769 100644 --- a/packagemanager +++ b/packagemanager @@ -109,7 +109,7 @@ TMP_DIR = "/tmp/packagemanager" CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf") # Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure # changes (so stale plans are ignored); TOOL_VERSION is the program version. -TOOL_VERSION = "1.10.1" +TOOL_VERSION = "1.11.6" def _build_id(): @@ -3391,6 +3391,77 @@ def _pip_owner_of_python(): return "python" +# One module, installed the package-user way. Shared by `packagemanager pip` +# and `packagemanager setup`, because they want exactly the same thing and a +# second copy of it is how the two would drift apart. +def pip_install_module(mod, site, grp, find_links=None, upgrade=False, + no_deps=False, target=None): + """Install ONE Python module as its own package user. True on success. + + `target` is an explicit file to install (a wheel in the sources directory); + without it pip is given the module NAME and finds it however it can. + """ + # THE ACCOUNT NAME COMES FROM THE CHOKEPOINT. + # + # This read `_sanitise_user_name(mod)` and used the result for every call + # below. create_package_user normalises internally, so the account it made + # was `p_requests` -- and then usermod and su were handed the bare + # `requests`, which does not exist: + # usermod: user 'requests' does not exist + # The same species as `chown: invalid user: 'wget:wget'`: half the code + # asking for the name, half assuming it. + user = pkgusr_name(_sanitise_user_name(mod)) + printHeader(f"{mod} -> package user '{user}'") + + if not _user_exists(user): + if not create_package_user(user): + printError(f"could not create {user}") + return False + report_created_user(user) + + # THE COLLECTOR GROUP IS GRANTED THROUGH ONE DOOR. + # + # lfs-helper owns the rules: join the group the directory already carries, + # ask which group should own it when there is none, never create one for + # root, never hand a package's tree to the install group. This did its own + # groupadd/chgrp -R/chmod -R instead -- a third implementation, after + # lfs-helper's and packagemanager_install's, and the only one that never + # asked what the directory already carried. + if shutil.which("lfs-helper"): + _run(["lfs-helper", "grant-dir", site, user, "--run"]) + else: + if not _group_exists(grp): + printInfo(f"creating collector group {grp}") + _run([_cmd_path("groupadd"), grp]) + _run([_cmd_path("usermod"), "-a", "-G", grp, user]) + if _group_exists(grp): + _run(["chgrp", "-R", grp, site]) + _run(["chmod", "-R", "g+w", site]) + + # PIP_USER=0 / PYTHONNOUSERSITE=1: without these pip quietly falls back + # to ~/.local when site-packages is not writable, and then reports + # "Requirement already satisfied" from there ever after -- so the module + # is installed for exactly one user and importable by nobody else. + env = ["env", "PIP_USER=0", "PYTHONNOUSERSITE=1"] + pip_args = ["pip3", "install"] + if upgrade: + pip_args.append("--upgrade") + if no_deps: + pip_args.append("--no-deps") + if find_links: + pip_args += ["--no-index", "--find-links", find_links] + pip_args.append(target or mod) + + cmd = ["su", "-s", "/bin/bash", user, "-c", + " ".join(shlex.quote(x) for x in env + pip_args)] + printInfo(" ".join(pip_args) + f" (as {user})") + if _run(cmd, show_output=True): + printSuccess(f"{mod} installed as package-user {user}") + return True + printError(f"{mod} failed -- see the output above") + return False + + def cmd_pip(args): """Install or update a Python module AS A PACKAGE USER. @@ -3404,8 +3475,7 @@ def cmd_pip(args): if not site: printError("no system site-packages directory found -- is Python installed?") return 1 - owner = _pip_owner_of_python() - grp = collector_prefix() + owner + grp = collector_prefix() + _pip_owner_of_python() modules = args.modules if not modules: @@ -3413,43 +3483,10 @@ def cmd_pip(args): return 1 for mod in modules: - user = _sanitise_user_name(mod) - printHeader(f"{mod} -> package user '{user}'") - - if not _user_exists(user): - if not create_package_user(user): - printError(f"could not create {user}") - continue - report_created_user(user) - - # the collector group is what lets this user write into Python's tree - if not _group_exists(grp): - printInfo(f"creating collector group {grp}") - _run([_cmd_path("groupadd"), grp]) - _run(["usermod", "-a", "-G", grp, user]) - if _group_exists(grp): - _run(["chgrp", "-R", grp, site]) - _run(["chmod", "-R", "g+w", site]) - - # PIP_USER=0 / PYTHONNOUSERSITE=1: without these pip quietly falls back - # to ~/.local when site-packages is not writable, and then reports - # "Requirement already satisfied" from there ever after -- so the module - # is installed for exactly one user and importable by nobody else. - env = ["env", "PIP_USER=0", "PYTHONNOUSERSITE=1"] - pip_args = ["pip3", "install"] - if args.upgrade: - pip_args.append("--upgrade") - if args.find_links: - pip_args += ["--no-index", "--find-links", args.find_links] - pip_args.append(mod) - - cmd = ["su", "-s", "/bin/bash", user, "-c", - " ".join(shlex.quote(x) for x in env + pip_args)] - printInfo(" ".join(pip_args) + f" (as {user})") - if _run(cmd, show_output=True): - printSuccess(f"{mod} installed as package-user {user}") - else: - printError(f"{mod} failed -- see the output above") + pip_install_module(mod, site, grp, + find_links=args.find_links, + upgrade=args.upgrade, + no_deps=getattr(args, "no_deps", False)) return 0 def cmd_add_user(args): @@ -3806,6 +3843,215 @@ def cmd_user_list(args): print(f"\n{len(names)} user(s).") +_SETUP_HELP = """\ +Bootstrap the tooling on a freshly booted system. + +LFS ships no download tool, so a new system cannot fetch even the sources for +its own next package. And `lfs` and `blfs` need `requests` and +`beautifulsoup4` to read the books, so without them the tools cannot run on the +system they just built. + + packagemanager setup --sources what it needs downloaded + packagemanager setup --run install it + +The tarballs must be on disk BEFORE the reboot, while the host still has a +network -- `lfs build-system get-sources` asks this command for the list. +""" + +# What a freshly booted system needs before it can fetch anything itself. +# +# wget comes from the BLFS BOOK: the book already knows its URL and keeps it +# current, and a second list of download links is a second thing to go stale. +_SETUP_BLFS_PACKAGES = ["wget"] + +# The Python modules `lfs` and `blfs` need to parse the books. +# +# WHEELS, not source tarballs, and not from the book. A modern sdist needs its +# build backend -- beautifulsoup4 wants hatchling, requests wants setuptools -- +# and with no package index reachable pip cannot fetch one: +# BackendUnavailable: Cannot import 'hatchling.build' +# A wheel is already built, so pip only unpacks it. All of these are pure +# Python, so the "any" wheel works on every architecture. +# +# Dependencies first: setup installs with --no-deps, so pip will not pull them. +_SETUP_WHEELS = [ + "https://files.pythonhosted.org/packages/py3/u/urllib3/urllib3-2.2.3-py3-none-any.whl", + "https://files.pythonhosted.org/packages/py3/c/charset_normalizer/charset_normalizer-3.4.0-py3-none-any.whl", + "https://files.pythonhosted.org/packages/py3/i/idna/idna-3.10-py3-none-any.whl", + "https://files.pythonhosted.org/packages/py3/c/certifi/certifi-2024.8.30-py3-none-any.whl", + "https://files.pythonhosted.org/packages/py3/s/soupsieve/soupsieve-2.6-py3-none-any.whl", + "https://files.pythonhosted.org/packages/py3/r/requests/requests-2.32.3-py3-none-any.whl", + "https://files.pythonhosted.org/packages/py3/b/beautifulsoup4/beautifulsoup4-4.12.3-py3-none-any.whl", +] + + +def setup_sources(): + """Every URL this command needs, resolved once.""" + import subprocess + urls = [] + for pkg in _SETUP_BLFS_PACKAGES: + try: + out = subprocess.run(["blfs", "sources", pkg], + capture_output=True, text=True, timeout=120) + except (OSError, subprocess.SubprocessError): + printWarning("could not run `blfs sources %s` -- is blfs installed?" + % pkg) + continue + got = [l.strip() for l in (out.stdout or "").splitlines() + if l.strip().startswith(("http://", "https://", "ftp://"))] + if not got: + printWarning("the BLFS book gave no download for '%s' " + "(try: blfs fetch)" % pkg) + continue + urls += got + return urls + _SETUP_WHEELS + + +# Where the tarballs and wheels are, on the booted system. +# +# Book 3.1's /sources, kept: `lfs build-system get-sources` put them there +# while the host still had a network, and that is the last moment there was +# one. Overridable, because a tree can be mounted anywhere. +def setup_sources_dir(): + return os.environ.get("LFS_SOURCES_DIR") or "/sources" + + +def _wheel_dist(url): + """(distribution, version, filename) from a wheel URL. + + PEP 427 fixes the filename: {dist}-{version}-{python}-{abi}-{platform}.whl, + so the first two fields are exactly what pip records in site-packages as + <dist>-<version>.dist-info. That is what makes the check below an + identity, not a guess. + """ + fn = url.rsplit("/", 1)[-1] + parts = fn[:-4].split("-") if fn.endswith(".whl") else fn.split("-") + if len(parts) < 2: + return fn, "", fn + return parts[0], parts[1], fn + + +def _wheel_installed(site, dist, version): + return bool(site) and os.path.isdir( + os.path.join(site, "%s-%s.dist-info" % (dist, version))) + + +def _setup_state(): + """What is already true, so setup can be run twice and do nothing twice.""" + site = _pip_site_dir() + wheels = [] + for url in _SETUP_WHEELS: + dist, ver, fn = _wheel_dist(url) + wheels.append((dist, ver, fn, url, _wheel_installed(site, dist, ver))) + return { + "site": site, + "wget": shutil.which("wget"), + "wheels": wheels, + } + + +def _setup_install_wget(yes): + """Install wget through the ordinary BLFS path. + + Invoked as a command rather than called as a function: `packagemanager + install wget --run` is a build plan, a script, a package user and a + manifest, and there is no version of "just the wget part" of that which is + not a second implementation of it. + """ + cmd = [sys.executable, os.path.abspath(__file__), "install", "wget", "--run"] + if yes: + cmd.append("--yes") + printInfo(" $ " + " ".join(cmd[2:])) + try: + return subprocess.run(cmd).returncode == 0 + except (OSError, subprocess.SubprocessError) as e: + printError("could not install wget: %s" % e) + return False + + +def cmd_setup(args): + if args.sources: + for u in setup_sources(): + print(u) + return + + st = _setup_state() + srcdir = getattr(args, "sources_dir", None) or setup_sources_dir() + force = getattr(args, "force", False) + todo_wget = not st["wget"] or force + todo_wheels = [w for w in st["wheels"] if force or not w[4]] + + printHeader("packagemanager setup") + print() + print(" site-packages : %s" % (st["site"] or "NOT FOUND -- is Python installed?")) + print(" sources : %s" % srcdir) + print() + print(" wget : %s" % ("installed at %s" % st["wget"] if st["wget"] + else "MISSING -- will build from the BLFS book")) + for dist, ver, fn, url, have in st["wheels"]: + print(" %-13s %s" % (dist, "%s installed" % ver if have + else "%s to install" % ver)) + + if not st["site"]: + printError("\nNo system site-packages directory: nothing can be installed.") + sys.exit(1) + + if not todo_wget and not todo_wheels: + print() + printSuccess("Nothing to do -- wget and every module are already installed.") + printInfo("Re-do it anyway with: packagemanager setup --run --force") + return + + if not args.run: + print() + printInfo("Dry run -- to apply, re-run with --run.") + return + + # The wheels are already on disk and pip must not reach for an index: this + # system has no CA certificates yet, so anything that tries is a hang, not + # an error. + missing = [fn for _d, _v, fn, _u, _h in todo_wheels + if not os.path.isfile(os.path.join(srcdir, fn))] + if missing: + printError("\nThese wheels are not in %s:" % srcdir) + for fn in missing: + print(" %s" % fn) + printWarning("They had to be downloaded before the reboot:") + printWarning(" lfs build-system get-sources --run (on the host)") + sys.exit(1) + + failed = [] + if todo_wget: + printHeader("wget") + if not _setup_install_wget(getattr(args, "yes", False)): + failed.append("wget") + else: + # A system with no certificates cannot verify anything it fetches. + # This is the same call the tool makes on every invocation; doing it + # here means the very next download works. + ensure_wget_workaround() + + if todo_wheels: + grp = collector_prefix() + _pip_owner_of_python() + # IN ORDER, with --no-deps: pip cannot reach an index to resolve + # anything, so the order in _SETUP_WHEELS is the dependency order and + # each module must already have what it imports. + for dist, ver, fn, url, _have in todo_wheels: + if not pip_install_module(dist, st["site"], grp, + find_links=srcdir, no_deps=True, + target=os.path.join(srcdir, fn)): + failed.append(dist) + + print() + if failed: + printError("setup finished with %d failure(s): %s" + % (len(failed), ", ".join(failed))) + printInfo("Fix them and run it again -- it skips what is already done.") + sys.exit(1) + printSuccess("setup complete.") + printInfo("Check it: python3 -c 'import requests, bs4' && wget --version | head -1") + + def cmd_user_create(args): u = _user_prefixed(args.name) if _user_exists(u): @@ -5502,6 +5748,24 @@ def main(): help="wait for it instead of running as a disowned task") p.set_defaults(func=cmd_reload_pkg_list) + p = sub.add_parser( + "setup", + help="bootstrap the tooling on a freshly booted system", + description=_SETUP_HELP, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--sources", action="store_true", + help="print the download URLs this needs, one per line, " + "and exit (used by `lfs build-system get-sources`)") + p.add_argument("--run", action="store_true", help="apply (default: dry run)") + p.add_argument("--force", action="store_true", + help="re-install everything, including what is already there") + p.add_argument("--sources-dir", dest="sources_dir", + help="where the downloaded tarballs and wheels are " + "(default: /sources)") + p.add_argument("--yes", action="store_true", + help="answer yes to the questions the wget build asks") + p.set_defaults(func=cmd_setup) + p = sub.add_parser("pip", help="install or update a Python module as a " "package user (not as root)") p.add_argument("action", choices=["install", "update", "upgrade"], @@ -5510,9 +5774,13 @@ def main(): p.add_argument("--find-links", metavar="DIR", help="install from local wheels in DIR instead of the network " "(there is no network inside the chroot)") + p.add_argument("--no-deps", action="store_true", + help="do not let pip pull dependencies -- name them yourself, " + "in order (what `setup` does, offline)") p.set_defaults(func=lambda a: cmd_pip( argparse.Namespace(modules=a.modules, upgrade=(a.action in ("update", "upgrade")), + no_deps=a.no_deps, find_links=a.find_links))) p = sub.add_parser("add-user", diff --git a/packagemanager_install b/packagemanager_install index fd513ad..87513e4 100644 --- a/packagemanager_install +++ b/packagemanager_install @@ -14,7 +14,7 @@ # packagemanager_install package_short_name path_to_install_file # -PACKAGEMANAGER_INSTALL_VERSION="1.10.1" +PACKAGEMANAGER_INSTALL_VERSION="1.11.6" # A fingerprint of this file's own contents, the same as the other four tools. # @@ -64,6 +64,15 @@ pkgusr_home_for() { case "$n" in "") printf '%s' "$PKGUSR_ROOT"; return ;; esac + # ASK, if there is anything to ask. lfs-helper owns this rule and the + # layout it depends on is configurable; a second copy of it here is right + # only until someone changes the layout. The rest of this function is the + # fallback for a system with no lfs-helper. + if command -v lfs-helper >/dev/null 2>&1; then + local _h + _h="$(lfs-helper pkgusr-home "$n" 2>/dev/null)" + case "$_h" in /*) printf '%s' "$_h"; return ;; esac + fi if [ -n "$PKGUSR_PREFIX" ]; then case "$n" in "$PKGUSR_PREFIX"*) ;; *) n="${PKGUSR_PREFIX}${n}" ;; esac fi @@ -99,8 +108,30 @@ home_user_dir="/home/${home_user}" # install -d on an existing directory -> succeed, change nothing # chmod on a directory we do not own -> skip, keep going # chown/chgrp on anything -> skip (a package user may not) -# Anything else is passed straight through. This is the same approach -# lfs-helper uses during the chroot build. +# Anything else is passed straight through. +# +# ONE SET OF WRAPPERS, NOT TWO. lfs-helper writes these to /usr/lib/pkgusr, +# where they outlive the build and where the package-user profile already +# points; this script wrote its OWN copy to a fresh /tmp directory on every +# run. Two implementations of "what may a package user do", free to disagree +# about it -- the species of bug that costs this project the most. Ask for +# them when lfs-helper is on the system, which it is on anything these tools +# built; the copy below is the fallback for a system without it. +_wrapper_dir_from_lfs_helper() { + command -v lfs-helper >/dev/null 2>&1 || return 1 + local d + d="$(lfs-helper wrapper-dir 2>/dev/null)" || return 1 + [ -n "$d" ] || return 1 + # Missing or incomplete -- write them. Idempotent, so this also repairs a + # tree whose wrappers predate a fix to them. + if [ ! -x "$d/install" ] || [ ! -x "$d/chown" ]; then + lfs-helper make-wrappers --run >/dev/null 2>&1 || return 1 + fi + [ -x "$d/install" ] || return 1 + printf '%s' "$d" +} + +# The temporary copy: only for a system with no lfs-helper. _make_build_wrappers() { local d="$1" mkdir -p "$d" || return 1 @@ -571,14 +602,16 @@ installPkg() { chown $user_name: $install_user_dir/install_* chown -R $user_name: $install_user_dir/log/ # pipefail so tee doesn't hide a failing build/install; capture the real code - _wrapdir="/tmp/pkgusr-wrappers.$$" - _make_build_wrappers "$_wrapdir" || _wrapdir="" - chmod 755 "$_wrapdir" 2>/dev/null || true + _wrapdir="$(_wrapper_dir_from_lfs_helper)" && _wraptmp=0 || { + _wrapdir="/tmp/pkgusr-wrappers.$$"; _wraptmp=1 + _make_build_wrappers "$_wrapdir" || _wrapdir="" + chmod 755 "$_wrapdir" 2>/dev/null || true + } su - -c "set -o pipefail; \ ${_wrapdir:+PATH='$_wrapdir':\$PATH; export PATH; } \ bash ~/install_$user_name $parameter 2>&1 | tee ~/log/packagemanager_install-$date_log.log" $user_name install_rc=${PIPESTATUS[0]} - [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir" + [ "${_wraptmp:-0}" = 1 ] && [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir" if echo "$user_name" | grep "^u_"; then gpasswd -d $user_name install # remove install group if nimgnu_group @@ -787,14 +820,16 @@ diagnose_install_failure() { auto_grant_from_log "$_log" "$user_name" || break _round=$((_round + 1)) echo -e "\033[0;33m# retrying $user_name ($parameter), round $_round ...\033[0m" - _wrapdir="/tmp/pkgusr-wrappers.$$" - _make_build_wrappers "$_wrapdir" || _wrapdir="" - chmod 755 "$_wrapdir" 2>/dev/null || true + _wrapdir="$(_wrapper_dir_from_lfs_helper)" && _wraptmp=0 || { + _wrapdir="/tmp/pkgusr-wrappers.$$"; _wraptmp=1 + _make_build_wrappers "$_wrapdir" || _wrapdir="" + chmod 755 "$_wrapdir" 2>/dev/null || true + } su - -c "set -o pipefail; \ ${_wrapdir:+PATH='$_wrapdir':\$PATH; export PATH; } \ bash ~/install_$user_name $parameter 2>&1 | tee ~/log/packagemanager_install-$date_log.log" $user_name install_rc=${PIPESTATUS[0]} - [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir" + [ "${_wraptmp:-0}" = 1 ] && [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir" done if [ "$install_rc" != "0" ]; then diff --git a/test_lfs_crosschain.sh b/test_lfs_crosschain.sh index baf9205..d13d604 100644 --- a/test_lfs_crosschain.sh +++ b/test_lfs_crosschain.sh @@ -1526,40 +1526,162 @@ fi # ---- what the built system needs before it can fetch anything --------------- # # LFS ships no download tool, so a freshly booted system cannot get the sources -# for its own next package. The tarball has to be on disk before the chroot is -# sealed, and get-sources is the last moment there is still a network. +# for its own next package. The tarballs have to be on disk before the chroot +# is sealed, and get-sources is the last moment there is still a network. # -# Resolved from the BLFS BOOK, not a hardcoded URL: the book already knows where -# wget comes from and keeps it current, and a second list of links is a second -# thing to go stale. +# The LIST belongs to `packagemanager setup`, which is what installs them. A +# copy in lfs would be a second thing to go stale -- the first version of this +# did exactly that, with a `bootstrap_packages` config key in lfs resolving wget +# separately from the tool that installs it. python3 - "$LFS_TOOL" <<'PYBOOT' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +import re, sys src = open(sys.argv[1]).read() problems = [] -for fn in ("bootstrap_packages", "bootstrap_sources"): - if not hasattr(lfs, fn): - problems.append("there is no %s" % fn) -if hasattr(lfs, "bootstrap_packages"): - lfs.load_config = lambda: {} - if lfs.bootstrap_packages() != ["wget"]: - problems.append("wget is not the default bootstrap package") - lfs.load_config = lambda: {"bootstrap_packages": ""} - if lfs.bootstrap_packages(): - problems.append("bootstrap packages cannot be turned off") - lfs.load_config = lambda: {"bootstrap_packages": "wget curl"} - if lfs.bootstrap_packages() != ["wget", "curl"]: - problems.append("more than one bootstrap package is not accepted") -if "blfs" not in src or '"sources"' not in src: - problems.append("the URLs are not resolved from the BLFS book") +if not re.search(r"^def bootstrap_sources\(", src, re.M): + problems.append("there is no bootstrap_sources") +m = re.search(r"^def bootstrap_sources\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + if "packagemanager" not in body or "--sources" not in body: + problems.append("lfs does not ask packagemanager what it needs") + # lfs must not keep its own copy of the list. CODE only -- the docstring + # explains why the list lives elsewhere, and naming it there is the point. + code = "\n".join("" if l.lstrip().startswith("#") else l + for l in body.splitlines()) + code = "".join(code.split('"""')[::2]) + if "wget" in code: + problems.append("lfs still names the packages itself") +if "bootstrap_packages" in src.replace("`bootstrap_packages`", ""): + problems.append("the old bootstrap_packages config key is still here") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS the bootstrap tarball is resolved from the book, not hardcoded") +print(" PASS lfs asks packagemanager what the new system needs downloaded") PYBOOT _count_rc $? +# and packagemanager must answer, without needing a booted system +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pm_src" ]; then + _p="" + _out="$(python3 "$pm_src" setup --sources 2>/dev/null)" + _n="$(printf '%s\n' "$_out" | grep -c '^https\?://')" + [ "${_n:-0}" -ge 7 ] \ + || _p="$_p;setup --sources listed only $_n urls (expected the wheels at least)" + # WHEELS, not sdists: a modern sdist needs its build backend and pip cannot + # fetch one with no index -- BackendUnavailable: Cannot import hatchling.build + printf '%s\n' "$_out" | grep -q 'requests-.*\.whl' \ + || _p="$_p;requests is not taken as a wheel" + printf '%s\n' "$_out" | grep -q 'beautifulsoup4-.*\.whl' \ + || _p="$_p;beautifulsoup4 is not taken as a wheel" + # and its dependencies, because it installs with --no-deps + for _d in urllib3 charset_normalizer idna certifi soupsieve; do + printf '%s\n' "$_out" | grep -q "$_d" \ + || _p="$_p;$_d is missing, and --no-deps will not pull it in" + done + # wget comes from the BOOK, not a hardcoded URL + grep -q '_SETUP_BLFS_PACKAGES' "$pm_src" \ + || _p="$_p;wget is not resolved from the BLFS book" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "packagemanager setup declares what it needs, wheels and all" + fi +fi + +# ---- setup INSTALLS what it declares (1.11.2) ------------------------------ # +# `--sources` worked and `--run` said "not implemented". Everything below is a +# guarantee the old last_build_step.sh held and dropped when it was removed. +if [ -f "$pm_src" ]; then +python3 - "$pm_src" <<'PYSETUP' +import sys, re, importlib.machinery as m +src = open(sys.argv[1]).read() +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +problems = [] + +def body(fn): + mm = re.search(r"^def %s\(.*?(?=\n\ndef |\n\n# )" % fn, src, re.S | re.M) + return mm.group(0) if mm else "" + +# ONE implementation. `pip` and `setup` want the same thing -- a module +# installed as its own package user -- and a second copy of it is how the two +# would drift apart, which is the species of bug this project keeps having. +if not re.search(r"^def pip_install_module\(", src, re.M): + problems.append("there is no shared pip_install_module") +if "pip_install_module" not in body("cmd_pip"): + problems.append("cmd_pip does not use the shared implementation") +if "pip_install_module" not in body("cmd_setup"): + problems.append("setup does not use the shared implementation") + +# THE ACCOUNT NAME IS ASKED FOR, NEVER ASSEMBLED. +# create_package_user normalises internally, so an unprefixed name creates +# `p_requests` and then hands `requests` to usermod and su: +# usermod: user 'requests' does not exist +# Same species as `chown: invalid user: 'wget:wget'` at step 104 of 105. +b = body("pip_install_module") +if not re.search(r"pkgusr_name\(\s*_sanitise_user_name", b): + problems.append("the pip account name skips the naming chokepoint") + +# offline, and in order. pip cannot reach an index on a fresh system: no CA +# certificates, so a lookup is a hang rather than an error. +sb = body("cmd_setup") +if "no_deps=True" not in sb: + problems.append("setup lets pip resolve dependencies it cannot fetch") +if "find_links" not in sb: + problems.append("setup does not install from the local wheels") +if "--no-index" not in b: + problems.append("pip is still allowed to reach for an index") + +# wget comes from the ordinary install path, not a private copy of it +if "install" not in sb or "wget" not in sb: + problems.append("setup no longer installs wget") +if re.search(r"^def _setup_install_wget\(", src, re.M): + wb = body("_setup_install_wget") + if '"install"' not in wb: + problems.append("wget is installed by some path other than `install`") + +# IDEMPOTENT. The old final step ran exactly once, at the end of a six-hour +# build; this one has to be safe to run again after fixing whatever failed. +if "dist-info" not in src: + problems.append("setup cannot tell what is already installed") +d, v, fn = pm._wheel_dist( + "https://x/packages/py3/c/charset_normalizer/charset_normalizer-3.4.0-py3-none-any.whl") +if (d, v) != ("charset_normalizer", "3.4.0"): + problems.append("the wheel filename is parsed as %r %r" % (d, v)) +if fn != "charset_normalizer-3.4.0-py3-none-any.whl": + problems.append("the wheel filename is not recovered from the url") + +if problems: + for x in problems: + print(" FAIL %s" % x) + sys.exit(1) +print(" PASS setup installs wget and the wheels, offline, as package users") +PYSETUP +_count_rc $? + +# It must REFUSE rather than half-run when the wheels were never downloaded. +# There is no network on a fresh system, so "try it and see" is a hang. +_sbx="$T/setup-empty"; mkdir -p "$_sbx" +_o="$(LFS_SOURCES_DIR="$_sbx" python3 "$pm_src" setup --run --yes 2>&1)"; _rc=$? +_p="" +[ "$_rc" -ne 0 ] || _p="$_p;setup --run succeeded with no wheels on disk" +printf '%s\n' "$_o" | grep -q 'get-sources' \ + || _p="$_p;it does not say where the wheels were supposed to come from" +printf '%s\n' "$_o" | grep -q 'requests-.*\.whl' \ + || _p="$_p;it does not name the files it could not find" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "setup stops when the wheels are missing, and says which" +fi +fi + + # the dry run must print EVERY url, not a sample grep -q "print the WHOLE list" "$LFS_TOOL" \ @@ -3553,6 +3675,255 @@ PY60 else bad "the local copy runs even when lfs-helper is available" fi + + # ---- one set of wrappers, not two (1.11.3) ------------------------------ # + # The wrappers ARE the rule about what a package user may do: chown skipped, + # chgrp skipped, `install -d` on an existing directory allowed to succeed. + # lfs-helper writes them to /usr/lib/pkgusr, where they outlive the build and + # where the package-user profile already points. This script wrote its own + # 480-line copy into a fresh /tmp directory on EVERY run: a second answer to + # the same question, and the profile pointed at neither of them. + _p="" + grep -q '^_wrapper_dir_from_lfs_helper()' "$pmi" \ + || _p="$_p;there is no way to ask lfs-helper for the wrappers" + grep -q 'lfs-helper wrapper-dir' "$pmi" \ + || _p="$_p;the wrapper directory is assumed rather than asked for" + # every call site must go through the resolver first + _n_ask="$(grep -c '_wrapper_dir_from_lfs_helper' "$pmi")" + _n_own="$(grep -c '_make_build_wrappers "\$_wrapdir"' "$pmi")" + [ "${_n_ask:-0}" -gt "${_n_own:-0}" ] \ + || _p="$_p;a call site still builds its own wrappers without asking" + # and the SHARED directory must never be deleted afterwards -- that would + # take the wrappers away from every package user on the system + grep -q '_wraptmp' "$pmi" \ + || _p="$_p;the cleanup does not distinguish the shared dir from a temporary one" + if grep -q 'rm -rf "$_wrapdir"' "$pmi" && ! grep -q '_wraptmp.*=.*1.*rm -rf "$_wrapdir"' "$pmi"; then + _p="$_p;the wrapper directory is removed unconditionally" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both tools use one set of wrappers, and only a temporary one is deleted" + fi + + # the account home is asked for, not re-derived from the layout + _hf="$(sed -n '/^pkgusr_home_for() {/,/^}/p' "$pmi")" + case "$_hf" in + *"lfs-helper pkgusr-home"*) ok "the account home comes from the chokepoint" ;; + *) bad "packagemanager_install re-derives where an account lives" ;; + esac +fi + +# ---- the kernel and the bootloader are yours (1.11.6) --------------------- # +# Both are decisions about the whole MACHINE, and getting either wrong costs +# the system you are building on. The build must not do them by default, and +# must not let you find that out after the reboot. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + grep -q '_say_what_is_yours_to_finish' "$_h" \ + || _p="$_p;the build never says the kernel and bootloader are yours" + _sf="$(sed -n '/^_say_what_is_yours_to_finish() {/,/^}/p' "$_h")" + printf '%s' "$_sf" | grep -qi 'kernel' \ + || _p="$_p;the closing note does not mention the kernel" + printf '%s' "$_sf" | grep -qi 'refind' \ + || _p="$_p;the closing note does not say how to add a bootloader entry" + # it is a note, not an action: nothing in it may write anything + printf '%s' "$_sf" | grep -qE '^\s*(cp|mv|rm|dd|mkfs|chmod|chown|install) ' \ + && _p="$_p;the closing note runs commands instead of printing them" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build ends by saying the kernel and bootloader are yours" + fi +fi + +# and no bootloader is set up unless it was asked for, by name +python3 - "$LFS_TOOL" <<'PYBL' +import sys, re, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +# the default must be "none" with NO config at all -- not "none" because some +# other key happened to be unset +lfs.load_config = lambda: {} +if lfs.bootloader() != "none": + problems.append("the default bootloader is %r, not 'none'" % lfs.bootloader()) +lfs.load_config = lambda: {"bootloader": ""} +if lfs.bootloader() != "none": + problems.append("an empty bootloader setting does not mean 'none'") +# and the prompt must say the machine keeps booting as it does now +src = open(sys.argv[1]).read() +m2 = re.search(r'\("bootloader",.*?\),\n\s*\("esp"', src, re.S) +if m2 and "DEFAULT" not in m2.group(0): + problems.append("the prompt does not say that doing nothing is the default") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS no bootloader is touched unless it was asked for by name") +PYBL +_count_rc $? + +# ---- $LFS may never be the running system (1.11.5) ------------------------ # +# Every path this tool writes is $LFS/something. With $LFS set to the host -- +# a config holding "/", an `export LFS=/` in the wrong shell, a typo in the +# interview -- `restart --run` deletes the top-level directories of the machine +# you are sitting on, and _verify_lfs_ownership runs `chown -R lfs /usr`. +# The chroot bind mounts were guarded; the tree itself was not. +python3 - "$LFS_TOOL" <<'PYHOST' +import sys, os, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] + +for fn in ("_is_host_path", "_refuse_host_tree"): + if not hasattr(lfs, fn): + problems.append("there is no %s" % fn) +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) + +# the host, by every spelling +for p in ("/", "/usr", "/etc", "/boot", "/var", "/home", "/bin", "/lib"): + if not lfs._is_host_path(p): + problems.append("%s is not recognised as the running system" % p) + +# by another name: a symlink to / +d = tempfile.mkdtemp() +link = os.path.join(d, "root-link") +try: + os.symlink("/", link) + if not lfs._is_host_path(link): + problems.append("a symlink to / is not recognised") +except OSError: + pass + +# a path INSIDE a system tree is the system too, however deep +for p in ("/usr/src/lfs", "/etc/lfs", "/var/tmp/lfs"): + if not lfs._is_host_path(p): + problems.append("%s is not recognised as part of the system" % p) + +# ...and a real build tree must still be allowed, even though it has the same +# layout. A check that refuses /mnt/lfs/usr refuses every build there is. +tree = os.path.join(d, "lfs") +os.makedirs(os.path.join(tree, "usr"), exist_ok=True) +os.makedirs(os.path.join(tree, "etc"), exist_ok=True) +for p in (tree, "/mnt/lfs", "/home/someone/lfs", "/media/disk/lfs"): + if lfs._is_host_path(p): + problems.append("a legitimate build tree is refused: %s" % p) + +# ONE rule, not two: the interview and the hard stop must agree, or a path +# refused by one is accepted by the other. +src = open(sys.argv[1]).read() +import re +body = re.search(r"def _refuse_host_tree\(lfs\):.*?\n\n\ndef ", src, re.S) +if body and "_is_host_path" not in body.group(0): + problems.append("the hard stop keeps its own copy of the rule") +if "_is_host_path" not in re.search(r"for k, desc, default in _SESSION_KEYS:.*?\n\n", src, re.S).group(0): + problems.append("the interview accepts a mount point the tools will refuse") + +# the destructive paths must go through it +for fn in ("require_lfs_for_root", "_verify_lfs_ownership"): + b = re.search(r"def %s\(.*?(?=\n\ndef )" % fn, src, re.S) + if b and "_refuse_host_tree" not in b.group(0): + problems.append("%s does not check the tree is not the host" % fn) + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS $LFS is refused when it is the running system, by any spelling") +PYHOST +_count_rc $? + +# ---- an answer that is collected must be acted on, or said (1.11.4) ------- # +# `strip` is asked for during the interview and nothing does it. The prompt +# says NOT YET IMPLEMENTED, but that was six hours and a whole build ago, and +# the build otherwise ends as though every answer had been acted on. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + grep -q '_warn_if_strip_was_asked_for' "$_h" \ + || _p="$_p;nothing tells you the strip answer was ignored" + # it must READ the answer, not assume it + _sf="$(sed -n '/^_warn_if_strip_was_asked_for() {/,/^}/p' "$_h")" + printf '%s' "$_sf" | grep -q 'LFS_STRIP' \ + || _p="$_p;the strip notice does not read LFS_STRIP" + # and only when it was actually asked for + printf '%s' "$_sf" | grep -q 'return 0' \ + || _p="$_p;the strip notice fires even when nobody asked for it" + # said at the END, with the other things you still have to do + _i1="$(grep -n '_warn_if_strip_was_asked_for$' "$_h" | tail -1 | cut -d: -f1)" + _i2="$(grep -n '^ _warn_if_no_login$' "$_h" | tail -1 | cut -d: -f1)" + [ -n "$_i1" ] && [ -n "$_i2" ] \ + || _p="$_p;the notice is not part of the build's last word" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a build that ignored your strip answer says so at the end" + fi +fi + +# ---- a suppressed failure is a bug waiting to be found (1.11.4) ----------- # +# `cmd 2>/dev/null || true` has cost this project more debugging than anything +# else: a tree came out with every package home root:root 755 and not one line +# said why. The pattern is not banned -- mkdir -p on a directory that exists is +# fine -- but a failure that CHANGES WHAT THE SYSTEM LOOKS LIKE must be said. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + grep -q '^soft() {' "$_h" || _p="$_p;there is no way to report a non-fatal failure" + # the ownership and permission calls are the ones that matter + _leak="$(grep -nE '(real_chmod|real_chown|real_chgrp|chown|chgrp) .*2>/dev/null \|\| true' "$_h" \ + | grep -v 'SUPPRESSED DELIBERATELY' | wc -l)" + # ANY comment above is not enough -- the surrounding prose explains what the + # call does, not why its failure may be thrown away, and a reintroduced + # suppression inherits whatever comment happened to be there. The line + # immediately above must say which of the two cases this is, in those words. + _undoc=0 + while IFS= read -r _ln; do + [ -n "$_ln" ] || continue + _n="${_ln%%:*}" + sed -n "$((_n-1))p" "$_h" \ + | grep -qE 'SUPPRESSED DELIBERATELY|benign:' || _undoc=$((_undoc+1)) + done <<< "$(grep -nE '(real_chmod|real_chown|real_chgrp) .*2>/dev/null \|\| true' "$_h")" + [ "${_undoc:-0}" = 0 ] \ + || _p="$_p;$_undoc suppressed ownership call(s) are not marked deliberate or benign" + # and the two that are deliberate must say so in those words + grep -q 'SUPPRESSED DELIBERATELY' "$_h" \ + || _p="$_p;the deliberate suppressions are not marked as such" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every suppressed ownership failure says why it is suppressed" + fi +fi + +# lfs-helper must actually offer the doors the other tool now knocks on. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + for _c in wrapper-dir make-wrappers grant-dir pkgusr-home owner-name; do + grep -qE "^ $_c\)" "$_h" \ + || _p="$_p;lfs-helper has no '$_c' command" + done + # `case` takes the first match, so a repeated entry is unreachable code + # pretending to be a second decision. Six of them were. + _dups="$(sed -n '/^case "${1:-}" in/,/^esac/p' "$_h" \ + | grep -oE '^ [a-z][a-z0-9|-]*\)' | sort | uniq -d)" + [ -z "$_dups" ] || _p="$_p;the dispatch repeats: $(echo $_dups | tr -d ')')" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every shared decision is reachable as one lfs-helper command" + fi fi # ---- editing the wrong install script fails silently ------------------------ # @@ -6073,7 +6444,7 @@ if [ -f "$helper_src" ]; then fi # nothing outside the chokepoints may build an account path by hand -for _f in packagemanager_install lfs-completion.bash last_build_step.sh; do +for _f in packagemanager_install lfs-completion.bash; do _p="$(dirname "$LFS_TOOL")/$_f" [ -f "$_p" ] || continue if grep -qE '"?/usr/src/\$[a-z_]+' "$_p"; then @@ -6872,6 +7243,7 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then _o="$( SNAP_ROOT=/ eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" eval "$(_slice_fn "$helper_src" is_never_claimed)" for f in /usr/share/info/dir /etc/ld.so.cache /var/cache/ldconfig/aux-cache; do is_never_claimed "$f" || echo "$f can still be claimed by a package" @@ -7457,6 +7829,7 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then _o="$( ETC=/etc; SNAP_ROOT=/ eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" eval "$(_slice_fn "$helper_src" is_never_claimed)" for f in /etc/passwd /etc/passwd- /etc/group /etc/group- \ /etc/shadow /etc/gshadow /etc/.pwd.lock; do @@ -7844,6 +8217,7 @@ if [ -f "$helper_src" ]; then _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr WRAPPERS=/usr/lib/pkgusr eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" eval "$(_slice_fn "$helper_src" is_never_claimed)" command -v is_never_claimed >/dev/null 2>&1 \ || { echo "there is no is_never_claimed"; exit 0; } @@ -7900,7 +8274,7 @@ _count_rc $? # ---- add-user names the account through the chokepoint ---------------------- # # It used its argument raw for the ACCOUNT and pkg_owner_name only for the HOME. # Fine while every caller passes an already-prefixed owner -- cmd_build does. -# last_build_step.sh does not: `lfs-helper add-user wget` produced an account +# last_build_step.sh did not: `lfs-helper add-user wget` produced an account # called `wget` living in /usr/src/pkgusr/p_wget, and the sanity report found # both halves without being able to connect them: # !! accounts without a known prefix: wget urllib3 requests ... @@ -7929,13 +8303,11 @@ if [ -f "$helper_src" ]; then ok "add-user gives the account and its home the same name" fi - # last_build_step.sh is the caller that exposed it -- it passes bare names - _lbs="$(dirname "$LFS_TOOL")/last_build_step.sh" - if [ -f "$_lbs" ]; then - grep -q 'lfs-helper add-user' "$_lbs" \ - && ok "the final step still creates its users through add-user" \ - || bad "the final step no longer goes through add-user" - fi + # last_build_step.sh was the caller that exposed it -- it passed bare names. + # It is gone (1.10.0) and this checked a file that no longer exists, so it + # never ran again. The requirement outlives it: `packagemanager setup` is + # the next caller to create accounts from outside the tool, and it must go + # through add-user rather than useradd. Assert that when setup lands. fi # ---- the sanity report reads the sorted state, and knows when a build is done # @@ -8003,6 +8375,7 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr; WRAPPERS=/usr/lib/pkgusr eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" eval "$(_slice_fn "$helper_src" is_never_claimed)" command -v is_never_claimed >/dev/null 2>&1 \ || { echo "there is no is_never_claimed"; exit 0; } @@ -8265,25 +8638,10 @@ if [ -f "$helper_src" ]; then fi fi -# and the final step must ASK rather than rebuild the name itself -_lbs="$(dirname "$LFS_TOOL")/last_build_step.sh" -if [ -f "$_lbs" ]; then - _p="" - _n="$(grep -c 'lfs-helper owner-name' "$_lbs")" - [ "${_n:-0}" -ge 2 ] \ - || _p="$_p;the final step still assumes the account name ($_n of 2 sites)" - # every chown/su in there must use the resolved name, never the bare argument - case "$(grep -c 'lfs-helper add-user' "$_lbs")" in - 0) _p="$_p;the final step no longer creates its users through add-user" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the final step asks for the account name it was given" - fi -fi +# The caller that exposed this was last_build_step.sh, which is gone (1.10.0). +# The rule is not: any script outside the tool that needs an account name must +# ASK -- `lfs-helper owner-name` -- instead of gluing a prefix on itself. +# `packagemanager setup` inherits it; assert it there when setup lands. # ---- the ESP hint prints a path you can actually type ----------------------- # # Two sources feed it and they disagree about the prefix: `lsblk -rno NAME` @@ -8459,6 +8817,210 @@ if [ -f "$helper_src" ]; then rm -rf "$_st" fi +# ---- unpacked sources are scratch, and no scan should walk them ------------- # +# Build trees moved from /build into each package user's home in 1.7.6. The +# snapshot scans followed; the ownership scans did not. So `lfs-helper verify` +# began walking every unpacked source tree in the system -- it ran for minutes +# with no output -- and a finished tree filled the sanity report with tcl's own +# documentation: +# +# !! files with no owner (first 40): +# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm +# +# A tarball can carry any uid it likes. Unpacked sources are not installed, +# nothing owns them, and asking who does has no answer. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" scan_prune_paths | grep -q 'PKGUSR_BUILD_SUBDIR' \ + || _p="$_p;the prune list does not cover the per-package build trees" + # ONE list -- there were five and they had already diverged + for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do + _slice_fn "$helper_src" "$_fn" | grep -q 'SCAN_PRUNE' \ + || _p="$_p;$_fn does not use the shared prune list" + done + # an ARRAY, not a string through eval: the patterns contain *, and eval lets + # the shell expand them before find ever sees them + case "$(_slice_fn "$helper_src" scan_prune_set)" in + *'SCAN_PRUNE+=('*) ;; + *) _p="$_p;the prune list is not built as an array" ;; + esac + if _slice_fn "$helper_src" _vfy_orphans | grep -q 'eval find'; then + _p="$_p;the orphan scan splices its prune list through eval" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every ownership scan skips the same scratch, from one list" + fi + + # end to end: a build tree is pruned, a real file is not + _pt="$T/prune" + mkdir -p "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html" "$_pt/usr/bin" + touch "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real" + if [ "$(id -u)" = 0 ]; then + chown 4321 "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real" + _r="$( SNAP_ROOT="$_pt"; STATE="$_pt/usr/src/lfs-pkgusr" + PKGUSR_BUILD_SUBDIR=src + PKGUSR_ROOT="$_pt/usr/src/pkgusr"; CFGUSR_ROOT="$_pt/usr/src/cfg" + pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; } + eval "$(_slice_fn "$helper_src" scan_prune_paths)" + eval "$(_slice_fn "$helper_src" scan_prune_set)" + scan_prune_set + find "$_pt" -xdev -nouser "${SCAN_PRUNE[@]}" 2>/dev/null \ + | sed "s|$_pt||" | sort | tr '\n' ' ' )" + case "$_r" in + "/usr/bin/real ") ok "an unpacked source tree is skipped, an installed file is not" ;; + *) bad "the prune list is wrong: got [$_r]" ;; + esac + fi + rm -rf "$_pt" +fi + +# the sanity report has to skip the same thing, or it reports what verify does not +san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$san" ]; then + _p="" + grep -q 'PKGROOT/\*/src' "$san" \ + || _p="$_p;the sanity report still walks the package build trees" + # a config step writing root-owned files is not an orphaned manifest + grep -q 'writes ROOT-owned files' "$san" \ + || _p="$_p;root-owned config files are still reported as missing an account" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the sanity report skips scratch and does not invent findings" + fi +fi + +# ---- verify must not work in silence, or one fork at a time ----------------- # +# `lfs-helper verify` ran for over five minutes printing nothing, on a tree it +# had already verified clean. Two separate faults: +# +# * a fork per path, twice over -- `stat -c %U` and a process substitution +# inside is_never_claimed -- across 66265 manifest paths +# * no output at all until every pass had finished +# +# A command that works for minutes in silence is indistinguishable from one that +# has hung, and was reported as one. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + # the never-claim list is read once, not per call + _slice_fn "$helper_src" is_never_claimed | grep -q '< <(never_claim_list)' \ + && _p="$_p;is_never_claimed forks for every path it is asked about" + _slice_fn "$helper_src" _never_claim_load | grep -q . \ + || _p="$_p;the never-claim list is not cached" + # owners come from one batched stat, not one per path + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in + *'xargs -0 -r stat'*) ;; + *) _p="$_p;the ownership pass still runs one stat per path" ;; + esac + # a REAL tab: stat does not expand \t, so '%U\tname' comes back as one field + # and the map silently stays empty -- correct but slow, the hardest wrong to see + # comments stripped: the fix NAMES the broken format while explaining it + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership \ + | grep -vE '^[[:space:]]*#')" in + *'%U\t%n'*) _p="$_p;the stat format uses a literal backslash-t" ;; + esac + # and it must say what it is doing, before it does it + _nd="$(_slice_fn "$helper_src" cmd_verify | grep -c 'detail "# ')" + [ "${_nd:-0}" -ge 5 ] \ + || _p="$_p;verify announces only $_nd of its passes" + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in + *'[%d/%d]'*) ;; + *) _p="$_p;the long pass shows no progress" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "verify says what it is doing, and does it in one pass not 66000" + fi + + # the batched lookup must actually map something -- an empty map is the bug + _pf="$T/statmap"; mkdir -p "$_pf" + touch "$_pf/a" "$_pf/b" "$_pf/c" + printf '%s\n' "$_pf/a" "$_pf/b" "$_pf/c" > "$_pf/man" + _n="$( _TAB=$'\t' + declare -A own=() + while IFS="$_TAB" read -r o nm; do + [ -n "$nm" ] && own["$nm"]="$o" + done < <(grep -v '^[[:space:]]*$' "$_pf/man" \ + | tr '\n' '\0' | xargs -0 -r stat -c "%U${_TAB}%n" 2>/dev/null) + echo "${#own[@]}" )" + [ "$_n" = 3 ] \ + && ok "the batched owner lookup maps every path it is given" \ + || bad "the batched owner lookup mapped $_n of 3 paths" + rm -rf "$_pf" +fi + +# ---- the build must not end quietly with no way to log in ------------------- # +# init-accounts asks for a root password, but it is skippable: a scripted run +# has no terminal, and an interactive one can be answered with a blank line. +# So the LAST thing build-all says is whether anyone can actually log in -- +# because it is the last moment it can still be fixed. After the reboot the +# chroot is gone, and the way back is booting the host and mounting the tree. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _warn_if_no_login | grep -q . \ + || _p="$_p;nothing checks whether the system can be logged into" + # it must be the LAST thing build-all does + _ba="$(_slice_fn "$helper_src" cmd_build_all)" + case "$_ba" in + *_warn_if_no_login*) ;; + *) _p="$_p;build-all can finish without saying whether you can log in" ;; + esac + _last="$(printf '%s\n' "$_ba" | grep -vE '^[[:space:]]*(#|$)' | tail -3)" + case "$_last" in + *_warn_if_no_login*) ;; + *) _p="$_p;the login check is not the last thing build-all does" ;; + esac + # it must check the RESULT, not that the step ran + case "$(_slice_fn "$helper_src" _warn_if_no_login)" in + *shadow*) ;; + *) _p="$_p;it trusts init-accounts instead of checking for a password" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build's last word is whether you can log in afterwards" + fi + + # all three outcomes, against a real shadow file + _lg="$T/login"; mkdir -p "$_lg" + _run_login() { # <shadow contents> -> "ok" or "warn" + ( ETC="$_lg"; say(){ :; }; ok(){ :; }; warn(){ echo warn; } + printf 'root:x:0:0::/root:/bin/bash\nbob:x:1000:1000::/home/bob:/bin/bash\n' > "$_lg/passwd" + printf '%s' "$1" > "$_lg/shadow" + eval "$(_slice_fn "$helper_src" _warn_if_no_login)" + _warn_if_no_login | head -1 ) 2>/dev/null + } + _bad="" + [ -z "$(_run_login 'root:$6$a:1:::::: +bob:!:1::::::')" ] || _bad="$_bad;a root password is not accepted" + [ -z "$(_run_login 'root:*:1:::::: +bob:$6$b:1::::::')" ] || _bad="$_bad;a user password is not accepted" + [ -n "$(_run_login 'root:*:1:::::: +bob:!:1::::::')" ] || _bad="$_bad;a system nobody can log into is not reported" + if [ -n "$_bad" ]; then + printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a locked-out system is reported; a usable one is not" + fi + rm -rf "$_lg" +fi + echo echo " PASS: $(_pass_total) FAIL: $(_fail_total) (final)" [ "$(_fail_total)" -eq 0 ]