Packagemanager-LFS-PackageUser-System git · main
Package-user based package manager for LFS/BLFS
Shell 63.7% Markdown 35.8%git clone https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System.gitwget https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System/archive/Packagemanager-LFS-PackageUser-System.tar.gzlfs run can compile all packages
HANDOFF.md | 872 ++++++++ README.md | 68 +- blfs | 2 +- collector-groups.export | 16 + lfs | 2051 +++++++++++++---- lfs-completion.bash | 18 +- lfs-helper | 4181 ++++++++++++++++++++++++++--------- lfs-sanity.sh | 269 +++ packagemanager | 269 ++- packagemanager_install | 66 +- skel-u_xdg/.bash_profile | 42 + test_lfs_crosschain.sh | 5508 ++++++++++++++++++++++++++++++++++++++++------ 12 files changed, 11196 insertions(+), 2166 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..c7307b1 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,872 @@ +# lfs-pkgusr — context for a new chat + +Five scripts that build and maintain an LFS/BLFS system where **every package +is owned by its own unprivileged user**. + +Attach: the five scripts (`lfs`, `lfs-helper`, `packagemanager`, +`packagemanager_install`, `blfs`), plus `test_lfs_crosschain.sh`, +`lfs-sanity.sh` and this file. + +**New here? "Start here" below says what to read and in what order.** + +## Current state + +All five tools at **1.10.1**. Build ids — check these match what is installed: + + lfs 8b2cb98 + lfs-helper c2f6fbc + packagemanager d051474 + blfs 36a4717 + packagemanager_install 1c068ed + +569 regression tests, 568 pass. + +**A full build on 1.9.3 verified clean: 66265 paths owned by the right +package, nothing else outstanding.** The one failure is `the XDG profile is not +shipped`, which looks for `skel-u_xdg/.bash_profile` next to the tool — +environment-only, it passes where the repo is laid out properly. + +**A full 104/104 build has completed** on the 1.8.2 code. Everything since is +either a fix for something that build revealed, or the 1.9.0 cleanup. + +**A fresh build is required and no tree is migrated.** Account names, home +directories, the build scratch and the state directory have all moved. + + make install + lfs build-system restart --run + lfs build-system session --reconfigure # several new questions + lfs run + # inside the chroot: + lfs-helper --version # MUST read c2f6fbc + lfs-helper build-all + +Afterwards `lfs-helper verify` is the real check — it reconciles ownership +across the whole tree. Then `check-toolchain`. + +Kernel and bootloader are still yours to finish. + +## Start here + +If you are picking this up cold, read in this order: + +1. **The core idea** — one paragraph, and everything else follows from it. +2. **Where everything lives** — the layout changed in 1.9.0. +3. **The ownership epoch** — the single moment the scheme turns on. +4. **One bug, twelve times** — every failure this project has had was the same + shape. Knowing the shape is worth more than knowing the twelve. +5. **Open items** — item 1 is the next job. + +## The core idea + +Each package gets a user that owns the files it installs. + +- Every file traces to the package that installed it. +- Removing a package is removing that user's files. +- A broken build cannot overwrite another package's files. + +Directories several packages install into are shared via **collector groups** +(`<prefix>_<owner>`, e.g. `nimgnu_python`). The install group (`install`, gid +9999) means "anyone may install here" and is only for genuinely shared dirs — +never a package's own tree, and never `/root` or `/home`. + +## The five scripts + +| Script | Runs | Purpose | +|---|---|---| +| `lfs` | host, Python | Drives the build: partition, sources, chapters 5-6, chroot, snapshots | +| `lfs-helper` | inside the chroot, **bash** | Builds each package as its user. Bash because there is no Python in the chroot yet | +| `packagemanager` | built system, Python | Installs/updates packages, application users, config | +| `packagemanager_install` | built system, bash | The install engine `packagemanager` calls | +| `blfs` | built system, Python | Parses the BLFS book, generates install scripts, resolves dependencies | + + +## Where everything lives + +Two trees under `/usr/src`: the accounts, and what the build knows about them. + +``` +/usr/src/pkgusr/p_gcc a package's account +/usr/src/pkgusr/p_gcc/src where gcc unpacks and builds +/usr/src/cfg/cfg_bootscripts a config step's account +/usr/src/u_firefox an application user +/usr/src/lfs-pkgusr/ what the build knows about all of them +``` + +`SRCROOT` (`/usr/src`) is the parent and stays whole-tree, **deliberately**: +every ownership pass excludes it as one path (`-not -path "$SRCROOT/*"`). +Splitting that exclusion would silently un-exclude half the accounts. + +That is also why the state directory moved here in 1.9.0. It was +`/.lfs-pkgusr` — a hidden directory at the root of the filesystem, sitting +beside `/boot` and `/etc` as if it were part of the system being built. It is +not; it is this toolchain's working notes *about* that system. Under +`/usr/src` the scans exclude one path instead of two. + +The state directory is sorted, so opening it tells you what is in it: + +| | | +|---|---| +| `scripts/` | one shell script per build step, generated from the book | +| `manifests/` | what each package installed (`.files`, `.dirs`) | +| `logs/` | one log per step, plus `verify.log` | +| `progress/` | `steps-built`, `steporder`, `adopted.list`, tree snapshots, the ownership marker | +| `groups/` | collector groups and the grants that created them | +| `config/` | `installdirs.lst`, `env` — settings you may edit | +| `stage/` `steps/` `tmp/` | transient, only during a build | + +Both tools name it once: `STATE` in `lfs-helper`, `PKGUSR_DIR` + +`pkgusr_state()` in `lfs`. They read each other's manifests, so a disagreement +here is a silent one. + +**Definition order matters.** `SRCROOT` must be set before anything derived +from it. Under `set -u` a forward reference is not a subtle bug — the tool +will not print its own version. There is a test on the ordering. + +## The build scratch is the package's + +A package unpacks and builds in `<its home>/src` — not in a shared `/build` +that every package writes into at once. + +The shared scratch was a permanent ownership fight. Every build's touched-file +scan swept it in, so each manifest claimed every other package's sources and +chowned them; the next package chowned them back. A real `verify` reported +3363 wrong-owner and 3348 of them were `/build`: + + /build/bash-5.3.tar.gz is p_shadow want p_acl + /build/bash-5.3.tar.gz is p_zstd want p_acl (after "repairing" it) + +Under the package user's home it is settled by construction: the user creates +the tree so the user owns it, `$SRCROOT` is already excluded from every scan so +it can never reach a manifest, and removing a package removes its build tree. +Steps with no account keep the shared `/build` — they run as root, which owns +everything there anyway. + +`~/src`, not `~/build`: `~/build` is already the hint's build helper script, +symlinked into every package user's home. `pkgusr_skel_links` names the +skeleton once and a test checks the build subdir is not one of them. + +## Names and places + +Three kinds of account share one passwd file, so each carries a prefix. +Without one, a package called `man` or `news` collides with a real system +account and nothing says which accounts belong to the build. + +| Thing | Looks like | Lives in | +|---|---|---| +| package user | `p_gcc` | `/usr/src/pkgusr/p_gcc` | +| config-step user | `cfg_bootscripts` | `/usr/src/cfg/cfg_bootscripts` | +| application user | `u_firefox` | `/usr/src/u_firefox` | +| collector group | `<prefix>_python` | — | + +**One account, one prefix — its own.** + +| Range | Used for | +|---|---| +| 9998 | `lfs` build user *(inside the chroot; the host's is unpinned)* | +| 9999 | `install` group | +| 10000+ | Package users, in build order (binutils first) | +| 90000+ | Collector groups | + +**Prefixing is idempotent, and that is load-bearing.** It is applied to names +a person typed (`gcc`), to directory names read back from disk (`p_gcc`), and +to values that already went through it. Because a second pass cannot produce +`p_p_gcc`, every call site can apply it without first knowing which kind of +name it holds — which is what made 24 call sites a mechanical change instead +of 24 judgement calls. + +**One chokepoint per tool.** Never build a home by hand: + +- `lfs-helper` — `pkgusr_kind()`, `pkgusr_prefix_for()`, `pkgusr_root_for()`, + `pkgusr_home_for()`, `pkgusr_roots()`, `unprefix_pkg_user()`, + `step_stage_dir()` +- `packagemanager` — `pkgusr_kind()`, `pkgusr_name()`, `unprefix_pkgusr()`, + `pkgusr_root()`, `pkgusr_roots()`, `pkgusr_home()` +- `lfs` — `pkgusr_prefix()`, `cfguser_prefix()`, `LAYOUT_DEFAULTS`, `layout()` + +`pkgusr_kind()` is the top of that chain: it decides the kind once and the +prefix and the root are both answered from it, so a name cannot be given one +kind's prefix and placed in another kind's root. + +`lfs-helper pkgusr-home <name>` exposes the rule to shell scripts, so nothing +else re-derives it. `packagemanager_install`, `lfs-completion.bash` and +`last_build_step.sh` all go through a chokepoint now; a test fails if any of +them builds `/usr/src/$something` by hand. + +**The chokepoint must resolve to the ACCOUNT home, not just the root.** +Callers pass a mixture and always will: `build_pkg` has the step name +(`man-pages`, `util-linux-tmp`), the staging code has the owner +(`p_man-pages`). `pkg_owner_name` maps one to the other — prefix, lowercase, +fold `-tmp`/`-passN` onto the real package — and it is idempotent. Routing by +kind without that mapping shipped once and broke the build at step 9: + + # created package user p_man-pages ... + line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory + +Chapter 7 did not catch it — root steps `mkdir -p` their own directory first, +so it surfaced at the first real package user. + +**A step that is not a package has no home.** `pkgusr_home_for` answers for +accounts; `step_stage_dir` answers for steps, and routes `init-*`, `last-step`, +`refind` and prose `cfg_*` sections to `$STATE/steps/<name>` because they never +get an account. Asking `pkgusr_home_for` for their home is what left +`/usr/src/pkgusr/p_init-dirs` behind. + +Note `SRCROOT` in `lfs-helper` is the **parent** (`/usr/src`), deliberately. +Tree scans exclude it as one path (`-not -path "$SRCROOT/*"`); splitting it +there would silently un-exclude half the accounts from every ownership pass. + + +## Two directories, two jobs + +| Directory | Job | Permissions | +|---|---|---| +| `$LFS/sources` | downloaded tarballs, never written by a build | `root:root 1777` | +| `<package home>/src` | where that package unpacks and builds | owned by its user | +| `$LFS/build` | scratch for steps with no account | `root:root 1777` | + +Book 3.1 settles the ownership — `chmod a+wt`, then +`chown root:root $LFS/sources/*` — and gives the reason: a host uid means +nothing in the built system. Here it is sharper than the book's case. The host +`lfs` account is created by book 4.3's own `useradd`, which pins no uid, so it +lands wherever the host had a gap — 10753 on a Debian host. Package users start +at 10000, so it does not merely show as a number, it **collides with a real +package user** and tarballs report as owned by something unrelated. + +`/sources` and `/build` are in `never_adopt_list` and NOT in +`install_dirs_list`. Being in both made `/sources` `root:install 775`, +dropping the `o+w` that lets the unprivileged build user unpack at all. + +**The symlink farm.** `unpack_pkg` links every downloaded file into +`BUILD_ROOT`, because the book reaches sibling tarballs through `..`: + + cd .. + tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 + +`cd ..` leaves the build subdirectory, so `..` is then `BUILD_ROOT`. Linking +everything rather than a known list means any future book works without a +second thing to keep in sync. + + +## The ownership epoch + +**Before book 7.6 there is no `/etc/passwd`, so ownership is not wrong — it is +meaningless.** No name resolves, not even root's; the prompt in the chroot +reads `I have no name!`. Every ownership question asked before that point gets +a confident wrong answer: + + chown root:install fails on every directory -> "# 0 install directories" + find -nouser matches every file in the tree -> 11802 "orphans", incl. / and /proc + +There is one gate: + +| | | +|---|---| +| `ownership_established()` | is it done? (`progress/ownership-established`) | +| `ownership_possible()` | can it be? (passwd + chown/chgrp/chmod) | +| `establish_ownership()` | do it | +| `_ownership_checkpoint()` | the one call site, after every step | + +`init-ownership` is a **step you can see**, inserted straight after +`init-files` (book 7.6). `lfs-helper` performs it itself — there is no book +script — and marks it done, so `list` and `next` agree with reality: + + 3. init-ownership built + +It does everything in order: install group and install directories, the build +user's leftovers handed back to root, package users for everything already +built, adoption, our own tools, then orphans. Idempotent, and the marker is +durable, so a resumed or restored build crosses it once. +`lfs-helper establish-ownership` runs it by hand. + +Everything that needs a user database refuses without one: `_vfy_orphans`, +`_has_orphaned_files`, `_vfy_build_user_leftovers`, `_vfy_install_dirs`. + +Install directories are set and read back **by number** (`0:$INSTALL_GID`) — +`root` need not be a name. + +## The two chowns that can destroy a tree + +Book 4.3 gives the tree to the build user; book 7.2 gives it back. Both are +recursive, and each is catastrophic on the wrong side of chapter 7. + +- **4.3 on a built tree** hands the finished system to the host's build + account. `_chown_tree_to_lfs` refuses when `_handover_is_safe` is false. + + There were **four doors to that chown** and only one had the guard: + `_chown_tree_to_lfs` (book 4.3), `session --run` re-applying it by hand, + the pre-build ownership check, and `fix-ownership --run`. All four now ask. + A test walks every `chown -R lfs` in the file and fails on any whose + enclosing function does not consult `_handover_is_safe` — the property, not + the four paths, because a fifth door would otherwise be silent too. +- **7.2 on a built tree** strips every package user in one pass. Refused for + the same reason. + +Refusing both left a tree with no way forward, so there is a third, precise +form: `_handover_build_user_only` selects on the **build user's uid**, taking +back exactly what 4.3 took and leaving every package user's files alone. Safe +at any stage. `chroot prepare` runs it instead of refusing, and `chroot enter` +runs it before deciding to stop. + +`chown -h root`, user only — 4.3 chowns the user, so the inverse does too; +taking `root:root` would strip `install` off every shared directory. And `-h`, +because `/bin`, `/lib` and `/sbin` are symlinks. + +`_handover_is_safe` asks the **tree's own passwd** which accounts are package +users (`_pkg_users_in_tree`), never a uid range. Book 4.3's `useradd` pins no +uid, so the host's `lfs` came out **10753** on one machine — inside the package +range — while the chroot's own `lfs` is 9998. Across that boundary only the +number means anything. + +## Saying things + +Four colours, one meaning each, identical in both tools — they print into the +same terminal one after the other, and a log where the same colour means two +things is worse than no colour at all. + +| | | +|---|---| +| green | something finished, and finished correctly | +| yellow | something needs your attention | +| red | something failed | +| dim | detail you can skip: provenance, and what to do next | + +Prose gets none. Colour on every other line stops carrying information. + +`lfs-helper`: `say` (prose) `ok` `warn` (stderr) `note` (stdout) `detail` +`hint` `fail` `die`. `lfs`: `warn` `note` `ok` `fail` `hint`. + +**Use the helper, never the variable.** A colour opened in one `echo` and +closed in another leaves the terminal painted if anything between them writes — +the failure banner was 22 lines inside a single open/close. The only direct use +left is the status column in `list`, where the colour *is* the information. + +## One bug, twelve times + +Every failure this project has had was the same shape: **one decision held in +more than one place, free to disagree.** They are worth reading as one list, +because the next one will look like them. + +| What disagreed | How it showed up | +|---|---| +| Package prefix applied regardless of kind | `p_cfg_bootscripts` — two prefixes, in the config root, claiming to be a package | +| Two wrapper directories | `make_wrappers` wrote `$STATE/wrappers`; the profile said `/usr/lib/pkgusr`, which nothing created | +| Two copy routines for the tools | `install_helper` claimed ownership, `_install_pkgusr_helpers` did not — eight files owned by nobody | +| `chgrp` vs `chown root:install` | `init-pkgusr` set the group, `verify` demanded owner+group; `/usr/share` sat at `lfs:install` | +| Install-dir rule in three places | `cmd_verify` carried inline copies without the guards the helpers had grown | +| **`cmd_verify` defined twice** | bash keeps the last; the tidy one had never run | +| Scratch scanned by two rules | the snapshot scans excluded `/build`, the build's own scan did not — 3348 false findings | +| Build tree named `build` | collided with `~/build`, the hint's helper script | +| Wrapper `chgrp` shadowing root's | reported 46 successes, changed nothing, and `verify --fix` could not repair it either | +| Handover guarded by one inode | `_owner_is_root` checks `$LFS/usr`; the pass it guards covers eight trees | +| uid range used to identify the build user | host `lfs` was 10753, inside the package range | +| Adoption reached from two directions | an up-front pass and a mid-loop "late" pass with its own flag | +| Generated scripts had no version stamp | the step order lives in the tree, the tools outside it — a tree generated by an older version went on running `last-step` under a lfs-helper that had removed it | +| Collector export written and read in two places | 1.9.0 sorted the state dir; the writer stayed at the top level, the reader moved into `groups/` | +| `pkgusr-home` was exposed, `owner-name` was not | a script could ask where a package lives but not what its user is called, so `last_build_step.sh` assumed — `chown: invalid user: 'wget:wget'` at step 104 of 105 | +| `add-user` named the account raw, the home prefixed | `add-user wget` gave account `wget` in `/usr/src/pkgusr/p_wget` — every caller but `last_build_step.sh` passed an already-prefixed name, so it never showed | +| Tree snapshot written and read in two places | same move, same shape: `lfs` kept `.snapshot`, `lfs-helper` read `progress/tree-snapshot` — the chapter 6→7 handover snapshot was lost | + +**Half a chokepoint is not a chokepoint.** `pkgusr_home_for` was reachable from +outside as `lfs-helper pkgusr-home`; `pkg_owner_name` was not. So a shell script +could ask where a package lives but had to *guess* what its user is called — +right by accident until `add-user` started applying the prefix, then wrong at +step 104 of a 105-step build. `lfs-helper owner-name <package>` is the other +half, and a test checks the two agree. + +The fix is always the same: **name the decision once, and make everything ask +that one thing.** `pkgusr_kind()`, `build_root_for()`, `never_claim_list()`, +`ownership_possible()`, `pkgusr_skel_links()`, `_real_tool()` all exist for +this reason. + +Two rules follow from it, and both have tests: + +- **No function may be defined twice.** A duplicate is invisible — both parse, + one silently wins. +- **Every `cmd_*` a tool calls must be defined in that tool.** `verify --fix` + called `cmd_fix_orphans`, which never existed, and died mid-repair after + chowning 3473 paths. + +## The collector export + +A previous build's group decisions, so a rebuild does not ask +"which group should share this directory?" for every package again. + + lfs-helper export-groups # write one out + collector_import_file = <path> # session setting on the host + +`lfs` copies it into `groups/collector-groups.import`; `lfs-helper` reads it +from there in `choose_collector_group`, **before** asking. Those two paths are +the whole mechanism, and nothing else connects them — when the state directory +was sorted in 1.9.0 the writer stayed at the top level while the reader moved, +and a rebuild with a perfectly good export asked about `/usr/lib/bfd-plugins` +anyway. Silent, because a question that gets asked looks exactly like a +question that was never answered. There is a test that the two paths match. + +It is re-copied on **every chroot entry**, so editing it on the host reaches +the next build without regenerating the scripts. It only prints when the +contents changed. + +`session` reports what is in the file (`N directory decision(s), M group(s)`) +rather than echoing a path — a file that was configured, present and never read +looked identical to one that was working. + +## One rule for /usr/src + +Everything under `/usr/src` is `root:install 1775`: `/usr/src` itself, the +account roots, and the state directory. + +The account roots were `root:root 0755` — defensible, since only root creates +an account home, but it made one subtree with two rules and a special case to +remember. **The sticky bit is what makes group-writable safe and is not +optional:** without it any member of `install` could delete or rename another +package's home. With it, only the owner can, so group write grants only the +ability to *create*, which nothing unprivileged does (`add-user` is root-only). + +Sharing the mode is not sharing the meaning. The account roots and the state +directory are still in `never_adopt_list`, so no package may ever take one as +its own, and `install_dirs_list` does not contain them. + +**The state directory is in `never_claim_list`.** It lives under `/usr/src`, +which IS an install directory, so `verify` walked its own manifests, scripts +and logs and reported each as a root-owned file no package claims. A clean +104/104 build came back with `448 root-owned file(s) that NO manifest claims` +and all 448 were this directory — including the two temp files the scan had +just written for itself. + +## Reading a sanity report + +Three of its findings used to be about the report, not the tree. All fixed, but +worth knowing the shape: + +- **The sticky bit means opposite things on either side of the finish line.** + During the build it is a bug — package users cannot replace the temporary + system's files. After `seal-install-dirs` it is the point, and its *absence* + is the bug. The report works out `BUILD_FINISHED` from `progress/steporder` + vs `progress/steps-built` and judges accordingly. +- **An empty manifest is not an orphan.** A prose config step — `cfg_clock`, + `cfg_hosts`, `cfg_locale` — writes into files another package owns, so it + records nothing and legitimately has no account. Fourteen of those were + reported on a perfect tree. +- **A step can install under a name that is not its own.** `libstdcpp`'s files + belong to `p_gcc`, because the book builds it out of the gcc tree. When the + name lookup fails, the report now asks the filesystem who owns the files + before calling it a finding. + +`packagemanager_install` has `--version` now, so section 0 can read all five +tools. A tool you cannot ask "which build are you?" is the one that quietly +goes stale. + +## comm needs one collation + +The snapshot diff is `comm -13 <(before) <(after)`. **comm compares line by +line and trusts both sides to be sorted the same way. It cannot detect +otherwise; it returns the wrong answer.** Two things broke that: + +- the two snapshots may have been written under different locales — a package + user's profile sets `LC_ALL=POSIX`, root's shell may not; +- `load_snapshot` rewrites only the lines carrying the host mount prefix, so a + file holding both forms comes back sorted under neither. + +`snapshot`, `snapshot_dirs` and `load_snapshot` all pin `LC_ALL=C sort`, and +`load_snapshot` re-sorts after the rewrite. The visible symptom is a package +installing forty files and being told it installed none. + +**A re-run over its own work is not a silent failure.** The zero-files check +counts what *this run* wrote, which is the wrong question the second time a +package is built: many installers skip a file that is already up to date -- +perl's ExtUtils::Install prints "Installing <path>" for every target and copies +only the stale ones. So the run writes nothing, the snapshot diff sees nothing +new, no mtime beats the stamp, and the count is 0 for the best possible reason. + +`xml-parser` hit exactly that: forty files on disk, every one owned by +`p_xml-parser`, and the step failed as a no-op. `_package_already_owns_files` +asks the tree instead of the counters. A package owning nothing is still a +failure. + +**"installed NO files" now shows its working.** There are exactly three ways to +reach zero and they need different fixes: nothing was written, it was written +somewhere excluded from tracking, or it was written over files another package +owns and dropped by the ownership filter. The message reports the snapshot-diff +count, the touched count, and which of the three it is. + +## A bootloader is opt-in + +`bootloader` defaults to **none**. It used to default to `refind`, on the +reasoning that rEFInd only ever adds files to an already-mounted ESP — true, +and still the safe choice, but it stopped a finished build to ask about a +bootloader on a machine that already had one. The machine booted in order to +build LFS at all. A bootloader is a decision about the whole machine, not a +package. + + lfs config bootloader refind + lfs config esp /boot/efi + lfs build-system gen-chroot-scripts --run --overwrite + +Skipping it is not silent — `gen-chroot-scripts` says so and prints those three +lines, because someone who *does* need a bootloader would otherwise reach a +finished, unbootable system without ever having been offered one. + +GRUB is unchanged and still withheld unless explicitly chosen: its book +instructions run `grub-install /dev/sda`, which writes a disk's boot sector. + +## The tree remembers which version generated it (1.10.1) + +`gen-chroot-scripts` writes `progress/generated-by` with the version, build id +and date, and deletes scripts for steps no longer in the order. +`_warn_if_scripts_are_stale` compares that against `lfs-helper`'s own version +before `build-all` runs anything. + +**The step order and the scripts live in the TREE; the tools live outside it.** +Nothing connected the two, so a tree generated before 1.10.0 kept running +`last-step` — a step 1.10.0 had removed entirely — under an `lfs-helper` that no +longer knew what it was. The chroot *tools* have carried a build id for exactly +this reason since 1.6; the generated scripts did not. + +Removing a step from the tools is therefore **two** changes: the code, and a +regeneration in every existing tree. The stamp is what makes the second one +visible instead of silent. + +## last_build_step.sh is gone (1.10.0) + +It was a script created once on the HOST and then owned by the person, run last +inside the chroot. Wrong home for everything it held: + +- a fix to it reached nobody who already had a copy — that is how + `chown: invalid user: 'wget:wget'` survived three releases; +- bootstrapping the tooling is `packagemanager setup`'s job; +- setting the root password is not a matter of taste. Book 8.5 says to do it. + +Split three ways: + +| What | Where now | +|---|---| +| root password, login account | **`init-accounts`**, a built-in step in `lfs-helper`, ordered last | +| wget, the Python wheels, the wgetrc weakening | **`packagemanager setup`** — not written yet | +| the `SOURCES` array | **`bootstrap_packages`**, resolved from the BLFS book | + +`init-accounts` could not move to `packagemanager setup`: setup runs on the +**booted** system, and you need to log in to run it. + +`bootstrap_packages` defaults to `wget` and its URL comes from `blfs sources`, +which runs on the host with its own book store. One place knows where wget +comes from, and it is the same place `packagemanager` will ask when it installs +it. Set it empty to turn it off. + +**Losing the "add your own steps here" hook is the real cost.** If it comes +back it should be a directory of drop-in scripts, not a template copied once +that then drifts from the tool that generated it. + +## packagemanager setup: what it owes + +Three guarantees left the suite with `last_build_step.sh` and belong to `setup`. +They are requirements, not history: + +1. **Each module installs as its OWN package user, from a WHEEL.** A modern + sdist needs its build backend, and with no index reachable pip cannot fetch + one: `BackendUnavailable: Cannot import 'hatchling.build'`. +2. **site-packages access goes through the normal collector-group rules** — + join the group that owns it, or ask which group should. Handing + site-packages to the `install` group would mean "anyone may install here", + which is what collector groups exist to avoid. +3. **Never rebuild an account name or home by hand** — `lfs-helper owner-name` + and `lfs-helper pkgusr-home`. This is the bug that started all of it. + +Also: wget needs `check_certificate = off` in `/etc/wgetrc` with the +`lfs-temporary-no-verify` marker, because a fresh system has no CA certificates +until BLFS `make-ca` — and you need wget to fetch make-ca. + +## The final step is yours, so fixes do not reach it + +`/etc/lfs/last_build_step.sh` is created once from a template embedded in `lfs` +and then **belongs to the person** — it is meant to be edited, so it is never +overwritten. Which means a fix to the template reaches nobody who already has a +copy, and the stale copy fails at step 104 of 105, after six hours, with an +error that looks like it came from the tools: + + chown: invalid user: 'wget:wget' + +`_LAST_STEP_STALE` lists known-broken patterns and +`_warn_if_last_step_is_stale` reports them at generation time with the exact +repair. A fixed copy stays quiet, or the warning becomes noise. **Fixing the +template alone is not fixing the bug** — there are two copies by design. + +## Things that belong to nobody + +`never_claim_list` — files no package may own, however many write to them: + +- `/usr/share/info/dir` — rewritten by every package that ships an info page +- `/etc/ld.so.cache`, `/var/cache/ldconfig/aux-cache` — regenerated by ldconfig +- **the user database** and Shadow's backups (`passwd-`, `group-`, `.pwd.lock`) + — every `cmd_add_user` during a build touches them, so whichever package + happened to be building swept them up. ncurses did not install `/etc/group-`; + useradd did, while ncurses was building. A package owning the file that says + which packages exist is the one ownership mistake with no way back. +- the wrapper directory — `root:root` **by design**, so a package user cannot + rewrite the rule that constrains it + +- **mail spools** (`/var/mail/<name>`) — Shadow creates one for every account + it makes, so they arrive as a side effect of `useradd` + +Recording any of them produces a conflict no repair can settle: the next +package takes the file straight back. + +**A repair must not create the next run's finding.** Two passes disagreed about +`/var/mail/lfs` and each undid the other — the build-user pass gave it to root, +the unclaimed pass then reported that root holds a file no manifest claims. +`_vfy_build_user_leftovers` consults `is_never_claimed` for exactly this reason, +and there is a test on the property, not just on the one path. + +## The wrappers + +`make_wrappers` ships `chown`, `chgrp`, `chmod`, `cp`, `install`, `mkdir` in +`/usr/lib/pkgusr`, so a **package user** cannot change ownership. They are +ordinary files on `$PATH`, and if that directory is ever ahead of `/usr/bin` +for **root**, root's own privileged calls resolve to them too — and they report +success while doing nothing. + +So every privileged ownership call goes through `real_chown` / `real_chgrp` / +`real_chmod`, which resolve the binary by absolute path via `_real_tool`. +**Ownership is root's business; no `$PATH` may come between the decision and +the filesystem.** `make_wrappers` itself deliberately uses plain `chmod` — it +is setting the mode on files it just wrote, and must stay self-contained. + +## Mechanisms that are not obvious from the code + +**Staged installs.** `make install DESTDIR=<stage>`, then each file is copied +beside its destination and `rename()`d over it. Never `cp -a` over a live +system: that truncates in-use shared libraries and kills running processes. +Only for `gcc glibc binutils bash coreutils`, and only for a lone +`--phase install` — see open item 4. + +**Earlier stages.** A package is built more than once (ncurses in chapter 6 as +`lfs`, `python-tmp` in chapter 7 as root, then the real one as its package +user). Before building X, `_claim_earlier_stages` hands X every path recorded +under X, X-tmp, X-pass1, X-pass2 — matching case-insensitively, because book +titles are capitalised and tarball names are not. + +**Adoption.** Chapter 5-6 packages get their users from the manifests. This +cannot run before book 7.6 creates `/etc/passwd`; it happens at the ownership +epoch (`init-ownership`), which is the one place that decides. Walks manifests in build order so uids read as the +build order. `adopted.list` records `name size` per package so it runs once. + +**verify.** Ownership is derived state: manifests + install-dir list + +collector map determine what the tree should look like. `lfs-helper verify +[--fix]` reconciles actual against expected. + +**Build ids.** Each tool prints an md5 of its own contents. Compare host and +chroot — a mismatch means the chroot is running old code, which wasted a lot of +debugging time. + +**Section resolution.** `_resolve_section` matches a book section by id **or** +title. Some sub-sections carry ids the book toolchain generated +(`idm139921653990176` = 8.5.2.1 Adding nsswitch.conf) which change on every +rebuild of the book. An ambiguous title fragment is refused, not guessed at. + +**Sections that ship a file.** 9.6.8 prints `/etc/sysconfig/rc.site` as a +`<pre class="auto">` block with no commands at all. `_auto_file_block` extracts +it into a quoted heredoc. Every line is commented; install it anyway — it +documents the boot-script knobs where the boot scripts look. + +**Steps that are not packages.** `last-step`, `init-*` and `refind` never get +a user. `cfg_*` is decided by READING the generated script: a versioned book +section produces a phased build with `pkg_glob`, prose produces a plain root +script. Book 9.2 is `LFS-Bootscripts-20250827`, a real package. + + +## Sanity script + +`lfs-sanity.sh` — read-only, run inside the chroot as root, writes one file: + +```sh +./lfs-sanity.sh # -> /tmp/lfs-sanity.txt +``` + +Eight sections: tool build ids, shared-directory modes (flags early sealing), +uid/gid collisions, unprefixed accounts, home ownership, files with no owner, +groups, and root-owned counts. Each says what a hit MEANS, so a report reads on +its own. + +Section 1 **asserts** the owner and group it prints, not just the mode — a tree +ran 34 packages with every install directory at `root:root 775` while the +report said nothing, because that column was decoration. + +Section 3 flags an account wearing **two** prefixes (`p_cfg_*`, `p_u_*`, +`p_p_*`). Nothing creates those any more, so one appearing means a tool is +still applying the package prefix regardless of kind. `tmp_` and `init_` are no +longer allowed either: there is no such kind of account. + + +## Watch on the next build + +Nothing below has run against the real book on 1.9.0. In rough order of when +you would see it: + +1. **`ls /usr/src`** should show `pkgusr/ cfg/ lfs-pkgusr/` and nothing loose. +2. **`init-ownership` appears in `lfs-helper list`** as step 3, and turns + `built` right after `init-files`. If it says `pending` afterwards, the + checkpoint ran but did not mark it. +3. **`# 27 install directories are now root:install`** — not `0`. Zero is now + an error, but see it green rather than trusting it. +4. **`ls -ald /usr/src/pkgusr/p_*` after the first few packages** — each home + owned by its own user, not `root:root`. +5. **`/usr/src/pkgusr/p_gettext/src`** exists and is a directory, not a + symlink. +6. **The first config step** (`cfg_nsswitch`) comes out as `cfg_nsswitch` + under `/usr/src/cfg`, with no `p_` on it. +7. **A collector-group prompt** offers `<prefix>_perl`, never `<prefix>_p_perl`. +8. **`lfs-helper verify`** at the end: expect nothing, or a stale manifest + entry for a file two packages both install (`/usr/bin/hostname`). + +Then `bash lfs-sanity.sh` and read the `!!` lines. + +## Open items + +1. **NEXT JOB: `packagemanager setup`.** `last_build_step.sh` does two + unrelated things: a *bootstrap* (builds `wget`, installs seven Python wheels + so `lfs` and `blfs` can parse the books at all) and *configuration* (root + password, login account). Only the second belongs in a file whose header + says "EDIT THIS FILE. It is yours." The bootstrap should become + `packagemanager setup`: idempotent, re-runnable, testable — today it runs + exactly once, at the end of a six-hour build. + + Requirements to preserve: wheels not sdists (a modern sdist needs its build + backend and no index is reachable — `BackendUnavailable: Cannot import + 'hatchling.build'`); dependency order with `--no-deps`; `PIP_USER=0 + PYTHONNOUSERSITE=1`; each module as its own package user; site-packages + reached via `lfs-helper grant-dir` joining the collector group the directory + already carries, never the `install` group; and the `/etc/wgetrc` + `lfs-temporary-no-verify` marker, which is how the tools later find and + remove that weakening once `make-ca` exists. + + **Trap:** `lfs build-system get-sources` parses the `SOURCES=()` array + *out of `last_build_step.sh`*. If the wheel list moves, `get-sources` must + still find it. There is a test on that path: *the final step's SOURCES are + parsed (never executed)*. + +2. **`strip` is asked for but does nothing.** The answer is collected and the + prompt says NOT YET IMPLEMENTED. Not a book copy-paste: stripping rewrites + each binary, and here a file's owner is the record of which package + installed it, so it must run as that owner or the record drifts. Book 8.85 + also removes libtool `.la` files, which can make later BLFS builds link + against stale paths. + +3. **Steps 88-99 have never completed.** Steps 1-87 are well proven — the + sources/build split ran across all of them. The last twelve are all `cfg_*` + steps, and the fixes to them are unit-tested but have never run against the + real book. + +4. **Staging is limited to `--phase install`,** so `build-all` does not stage + gcc or glibc. `--phase all` runs the book's post-install commands in the + same script, and those touch live paths (`mv /usr/bin/chroot /usr/sbin`), + which fails with DESTDIR set. Only bites on rebuilds of a *running* system. + Deliberately deferred to item 6 — fixing it separately means restructuring + the build loop twice. + +5. **Root password** — the one decision the interview cannot hold; a plaintext + config would be worse. `last_build_step.sh` still prompts, and prints + `chroot /mnt/lfs /usr/bin/passwd root` when non-interactive. + +6. **`lfs-helper` and `packagemanager_install` are two implementations of one + thing.** Nearly every bug this project has had was the same species: one + decision held in several places, free to disagree — `cfg_bootscripts`, + `adopted.list`, the package-user prefix, the `/sources` permissions, the + account's kind. `install-as` is the entry point that makes unifying them + possible. Highest value, highest risk. + +7. **`migrate-layout`** — layout is configurable but only takes effect on a + fresh build. Moving an existing system's users is not written, and the + naming split widened the gap again: an existing tree holds + `p_cfg_bootscripts` accounts and directories. The tools *read* those + correctly — `pkgusr_kind` strips the package prefix before it looks, so a + legacy name resolves to the right kind and root — but nothing renames the + account or moves the directory. A fresh build is still the answer. + +8. ~~`README.md` is stale~~ — done, and kept current. Build ids are + deliberately not in it; they are in "Current state" above, which is the + file that gets handed over. + +9. **`lfs-sanity.sh` still reads the old state path on an old tree.** It + takes `LFS_PKGUSR_DIR`, so it is right on anything built by 1.9.0, but it + has no fallback for a tree built before the move. Harmless given that no + tree is migrated; worth deleting the ambiguity if migration is ever written. + +10. **A sweep for `2>/dev/null || true` is still overdue.** Six suppressed + errors were made visible in an earlier session and they were the cause of + nearly every bug that session. `ensure_pkgusr_roots` adds two more + suppressed `chown`/`chmod` calls — deliberately, because the roots being + wrong is not fatal and `verify --fix` repairs it, but it is the same + pattern and should be counted in the sweep. + + +## Useful commands + +```sh +lfs build-system verify # what state is this tree in? +lfs build-system run # continue from wherever it stopped +lfs snapshot save <n> --run # records the toolchain state in metadata +lfs-helper verify --fix # reconcile ownership +lfs-helper which-package <p> # which package installed this +lfs-helper install-as <n> -- <cmd> # install anything as its own package user +lfs-helper pkgusr-info # where each package came from +``` + + +## Tests + +```sh +bash test_lfs_crosschain.sh ./lfs # 569 tests, 568 pass +``` + +Counts are kept in files, not shell variables, so subshell increments survive. +A few Python blocks print several PASS lines but count once. + +"Nothing can fail silently" was not true: a test whose helper died before its +first assertion reported PASS. Use `_slice_fn` to lift a function out of a +script — never `sed -n "/^fn() {/,/^}/p"`, which cannot handle a one-liner. + +**Slice the whole function, never a fixed window.** Several tests used +`grep -B 14` or `sed -n '/marker/,+6p'` around a message. Adding a branch above +the marker pushed what they were asserting out of the window, and they failed +without anything being wrong. Use `_slice_fn`. + +**Do not define a function inside another one.** `_package_already_owns_files` +was first written inside `cmd_build`. bash accepts it -- and it is then defined +only after `cmd_build` has run once -- but every `sed -n '/^cmd_build() {/,/^}/p'` +in the suite stopped at the inner closing brace, silently truncating what those +tests examined. + +**Match invocations, not message text.** Several tests assert that a function +no longer calls something. The comment explaining *why* it no longer calls it +names the thing, so a naive grep matches the explanation and the test can never +pass. Strip comments first, or anchor on the call syntax. + +Run it from a directory holding ALL five scripts plus `last_build_step.sh` — +about 100 tests silently skip if the others are not beside `lfs`. + + +## Working style that worked + +- Diagnose from evidence in the output, not from the most recent theory. +- Every bug fixed gets a regression test with a comment naming what broke. +- Verify a new test **fails on the old code**. A test that cannot fail is + worse than no test. One nearly shipped because its slice boundary was wrong + and it reported four false failures; another shipped and sat green for + months, its four assertions never reached, because `sed -n "/^fn() {/,/^}/p"` + cannot slice a one-line function and the over-captured text died under + `set -u` before the first check. If a test has never been seen red, it has + not been tested. Use `_slice_fn`. +- **An empty result is not a pass.** Both of those failed the same way: a + helper was missing, the output was empty, and empty read as "no findings". + Assert the helper exists before trusting that it found nothing. +- Assert intent, not implementation. Three tests broke on the rename because + they grepped for literal paths rather than the property that mattered. +- Fix causes, not symptoms. Remove workarounds once the cause is fixed. +- Say plainly when something is uncertain or untested. +- **Before any restructuring, diff the command list before and after.** A + deleted command passes every test that does not mention it — that is how + `adopt-existing`, `sort-users` and `seal-install-dirs` silently disappeared. + + +## Known limits + +x86_64 only. Kernel and bootloader are left to the user. diff --git a/README.md b/README.md index da3f241..3d76163 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Build and maintain an LFS/BLFS system where **every package is owned by its own user**. > **100% vibecode, but tested.** Prompted into existence rather than hand -> written. 387 regression tests pass. It builds my own system — read it before +> written. 568 regression tests. It builds my own system — read it before > you point it at yours. --- @@ -106,6 +106,22 @@ once for your own account; every shared user then joins whatever is there. --- +## When ownership starts + +Book 7.6 creates `/etc/passwd`. Before that no name resolves, so ownership has +no meaning and nothing tries to set it. + +`init-ownership` is a step in the build, right after `init-files` (book 7.6). +It creates the install group, sets the install directories, gives package users +to everything already built, and adopts it. From there on every install is owned +by the package that made it. + +```sh +lfs-helper establish-ownership # by hand, or to check whether it happened +``` + +--- + ## Repair Everything is dry-run by default. `--run` applies. @@ -121,6 +137,45 @@ Everything is dry-run by default. `--run` applies. --- +## Names and places + +Three kinds of account share one passwd file, so each carries a prefix. +Without one a package called `man` or `news` collides with a real account. + +**One account, one prefix — its own.** + +| Thing | Looks like | Lives in | +|---|---|---| +| package user | `p_gcc` | `/usr/src/pkgusr/p_gcc` | +| config-step user | `cfg_bootscripts` | `/usr/src/cfg/cfg_bootscripts` | +| application user | `u_firefox` | `/usr/src/u_firefox` | + +There is no `tmp_` and no `init_`. A temporary step is the same package +chapter 8 rebuilds, so it is built as that package's user. An init step owns no +files and gets no account. + +Prefixes and locations are set once, in `lfs build-system session`. + +Sources, build trees and state are three different things: + +| Directory | Job | +|---|---| +| `$LFS/sources` | downloaded tarballs, never written by a build | +| `<package home>/src` | where that package unpacks and builds, owned by its user | +| `$LFS/build` | scratch for steps with no account. Deleted whole | + +Everything the build knows about the tree lives in `/usr/src/lfs-pkgusr`, +beside the accounts it describes: + +| | | +|---|---| +| `scripts/` | one shell script per build step | +| `manifests/` | what each package installed | +| `logs/` | one log per step, plus `verify.log` | +| `progress/` | what has been built, and how far | +| `groups/` | collector groups | +| `config/` | settings you may edit | + ## Conventions | Range | Used for | @@ -133,6 +188,10 @@ Everything is dry-run by default. `--run` applies. Install directories are group-writable during the build and become sticky (`o+t`) at the end, so packages can no longer overwrite each other. +The account roots are **not** install directories. Only root creates a home +there, so they stay `root:root 0755` — group-writable with no sticky bit would +let any package user delete another package's home. + --- ## Checking your version @@ -140,7 +199,7 @@ Install directories are group-writable during the build and become sticky The tools print a fingerprint of their own contents: ```sh -lfs --version # lfs 1.7.0 (build d64e55c) +lfs --version # lfs 1.7.1 (build 4f5f344) lfs-helper --version # inside the chroot ``` @@ -155,9 +214,12 @@ enter it. bash test_lfs_crosschain.sh ./lfs ``` -387 tests. Each encodes a bug that actually happened, with a comment explaining +568 tests. Each encodes a bug that actually happened, with a comment explaining what broke. +Run it from a directory holding all five scripts — about 100 tests skip +without them. + --- ## Caveats diff --git a/blfs b/blfs index 99ffc4a..f851ef0 100644 --- a/blfs +++ b/blfs @@ -118,7 +118,7 @@ CACHE_VERSION = 4 # tool + generated-script format versions. Bump SCRIPT_VERSION when the shape # of generated install scripts changes (phases, dispatcher, ...), and # CACHE_VERSION when the parsed/cached data structure changes. -BLFS_VERSION = "1.7.0" +BLFS_VERSION = "1.10.1" def _build_id(): diff --git a/collector-groups.export b/collector-groups.export new file mode 100644 index 0000000..9b6404b --- /dev/null +++ b/collector-groups.export @@ -0,0 +1,16 @@ +# lfs collector groups -- import with: lfs-helper import-groups <file> +# prefix|nimgnu +dir|/usr/lib/bfd-plugins|nimgnu_bfd-plugins +dir|/usr/lib/perl5/5.42/core_perl|nimgnu_perl +dir|/usr/lib/perl5/5.42/core_perl/ExtUtils|nimgnu_perl +dir|/usr/lib/perl5/5.42/site_perl|nimgnu_perl +dir|/usr/lib/python3.13/site-packages|nimgnu_python +dir|/usr/lib/cmake|nimgnu_cmake +dir|/usr/lib/udev/rules.d|nimgnu_udev +dir|/usr/share/zsh/site-functions|nimgnu_zsh +group|nimgnu_bfd-plugins|p_binutils,p_gcc +group|nimgnu_perl|p_perl,p_xml-parser +group|nimgnu_cmake|p_expat,p_openssl +group|nimgnu_python|p_python,p_flit-core,p_packaging,p_wheel,p_setuptools,p_meson,p_markupsafe,p_jinja2,urllib3,charset-normalizer,idna,certifi,soupsieve,requests,beautifulsoup4 +group|nimgnu_zsh|p_ninja,p_meson,p_kmod +group|nimgnu_udev|p_udev,p_e2fsprogs diff --git a/lfs b/lfs index 2b33f81..721b1a1 100644 --- a/lfs +++ b/lfs @@ -39,6 +39,7 @@ try: import readline # noqa: F401 except ImportError: pass +import stat import subprocess import sys import time @@ -59,7 +60,7 @@ except ImportError: " packagemanager pip install beautifulsoup4\n") sys.exit(2) -LFS_VERSION = "1.7.0" +LFS_VERSION = "1.10.1" def _build_id(): @@ -89,19 +90,61 @@ def _tty(): return sys.stderr.isatty() and os.environ.get("NO_COLOR") is None -_C_WARN = "\033[0;33m" # orange: something did NOT happen +def _tty_out(): + return sys.stdout.isatty() and os.environ.get("NO_COLOR") is None + + +# Four colours, one meaning each, matching lfs-helper exactly -- the two tools +# print into the same terminal one after the other, and a build log where the +# same colour means two different things is worse than no colour at all. +# +# green something finished, and finished correctly +# yellow something needs your attention -- including "nothing happened" +# red something failed +# dim detail you can skip: provenance, and what to do next +# +# Prose gets none. Colour that appears on every other line stops carrying +# information. +_C_WARN = "\033[0;33m" _C_ERR = "\033[0;31m" _C_OK = "\033[0;32m" +_C_DIM = "\033[2m" _C_OFF = "\033[0m" -def warn(msg): - """A dry run, a skip, a refusal -- anything the user might otherwise read - as 'done'. Coloured, because the whole point is that nothing happened.""" - if _tty(): - sys.stderr.write("%s%s%s\n" % (_C_WARN, msg, _C_OFF)) +def _paint(msg, colour, stream): + if (stream is sys.stderr and _tty()) or (stream is sys.stdout and _tty_out()): + stream.write("%s%s%s\n" % (colour, msg, _C_OFF)) else: - sys.stderr.write("%s\n" % msg) + stream.write("%s\n" % msg) + + +def warn(msg): + """Yellow, on stderr. A dry run, a skip, a refusal -- anything the reader + might otherwise take for 'done'. Coloured because the point is that + nothing happened.""" + _paint(msg, _C_WARN, sys.stderr) + + +def note(msg): + """Yellow, on stdout. Needs attention, but belongs in sequence with the + build output rather than beside it.""" + _paint(msg, _C_WARN, sys.stdout) + + +def ok(msg): + """Green. Something finished, and finished correctly.""" + _paint(msg, _C_OK, sys.stdout) + + +def fail(msg): + """Red, on stderr. Something failed.""" + _paint(msg, _C_ERR, sys.stderr) + + +def hint(msg): + """Dim. What to do next, or provenance -- never the finding itself.""" + _paint(msg, _C_DIM, sys.stdout) def dry_run_note(what="apply"): @@ -146,10 +189,6 @@ def _colour_epilog(text): return "\n".join(out) -def _tty_out(): - return sys.stdout.isatty() and os.environ.get("NO_COLOR") is None - - _C_HEAD = "\033[1m" # bold _C_GROUP = "\033[1;36m" # bold cyan -- the group you are scanning for _C_CMD = "\033[0;32m" # green -- the thing you actually type @@ -175,6 +214,52 @@ def _wrap(text, width): LFS_HANDOVER_DIRS = ("usr", "lib", "lib64", "var", "etc", "tools", "bin", "sbin") +# Where package users and application users live. +# +# TWO SETS, deliberately. The `target_*` values describe the system being +# BUILT and are written into its config; the plain ones describe THIS machine, +# if these tools also manage it. Without the split, configuring your own +# host's layout would silently change the layout of every system you build, +# which is exactly backwards -- the point of the build is that the new system +# is not this one. +# +# Defaults are the current layout, so nothing moves on an existing tree. Set +# them before a fresh build to get the tidier arrangement. +# Accounts live in subdirectories by kind, so `ls /usr/src` says what a thing +# IS rather than listing a hundred packages and a few config steps as one flat +# set: +# /usr/src/pkgusr/p_gcc a package +# /usr/src/cfg/cfg_bootscripts a config step that installs files +# /usr/src/u_firefox an application user +LAYOUT_DEFAULTS = { + # the system being built + "target_pkgusr_home": "/usr/src/pkgusr", + "target_cfguser_home": "/usr/src/cfg", + "target_appuser_home": "/usr/src", # or /home/shared_users + # this machine + "pkgusr_home": "/usr/src/pkgusr", + "cfguser_home": "/usr/src/cfg", + "appuser_home": "/usr/src", +} + + +def layout(key, target=False): + """One layout setting, target or host. + + `target=True` asks about the system being built, and falls back to the + host value only when no target value is set -- so a half-configured setup + behaves like the old one rather than mixing the two.""" + cfg = load_config() + if target: + v = cfg.get("target_" + key) + if v: + return v + v = cfg.get(key) + if v: + return v + return LAYOUT_DEFAULTS["target_" + key if target else key] + + def store_dir(): """One shared location so root and the lfs user see the SAME books. Override with $LFS_STORE.""" @@ -450,7 +535,13 @@ def parse_sections(soup): if cur is None: continue classes = tag.get("class") or [] - if "userinput" in classes: + # `userinput` is the usual class, but not the only one the book uses + # for commands. 9.9's `cat > /etc/shells` block is class "root", so + # cfg_shells resolved to ZERO commands and /etc/shells was never + # written -- silently, because an empty step looked like a finished + # one. "install" covers 9.4.1.2's optional udev rules. + # NOT "screen": that class is output samples (65 of them here). + if any(c in classes for c in ("userinput", "root", "install")): # Skip commands inside admonitions (note/tip/caution/warning). # Those are conditional examples, not part of the build -- GMP's # note shows "ABI=32 ./configure ..." for 32-bit hosts, and running @@ -587,6 +678,37 @@ def _tar_compressor(): return "--gzip", ".tar.gz" +def _toolchain_state(lfs): + """What state is the toolchain in, judged from the tree alone? + + Checkable from the host, without entering the chroot: the C++ headers must + exist under the target the tree was built for, and a NATIVE gcc directory + must not be present -- chapter 6 only ever installs under $LFS_TGT, so a + native one means a chapter-8 gcc was installed on top. + + Returns a short phrase for the snapshot metadata. + """ + import glob as _glob + tgt = lfs_tgt() + inc = _glob.glob(os.path.join(lfs, "usr", "include", "c++", "*")) + if not inc: + return "chapter 6 unfinished (no C++ headers)" + have = [os.path.basename(d) for p in inc + for d in _glob.glob(os.path.join(p, "*")) + if os.path.isfile(os.path.join(d, "bits", "c++config.h"))] + native = _glob.glob(os.path.join(lfs, "usr", "lib", "gcc", + "*-pc-linux-gnu")) + if native and tgt not in [os.path.basename(n) for n in native]: + return ("BROKEN: a native gcc (%s) is installed over the cross one" + % os.path.basename(native[0])) + if not have: + return "chapter 6 unfinished (no c++config.h)" + if tgt not in have: + return "BROKEN: headers are under %s, but LFS_TGT is %s" % ( + ", ".join(have), tgt) + return "ok (C++ headers under %s)" % tgt + + def cmd_snapshot(args): action = args.action or "list" if action == "list": @@ -603,6 +725,8 @@ def cmd_snapshot(args): print(" %-24s %s" % ("", info["note"])) if info.get("progress"): print(" %-24s %s" % ("", info["progress"])) + if info.get("toolchain"): + print(" %-24s %s" % ("", info["toolchain"])) print("\nRestore one with: %slfs snapshot restore <name> --run" % sudo_prefix()) return @@ -657,17 +781,26 @@ def cmd_snapshot(args): sys.exit(rc) prog = _load_crosschain_progress(lfs) or {} done = len(prog.get("done", [])) + state = _toolchain_state(lfs) meta = { "when": time.strftime("%Y-%m-%d %H:%M:%S"), "lfs": lfs, "book": build_version(), "note": args.note or "", + "toolchain": state, + "state": "; ".join("%s %s" % (k, v) for k, v in _tree_state(lfs) + if k != "toolchain"), "progress": "chapters 5-6: %d/%d steps" % (done, len(CROSSCHAIN_STEPS)) if prog else "", } with open(os.path.join(snapshots_dir(), name + ".json"), "w") as f: json.dump(meta, f, indent=2) print("\nsaved %s (%s)" % (name, _human_size(os.path.getsize(dest)))) + print(" toolchain: %s" % state) + if state.startswith("BROKEN"): + warn("\n! this snapshot is of a tree whose toolchain is already " + "broken.") + warn(" Restoring it later will restore the breakage too.") return if action == "restore": @@ -776,7 +909,7 @@ def cmd_reset(args): ] lfs_mount = load_config().get("lfs_mount") or os.environ.get("LFS") or "" if lfs_mount: - targets.append((os.path.join(lfs_mount, ".lfs-pkgusr"), + targets.append((os.path.join(lfs_mount, PKGUSR_DIR), "the in-chroot step list and generated scripts")) present = [(p, d) for p, d in targets if os.path.exists(p)] @@ -1047,6 +1180,13 @@ def _lfs_env_prefix(lfs): tgt = cfg.get("lfs_tgt") or "$(uname -m)-lfs-linux-gnu" mkf = cfg.get("makeflags") or "-j$(nproc)" return ( + # Book 4.4's ~/.bashrc opens with these two, and run_as_lfs embeds this + # prefix instead of sourcing that file -- so without them the build + # runs with bash's command hash on (stale paths to tools that have just + # been replaced) and with whatever umask root's shell had. On a host + # with umask 077 the toolchain installs unreadable. + f'set +h\n' + f'umask 022\n' f'export LFS="{lfs}"\n' f'export LC_ALL=POSIX\n' f'LFS_TGT={tgt}\n' @@ -1055,6 +1195,8 @@ def _lfs_env_prefix(lfs): f'if [ ! -L /bin ]; then PATH=/bin:$PATH; fi\n' f'PATH="$LFS/tools/bin:$PATH"\n' f'export PATH\n' + f'LFS_SRC_ROOT="{layout("pkgusr_home", target=True)}"\n' + f'export LFS_SRC_ROOT\n' f'CONFIG_SITE="$LFS/usr/share/config.site"\n' f'export CONFIG_SITE\n' f'export MAKEFLAGS="{mkf}"\n' @@ -1124,6 +1266,18 @@ _SESSION_KEYS = [ ("collector_prefix", "prefix for collector groups (packages that install " "into another package's directory join <prefix>_<owner>)", "sysgroup"), + # Three kinds of account share one passwd file: package users, application + # users (u_) and collector groups. Without a prefix a package called + # `man` or `news` collides with a real system account, and nothing in the + # passwd file says which accounts belong to the build. Blank = no prefix. + ("pkgusr_prefix", "prefix for package users (p gives p_gcc, p_zlib; " + "blank for none)", "p"), + # Config steps carry their OWN prefix, not the package one stacked on top + # of it. `p_cfg_bootscripts` wore both: `p_` claimed it was a package + # while the account lived in the config root, and every rule about + # prefixes had to special-case it. One account, one prefix. + ("cfguser_prefix", "prefix for config-step users (cfg gives " + "cfg_bootscripts; blank for none)", "cfg"), # Asked here with the rest of the session settings: a groups export from a # previous build saves answering the same sharing questions again, and it # has to be known BEFORE the chroot scripts are generated (that is when the @@ -1138,8 +1292,40 @@ _SESSION_KEYS = [ "%s-lfs-linux-gnu" % os.uname().machine), ("makeflags", "parallel jobs for builds", "-j%s" % (os.cpu_count() or 1)), ("hostname", "hostname for the built system", "lfs"), + ("target_pkgusr_home", + "where package users live on the BUILT system", + "/usr/src"), + ("target_appuser_home", + "where application users live on the BUILT system", + "/usr/src"), ("locale", "system locale (see 'locale -a')", "en_US.UTF-8"), ("paper_size", "paper size for groff: A4 or letter", "A4"), + # Everything below is a decision the build used to discover halfway + # through, or at the very end -- `last_build_step.sh` blocked on `passwd + # root` after a six-hour build, and its own comment admits the + # non-interactive path leaves the system "unbootable-but-known". Asking + # here is what lets `lfs run` and `build-all` finish without a person. + ("timezone", "time zone for the built system (e.g. Europe/Berlin, " + "or UTC)", "UTC"), + # Deleting a working cross-toolchain should never happen by default, so + # the default here is no. It is removed at the END of the build rather + # than at book 7.13, which keeps it available if chapter 8 has to be redone. + ("remove_tools", "delete /tools when the build finishes? It is the " + "cross-toolchain that built the system: 1-3 GB, and " + "your way back if chapter 8 needs redoing (yes/no)", + "no"), + # Off by default and deliberately so: stripping REWRITES each binary, and + # in this system a file's owner is the record of which package installed + # it, so stripping has to run as that owner or the record drifts. + ("strip", "strip debug symbols at the end? Saves several GB, " + "but rewrites every binary -- slower, and harder to " + "debug a crash afterwards. NOT YET IMPLEMENTED: the " + "answer is stored and nothing acts on it (yes/no)", + "no"), + # The account you will actually log in as. last_build_step.sh reads this; + # without it, a freshly booted system has only root, with no password. + ("main_user", "login account to create on the built system " + "(blank for none)", ""), ] @@ -1176,6 +1362,24 @@ def _prompt_session_config(cfg, force=False): val = "" else: val = os.path.abspath(p) + # Say what is actually in it. The prompt echoed a path and + # nothing else, so a file that was configured, present and + # never read looked exactly like one that was working. + try: + txt = open(val).read().splitlines() + except OSError: + txt = [] + n_dirs = sum(1 for l in txt if l.startswith("dir|")) + n_groups = sum(1 for l in txt if l.startswith("group|")) + if n_dirs or n_groups: + print(" %d directory decision(s), %d group(s) -- the " + "build will not ask about those again" + % (n_dirs, n_groups)) + else: + sys.stderr.write(" that file holds no group decisions " + "(no dir| or group| lines)\n") + print(" it is re-read on every chroot entry, so editing it " + "here reaches the next build") cfg[k] = val save_config(cfg) print(f" saved session config -> {config_path()}") @@ -1247,21 +1451,51 @@ def cmd_bs_layout(args): # so without this the tree stays root-owned and chapter 5 dies with # permission errors it blames on chapter 7 having already happened. _chown_tree_to_lfs(require_mounted_lfs()) + # and the scratch directory chapter 5 unpacks into: $LFS is root-owned, + # so the build user cannot create it itself + _ensure_build_root(require_mounted_lfs()) def _chown_tree_to_lfs(lfs, quiet=False): """Hand the build tree to the lfs user (book 4.3). A no-op if there is no - lfs user yet -- add-user calls this again once there is.""" + lfs user yet -- add-user calls this again once there is. + + /sources is NOT in the set. Book 4.3 chowns the handover directories and + nothing else; /sources is made world-writable in 3.1 instead, and its + contents are explicitly put back to root:root there. Chowning it to `lfs` + was writing a host uid into the tree -- see _normalize_sources. + """ try: import pwd pwd.getpwnam("lfs") except (KeyError, ImportError): return False - # the handover set, plus /sources which the lfs user keeps - dirs = [d for d in LFS_HANDOVER_DIRS + ("sources",) + dirs = [d for d in LFS_HANDOVER_DIRS if os.path.exists(os.path.join(lfs, d))] if not dirs: return False + # NOT on a tree that is past chapter 7. + # + # This is `chown -R lfs` over /usr, /etc, /var and the rest. Before + # chapter 5 that is book 4.3 and exactly right. On a BUILT tree it hands + # the finished system to the host's build account in one pass and every + # package's ownership is gone -- recoverable only from the manifests. + # + # It happened: `lfs build-system run` on a 104/104 tree replayed step 4.2, + # and /usr came back + # drwxrwxr-t 1 lfs install /mnt/lfs/usr + # with the host uid 10753 written through the tree, which inside the chroot + # resolves to nothing at all -- the chroot's own `lfs` is 9998. + # + # _handover_is_safe already answers the question, from the other side: it + # is True exactly while no package user owns anything. + if not _handover_is_safe(lfs): + print("# 4.3 SKIPPED: package users own files here, so this tree is") + print("# past chapter 7. `chown -R lfs` would hand the finished") + print("# system to the build user and lose every package's") + print("# ownership. Nothing was changed.") + print("# If you meant to start over: lfs build-system restart --run") + return False _run_bash('chown -R lfs "%s"/{%s} 2>/dev/null || true' % (lfs, ",".join(dirs))) if not quiet: @@ -1381,6 +1615,13 @@ def _verify_lfs_ownership(): if not bad: print(f"# ownership check: {lfs} build dirs belong to lfs -- OK") return + # Not on a tree that is past chapter 7. This runs before a chapter 5-6 + # build to make sure the lfs user can write; on a BUILT tree the same + # recursive chown destroys every package's ownership. + if not _handover_is_safe(lfs): + warn("# ownership check: package users own files here, so this tree is") + warn("# past chapter 7 -- not giving it back to the build user.") + return print(f"# ownership check: fixing {len(bad)} dir(s) still owned by root ...") for p in bad: _run_bash(f'chown -R lfs "{p}"') @@ -1444,6 +1685,11 @@ def cmd_bs_session(args): print(f" /sbin/swapon -v {cfg['lfs_swap']}") print(f" chown lfs (build dirs under {mount})") print(f"\nexport LFS={mount} # <-- run this in your shell") + print() + print("Next:") + print(f" {sudo_prefix()}lfs build-system session --run # apply the above") + print(f" {sudo_prefix()}lfs run # then build") + hint(f" {sudo_prefix()}lfs build-system next # just say what comes next") return require_lfs_for_root() @@ -1466,18 +1712,25 @@ def cmd_bs_session(args): os.path.join(mount, "home")], cwd="/") if cfg.get("lfs_swap"): subprocess.run(["/sbin/swapon", "-v", cfg["lfs_swap"]], cwd="/") - # re-apply the 4.3 chown so lfs owns the build tree (harmless if already so) - try: - import pwd - pwd.getpwnam("lfs") - _run_bash(f'chown -R lfs "{mount}"/{{usr,lib,lib64,var,etc,tools}} ' - f'2>/dev/null || true') - except (KeyError, ImportError): - pass - print(f"\nSession ready. $LFS={mount}") + # Re-apply the 4.3 chown through _chown_tree_to_lfs, NOT by hand. + # + # This was a second copy of `chown -R lfs` over the build tree, and it had + # neither of the things the real one grew: the refusal on a tree that is + # past chapter 7, and the same directory list. This copy also omitted bin + # and sbin, so the two disagreed about what "the build tree" even means. + # + # `session --reconfigure --run` on a finished system would have handed it + # to the build user in one pass -- the same damage the guard exists to + # prevent, through a door the guard was not on. + _chown_tree_to_lfs(mount, quiet=True) + + ok(f"\nSession ready. $LFS={mount}") print("Put this in your shell (root needs it too):") print(f" export LFS={mount}") - print("Then become the build user: su - lfs") + print() + print("Next:") + print(f" {sudo_prefix()}lfs run # build everything, step by step") + hint(f" {sudo_prefix()}lfs build-system next # just say what comes next") # ---- chapter 3: sources ---------------------------------------------------- # @@ -1526,11 +1779,17 @@ def cmd_bs_get_sources(args): urls = list(dict.fromkeys(urls + sorted(_book_patch_urls(args)))) lfs = require_mounted_lfs() if args.run else (_lfs_dir() or "$LFS") srcdir = os.path.join(lfs, "sources") - # get-sources usually runs BEFORE gen-chroot-scripts, so make sure the - # final-step script exists by now -- otherwise its SOURCES are silently - # skipped and the packages it needs are missing when it runs. - ensure_last_step_script() - extra = last_step_sources() + # Whatever the finished system needs before it can fetch anything itself. + # + # LFS ships no download tool, so a freshly booted system cannot get the + # sources for its own next package. `packagemanager setup` installs those, + # but the tarballs have to be here BEFORE the chroot is sealed -- and this + # is the only moment there is still a network. + # + # Resolved from the BLFS BOOK rather than a hardcoded list: the book already + # knows wget's URL and keeps it current, and a second list of download links + # is a second thing to go stale. + extra = bootstrap_sources() if extra: urls = list(urls) + [u for u in extra if u not in urls] npatch = sum(1 for u in urls if u.endswith(".patch")) @@ -1577,13 +1836,28 @@ def cmd_bs_get_sources(args): fail += 1 print(f"FAILED: {err}", flush=True) print(f"\ndownloaded {got}, skipped {skip}, failed {fail}") + # Book 3.1, second half: `chown root:root $LFS/sources/*`. Whatever ran + # the download owns the files, and that uid means nothing in the built + # system -- worse, it can collide with a real package user. Do it now, + # while the set of files is exactly what was just fetched. + _normalize_sources(lfs, run=True) md5 = md5sums_path(ver) if os.path.isfile(md5): import shutil shutil.copy(md5, os.path.join(srcdir, "md5sums")) print("verifying md5sums...", flush=True) - _run_bash(f'cd "{srcdir}" && md5sum -c md5sums | grep -v ": OK" || ' - f'echo "all md5sums OK"') + # `md5sum -c … | grep -v ": OK"` reported success on a CORRUPTED tree: + # grep inverts the verdict and the pipe throws away md5sum's status. + # Verified both ways -- rc was 0 either way. Check the status itself. + rc = _run_bash(f'cd "{srcdir}" && md5sum -c md5sums --quiet') + if rc != 0: + sys.stderr.write( + "\n! some downloads do not match their md5sum.\n" + " The lines above name them. A mirror returning an error\n" + " page with status 200 looks exactly like this.\n" + " Delete those files and run get-sources again.\n") + sys.exit(1) + print("all md5sums OK") if fail: sys.exit(1) @@ -1620,22 +1894,6 @@ CROSSCHAIN_STEPS = [ CROSSCHAIN_MARK = "### lfs-crosschain-script" -def _pkg_glob_for(pkgver, tarpfx): - """Candidate tarball patterns for a package. Book titles are capitalised - ("Python-3.13.7") and so is the real tarball, but plenty of packages ship - lowercase -- so try BOTH spellings rather than guessing (a lowercase-only - glob silently misses Python-3.13.7.tar.xz).""" - stems = [] - if pkgver: - stems.append(pkgver) - if pkgver.lower() != pkgver: - stems.append(pkgver.lower()) - else: - stems.append(f"{tarpfx}-*") - exts = ("tar.*", "tgz", "tbz2", "zip") - return " ".join(f"{st}.{e}" for st in stems for e in exts) - - def _title_pkgver(title): """The Name-Version token from a book section title. @@ -1723,8 +1981,39 @@ def _fill_placeholders(text, name): text = re.sub(r'^(<\d+\.\d+\.\d+\.\d+>.*)$', r'# \1 <-- uncomment and edit if this host has a static IP', text, flags=re.M) + # 8.5.2.2 ends with + # tzselect + # ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime + # tzselect is an INTERACTIVE menu whose only output is the name to put in + # <xxx>. We asked for the timezone in `session`, so both the menu and the + # placeholder are already answered -- running the menu inside a scripted + # build just blocks forever. + tz = cfg.get("timezone") or "UTC" + if re.search(r'^\s*tzselect\s*$', text, flags=re.M): + text = re.sub(r'^\s*tzselect\s*$', + '# tzselect -- answered by `lfs config timezone %s`' % tz, + text, flags=re.M) + text = text.replace('/usr/share/zoneinfo/<xxx>', '/usr/share/zoneinfo/%s' % tz) + + # The same section unpacks tzdata with a path relative to GLIBC's build + # directory: + # tar -xf ../../tzdata2025b.tar.gz + # Lifted out of glibc and run as its own step, ".." is somewhere else + # entirely and the tarball is simply not found. Take it from the sources + # directory by name, and unpack into scratch rather than beside it. + def _tzdata(m): + return ('_tzd="$(ls -1 "${SOURCES_DIR:-/sources}"/%s* 2>/dev/null ' + '| head -n1)"\n' + '[ -n "$_tzd" ] || { echo "no %s tarball in ' + '${SOURCES_DIR:-/sources}" >&2; exit 1; }\n' + '_tzw="${BUILD_ROOT:-/build}/tzdata"\n' + 'rm -rf "$_tzw"; mkdir -p "$_tzw"; cd "$_tzw"\n' + 'tar -xf "$_tzd"' % (m.group(1), m.group(1))) + text = re.sub(r'tar -xf \.\./(?:\.\./)?(tzdata)[^\s]*', _tzdata, text) + known = { "paper_size": cfg.get("paper_size") or "A4", + "xxx": tz, "lfs": cfg.get("hostname") or "lfs", "locale name": loc, "charmap": cfg.get("charmap") or "UTF-8", @@ -1851,7 +2140,8 @@ _TEST_DIAG_RE = re.compile( # test logs that do not exist and exits non-zero, and `set -e` then fails # the build -- reported as "phase 'build' FAILED" with make having # succeeded. - r'|\btestsuite\b|\btest_summary\b|^\s*ulimit\b') + r'|\btestsuite\b|\btest_summary\b|^\s*ulimit\b', + re.M) # Scaffolding that exists only so the test suite can run: the books create a # `tester` user and a throwaway group, hand the build tree to them, and tear it @@ -1888,6 +2178,21 @@ def _is_test_block(b): # the build silently, so such lines get an explicit `|| true`. _LONE_GREP_RE = re.compile(r'^\s*grep\b[^\n]*$') +# Greps that are the book's TOOLCHAIN GATE, not diagnostics. +# +# In 5.5 and 8.29 these must each produce output: no match means the toolchain +# is still linked against the host, and the book says to stop. Appending +# `|| true` to them -- as _soften_diagnostics does to every other lone grep -- +# turned the only real check in the build into a guaranteed pass. +_TOOLCHAIN_GATE_RE = re.compile( + r'crt\[1in\].*succeeded' + r'|\^ ?/?(usr/)?include' + r'|SEARCH.*/usr/lib' + r'|/lib.*/libc\.so\.6' + r'|^\s*grep\s+found\s+dummy\.log' + r'|grep\s+--version', + re.M) + # Re-running a phase must not fail just because it already did its job. The # books write `ln -sv target /usr/lib` with no -f, so the second attempt dies @@ -1909,7 +2214,11 @@ def _idempotent_links(block): def _soften_diagnostics(block): out = [] for ln in block.split("\n"): - if _LONE_GREP_RE.match(ln) and not ln.rstrip().endswith(("\\", "|| true")): + if _TOOLCHAIN_GATE_RE.search(ln): + # the book's toolchain gate: no match means the build is linked + # against the HOST, and it must fail loudly rather than pass + out.append(ln) + elif _LONE_GREP_RE.match(ln) and not ln.rstrip().endswith(("\\", "|| true")): out.append(ln.rstrip() + " || true # informational: no match is fine") else: out.append(ln) @@ -2010,10 +2319,48 @@ def _crosschain_script_body(name, sid, pkgver, glob, cmds): w('# so anything they reference must be in the ENVIRONMENT, not just a shell') w('# variable of this script.') w('export LFS') - w('BUILD_ROOT="${BUILD_ROOT:-$LFS/sources}"') - w('export BUILD_ROOT') + w('# Two directories, two jobs.') + w('# SOURCES_DIR the downloaded tarballs and patches. Written once by') + w('# get-sources, read-only to every build, root:root 1777.') + w('# BUILD_ROOT scratch: unpacked trees, build markers, anything a') + w('# book command drops beside the source. Deleted whole.') + w('# They used to be one directory, which is why a rebuild had to pick') + w('# stale markers and foreign ownership out of the tarball store.') + w('SOURCES_DIR="${SOURCES_DIR:-$LFS/sources}"') + w('BUILD_ROOT="${BUILD_ROOT:-$LFS/build}"') + w('export SOURCES_DIR BUILD_ROOT') w('pkg_dir=""') w("") + w("# Make BUILD_ROOT look like the sources directory to anything that") + w("# walks up out of an unpacked tree. The book writes commands like") + w("# tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1") + w("# from INSIDE the unpacked directory, so '..' has to hold the") + w("# downloaded files or the command fails.") + w("#") + w("# Link EVERY downloaded file rather than the ones some package is known") + w("# to reach for. A list of those would be a second thing to keep in") + w("# sync with the book, and it would be wrong the first time a new") + w("# release adds a package that does the same trick. Linking the lot") + w("# means any '../<anything-we-downloaded>' resolves, for any package,") + w("# in any future book -- and it costs a few hundred symlinks.") + w("_link_sources() {") + w('\t[ -d "$SOURCES_DIR" ] || return 0') + w('\tmkdir -p "$BUILD_ROOT" || return 1') + w('\tlocal f b') + w('\t# a link whose target went away (re-downloaded, renamed) would make') + w('\t# the glob in unpack_pkg match a file that cannot be read') + w('\tfor b in "$BUILD_ROOT"/*; do') + w('\t\t[ -L "$b" ] || continue') + w('\t\t[ -e "$b" ] || rm -f "$b"') + w('\tdone') + w('\tfor f in "$SOURCES_DIR"/*; do') + w('\t\t[ -f "$f" ] || continue') + w('\t\tb="$BUILD_ROOT/${f##*/}"') + w('\t\t# never shadow real content: an unpacked tree wins over a link') + w('\t\tif [ -e "$b" ] && [ ! -L "$b" ]; then continue; fi') + w('\t\tln -sfn "$f" "$b" 2>/dev/null || true') + w('\tdone') + w("}\n") w("# Run a phase body as its OWN process, from a FILE -- never piped in on") w("# stdin. A book command that reads stdin (Expect's PTY check does:") w("# python3 -c 'from pty import spawn; spawn([\"echo\", \"ok\"])' )") @@ -2062,6 +2409,7 @@ def _crosschain_script_body(name, sid, pkgver, glob, cmds): w("unpack_pkg() {") w("#### UNPACK ####") + w('\t_link_sources') w('\tcd "$BUILD_ROOT" || return 1') w('\t# Pick the source tarball. Two traps here:') w('\t# * do NOT use `ls $pkg_glob` -- patterns that match nothing make ls') @@ -2083,7 +2431,7 @@ def _crosschain_script_body(name, sid, pkgver, glob, cmds): w('\t\t\tpkg="$_c"; break') w('\t\tdone') w('\tfi') - w('\t[ -n "$pkg" ] || { echo "no $pkg_glob in $BUILD_ROOT -- run: ' + w('\t[ -n "$pkg" ] || { echo "no $pkg_glob in $SOURCES_DIR -- run: ' 'lfs build-system get-sources --run" >&2; return 1; }') w('\tdir=$(tar tf "$pkg" | head -n1 | cut -d/ -f1)') w('\t# --no-same-owner: as root, tar would restore the UIDs recorded IN the') @@ -2234,7 +2582,7 @@ def ensure_crosschain_script(args, ver, name, sid, tarpfx, force_regen=False, # ---- crosschain resume: remember progress across steps so a cancelled run can # continue where it left off instead of restarting from step 1 ---------------- def _crosschain_progress_path(lfs): - d = os.path.join(lfs, PKGUSR_DIR) + d = pkgusr_state(lfs, "progress") os.makedirs(d, exist_ok=True) return os.path.join(d, "crosschain-progress.json") @@ -2277,8 +2625,31 @@ def _mark_crosschain_done(lfs, ver, name): prog = {"book": ver, "done": []} done = [d for d in prog.get("done", []) if d != name] done.append(name) + # Record the target triplet with the progress. gcc-pass2 installs its + # headers under $LFS_TGT; change LFS_TGT afterwards and the compiler looks + # under the new name while the headers sit under the old one, which shows + # up only much later as + # fatal error: bits/c++config.h: No such file or directory with open(_crosschain_progress_path(lfs), "w") as f: - json.dump({"book": ver, "done": done}, f, indent=2) + json.dump({"book": ver, "done": done, "lfs_tgt": lfs_tgt()}, + f, indent=2) + + +def lfs_tgt(): + cfg = load_config() + return cfg.get("lfs_tgt") or "%s-lfs-linux-gnu" % os.uname().machine + + +def _crosschain_tgt_mismatch(lfs): + """Was the tree built with a different LFS_TGT than is configured now? + + Returns (recorded, current) when they differ, else None.""" + prog = _load_crosschain_progress(lfs) or {} + was = prog.get("lfs_tgt") + now = lfs_tgt() + if was and was != now: + return (was, now) + return None def _clear_crosschain_progress(lfs): @@ -2380,9 +2751,21 @@ def cmd_bs_fix_ownership(args): if os.geteuid() != 0: sys.stderr.write("! must be run as root.\n") sys.exit(2) + # The third copy of this chown, and the third guard. `fix-ownership` is a + # chapter 5-6 repair: it hands the tree to the build user so packages can + # install as `lfs`. After chapter 7 the packages ARE the users, and doing + # it would strip all of them in one pass. + if not _handover_is_safe(lfs): + sys.stderr.write( + "! not doing that: package users own files here, so this tree is\n" + " past chapter 7. chown -R lfs would strip every package's\n" + " ownership, and only the manifests could rebuild it.\n" + " To repair ownership on a built tree, inside the chroot:\n" + " lfs-helper verify --fix\n") + sys.exit(2) for p in targets: _run_bash(f'chown -R lfs "{p}"') - print("# ownership restored to lfs.") + ok("# ownership restored to lfs.") # Every user-settable key, with a description and how the default is derived. @@ -2401,7 +2784,7 @@ CONFIG_SECTIONS = [ ]), ("building a system: how it is built", [ "build_book", "lfs_tgt", "makeflags", "collector_prefix", - "collector_import_file", "last_step_script", + "collector_import_file", ]), ("the system being built: how it is set up", [ "hostname", "locale", "charmap", "paper_size", "locales", "network", @@ -2415,10 +2798,16 @@ CONFIG_SECTIONS = [ # a machine afterwards. Printing them in one flat list made it impossible to # tell which setting affected which -- and "set-default" ambiguous. CONFIG_SECTIONS = [ + ("layout of the system being built", [ + "target_pkgusr_home", "target_appuser_home", + ]), + ("layout of this machine", [ + "pkgusr_home", "appuser_home", + ]), ("building a system", [ "build_book", "lfs_mount", "lfs_device", "lfs_fstype", "lfs_swap", "lfs_home_device", "lfs_tgt", "makeflags", "bootloader", "esp", - "last_step_script", "snapshot_dir", + "snapshot_dir", ]), ("the system being built (its settings, not yours)", [ "locale", "charmap", "paper_size", "hostname", "network", @@ -2449,8 +2838,14 @@ CONFIG_KEYS = [ ("editor", "editor for --edit; default: $EDITOR or vim"), ("sudo", "prefix suggested root commands with 'sudo' " "(on/off; default off)"), - ("bootloader", "which bootloader to set up: refind (default), grub, " - "or none. GRUB's book instructions run " + ("bootstrap_packages", + "BLFS packages the built system needs before it can " + "fetch anything at all (default: wget). Their tarballs " + "are downloaded by get-sources, while there is still a " + "network; `packagemanager setup` installs them on the " + "booted system. Blank for none"), + ("bootloader", "which bootloader to set up: none (default), refind, " + "or grub. GRUB's book instructions run " "'grub-install /dev/sda', which writes to a disk's boot " "sector -- with 'refind' that section is skipped " "entirely and a rEFInd script is generated instead"), @@ -2471,11 +2866,6 @@ CONFIG_KEYS = [ "message catalogues (space separated; English is " "always kept). Used by 'lfs-helper prune-locales'. " "Default: en de"), - ("last_step_script", "your own script, run as the LAST step of the build " - "(default /etc/lfs/last_build_step.sh). Its SOURCES " - "array is downloaded by get-sources, and the rest runs " - "as root in the chroot after every book package. It " - "ships with wget, since LFS has no download tool"), ("network", "how the built system gets its network: dhcp " "(default) or static. With dhcp no static interface or " "resolv.conf files are generated -- you install a DHCP " @@ -2532,13 +2922,59 @@ def network_mode(): def bootloader(): - return (load_config().get("bootloader") or "refind").lower() + """Which bootloader step to generate: none (default), refind, or grub. + + NONE by default, and that is the important part. Most people running this + already have a working EFI partition and a bootloader that finds it -- the + machine booted in order to build LFS at all. Generating a rEFInd step for + them turns the last stretch of a six-hour build into a stop: + + == rEFInd needs you to do something first == + /boot/efi does not exist, so there is nowhere to install to. + + which is a question nobody asked to be asked. A bootloader is a decision + about the whole machine, not a package, so it is opt-in: say so in + `session`, or `lfs config bootloader refind` later. + """ + return (load_config().get("bootloader") or "none").lower() def collector_prefix(): return (load_config().get("collector_prefix") or "sysgroup").rstrip("_") +def pkgusr_prefix(): + """Prefix on package users: `p_` gives p_gcc. Blank means none. + + `lfs-helper` has always READ this (falling back to `p`), but until now + nothing ever wrote it, so the default applied by accident and + `packagemanager` on the built system did not implement it at all -- the + build would create p_gcc and the built system would look for gcc. + Writing it here is what makes the three tools agree. + + An explicitly empty value means "no prefix" and is not the same as unset. + """ + cfg = load_config() + v = cfg.get("pkgusr_prefix", "p") if "pkgusr_prefix" in cfg else "p" + v = (v or "").rstrip("_") + return (v + "_") if v else "" + + +def cfguser_prefix(): + """Prefix on config-step users: `cfg_` gives cfg_bootscripts. + + Its own prefix, not the package one on top of it. Written into the chroot + env for the same reason as pkgusr_prefix: three tools deciding an account's + name separately is how they end up building two parallel sets of accounts. + + An explicitly empty value means "no prefix" and is not the same as unset. + """ + cfg = load_config() + v = cfg.get("cfguser_prefix", "cfg") if "cfguser_prefix" in cfg else "cfg" + v = (v or "").rstrip("_") + return (v + "_") if v else "" + + def sudo_prefix(): """'sudo ' or '' -- some systems (and root shells) have no sudo at all, so suggested commands are shown bare unless you turn this on.""" @@ -2546,6 +2982,31 @@ def sudo_prefix(): return "sudo " if v in ("1", "on", "true", "yes") else "" +def remove_tools(): + """Delete /tools at the end of the build (book 7.13)? + + Default off. The book removes it as soon as chapter 7 is over, but /tools + is the cross-toolchain that BUILT chapter 8 -- keeping it costs 1-3 GB and + buys a way back if chapter 8 has to be redone. Deleting it at the END + rather than at 7.13 keeps that fallback for the part of the build most + likely to need it. + """ + v = str(load_config().get("remove_tools", "")).lower() + return v in ("1", "on", "true", "yes") + + +def strip_binaries(): + """Strip debug symbols at the end of the build (book 8.85)? + + Default off, and deliberately so. Stripping REWRITES each binary, so in a + package-user system it has to run as the file's owner or ownership drifts + away from the package that installed it -- which is the one thing this + whole scheme exists to keep true. + """ + v = str(load_config().get("strip", "")).lower() + return v in ("1", "on", "true", "yes") + + def cmd_config(args): """Show or change every setting, and show where things live on disk.""" cfg = load_config() @@ -2673,6 +3134,153 @@ def _owner_is_root(lfs): return False +# Package users start here; anything below is the host's own accounts. +PKG_UID_MIN = 10000 + + +def _find_uid(lfs, expr, limit=1): + """First path under the handover trees matching a uid expression. + + `find ... -print -quit` stops at the first hit, so this costs almost + nothing even on a full /usr. + """ + import subprocess + dirs = [os.path.join(lfs, d) for d in LFS_HANDOVER_DIRS + if os.path.exists(os.path.join(lfs, d))] + if not dirs: + return None + try: + out = subprocess.run(["find"] + dirs + ["-xdev"] + expr + + ["-print", "-quit"], + capture_output=True, text=True, timeout=120) + return (out.stdout or "").strip() or None + except (OSError, subprocess.SubprocessError): + return None + + +def _build_user_uid(): + """The HOST's lfs account, by number. + + Book 4.3's useradd pins no uid, so this is whatever the host had free -- + 10753 on one Debian machine. The chroot has its own `lfs` at 9998, so the + same file reads as two different owners depending on which side you are on, + and inside the chroot the host's uid resolves to no name at all. Only the + number is meaningful across that boundary. + """ + try: + import pwd + return pwd.getpwnam("lfs").pw_uid + except (KeyError, ImportError): + return None + + +def _handover_build_user_only(lfs, run=True): + """Give back exactly what the BUILD USER holds, and nothing else. + + Book 7.2 is `chown -R root:root` over eight trees, and on a chapter-8 tree + that is destructive -- it strips every package user in one pass. So the + recursive form is refused there, which left a tree with no way forward at + all: 4.3 had taken /usr, /etc, /var and the root symlinks, 7.2 would not + give them back, and `chroot enter` refused because /usr was not root's. + + This is the precise version. Selecting on the build user's uid touches + what 4.3 took and leaves every package user's files alone, so it is safe at + any stage of the build. + + `chown -h root`, user only: 4.3 chowned the user only, so the inverse does + too. Taking the group as well would strip `install` from every shared + directory and undo the ownership epoch's work. And `-h`, because /bin, + /lib and /sbin are symlinks -- following them would retarget /usr/bin. + """ + uid = _build_user_uid() + if uid is None: + return 0 + dirs = [os.path.join(lfs, d) for d in LFS_HANDOVER_DIRS + if os.path.exists(os.path.join(lfs, d))] + if not dirs: + return 0 + import subprocess + try: + out = subprocess.run(["find"] + dirs + ["-xdev", "-uid", str(uid)], + capture_output=True, text=True, timeout=600) + paths = [p for p in (out.stdout or "").splitlines() if p] + except (OSError, subprocess.SubprocessError): + return 0 + if not paths: + return 0 + if run: + _run_bash('find %s -xdev -uid %d -exec chown -h root {} + 2>/dev/null' + % (" ".join('"%s"' % d for d in dirs), uid)) + return len(paths) + + +def _handover_needed(lfs): + """Is anything in the handover trees still owned by the BUILD user? + + `_owner_is_root` alone decided this, and it looks at one inode -- + $LFS/usr. The pass it guards is `chown -h -R root:root` over eight trees. + So a tree whose /usr was already root-owned got no handover at all, while + everything chapters 5-6 created below it kept the host uid: + + drwxr-xr-x 1 lfs lfs /usr/share/gcc-15.2.0/python/libstdcxx + lfs:install drwxrwxr-x /usr/share + lfs:install drwxrwxr-x /lib64 + + Ask the tree, not one directory in it. + """ + if not _owner_is_root(lfs): + return True + # Anything not owned by root. NOT "a uid outside the package range" -- + # that was the first attempt and it was wrong for the oldest reason in this + # project: book 4.3's useradd pins no uid, so the host's `lfs` account lands + # wherever the host had a gap. On a Debian host that was 10753, which is + # inside the package-user range, so the filter excluded the very user it was + # looking for. The tree then reported "7.2 already done" while + # lrwxrwxrwx 1 lfs lfs /bin -> usr/bin + # was still sitting there. _handover_is_safe already establishes that no + # package owns anything, so here anything non-root is the build user's. + return _find_uid(lfs, ["!", "-uid", "0"]) is not None + + +def _handover_is_safe(lfs): + """True while no package user owns anything. + + `chown -R root:root` over $LFS/usr is exactly right before chapter 8 and + catastrophic after it: it would strip every package user's ownership in one + pass and there is no record left to rebuild it from. + + Asked of the TREE's own passwd, not of a uid range. A uid range cannot tell + a package user from the host's build account -- book 4.3 pins no uid for + `lfs`, so on one host it came out 10753, inside the package range. The + tree's passwd is unambiguous: these are the accounts that exist in there. + """ + users = _pkg_users_in_tree(lfs) + if not users: + return True # no package user exists; nothing to strip + for uid in users: + if _find_uid(lfs, ["-uid", str(uid)]): + return False + return True + + +def _pkg_users_in_tree(lfs): + """uids of the package users in the TREE's passwd, by name prefix.""" + pfx = pkgusr_prefix() + out = [] + try: + with open(os.path.join(lfs, "etc", "passwd")) as f: + for line in f: + parts = line.split(":") + if len(parts) > 2 and (not pfx or parts[0].startswith(pfx)): + try: + out.append(int(parts[2])) + except ValueError: + pass + except OSError: + return [] + return out + + def _owner_name_of(path): """Who owns this, by name where possible. @@ -2708,25 +3316,119 @@ CHROOT_INIT_STEPS = [ ("init-files", "ch-tools-createfiles"), ] +def _auto_file_block(soup_or_raw, sid, dest_path): + """Some sections ship a FILE, not commands. + + Book 9.6.8 (The rc.site File) has no `<kbd class="command">` at all -- it + prints /etc/sysconfig/rc.site as a `<pre class="auto">` block with every + line commented out. A reader is meant to copy it and uncomment what they + want. Because the parser only harvests command blocks, cfg_rcsite + resolved to zero blocks and the file was simply never created. + + An all-commented file is still worth installing: it is the documentation + of every knob the boot scripts have, in the place the boot scripts look. + Without it you have to go back to the book to find out what exists. + + Returns a single shell command that writes the file, or None. + """ + import re as _re + raw = soup_or_raw if isinstance(soup_or_raw, str) else str(soup_or_raw) + i = raw.find('id="%s"' % sid) + if i < 0: + return None + # stop at the next section so a later <pre> cannot be picked up by mistake + j = _re.search(r'<div class="(?:sect1|sect2)\b', raw[i + 1:]) + seg = raw[i:i + 1 + j.start()] if j else raw[i:i + 40000] + m = _re.search(r'<pre[^>]*class="auto"[^>]*>(.*?)</pre>', seg, _re.S) + if not m: + return None + import html as _html + body = _html.unescape(_re.sub(r'<[^>]+>', '', m.group(1))).strip("\n") + if not body.strip(): + return None + # A quoted heredoc: the body is full of #comments and shell metacharacters + # that must land in the file exactly as the book prints them. + return ('mkdir -pv %s\ncat > %s << "EOF"\n%s\nEOF' + % (os.path.dirname(dest_path), dest_path, body)) + + +# Sections that ship a file body instead of commands: name -> (section, path). +CONFIG_FILE_SECTIONS = { + "cfg_rcsite": ("ch-config-site", "/etc/sysconfig/rc.site"), +} + + +def _resolve_section(secs, sid): + """Find a book section by id, falling back to its title. + + Most sections carry a stable id (`ch-config-hostname`), but some carry an + id the toolchain GENERATED when the book was built: + + idm139921653990176 8.5.2.1. Adding nsswitch.conf + idm139921653984688 8.5.2.2. Adding Time Zone Data + + Those change every time the book is regenerated -- including between point + releases of the same edition -- so an entry that names one works exactly + once and then silently resolves to nothing. Silently is the problem: + `/etc/shells` went missing from finished systems because cfg_shells + resolved to zero blocks and was skipped without a word. + + So a step may name either an id or a title fragment. A fragment is matched + case-insensitively against section titles, and must match exactly ONE + section -- an ambiguous fragment is a mistake in the table, not something + to resolve by picking the first. + """ + if sid in secs: + return secs[sid] + want = sid.lower() + hits = [(k, v) for k, v in secs.items() if want in v[0].lower()] + if len(hits) == 1: + return hits[0][1] + if len(hits) > 1: + warn(" '%s' matches %d sections (%s) -- make it more specific" + % (sid, len(hits), ", ".join(h[1][0][:28] for h in hits[:3]))) + return None + + +# Chapter 8 configuration the book performs mid-chapter, between packages. +# +# These are files, not software: nothing built in chapter 8 reads them, so +# running them in the configuration phase with the rest is equivalent and +# keeps all configuration in one place. +# +# Two of the three sit in sub-sections whose ids are generated, so they are +# named by TITLE -- see _resolve_section. +CHROOT_CONFIG_STEPS_8 = [ + # glibc leaves /etc/nsswitch.conf absent; without it name resolution falls + # back to compiled-in defaults and getent behaves inconsistently. + ("cfg_nsswitch", "Adding nsswitch.conf"), + # /etc/localtime -- without it the system runs UTC and every log line, + # timestamp and `date` is offset. + ("cfg_timezone", "Adding Time Zone Data"), + # The glibc step already does `touch /etc/ld.so.conf`; this is the rest of + # 8.5.2.3, which matters as soon as anything installs outside /usr/lib. + ("cfg_ld", "conf-ld"), +] + # Chapter 9: configuring the system. These are root housekeeping steps that # write config files -- no package is built -- so they are generated as plain # root scripts, like 7.5 and 7.6. Several ask for a decision (hostname, network, # locale), so they are placed after the packages and flagged where the book # leaves a placeholder for you to fill in. CHROOT_CONFIG_STEPS_9 = [ - ("cfg-bootscripts", "ch-config-bootscripts"), - ("cfg-network", "ch-config-network"), - ("cfg-resolv", "resolv.conf"), - ("cfg-hostname", "ch-config-hostname"), - ("cfg-hosts", "ch-config-hosts"), - ("cfg-sysvinit", "conf-sysvinit"), - ("cfg-clock", "ch-config-clock"), - ("cfg-console", "ch-config-console"), - ("cfg-sysklogd", "ch-config-sysklogd"), - ("cfg-rcsite", "ch-config-site"), - ("cfg-locale", "ch-config-locale"), - ("cfg-inputrc", "ch-config-inputrc"), - ("cfg-shells", "ch-config-shells"), + ("cfg_bootscripts", "ch-config-bootscripts"), + ("cfg_network", "ch-config-network"), + ("cfg_resolv", "resolv.conf"), + ("cfg_hostname", "ch-config-hostname"), + ("cfg_hosts", "ch-config-hosts"), + ("cfg_sysvinit", "conf-sysvinit"), + ("cfg_clock", "ch-config-clock"), + ("cfg_console", "ch-config-console"), + ("cfg_sysklogd", "ch-config-sysklogd"), + ("cfg_rcsite", "ch-config-site"), + ("cfg_locale", "ch-config-locale"), + ("cfg_inputrc", "ch-config-inputrc"), + ("cfg_shells", "ch-config-shells"), ] CHROOT_STEPS_7 = [ @@ -2747,7 +3449,8 @@ def write_chroot_env(lfs, quiet=False): cfg = load_config() tgt = cfg.get("lfs_tgt") or f"{os.uname().machine}-lfs-linux-gnu" mkf = cfg.get("makeflags") or "-j$(nproc)" - d = os.path.join(lfs, PKGUSR_DIR) + # config/, because it holds settings; lfs-helper sources it from there. + d = pkgusr_state(lfs, "config") os.makedirs(d, exist_ok=True) path = os.path.join(d, "env") with open(path, "w") as f: @@ -2761,43 +3464,151 @@ def write_chroot_env(lfs, quiet=False): f'export LFS_HOST_MOUNT="{lfs}"\n' f'# prefix for collector groups -- see `lfs config`\n' f'export LFS_COLLECTOR_PREFIX="{collector_prefix()}"\n' + f'# prefix for package users. lfs-helper has always READ this;\n' + f'# nothing wrote it, so the default applied by accident.\n' + f'export LFS_PKGUSR_PREFIX="{pkgusr_prefix()}"\n' + f'# prefix for config-step users. Its own, NOT the package\n' + f'# prefix stacked on top -- one account, one prefix.\n' + f'export LFS_CFGUSR_PREFIX="{cfguser_prefix()}"\n' + f'# where accounts live, by kind -- see LAYOUT_DEFAULTS\n' + f'export LFS_PKGUSR_ROOT="{layout("pkgusr_home", target=True)}"\n' + f'export LFS_CFGUSR_ROOT="{layout("cfguser_home", target=True)}"\n' f'# languages to keep -- see `lfs-helper prune-locales`\n' - f'export LFS_LOCALES="{cfg.get("locales") or "en de"}"\n') + f'export LFS_LOCALES="{cfg.get("locales") or "en de"}"\n' + f'# answers collected by `lfs build-system session`, so the\n' + f'# chroot side never has to stop and ask. last_build_step.sh\n' + f'# reads LFS_MAIN_USER instead of blocking on a prompt after\n' + f'# a six-hour build.\n' + f'export LFS_TIMEZONE="{cfg.get("timezone") or "UTC"}"\n' + f'export LFS_MAIN_USER="{cfg.get("main_user") or ""}"\n' + f'export LFS_STRIP="{"1" if strip_binaries() else "0"}"\n' + f'export LFS_REMOVE_TOOLS="{"1" if remove_tools() else "0"}"\n') _make_world_readable(path) if not quiet: print(f"# build env -> {path} (LFS_TGT={tgt}, MAKEFLAGS={mkf})") return path -def _find_helper_source(): - """The lfs-helper script shipped next to this tool.""" +def _find_helper_source(name="lfs-helper"): + """One of our own scripts, shipped next to this tool.""" here = os.path.dirname(os.path.abspath(__file__)) - for cand in (os.path.join(here, "lfs-helper"), - os.path.join(here, "..", "lfs-helper"), - "/usr/bin/lfs-helper", "/usr/share/lfs/lfs-helper"): + for cand in (os.path.join(here, name), + os.path.join(here, "..", name), + "/usr/bin/" + name, "/usr/share/lfs/" + name): if os.path.isfile(cand): return os.path.abspath(cand) return None +# Our own scripts, copied into the tree beside lfs-helper. +# +# Only lfs-helper used to go in, because only lfs-helper runs during the build. +# But the built system needs the rest -- `packagemanager` cannot be installed +# by `packagemanager` -- and putting them in at build time is free: they are a +# few files, they refresh on every `lfs run`, and a system that boots without +# them has no way to install anything at all. +# +# They cannot be USED until the last build step provides Python and the two +# book-parsing modules. That is fine: they are there, and `packagemanager +# setup` completes them. +TOOLCHAIN_SCRIPTS = ("lfs-helper", "packagemanager", "packagemanager_install", + "blfs", "lfs") + +# The package user that owns them. +# +# They are software installed into /usr/bin like anything else, so they get an +# owner like anything else -- otherwise they show up forever as +# 1516 root-owned file(s) that NO manifest claims: +# /usr/bin/lfs-helper +# /usr/bin/list_package +# which is true, unhelpful, and buries the cases that matter. One user for the +# whole repo: they are one thing, versioned and installed together. +PKGUSR_TOOLS_USER = "pkgusr" + + def install_helper(lfs, quiet=False): - """Put lfs-helper inside the LFS tree so it's on PATH in the chroot.""" - src = _find_helper_source() - if not src: - if not quiet: - sys.stderr.write("! lfs-helper not found next to this tool -- the " - "chroot won't have it.\n") - return None + """Put our scripts inside the LFS tree so they are on PATH in the chroot. + + Called on every `lfs run` and before entering the chroot, so the copies + track the host's -- a stale chroot copy running old code has cost more + debugging time here than any other single thing, which is why each tool + prints a build id. + """ + import shutil dest_dir = os.path.join(lfs, "usr", "bin") os.makedirs(dest_dir, exist_ok=True) - dest = os.path.join(dest_dir, "lfs-helper") - import shutil - shutil.copy(src, dest) - os.chmod(dest, 0o755) + helper_dest, copied, missing = None, [], [] + + for name in TOOLCHAIN_SCRIPTS: + src = _find_helper_source(name) + if not src: + missing.append(name) + continue + dest = os.path.join(dest_dir, name) + shutil.copy(src, dest) + os.chmod(dest, 0o755) + copied.append(name) + if name == "lfs-helper": + helper_dest = dest + + # Give them an owner. The account may not exist yet -- chapter 7 creates + # /etc/passwd -- so this is best-effort now and repeated on the next run, + # by which time it will exist. Failing to chown is not worth stopping a + # build over, but silence is: that is how a whole tree came out root-owned. + _chown_tools(lfs, dest_dir, copied, quiet=quiet) + if not quiet: - print(f"# installed lfs-helper -> {dest} (available as 'lfs-helper' " - f"in the chroot)") - return dest + if copied: + print("# installed %s -> %s (on PATH in the chroot)" + % (", ".join(copied), dest_dir)) + for name in missing: + sys.stderr.write("! %s not found next to this tool -- the chroot " + "won't have it.\n" % name) + if helper_dest is None and not quiet: + sys.stderr.write("! lfs-helper is missing: the chroot cannot build.\n") + return helper_dest + + +def _chown_tools(lfs, dest_dir, names, quiet=False): + """Hand our scripts to their package user inside the tree.""" + if not names: + return + user = pkgusr_prefix() + PKGUSR_TOOLS_USER + uid = _uid_in_tree(lfs, user) + if uid is None: + if not quiet: + print("# (%s does not exist yet -- ownership set on a later run)" + % user) + return + n = 0 + for name in names: + p = os.path.join(dest_dir, name) + try: + os.chown(p, uid, uid) + n += 1 + except OSError as e: + sys.stderr.write("! could not give %s to %s: %s\n" % (p, user, e)) + if n and not quiet: + print("# gave %d tool(s) to '%s'" % (n, user)) + + +def _uid_in_tree(lfs, user): + """Look a user up in the TREE's passwd file, not this machine's. + + The host has its own accounts and they mean nothing inside the chroot; + reading the host's would write a host uid into the tree, which is exactly + the mistake book 3.1 warns about for /sources. + """ + pw = os.path.join(lfs, "etc", "passwd") + try: + with open(pw) as f: + for line in f: + parts = line.split(":") + if len(parts) > 2 and parts[0] == user: + return int(parts[2]) + except (OSError, ValueError): + pass + return None def _chrootify(text): @@ -2805,8 +3616,10 @@ def _chrootify(text): final system is simply '/', so $LFS must not be required (the cross-build guard would refuse to run), sources live at /sources, and there's no cross-compiler prefix to put on PATH.""" - text = text.replace('BUILD_ROOT="${BUILD_ROOT:-$LFS/sources}"', - 'BUILD_ROOT="${BUILD_ROOT:-/sources}"') + text = text.replace('SOURCES_DIR="${SOURCES_DIR:-$LFS/sources}"', + 'SOURCES_DIR="${SOURCES_DIR:-/sources}"') + text = text.replace('BUILD_ROOT="${BUILD_ROOT:-$LFS/build}"', + 'BUILD_ROOT="${BUILD_ROOT:-/build}"') text = text.replace('export LFS\n', 'export LFS="${LFS:-/}"\n') # drop the "refusing to guess without $LFS" guard: correct outside, wrong here text = re.sub(r'if \[ -z "\$LFS" \]; then\n.*?\nfi\n', '', text, count=1, @@ -2916,7 +3729,14 @@ suggest_esp() {{ fi if [ -n "$found" ]; then echo " FAT partitions on this machine (your ESP is almost certainly one):" >&2 - printf '%s\n' "$found" | sed 's|^| /dev/|' >&2 + # The two sources disagree about the prefix: `lsblk -rno NAME` gives a + # bare name (nvme0n1p1) while `blkid -o device` gives a full path + # (/dev/nvme0n1p1). Prefixing both produced + # /dev//dev/nvme0n1p1 + # in the one message whose whole job is to tell you what to type. + # Add /dev/ only where it is missing. + printf '%s\n' "$found" \ + | sed -e 's|^\([^/]\)|/dev/\1|' -e 's|^| |' >&2 else echo " Could not list partitions from in here. Run this OUTSIDE the chroot:" >&2 echo " lsblk -o NAME,SIZE,FSTYPE,PARTTYPENAME,MOUNTPOINT" >&2 @@ -3161,14 +3981,37 @@ def _install_collector_groups(lfs): "exist\n" % src) return import shutil - dst_dir = os.path.join(lfs, PKGUSR_DIR) + # groups/, where lfs-helper looks for it. These two paths are the whole + # mechanism: `lfs` puts the file where `lfs-helper` reads it, and nothing + # else connects them. When the state directory was sorted in 1.9.0 this + # one kept writing to the top level while the reader moved into groups/, + # so a rebuild with a perfectly good export asked + # 'p_gcc' needs to install into: /usr/lib/bfd-plugins + # Which group should share it? + # for a directory the file already answered. + dst_dir = pkgusr_state(lfs, "groups") os.makedirs(dst_dir, exist_ok=True) dst = os.path.join(dst_dir, "collector-groups.import") + # Only say anything when it CHANGES. This runs on every chroot entry, so + # that an edit on the host reaches the build without regenerating the + # scripts -- but three unchanging lines on every entry is noise, and noise + # is what stopped anyone noticing the file was going to the wrong place. + same = (os.path.isfile(dst) + and open(dst, "rb").read() == open(src, "rb").read()) shutil.copy(src, dst) _make_world_readable(dst) - print("# collector groups -> %s" % dst) - print("# import them inside the chroot with:") - print("# lfs-helper import-groups --run") + if same: + return + n_dirs = n_groups = 0 + for line in open(dst): + if line.startswith("dir|"): + n_dirs += 1 + elif line.startswith("group|"): + n_groups += 1 + ok("# collector groups: %d directory decision(s), %d group(s) from %s" + % (n_dirs, n_groups, src)) + hint("# the build will not ask about those directories again") + hint("# apply them now with: lfs-helper import-groups --run") def _gen_blfs_extra(outdir, order, unresolved): @@ -3236,312 +4079,60 @@ def _gen_blfs_extra(outdir, order, unresolved): return written -LAST_STEP_DEFAULT = r'''#!/bin/bash -# last_build_step.sh -- the final step of an LFS build. -# -# EDIT THIS FILE. It is yours: whatever a freshly built system should have -# that the book does not provide goes here, and it runs as the last step of -# `lfs-helper build-all`. -# -# * SOURCES below is read by `lfs build-system get-sources`, so anything you -# add is downloaded along with the book's own packages. -# * The rest runs as root inside the chroot, after every book package. -# * It is copied into the chroot by `lfs build-system gen-chroot-scripts`, -# so edit it HERE (on the host) and regenerate -- not in the chroot. -# -# It ships with wget because LFS has no download tool at all: without one a -# freshly booted system cannot fetch even the sources for its next package. - -# --- extra sources -------------------------------------------------------- # -# Downloaded into $LFS/sources by `lfs build-system get-sources`. -SOURCES=( - # wget, so the new system can fetch anything at all - "https://ftpmirror.gnu.org/wget/wget-1.25.0.tar.gz" - - # What `lfs` and `blfs` need to parse the books. WHEELS, not source - # tarballs: a modern sdist needs its build backend (beautifulsoup4 wants - # hatchling, requests wants setuptools) and with no package index reachable - # pip cannot fetch one -- - # BackendUnavailable: Cannot import 'hatchling.build' - # A wheel is already built, so pip just unpacks it. Both are pure Python, - # so the "any" wheel works on every architecture. - "https://files.pythonhosted.org/packages/py3/r/requests/requests-2.32.3-py3-none-any.whl" - "https://files.pythonhosted.org/packages/py3/b/beautifulsoup4/beautifulsoup4-4.12.3-py3-none-any.whl" - # urllib3, charset-normalizer, idna and certifi are what requests imports - "https://files.pythonhosted.org/packages/py3/u/urllib3/urllib3-2.2.3-py3-none-any.whl" - "https://files.pythonhosted.org/packages/py3/c/charset_normalizer/charset_normalizer-3.4.0-py3-none-any.whl" - "https://files.pythonhosted.org/packages/py3/i/idna/idna-3.10-py3-none-any.whl" - "https://files.pythonhosted.org/packages/py3/c/certifi/certifi-2024.8.30-py3-none-any.whl" - "https://files.pythonhosted.org/packages/py3/s/soupsieve/soupsieve-2.6-py3-none-any.whl" -) - -# --- the work ------------------------------------------------------------- # -set -e -# Build one of the SOURCES tarballs as its own package user. -# build_as_package_user <user> <tarball-glob> <configure options...> -build_as_package_user() { - local user="$1" glob="$2"; shift 2 - local src="/sources" - - local tarball="" - local c - for c in $src/$glob; do [ -e "$c" ] && { tarball="$c"; break; }; done - if [ -z "$tarball" ]; then - echo "!! no $glob in $src -- add it to SOURCES and run:" >&2 - echo " lfs build-system get-sources --run" >&2 - return 1 - fi - - lfs-helper add-user "$user" >/dev/null 2>&1 || true - - local dir - dir="$(tar tf "$tarball" | head -n1 | cut -d/ -f1)" - rm -rf "${src:?}/${dir:?}" - tar --no-same-owner -xf "$tarball" -C "$src" - chown -R "$user:$user" "$src/$dir" - - echo "# building $user from $(basename "$tarball") as package user '$user'" - su - "$user" -c "cd '$src/$dir' && ./configure $* && make && make install" - - # record what was installed, the same way lfs-helper does - printf '%s\n# installed %s by last_build_step.sh\n' \ - "$dir" "$(date '+%Y-%m-%d %H:%M:%S')" > "/usr/src/$user/VERSION" \ - 2>/dev/null || true - chown "$user:$user" "/usr/src/$user/VERSION" 2>/dev/null || true -} -# Install a Python module as ITS OWN package user, the same as any other -# package. requests and beautifulsoup4 are what `lfs` and `blfs` need to parse -# the books, so without them the tools cannot run on the new system. -# Let a package user install into site-packages. -# -# site-packages belongs to the python package and is shared through a collector -# group. Ask lfs-helper for access, exactly as a failed build would: it joins -# the collector group the directory already carries, or asks which group should -# own it. It must NOT be handed to the install group -- that means "anyone may -# install here", and site-packages is python's tree, not shared infrastructure. -_grant_site_packages() { - local user="$1" sp - sp="$(python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' \ - 2>/dev/null)" - if [ -z "$sp" ] || [ ! -d "$sp" ]; then - echo "note: cannot find site-packages -- skipping $user" >&2 - return 1 - fi - lfs-helper grant-dir "$sp" "$user" --run -} +def bootstrap_packages(): + """BLFS packages the built system needs before it can fetch anything. -install_python_module() { - local user="$1" module="$2" glob="$3" - local src="/sources" wheel="" c - for c in $src/$glob; do [ -e "$c" ] && { wheel="$c"; break; }; done - if [ -z "$wheel" ]; then - echo "note: no $glob in $src -- skipping $module" >&2 - echo " add it to SOURCES and run: lfs build-system get-sources --run" >&2 - return 0 - fi - command -v python3 >/dev/null 2>&1 || { - echo "note: no python3 yet -- skipping $module" >&2; return 0; } - - lfs-helper add-user "$user" >/dev/null 2>&1 || true - chown "$user:$user" "$wheel" 2>/dev/null || true - _grant_site_packages "$user" || return 0 - - echo "# installing $module from $(basename "$wheel") as package user '$user'" - # A wheel needs no build backend and no network: pip unpacks it. - # PIP_USER=0 / PYTHONNOUSERSITE keep it out of the package user's ~/.local, - # where nothing else would ever find it. - su - "$user" -c "PIP_USER=0 PYTHONNOUSERSITE=1 \ - python3 -m pip install --no-index --no-deps '$wheel'" \ - || echo "!! $module did not install -- do it later with: - packagemanager pip install $module" >&2 -} + Just wget by default: it is the one thing LFS leaves out that makes a fresh + system unable to download its own next package. `packagemanager setup` + installs it; this is only about having the tarball on disk in time. + """ + v = load_config().get("bootstrap_packages") + if v is None: + return ["wget"] + return [p for p in str(v).replace(",", " ").split() if p] -# wget: the one thing a new system cannot bootstrap without. -# Deliberately minimal -- reinstall it properly once the system boots: -# packagemanager install wget --recursive --run -build_as_package_user wget "wget-*.tar.*" \ - --prefix=/usr --sysconfdir=/etc --with-ssl=openssl - -# The two Python modules the tools themselves need. Each gets its own package -# user, exactly like every other package -- there is no "last-step" package and -# nothing here is owned by one. -# Dependencies first -- --no-deps means pip will not pull them in itself. -install_python_module urllib3 urllib3 "urllib3-*.whl" -install_python_module charset-normalizer charset-normalizer "charset_normalizer-*.whl" -install_python_module idna idna "idna-*.whl" -install_python_module certifi certifi "certifi-*.whl" -install_python_module soupsieve soupsieve "soupsieve-*.whl" -install_python_module requests requests "requests-*.whl" -install_python_module beautifulsoup4 beautifulsoup4 "beautifulsoup4-*.whl" - -# --- wget without certificates -------------------------------------------- # -# A freshly built LFS has no CA certificates -- they come from BLFS' make-ca -- -# so every HTTPS fetch fails with "cannot verify ... certificate". That is a -# chicken-and-egg problem: you need wget to fetch make-ca. -# -# Turn verification off in /etc/wgetrc rather than with a wrapper script in -# $PATH. wget reads /etc/wgetrc no matter how it is invoked or what PATH the -# caller has, so this works for builds run through `su -` with the package -# user's own environment -- where a $PATH wrapper is simply never seen. -# -# This is a deliberate, TEMPORARY weakening: downloads are unverified until -# make-ca is installed. The marker below is how the tools find and remove it. -disable_wget_verification() { - local rc=/etc/wgetrc - grep -q 'lfs-temporary-no-verify' "$rc" 2>/dev/null && return 0 - cat >> "$rc" <<'WRC' - -# --- lfs-temporary-no-verify --------------------------------------------- -# This system has no CA certificates yet, so wget cannot verify anything. -# Downloads are UNVERIFIED until you install make-ca: -# packagemanager install make-ca --recursive --run -# The tools remove these two lines automatically once certificates exist. -check_certificate = off -# --- end lfs-temporary-no-verify ----------------------------------------- -WRC - echo "# certificate checking disabled in $rc until make-ca is installed" - echo "# (downloads are UNVERIFIED until then -- the tools re-enable it" - echo "# automatically as soon as certificates exist)" -} -disable_wget_verification - -# --- add your own steps below --------------------------------------------- # -# e.g. a package the book does not carry, a config file, a user account. - -# --- accounts you can actually log in with -------------------------------- # -# A freshly built system has root with NO password. Depending on the login -# manager that is either "anyone can log in as root" or "nobody can log in at -# all" -- and it is discovered at the worst moment, after a reboot, with no -# way in. Ask now, while there is still a working shell here. -set_up_login_accounts() { - # No terminal (a scripted or resumed run): say what is missing rather than - # blocking, and leave the system unbootable-but-known instead of hanging. - if [ ! -t 0 ]; then - echo - echo "!! Not running interactively, so no passwords were set." - echo " Before rebooting, set at least the root password:" - echo " chroot /mnt/lfs /usr/bin/passwd root" - return 0 - fi +def bootstrap_sources(): + """Their download URLs, asked of the BLFS book. - echo - echo "--- accounts ------------------------------------------------" + `blfs` runs on the host and keeps its own book store, so this works before + the chroot exists. Asking the book rather than hardcoding URLs means there + is one place that knows where wget comes from, and it is the same place + `packagemanager` will ask when it installs it. + """ + import subprocess + urls = [] + for pkg in bootstrap_packages(): + try: + out = subprocess.run(["blfs", "sources", pkg], + capture_output=True, text=True, timeout=120) + except (OSError, subprocess.SubprocessError): + warn("! could not run `blfs sources %s` -- is blfs installed?" % pkg) + continue + got = [l.strip() for l in (out.stdout or "").splitlines() + if l.strip().startswith(("http://", "https://", "ftp://"))] + if not got: + warn("! the BLFS book gave no download for '%s'." % pkg) + warn(" A fresh system will have no way to fetch anything. Either") + warn(" fetch a book (blfs fetch) or drop it:") + warn(" lfs config bootstrap_packages ''") + continue + urls += got + return urls - # root - if grep -qE '^root:[^:]*:' /etc/shadow 2>/dev/null && - ! grep -qE '^root:(\*|!|)?:' /etc/shadow 2>/dev/null; then - echo "root already has a password." - else - echo "root has NO password yet. Without one you may not be able to" - echo "log in after rebooting." - passwd root || echo "!! setting the root password failed -- do it before rebooting" >&2 - fi - # a login account for a person: the name is already configured, so this - # only has to create it and set a password - local main - main="$(sed -n 's/^main_user=\(.*\)$/\1/p' /etc/pkgusr/packagemanager.conf \ - 2>/dev/null | head -n1)" - if [ -z "$main" ]; then - echo - echo "No login account is configured. You can add one now, or later" - echo "with: useradd -m -G users <name> && passwd <name>" - printf "Create a login account now? [name, or blank to skip]: " - read -e -r main - fi - [ -n "$main" ] || { echo; return 0; } - if id "$main" >/dev/null 2>&1; then - echo "Account '$main' already exists." - else - echo "Creating login account '$main' ..." - useradd -m -k /etc/skel -G users -s /bin/bash "$main" \ - || { echo "!! could not create $main" >&2; return 0; } - fi - if grep -qE "^$main:(\*|!|)?:" /etc/shadow 2>/dev/null; then - echo "Set a password for '$main':" - passwd "$main" || echo "!! setting the password for $main failed" >&2 - fi - # so the account can reach the application users created later - echo - echo " '$main' can be given access to application users with:" - echo " packagemanager user create <app> --shared --launcher" -} -set_up_login_accounts - -echo -echo "==============================================================" -echo " The base system is built." -echo -echo " wget was installed with a minimal configure line so the system" -echo " can fetch things at all. Reinstall it properly -- with its" -echo " dependencies, from the BLFS book -- once you have booted:" -echo -echo " packagemanager install wget --recursive --run" -echo -echo " Anything else you want on a fresh system belongs in" -echo " last_build_step.sh on the host." -echo "==============================================================" -''' -def last_step_path(): - """Where the user's final-step script lives ON THE HOST.""" - p = load_config().get("last_step_script") - if p: - return os.path.expanduser(p) - return "/etc/lfs/last_build_step.sh" -def ensure_last_step_script(verbose=True): - """Create the final-step script if it does not exist yet.""" - path = last_step_path() - if os.path.isfile(path): - return path - try: - os.makedirs(os.path.dirname(path), exist_ok=True) - with open(path, "w") as f: - f.write(LAST_STEP_DEFAULT) - os.chmod(path, 0o755) - if verbose: - print("# created %s -- edit it to change what a finished system" - " gets" % path) - except OSError as e: - sys.stderr.write("! could not create %s: %s\n" % (path, e)) - return None - return path -def last_step_sources(): - """The SOURCES array from the final-step script. - Parsed rather than executed: this file is meant to be edited freely, and - running it just to read a list of URLs would be a poor trade.""" - path = last_step_path() - if not os.path.isfile(path): - return [] - try: - text = open(path).read() - except OSError: - return [] - m = re.search(r'^SOURCES=\(\s*(.*?)^\)', text, re.M | re.S) - if not m: - return [] - urls = [] - for line in m.group(1).splitlines(): - line = line.strip() - if not line or line.startswith("#"): - continue - for tok in re.findall(r'"([^"]+)"|\'([^\']+)\'|(\S+)', line): - u = tok[0] or tok[1] or tok[2] - if u.startswith(("http://", "https://", "ftp://")): - urls.append(u) - return urls def _write_tool_stamps(lfs, tools): @@ -3775,6 +4366,155 @@ def cmd_bs_run(args): sys.exit(1) +# --------------------------------------------------------------------------- # +# $LFS/sources: ONE convention, and it is the book's own +# --------------------------------------------------------------------------- # +# Book 3.1 says exactly two things about it: +# +# mkdir -v $LFS/sources +# chmod -v a+wt $LFS/sources # root:root, 1777 +# chown root:root $LFS/sources/* # the CONTENTS too +# +# and gives the reason in full: +# +# "The file system records the owner by its UID, and the UID of a normal +# user in the host distro is not assigned in LFS. So the files will be +# left owned by an unnamed UID in the final LFS system." +# +# That is not hypothetical here, it is worse than the book's version. The host +# `lfs` account is created by book 4.3's own `useradd`, which does not pin a +# uid, so it gets whatever the host had free -- 10753 on a Debian box. Package +# users in the built system start at PKG_UID_MIN=10000. A host uid in that +# range does not merely show as a number inside the chroot: it collides with a +# REAL package user, and every tarball in /sources reports as owned by, say, +# `iana-etc`. Ownership is this scheme's source of truth, so that is a lie the +# tools will act on. +# +# Three rules had drifted apart, and the mixture is what you see on disk: +# * `_chown_tree_to_lfs` chowned /sources to `lfs` (host uid); +# * lfs-helper's install_dirs_list carried /sources, so `verify --fix` made +# it root:install 775 and the seal made it 1775 -- dropping the o+w the +# book asks for, which is what lets the unprivileged build user unpack; +# * cmd_clean_sources reset unknown owners to root:root. +# +# So: root:root, 1777, contents root:root. World-writable is what grants the +# build user access, so no group is needed and none is set. /sources is +# scratch, NOT an install directory. +def _ensure_build_root(lfs, quiet=False): + """Create $LFS/build -- the scratch directory builds unpack into. + + Same permissions as /sources and for the same reason: chapters 5-6 run on + the host as the unprivileged `lfs` user and chapters 7+ run as root or as + package users, so every one of them has to be able to create files here. + World-writable is what grants that; the sticky bit is what stops one + package user deleting another's tree. No group is involved. + + $LFS itself is root-owned, so the build user cannot create this -- it has + to exist before chapter 5 starts. + """ + p = os.path.join(lfs, "build") + try: + os.makedirs(p, exist_ok=True) + os.chown(p, 0, 0) + os.chmod(p, stat.S_ISVTX | 0o777) + except OSError as e: + sys.stderr.write(" could not create %s: %s\n" % (p, e)) + return None + if not quiet: + print("# build scratch: %s (root:root 1777)" % p) + return p + + +def _normalize_sources(lfs, run=True, quiet=False): + """Apply book 3.1 to $LFS/sources. Returns (n_chowned, mode_fixed).""" + srcdir = os.path.join(lfs, "sources") + if not os.path.isdir(srcdir): + return 0, False + mode_fixed = False + try: + st = os.stat(srcdir) + want = stat.S_ISVTX | 0o777 + if stat.S_IMODE(st.st_mode) != want or st.st_uid != 0 or st.st_gid != 0: + mode_fixed = True + if run: + os.chown(srcdir, 0, 0) + os.chmod(srcdir, want) + except OSError as e: + sys.stderr.write(" could not set %s to root:root 1777: %s\n" + % (srcdir, e)) + + # `chown root:root $LFS/sources/*` -- non-recursive, exactly as the book + # writes it. Directories are skipped: at book 3.1 nothing is unpacked yet, + # so the glob matches only downloaded files and this is equivalent there. + # Later it is not -- an unpacked tree belongs to whoever is building it, + # and taking it away mid-build would stop that build. Those trees are + # scratch: deleted and re-extracted, never corrected. + n = 0 + try: + names = os.listdir(srcdir) + except OSError: + return n, mode_fixed + for name in names: + p = os.path.join(srcdir, name) + try: + st = os.lstat(p) + except OSError: + continue + if stat.S_ISDIR(st.st_mode): + continue + if st.st_uid == 0 and st.st_gid == 0: + continue + n += 1 + if run: + try: + os.chown(p, 0, 0, follow_symlinks=False) + except OSError as e: + sys.stderr.write(" could not chown %s: %s\n" % (p, e)) + n -= 1 + if n and not quiet: + print("# gave %d file(s) in %s back to root:root (book 3.1)" + % (n, srcdir)) + return n, mode_fixed + + +def _restart_source_debris(srcdir): + """Anything in $LFS/sources that is not a downloaded file. + + With the build scratch split out into $LFS/build there should be nothing + here at all: no unpacked trees, no `.cc-*` markers, no files owned by the + last build. This is the check that says so, and it is deliberately not a + cleanup pass for old layouts -- a tree built by an earlier version is + rebuilt, not repaired. + + It stays because the guarantee is worth asserting: /sources holds only + what get-sources downloaded, so a rebuild cannot inherit anything from the + build before it. + + Returns (markers, foreign) as sorted absolute paths. + """ + markers, foreign = [], [] + if not os.path.isdir(srcdir): + return markers, foreign + try: + names = os.listdir(srcdir) + except OSError: + return markers, foreign + for n in names: + p = os.path.join(srcdir, n) + if n.startswith(".cc-"): + markers.append(p) + continue + try: + st = os.lstat(p) + except OSError: + continue + if stat.S_ISDIR(st.st_mode): + continue + if st.st_uid != 0 or st.st_gid != 0: + foreign.append(p) + return sorted(markers), sorted(foreign) + + def cmd_bs_restart(args): """Throw away the built system and start the build again from scratch. @@ -3797,6 +4537,18 @@ def cmd_bs_restart(args): sys.exit(2) keep_sources = not args.wipe_sources + + # Unpacked source trees are build scratch: a rebuild re-extracts every one + # of them, and leaving hundreds of thousands of stale files behind makes + # every later scan (ownership, orphans, manifests) slow and noisy. The + # TARBALLS are kept -- those are the expensive part. + # $LFS/build is scratch and is deleted whole -- it is in `doomed` below + # like any other top-level directory, so nothing here has to pick unpacked + # trees or markers out of the tarball store. That is the entire point of + # the split. /sources should hold only what get-sources downloaded; the + # survey asserts it rather than repairing an older layout. + srcdir = os.path.join(lfs, "sources") + markers, foreign = _restart_source_debris(srcdir) if keep_sources else ([], []) entries = [] try: entries = sorted(os.listdir(lfs)) @@ -3823,10 +4575,33 @@ def cmd_bs_restart(args): if not doomed: print(" (nothing -- the tree is already empty)") print() + if markers: + print(" from %s (build markers -- this tree predates the " + "sources/build split):" % srcdir) + for p in markers[:6]: + print(" %s" % os.path.basename(p)) + if len(markers) > 6: + print(" ... and %d more" % (len(markers) - 6)) + print(" Markers belong in %s now. The build rewrites them." + % os.path.join(lfs, "build")) + print() + + if foreign: + print(" in %s: %d file(s) not owned by root -> root:root (book 3.1)" + % (srcdir, len(foreign))) + for p in foreign[:6]: + print(" %-40s (%s)" + % (os.path.basename(p), _owner_name_of(p))) + if len(foreign) > 6: + print(" ... and %d more" % (len(foreign) - 6)) + print(" A host uid means nothing in the built system, and one") + print(" above 10000 collides with a real package user.") + print() + print(" kept:") if keep_sources: - print(" sources the downloaded tarballs " - "(--wipe-sources removes these too)") + print(" the downloaded tarballs in %s" % srcdir) + print(" (--wipe-sources removes those too)") print(" the books, your settings and any snapshots (outside the tree)") print() print(" Package users and collector groups live in %s/etc, so they go" @@ -3879,6 +4654,28 @@ def cmd_bs_restart(args): sys.stderr.write(" could not remove %s: %s\n" % (p, err)) print("\nremoved %d item(s) from %s" % (removed, lfs)) + if keep_sources: + markers, _ = _restart_source_debris(srcdir) + n = 0 + for p in markers: + try: + os.remove(p) + n += 1 + except OSError as err: + sys.stderr.write(" could not remove %s: %s\n" % (p, err)) + if n: + print("removed %d build marker(s); the next build writes its own" + % n) + + # Book 3.1: /sources is root:root 1777 and so are its contents. The + # tarballs survived the wipe, so their ownership must be reset here or + # the fresh build inherits the previous system's uids. + n, mode_fixed = _normalize_sources(lfs, run=True, quiet=True) + if n: + print("gave %d source file(s) back to root:root (book 3.1)" % n) + if mode_fixed: + print("reset %s to root:root 1777 (book 3.1)" % srcdir) + # the build's own progress lives outside the tree as well for f in ("crosschain",): d = os.path.join(store_dir(), f) @@ -3892,6 +4689,74 @@ def cmd_bs_restart(args): next_step("%slfs build-system run" % sudo_prefix(), label="start again with:") +def _tree_state(lfs): + """Everything worth knowing about a tree, as (label, value) pairs. + + Judged from the tree itself, from the host, without entering the chroot. + This exists because a snapshot labelled "chapters 5-6: 22/22" said nothing + about chapter 8 -- so a tree that had already been damaged by a chapter-8 + build looked pristine, and restoring it restored the damage. + """ + import glob as _glob + out = [] + + prog = _load_crosschain_progress(lfs) or {} + done = [d for d in prog.get("done", [])] + out.append(("chapters 5-6", + "%d of %d steps" % (len(done), len(CROSSCHAIN_STEPS)))) + if prog.get("lfs_tgt"): + out.append(("built for", prog["lfs_tgt"])) + + chroot_done = _chroot_progress(lfs) + order = _chroot_steporder(lfs) + if chroot_done or order: + out.append(("chapters 7-9", + "%d of %d steps" % (len(chroot_done), len(order) or 98))) + if chroot_done: + out.append(("last built", chroot_done[-1])) + + out.append(("toolchain", _toolchain_state(lfs))) + + # package users are the clearest sign chapter 8 actually ran + try: + n = len([d for d in os.listdir(os.path.join(lfs, "usr", "src")) + if os.path.isdir(os.path.join(lfs, "usr", "src", d))]) + out.append(("package users", "%d" % n)) + except OSError: + pass + + native = _glob.glob(os.path.join(lfs, "usr", "lib", "gcc", "*-pc-linux-gnu")) + if native: + out.append(("WARNING", + "a native gcc is present (%s) -- chapter 8 has run" + % os.path.basename(native[0]))) + return out + + +def cmd_bs_verify(args): + """What state is this tree in? + + Answers the question a snapshot label cannot: how far did the build get, + for which target, and is the toolchain usable.""" + lfs = require_mounted_lfs() + print("State of %s\n" % lfs) + for label, value in _tree_state(lfs): + marker = " ! " if label == "WARNING" else " " + if label == "WARNING": + warn("%s%s" % (marker, value)) + else: + print(" %-16s %s" % (label + ":", value)) + print() + st = dict(_tree_state(lfs)).get("toolchain", "") + if st.startswith("BROKEN") or "unfinished" in st: + warn("This tree cannot build chapter 8 as it stands.") + print(" A snapshot taken now would capture that -- check with:") + print(" %slfs snapshot list" % sudo_prefix()) + else: + ok_ = " The toolchain looks usable." + print(ok_) + + def cmd_bs_set_book(args): """Choose the book the SYSTEM is built from (config key build_book). @@ -4069,12 +4934,16 @@ def _fstab_script(cfg): def cmd_bs_gen_chroot_scripts(args): require_complete_config() """Write phased install scripts for the in-chroot packages (chapter 7's - remainder and chapter 8) into $LFS/.lfs-pkgusr/scripts, where lfs-helper + remainder and chapter 8) into $LFS/usr/src/lfs-pkgusr/scripts, where lfs-helper runs them. Same phased format as the packagemanager scripts, so packagemanager can reuse them to update these packages later.""" ver, path = resolve_book(args) lfs = require_mounted_lfs() soup = load_soup(path) + # The parser harvests command blocks only. A few sections ship a FILE + # instead (9.6.8 prints rc.site as <pre class="auto">), so keep the raw + # text for _auto_file_block. + book_raw = str(soup) secs = {sid: (title, sec) for sid, title, sec in iter_sections(soup)} steps = list(CHROOT_INIT_STEPS) + list(CHROOT_STEPS_7) @@ -4093,7 +4962,7 @@ def cmd_bs_gen_chroot_scripts(args): steps = [(n, sid) for n, sid in steps if "grub" not in n.lower() and "grub" not in sid.lower()] - outdir = os.path.join(lfs, PKGUSR_DIR, "scripts") + outdir = pkgusr_state(lfs, "scripts") print(f"{len(steps)} script(s) -> {outdir}") if not args.run: for name, sid in steps[:10]: @@ -4136,22 +5005,38 @@ def cmd_bs_gen_chroot_scripts(args): written += 1 # chapter 9 configuration, after all the packages pkg_like_cfg = [] - cfg_steps_9 = list(CHROOT_CONFIG_STEPS_9) + cfg_steps_9 = list(CHROOT_CONFIG_STEPS_8) + list(CHROOT_CONFIG_STEPS_9) if network_mode() == "dhcp": # The book's network section writes a static interface config and a # hand-made resolv.conf. With DHCP both are wrong -- the client # supplies them -- so don't generate scripts that would overwrite what # dhcpcd sets up. cfg_steps_9 = [(n, sid) for n, sid in cfg_steps_9 - if n not in ("cfg-network", "cfg-resolv")] + if n not in ("cfg_network", "cfg_resolv")] if not args.chapter7_only: secs_all = secs for cname, csid in cfg_steps_9: - entry = secs_all.get(csid) + entry = _resolve_section(secs_all, csid) + # A named step that produces nothing looked exactly like one that + # worked. /etc/shells was missing from finished systems for + # exactly this reason: cfg_shells resolved to zero blocks and was + # skipped without a word. Say so -- some of these are genuinely + # prose (cfg_rcsite), but silence cannot tell the two apart. if not entry: + warn(f" {cname}: section '{csid}' not found in this book " + f"-- nothing generated") continue ctitle, csec = entry - if not csec["commands"]: + ccmds = csec["commands"] + if not ccmds and cname in CONFIG_FILE_SECTIONS: + # no commands, but the section prints a file -- write that + _fsid, _fpath = CONFIG_FILE_SECTIONS[cname] + _blk = _auto_file_block(book_raw, _fsid, _fpath) + if _blk: + ccmds = [_blk] + if not ccmds: + warn(f" {cname}: '{csid}' has no command blocks in this book " + f"-- nothing generated") continue dest = os.path.join(outdir, f"{cname}.sh") if os.path.isfile(dest) and not args.overwrite: @@ -4185,13 +5070,13 @@ def cmd_bs_gen_chroot_scripts(args): # /etc/fstab, before the bootloader -- the system cannot boot read-write # without it, and the bootloader step is where booting starts to matter if not args.chapter7_only: - dest = os.path.join(outdir, "cfg-fstab.sh") + dest = os.path.join(outdir, "cfg_fstab.sh") if not (os.path.isfile(dest) and not args.overwrite): with open(dest, "w") as f: f.write(_fstab_script(load_config())) os.chmod(dest, 0o755) written += 1 - order.append("cfg-fstab") + order.append("cfg_fstab") if bl == "refind": dest = os.path.join(outdir, "refind.sh") @@ -4200,41 +5085,118 @@ def cmd_bs_gen_chroot_scripts(args): os.chmod(dest, 0o755) order.append("refind") written += 1 + elif bl == "none": + # Say that it was skipped, and how to change your mind. Silence here + # would be its own trap: someone who DOES need a bootloader would reach + # a finished, unbootable system without ever having been offered one. + hint("# no bootloader step: your machine already boots, so this build") + hint("# does not touch that. To add rEFInd to your existing ESP:") + hint("# lfs config bootloader refind") + hint("# lfs config esp /boot/efi") + hint("# lfs build-system gen-chroot-scripts --run --overwrite") + + + + # /tools -- the cross-toolchain that built chapter 8. Book 7.13 removes it + # right after chapter 7; this runs it at the END instead, so the fallback + # survives the part of the build most likely to need redoing. Off unless + # asked for: deleting a working toolchain should never be a default. + if not args.chapter7_only and remove_tools(): + entry = _resolve_section(secs, "ch-tools-cleanup") + if entry: + ctitle, csec = entry + cmds = [c for c in csec["commands"] if "/tools" in c] + if cmds: + dest = os.path.join(outdir, "cfg_rm-tools.sh") + if not (os.path.isfile(dest) and not args.overwrite): + text = _plain_root_script("cfg_rm-tools", "ch-tools-cleanup", + ctitle, cmds) + with open(dest, "w") as f: + f.write(text) + os.chmod(dest, 0o755) + written += 1 + order.append("cfg_rm-tools") + else: + warn(" cfg_rm-tools: 7.13 has no /tools command in this book") - # the user's own final step, copied in and run last - lstep = ensure_last_step_script() - if lstep and os.path.isfile(lstep): - import shutil as _sh - dest = os.path.join(outdir, "last-step.sh") - _sh.copy(lstep, dest) - os.chmod(dest, 0o755) - if "last-step" in order: - order.remove("last-step") - order.append("last-step") # always last - written += 1 - print("# last build step <- %s" % lstep) - extra_urls = last_step_sources() - if extra_urls: - print("# extra sources it wants (get-sources downloads these):") - for u in extra_urls: - print("# %s" % u) - - - with open(os.path.join(lfs, PKGUSR_DIR, "steporder"), "w") as f: + # The ownership epoch, as a STEP you can see. + # + # It ran as an invisible checkpoint inside build-all -- correct, but it + # never appeared in `lfs-helper list`, so the one place the whole scheme + # turns on was the one place you could not point at. It goes straight after + # init-files, which is book 7.6: the step that creates /etc/passwd, and + # therefore the first moment ownership means anything. + if "init-files" in order and "init-ownership" not in order: + order.insert(order.index("init-files") + 1, "init-ownership") + + # Accounts you can log in with -- LAST, after everything else. + # + # A BUILT-IN step: lfs-helper performs it, there is no script. It sets + # the root password, so everything that could still fail should have + # failed by now -- and it is the last thing standing between a built + # system and a reboot you cannot log in after. Appended here rather + # than beside the bootloader, because cfg_rm-tools comes after that. + # + # This replaces last_build_step.sh entirely. wget and the Python modules + # move to `packagemanager setup`, which is where bootstrapping the tooling + # belongs; the root password stays here because setup runs on the BOOTED + # system and you need to log in to run it. + # + # It is a built-in rather than a user-editable script for the reason the + # old one kept failing: a file created once on the host never receives a + # fix, and `chown: invalid user: 'wget:wget'` survived three releases that + # way. Book 8.5 says to run `passwd root`; that is not a matter of taste. + if "init-accounts" in order: + order.remove("init-accounts") + order.append("init-accounts") + + with open(pkgusr_state(lfs, "progress", "steporder"), "w") as f: f.write("\n".join(order) + "\n") + + # Stamp who generated this, and delete scripts for steps that are no + # longer in the order. + # + # The step order and the scripts live in the tree; the tools live outside + # it. Nothing connected the two, so a tree generated by an older version + # kept running steps that version had -- `last-step` was removed from the + # tools in 1.10.0 and went on failing in an existing tree, under a + # lfs-helper that no longer knew what it was. The chroot tools already + # carry a build id for exactly this reason; the generated scripts did not. + import time as _t + with open(pkgusr_state(lfs, "progress", "generated-by"), "w") as f: + f.write("version=%s\nbuild=%s\ndate=%s\n" + % (LFS_VERSION, _build_id(), + _t.strftime("%Y-%m-%d %H:%M:%S"))) + stale = [f for f in sorted(os.listdir(outdir)) + if f.endswith(".sh") and f[:-3] not in order] + for f in stale: + try: + os.remove(os.path.join(outdir, f)) + except OSError: + continue + if stale: + note("# removed %d script(s) for steps no longer in the order:" + % len(stale)) + for f in stale: + note("# %s" % f[:-3]) + hint("# clear them from the progress file too, if they were built:") + for f in stale: + hint("# lfs-helper undone %s" % f[:-3]) # chapter 7's temporary tools are built as root, like the book does; the # package-user system takes over for the real chapter 8 packages. root_steps = [n for n, _s in CHROOT_INIT_STEPS + CHROOT_STEPS_7 if n in order] root_steps += [n for n, _s in cfg_steps_9 if n in order and n not in pkg_like_cfg] - if "last-step" in order: - root_steps.append("last-step") # the user's own final step, as root - if "cfg-fstab" in order: - root_steps.append("cfg-fstab") # writes /etc/fstab + # init-accounts is performed by lfs-helper itself, not by a script, so it + # does not go in root_steps -- there is nothing to run as root. + if "cfg_fstab" in order: + root_steps.append("cfg_fstab") # writes /etc/fstab + if "cfg_rm-tools" in order: + root_steps.append("cfg_rm-tools") # deletes /tools, owned by root if "refind" in order: root_steps.append("refind") # writes to the ESP, so root - with open(os.path.join(lfs, PKGUSR_DIR, "rootsteps"), "w") as f: + with open(pkgusr_state(lfs, "progress", "rootsteps"), "w") as f: f.write("\n".join(root_steps) + "\n") _run_bash(f'chmod -R a+rX "{os.path.join(lfs, PKGUSR_DIR)}" 2>/dev/null; true') print(f"# wrote {written}, kept {skipped} existing " @@ -4246,7 +5208,7 @@ def cmd_bs_gen_chroot_scripts(args): for nm, left in unresolved: sys.stderr.write(" %-18s %s\n" % (nm, ", ".join("<%s>" % x for x in left))) - print(f"# step order -> {os.path.join(lfs, PKGUSR_DIR, 'steporder')}") + print(f"# step order -> {pkgusr_state(lfs, "progress", "steporder")}") write_chroot_env(lfs) install_helper(lfs) _install_collector_groups(lfs) @@ -4363,6 +5325,27 @@ def cmd_bs_chroot(args): # no longer build, and chapters 5-6 cannot be resumed. Refuse to do # that on top of a half-built toolchain -- an interrupted chain used to # sail straight through here into a chroot that could build nothing. + # A changed target triplet invalidates the whole toolchain: the + # headers were installed under the old one and the compiler will look + # under the new one. Catch it here, before the chroot, rather than + # forty packages later inside gcc. + mism = _crosschain_tgt_mismatch(lfs) + if mism and not getattr(args, "force", False): + was, now = mism + sys.stderr.write( + "\n! this tree was built for a different target:\n" + " built with : %s\n" + " configured : %s\n" + " The C++ headers are installed under the first name and the\n" + " compiler will look under the second, so chapter 8's gcc\n" + " will fail with 'bits/c++config.h: No such file or" + " directory'.\n\n" + " Put it back:\n %slfs config lfs_tgt %s\n" + " or rebuild chapters 5-6 for the new target:\n" + " %slfs build-system crosschain --run --restart\n" + % (was, now, sudo_prefix(), was, sudo_prefix())) + sys.exit(2) + cc_ok, cc_n = _crosschain_looks_done(lfs) if not cc_ok and not getattr(args, "force", False): sys.stderr.write( @@ -4379,7 +5362,45 @@ def cmd_bs_chroot(args): sys.exit(2) steps = [] - if not _owner_is_root(lfs): + _need = _handover_needed(lfs) + if _need and not _handover_is_safe(lfs): + # A package user owns files here, so this is a chapter-8 tree and + # the RECURSIVE chown would strip every one of them. Refusing and + # stopping there left the tree with no way forward: 4.3 had taken + # /usr, /etc, /var and the root symlinks, 7.2 would not give them + # back, and `chroot enter` refused because /usr was not root's. + # + # Reclaim precisely what the build user holds instead. Selecting on + # its uid leaves every package user's files untouched, so this is + # safe here in a way `chown -R root:root` is not. + print("# 7.2: package users own files here, so this is a chapter-8") + print("# tree and chown -R root:root would strip all of them.") + print("# Taking back only what the build user holds:") + _n = _handover_build_user_only(lfs, run=True) + if _n: + print("# gave %d path(s) back to root (group left alone)" % _n) + else: + print("# nothing of the build user's left in the tree") + _need = False + if _need: + # Book 7.2 hands over the directories it created in 4.2, but the + # tree ROOT and the mount points are not in that list -- they were + # created before the handover and never came back. The result is + # a system where /, /dev, /proc, /sys, /run, /media, /opt and + # /tools belong to a user that does not exist in the chroot: + # drwxr-xr-x 1 lfs lfs 178 / + # Nothing owned by the build user should survive into the finished + # system. NOT recursive. + # + # /sources is skipped HERE, not exempted: a blanket `chown -h + # root:root` would take the directory but not the files inside it, + # and would say nothing about the 1777 mode. _normalize_sources + # below owns that rule in one place and applies all of it. + steps.append(("give the top level to root", + f'for d in "{lfs}" "{lfs}"/*; do ' + f'case "$d" in *"/sources") continue ;; esac; ' + f'[ -e "$d" ] || continue; ' + f'chown -h root:root "$d" 2>/dev/null; done; true')) steps.append(("give the tree to root", f'for d in {" ".join(LFS_HANDOVER_DIRS)}; do ' f'[ -e "{lfs}/$d" ] && ' @@ -4425,10 +5446,11 @@ def cmd_bs_chroot(args): if rc != 0: sys.stderr.write(f"\n! failed: {label} (exit {rc})\n") sys.exit(rc) + _normalize_sources(lfs, run=True) write_chroot_env(lfs) install_helper(lfs) _copy_resolv_conf(lfs) - have_scripts = os.path.isdir(os.path.join(lfs, PKGUSR_DIR, "scripts")) + have_scripts = os.path.isdir(pkgusr_state(lfs, "scripts")) print("\n# chroot environment ready.") if not have_scripts: next_step(f"{sudo_prefix()}lfs build-system gen-chroot-scripts --run", @@ -4467,6 +5489,23 @@ def cmd_bs_chroot(args): # same reason as in `run`: entering with a stale lfs-helper means # debugging a version that no longer exists out here _refresh_chroot_tools(lfs) + # and the collector-group export, so editing it on the host reaches the + # next build without regenerating the chroot scripts. It only speaks + # up when the contents changed. + _install_collector_groups(lfs) + # Book 3.1 on every entry, not only on get-sources and restart. + # + # The host `lfs` account comes from book 4.3's own useradd, which pins + # no uid, so it lands wherever the host had a gap -- 10753 on a Debian + # host. Package users start at 10000, so that uid does not merely read + # as a number inside the chroot, it COLLIDES with a real package user + # and every tarball reports as owned by something unrelated. + # + # A tree prepared before _normalize_sources existed keeps that uid + # forever unless something applies the rule again. This is that + # something: entry is the one path every build goes through, whatever + # it did before. + _normalize_sources(lfs, run=True) live = _chroot_mounts(lfs) missing = [m for m in _KERNFS if m not in live] if missing: @@ -4475,17 +5514,32 @@ def cmd_bs_chroot(args): f"({', '.join(missing)}).\n" f" Run first: {sudo_prefix()}lfs build-system chroot prepare --run\n") sys.exit(2) + if not _owner_is_root(lfs): + # Repair it rather than sending you back to a command that just + # declined to do it. This guard refused while `chroot prepare` + # also refused -- correctly, since the recursive chown would have + # stripped every package user -- so the tree had no way forward and + # the message pointed at the command that had already said no. + # + # Only the build user's own paths are touched, so this is safe on a + # chapter-8 tree. If something else owns $LFS/usr, that is not + # ours to reassign and the guard still stops. + _n = _handover_build_user_only(lfs, run=True) + if _n: + print("# 7.2: took %d path(s) back from the build user " + "(group left alone)" % _n) if not _owner_is_root(lfs): sys.stderr.write( - "! the tree is not owned by root yet (book 7.2): %s owns " - "%s/usr.\n" % (_owner_name_of(os.path.join(lfs, "usr")), lfs) + "! %s/usr is owned by '%s', not root.\n" + % (lfs, _owner_name_of(os.path.join(lfs, "usr"))) + + " Book 7.2 hands the tree to root before the chroot.\n" + f" Run first: {sudo_prefix()}lfs build-system chroot prepare --run\n") sys.exit(2) cfg = load_config() mkf = cfg.get("makeflags") or "-j$(nproc)" install_helper(lfs, quiet=True) write_chroot_env(lfs, quiet=True) - scripts_dir = os.path.join(lfs, PKGUSR_DIR, "scripts") + scripts_dir = pkgusr_state(lfs, "scripts") print(f"# entering chroot at {lfs} (type 'exit' to leave)") print("#") print("# There's no Python in here yet -- use 'lfs-helper' (bash) to") @@ -4717,11 +5771,38 @@ def cmd_bs_crosschain(args): # subdirs there (glibc's var/lib/nss_db fails with 'Permission denied'). It's # our bookkeeping, not part of the LFS system being built, so it goes in its own # top-level dir that we own outright. -PKGUSR_DIR = ".lfs-pkgusr" +# Where this toolchain keeps its working notes about the tree, RELATIVE to +# $LFS. Must match lfs-helper's $STATE exactly: both tools read and write the +# same manifests, the same step order and the same progress. +# +# Under /usr/src, beside the accounts it describes -- not in a hidden directory +# at the root of the filesystem, where it sat next to /boot and /etc as if it +# were part of the system being built. It is not; it is what the build knows +# about that system. +# +# It also means the tree scans exclude ONE directory instead of two, because +# everything under /usr/src already is. +PKGUSR_DIR = "usr/src/lfs-pkgusr" + +# The sorted layout, so a name is written once and both tools agree. +PKGUSR_SUBDIRS = { + "scripts": "scripts", # one shell script per build step + "manifests": "manifests", # what each package installed + "logs": "logs", # one log per step, plus verify.log + "progress": "progress", # what has been built, and how far + "groups": "groups", # collector groups and their grants + "config": "config", # settings you may edit +} + + +def pkgusr_state(lfs, *parts): + """A path inside the state directory: pkgusr_state(lfs, "progress", + "steporder").""" + return os.path.join(lfs, PKGUSR_DIR, *parts) def _pkgusr_dir(lfs): - d = os.path.join(lfs, PKGUSR_DIR, "manifests") + d = pkgusr_state(lfs, "manifests") existed = os.path.isdir(d) os.makedirs(d, exist_ok=True) if not existed: @@ -4731,7 +5812,10 @@ def _pkgusr_dir(lfs): def _snapshot_cmd(lfs): - x = PKGUSR_DIR + # /usr/src wholesale, not just the state directory inside it: the package + # users' homes and their build trees are under there too, and lfs-helper's + # own scans already exclude it. Two tools, one rule. + x = "usr/src" return (f'find "{lfs}" -xdev \\( -type f -o -type l \\) ' f'-not -path "{lfs}/sources/*" -not -path "{lfs}/{x}/*" ' f'-not -path "{lfs}/dev/*" -not -path "{lfs}/proc/*" ' @@ -4740,7 +5824,7 @@ def _snapshot_cmd(lfs): def _snapshot_dirs_cmd(lfs): - x = PKGUSR_DIR + x = "usr/src" return (f'find "{lfs}" -xdev -type d ' f'-not -path "{lfs}/sources/*" -not -path "{lfs}/{x}/*" ' f'-not -path "{lfs}/dev/*" -not -path "{lfs}/proc/*" ' @@ -4765,8 +5849,18 @@ def _run_tracked(name, script, as_lfs, lfs): silently erase a correct manifest from an earlier successful run. Returns the install's exit code.""" import subprocess - d = _pkgusr_dir(lfs) - snap = os.path.join(d, ".snapshot") + # progress/tree-snapshot -- the SAME file lfs-helper reads. + # + # These two tools hand the tree over to each other at the end of chapter 6: + # `lfs` takes the last snapshot outside the chroot, `lfs-helper` diffs + # against it for the first package inside. When the state directory was + # sorted in 1.9.0 the reader moved to progress/tree-snapshot and this + # writer kept using .snapshot at the top level, so the handover snapshot + # was silently lost and the first chroot package diffed against nothing. + d = _pkgusr_dir(lfs) # state root; manifests hang off it + _prog = pkgusr_state(lfs, "progress") + os.makedirs(_prog, exist_ok=True) + snap = os.path.join(_prog, "tree-snapshot") before = (set(open(snap).read().splitlines()) if os.path.isfile(snap) else _take_snapshot(lfs)) # The directory snapshot must be taken HERE, before the build runs -- the @@ -4774,7 +5868,7 @@ def _run_tracked(name, script, as_lfs, lfs): # was no saved snapshot, so on a first run every package recorded zero # directories. Its files got the package user while the directories # holding them stayed root-owned. - snapd = os.path.join(d, ".snapshot-dirs") + snapd = os.path.join(_prog, "tree-snapshot-dirs") before_d = (set(open(snapd).read().splitlines()) if os.path.isfile(snapd) else _take_snapshot(lfs, dirs=True)) if as_lfs: @@ -4814,7 +5908,7 @@ def cmd_bs_manifests(args): if not lfs: sys.stderr.write("no LFS mount set (lfs build-system session)\n") sys.exit(2) - mdir = os.path.join(lfs, PKGUSR_DIR, "manifests") + mdir = pkgusr_state(lfs, "manifests") if not os.path.isdir(mdir): print("no manifests yet.") return @@ -4870,6 +5964,13 @@ def cmd_bs_manifests(args): TOOL_PY_DEPS = [("bs4", "beautifulsoup4"), ("requests", "requests")] +# The command wrappers, at the ONE location lfs-helper writes them to and +# /etc/pkgusr/bash_profile points at. Named here so this tool does not become a +# fourth place that spells the path out: the wrapper directory has already been +# two different directories once. +WRAPPERS = os.environ.get("LFS_WRAPPERS", "/usr/lib/pkgusr") + + # The package-users hint ships helper scripts and command wrappers. A system # built with this tooling should have the same ones, so `add_package_user`, # `install_package` and friends work in the chroot exactly as they do outside. @@ -4877,8 +5978,8 @@ PKGUSR_HELPERS = [ # (name, where it belongs) ("add_package_user", "usr/sbin"), ("install_package", "usr/sbin"), - ("useradd", "usr/lib/pkgusr"), - ("groupadd", "usr/lib/pkgusr"), + ("useradd", WRAPPERS.lstrip("/")), + ("groupadd", WRAPPERS.lstrip("/")), ("list_package", "usr/bin"), ("uninstall_package", "usr/bin"), ("list_suspicious_files", "usr/bin"), @@ -4886,11 +5987,11 @@ PKGUSR_HELPERS = [ ("grep_all_regular_files_for", "usr/bin"), ("forall_direntries_from", "usr/bin"), # the five command wrappers, which live first in a package user's PATH - ("chgrp", "usr/lib/pkgusr"), - ("chown", "usr/lib/pkgusr"), - ("chmod", "usr/lib/pkgusr"), - ("mkdir", "usr/lib/pkgusr"), - ("install", "usr/lib/pkgusr"), + ("chgrp", WRAPPERS.lstrip("/")), + ("chown", WRAPPERS.lstrip("/")), + ("chmod", WRAPPERS.lstrip("/")), + ("mkdir", WRAPPERS.lstrip("/")), + ("install", WRAPPERS.lstrip("/")), ] _HELPER_SEARCH = ["/usr/lib/pkgusr", "/usr/sbin", "/usr/bin", "/sbin", "/bin", @@ -4957,7 +6058,7 @@ def _install_pkgusr_skel(lfs, verbose=True): def _install_pkgusr_helpers(lfs, verbose=True): """Copy the hint's helper scripts into the chroot, where present.""" import shutil - copied, missing = [], [] + copied, missing, claimed = [], [], [] for name, subdir in PKGUSR_HELPERS: src = _find_host_helper(name, subdir) if not src: @@ -4970,8 +6071,29 @@ def _install_pkgusr_helpers(lfs, verbose=True): shutil.copy(src, dst) os.chmod(dst, 0o755) copied.append("%s -> /%s" % (name, subdir)) + claimed.append((name, subdir)) except OSError as e: sys.stderr.write("! %s: %s\n" % (name, e)) + # Give them an owner, the same as install_helper does for lfs/lfs-helper/ + # packagemanager/blfs. + # + # This path copied and chmod'd but never chowned, so the hint's helpers came + # out root:root while the four tools beside them in /usr/bin belonged to + # p_pkgusr. Two copy routines, one claiming ownership and one not -- the + # same species as the two wrapper directories and the two chgrp rules. + # + # Everything in the WRAPPER directory is excluded and stays root:root. A + # package user that could rewrite chown, chgrp or install could rewrite the + # rule that constrains it, and useradd/groupadd live there too. + wrapper_sub = WRAPPERS.lstrip("/") + by_dir = {} + for name, subdir in claimed: + if subdir.rstrip("/") == wrapper_sub: + continue + by_dir.setdefault(subdir, []).append(name) + for subdir, names in sorted(by_dir.items()): + _chown_tools(lfs, os.path.join(lfs, subdir), names, quiet=not verbose) + if verbose: if copied: print("# package-user helpers copied from this machine:") @@ -5128,6 +6250,9 @@ def cmd_bs_install_tools(args): f.write("# written by lfs build-system install-tools\n") f.write("collector_prefix=%s\n" % collector_prefix()) f.write("user_prefix=u\n") + # the layout of the system being BUILT, not this machine's + f.write("pkgusr_home=%s\n" % layout("pkgusr_home", target=True)) + f.write("appuser_home=%s\n" % layout("appuser_home", target=True)) _install_pkgusr_skel(lfs) _install_pkgusr_helpers(lfs) _write_tool_stamps(lfs, tools) @@ -5175,7 +6300,7 @@ def _blfs_knows(name): def _lfs_script_path(lfs, name): - return os.path.join(lfs, PKGUSR_DIR, "scripts", "%s.sh" % name) + return pkgusr_state(lfs, "scripts", "%s.sh" % name) def _find_lfs_package(args, name): @@ -5203,7 +6328,7 @@ def cmd_install(args): file tracking -- so this generates a current script and hands over, rather than growing a second, subtly different build path.""" # Are we running INSIDE the chroot? The state directory sits at the root of - # the built system, so it only exists at /.lfs-pkgusr when we are in there. + # the built system, so it only exists at /usr/src/lfs-pkgusr when we are in there. inside = os.path.isdir(os.path.join("/", PKGUSR_DIR)) lfs = "/" if inside else require_mounted_lfs() reinstall = args.reinstall @@ -5304,7 +6429,7 @@ def cmd_update(args): ver, path = resolve_book(args) soup = load_soup(path) - manifests = os.path.join(lfs, PKGUSR_DIR, "manifests") + manifests = pkgusr_state(lfs, "manifests") installed = set() if os.path.isdir(manifests): installed = {f[:-6] for f in os.listdir(manifests) if f.endswith(".files")} @@ -5374,14 +6499,14 @@ def cmd_update(args): def _chroot_progress(lfs): """Steps already built INSIDE the chroot (written by lfs-helper).""" - p = os.path.join(lfs, PKGUSR_DIR, "chroot-progress") + p = pkgusr_state(lfs, "progress", "steps-built") if os.path.isfile(p): return [l.strip() for l in open(p) if l.strip()] return [] def _chroot_steporder(lfs): - p = os.path.join(lfs, PKGUSR_DIR, "steporder") + p = pkgusr_state(lfs, "progress", "steporder") if os.path.isfile(p): return [l.strip() for l in open(p) if l.strip()] return [] @@ -5498,7 +6623,7 @@ def _next_steps(): steps.append(_step(len(live) == len(_KERNFS) and _owner_is_root(lfs), "prepare the chroot (chapter 7)", "%slfs build-system chroot prepare --run" % sudo_prefix())) - steps.append(_step(os.path.isdir(os.path.join(lfs, PKGUSR_DIR, "scripts")), + steps.append(_step(os.path.isdir(pkgusr_state(lfs, "scripts")), "generate the in-chroot build scripts", "%slfs build-system gen-chroot-scripts --run" % sudo_prefix())) @@ -5636,7 +6761,7 @@ def inside_chroot(): """Are we running inside the LFS chroot? The state directory lives at the root of the built system, so it is only at - /.lfs-pkgusr when we are in there.""" + /usr/src/lfs-pkgusr when we are in there.""" return os.path.isdir(os.path.join("/", PKGUSR_DIR)) @@ -5888,6 +7013,10 @@ def main(): help="delete the downloaded tarballs too (they re-download)") q.set_defaults(func=cmd_bs_restart) + q = bsub.add_parser("verify", help="what state is this tree in? how far " + "did the build get, and is the toolchain usable") + q.set_defaults(func=cmd_bs_verify) + q = bsub.add_parser("set-book", help="which book to BUILD THE SYSTEM from") q.add_argument("version") q.set_defaults(func=cmd_bs_set_book) diff --git a/lfs-completion.bash b/lfs-completion.bash index c88f12e..2574939 100644 --- a/lfs-completion.bash +++ b/lfs-completion.bash @@ -45,12 +45,18 @@ _lfs_tool_subcommands() { } _lfs_packages() { - # package users own /usr/src/<name>; that is the cheapest list of what is - # installed, and it works without invoking python - local d - for d in /usr/src/*/; do - [ -d "$d" ] || continue - basename "$d" + # Accounts own a directory under /usr/src, grouped by kind, and carry a + # prefix (p_gcc). Completing on the raw directory names would offer + # "p_gcc" where every command wants "gcc", so strip it back off. + # Still the cheapest list of what is installed, and no python needed. + local d n root + for root in "${LFS_PKGUSR_ROOT:-/usr/src/pkgusr}" \ + "${LFS_CFGUSR_ROOT:-/usr/src/cfg}"; do + for d in "$root"/*/; do + [ -d "$d" ] || continue + n="$(basename "$d")" + printf '%s\n' "${n#${LFS_PKGUSR_PREFIX-p_}}" + done done } diff --git a/lfs-helper b/lfs-helper index 4c0aa9c..c8cf943 100644 --- a/lfs-helper +++ b/lfs-helper @@ -7,7 +7,7 @@ # package-user system, creates package users, runs the phased install scripts, # and records which files each package installed. # -# State lives in /.lfs-pkgusr (the same dir `lfs` uses from outside, so both +# State lives under /usr/src (the same dir `lfs` uses from outside, so both # tools see one shared session). # # The install scripts it runs are the SAME phased format `packagemanager` uses @@ -22,7 +22,7 @@ set -u # Bump on every change that alters behaviour. `lfs-helper --version` makes it # possible to tell at a glance whether the copy inside the chroot is the one # that was just fixed -- guessing at that has wasted a lot of time. -LFS_HELPER_VERSION="1.7.0" +LFS_HELPER_VERSION="1.10.1" # A fingerprint of this file's own contents. A hand-maintained version number # goes stale the moment someone forgets to bump it -- and the reason for @@ -37,18 +37,82 @@ _build_id() { fi } -STATE="${LFS_PKGUSR_DIR:-/.lfs-pkgusr}" -SCRIPTS="$STATE/scripts" -MANIFESTS="$STATE/manifests" -PROGRESS="$STATE/chroot-progress" -SNAPSHOT="$STATE/.snapshot" -SNAPDIRS="$STATE/.snapshot-dirs" -INSTALLDIRS="$STATE/installdirs.lst" -SRCROOT="${LFS_SRC_ROOT:-/usr/src}" +# --------------------------------------------------------------------------- # +# where everything lives +# --------------------------------------------------------------------------- # + # Root of the filesystem being tracked. Inside the chroot that is "/"; the -# test harness points it elsewhere. Defined here because several paths below -# are derived from it. +# test harness points it elsewhere. Defined first because the paths below are +# derived from it. SNAP_ROOT="${LFS_SNAP_ROOT:-/}" + +# Where accounts live. SRCROOT is the PARENT and stays whole-tree, because +# tree scans exclude it as one path; the accounts themselves live in +# subdirectories by kind, so `ls /usr/src` says what a thing IS: +# +# /usr/src/pkgusr/p_gcc a package +# /usr/src/cfg/cfg_bootscripts a configuration step that installs files +# /usr/src/u_firefox an application user +# +# Flat under /usr/src, a hundred package users and a handful of config steps +# were one undifferentiated list. +SRCROOT="${LFS_SRC_ROOT:-/usr/src}" +PKGUSR_ROOT="${LFS_PKGUSR_ROOT:-$SRCROOT/pkgusr}" +CFGUSR_ROOT="${LFS_CFGUSR_ROOT:-$SRCROOT/cfg}" + +# What the build knows about those accounts -- under /usr/src beside them, not +# in a hidden directory at the root of the filesystem. /.lfs-pkgusr sat next to +# /boot and /etc as if it were part of the system being built. It is not: it is +# this toolchain's working notes ABOUT that system. +# +# It also means the tree scans exclude ONE path instead of two, because +# everything under $SRCROOT already is. +# +# Sorted, so that opening it tells you what is in it: +# +# scripts/ one shell script per build step, generated from the book +# manifests/ what each package installed +# logs/ one log per step, plus verify.log +# progress/ what has been built, and how far +# groups/ collector groups and the grants that created them +# config/ settings you may edit +# stage/ staged installs -- only during a build +# steps/ staging for steps that have no account +# tmp/ scratch -- only during a build +STATE="${LFS_PKGUSR_DIR:-${SRCROOT%/}/lfs-pkgusr}" + +SCRIPTS="$STATE/scripts" # generated build scripts +MANIFESTS="$STATE/manifests" # what each package installed +LOGS="$STATE/logs" # one log per step, plus verify.log +STATE_PROGRESS="$STATE/progress" # how far the build has got +STATE_GROUPS="$STATE/groups" # collector groups and their grants +STATE_CONF="$STATE/config" # settings a person may edit + +PROGRESS="$STATE_PROGRESS/steps-built" +SNAPSHOT="$STATE_PROGRESS/tree-snapshot" +SNAPDIRS="$STATE_PROGRESS/tree-snapshot-dirs" +INSTALLDIRS="$STATE_CONF/installdirs.lst" + +# ONE wrapper directory, and it is the one the profiles already name. +# +# There were two. make_wrappers wrote to $STATE/wrappers, while +# /etc/pkgusr/bash_profile -- and packagemanager's copy of it -- put +# /usr/lib/pkgusr first on a package user's PATH. Nothing created that, so a +# package user's login shell led with a directory that did not exist, and the +# wrappers only ever applied because cmd_build overrides PATH on the command +# line. Two names for one decision, which is the species of bug this project +# keeps hitting. +# +# /usr/lib/pkgusr wins over $STATE/wrappers for one reason: it has to outlive +# the build. $STATE is build state, and `packagemanager` on the finished system +# hands its package users the same profile -- pointing at a path that would no +# longer be there. +# +# It lives under /usr/lib but is NOT an install directory: root:root 0755, never +# group-writable, never adopted. A package user that could rewrite the wrappers +# could rewrite the rule that constrains it. +WRAPPERS="${LFS_WRAPPERS:-${SNAP_ROOT%/}/usr/lib/pkgusr}" + ETC="${LFS_ETC:-/etc}" # overridable so the test suite can exercise # user creation without touching the real /etc @@ -95,21 +159,65 @@ create_collector_group() { getent group "$grp" >/dev/null 2>&1 } +# --------------------------------------------------------------------------- # +# saying things +# --------------------------------------------------------------------------- # +# +# Four colours, one meaning each, and prose gets none. Colour that appears on +# every other line stops carrying information -- if the whole build log is +# yellow, nothing in it is a warning. +# +# green something finished, and finished correctly +# yellow something needs your attention +# red something failed +# dim detail you can skip: provenance, and what to do next +# +# Use the helper, never the variable. A colour opened in one echo and closed +# in another leaves the terminal coloured when the line between them is a +# command's own output. C_OK=$'\033[0;32m'; C_WARN=$'\033[0;33m'; C_ERR=$'\033[0;31m' C_DIM=$'\033[2m'; C_OFF=$'\033[0m' [ -t 1 ] || { C_OK=; C_WARN=; C_ERR=; C_DIM=; C_OFF=; } -say() { echo "$*"; } -ok() { echo "${C_OK}$*${C_OFF}"; } -warn() { echo "${C_WARN}$*${C_OFF}" >&2; } -die() { echo "${C_ERR}!! $*${C_OFF}" >&2; exit 1; } +# prose -- the ordinary running commentary of a build +say() { echo "$*"; } +# green: something succeeded +ok() { echo "${C_OK}$*${C_OFF}"; } +# yellow on stderr: something needs attention, and is not part of normal flow +warn() { echo "${C_WARN}$*${C_OFF}" >&2; } +# yellow on stdout: needs attention, but IS part of the flow -- a retry, a +# question, a directory being granted. Same colour because it means the same +# thing; stdout because it belongs in sequence with the build output. +note() { echo "${C_WARN}$*${C_OFF}"; } +# dim: provenance -- which script, which book section, which user +detail() { echo "${C_DIM}$*${C_OFF}"; } +# dim: what to do next. Never the finding itself, only the remedy. +hint() { echo "${C_DIM}$*${C_OFF}"; } +# red on stderr, and stop +die() { echo "${C_ERR}!! $*${C_OFF}" >&2; exit 1; } +# red on stdout: a failure banner that belongs in sequence with the build +fail() { echo "${C_ERR}$*${C_OFF}"; } need_root() { [ "$(id -u)" = 0 ] || die "must be run as root (inside the chroot)"; } # --------------------------------------------------------------------------- # # state # --------------------------------------------------------------------------- # -mkstate() { mkdir -p "$STATE" "$SCRIPTS" "$MANIFESTS"; } +# Every state directory, created together so none of them can be missing on +# the one path that happens not to create it. +# +# The top of it gets the same root:install 1775 as everything else under +# /usr/src -- one rule for the subtree. What is INSIDE stays root-owned and +# unreadable-by-nobody-in-particular: these are the build's own notes, and a +# package user has no reason to write to them. +mkstate() { + mkdir -p "$STATE" "$SCRIPTS" "$MANIFESTS" "$LOGS" \ + "$STATE_PROGRESS" "$STATE_GROUPS" "$STATE_CONF" || return 0 + if [ "$(id -u)" = 0 ]; then + set_install_dir_owner "$STATE" 2>/dev/null || true + real_chmod 1775 "$STATE" 2>/dev/null || true + fi +} is_done() { [ -f "$PROGRESS" ] && grep -qxF "$1" "$PROGRESS"; } mark_done() { mkstate; is_done "$1" || echo "$1" >> "$PROGRESS"; } @@ -118,8 +226,8 @@ unmark() { [ -f "$PROGRESS" ] && sed -i "\|^$1\$|d" "$PROGRESS"; } # The ordered list of steps. Each is "name" and the script is # $SCRIPTS/<name>.sh -- generated by `lfs build-system gen-chroot-scripts`. steps() { - if [ -f "$STATE/steporder" ]; then - grep -v '^[[:space:]]*$' "$STATE/steporder" + if [ -f "$STATE_PROGRESS/steporder" ]; then + grep -v '^[[:space:]]*$' "$STATE_PROGRESS/steporder" else ls -1 "$SCRIPTS" 2>/dev/null | sed -n 's/\.sh$//p' fi @@ -194,10 +302,137 @@ raw_useradd() { # belong to root:install and are group-writable so any package user can add # files; they are NOT owned by individual packages. (Modelled on # more_control_helpers/installdirs.lst from the package-users hint.) +# Directories no package may ever own -- but which are NOT install dirs. +# +# Two different ideas were conflated. An install directory is shared: it gets +# the install group and g+w so any package user can add files. These are +# something else entirely -- scratch and people's home directories -- and must +# simply never be adopted by a package. Putting them in install_dirs_list made +# /root root:install drwxrwx---, handing every package user access to root's +# home, which is the opposite of what was wanted. +# +# The account roots belong here too, and by decision rather than by omission. +# They carry the same root:install 1775 as `/usr/src` above them -- one rule for +# the subtree rather than a special case to remember. The sticky bit is what +# makes that safe: without it any member of `install` could delete or rename +# another package's home. Being in never_adopt_list is the other half -- no +# package may ever take one of these as its own. See ensure_pkgusr_roots. +# +# They come from pkgusr_roots, not from a second list of literal paths: the +# naming split adds two more roots, and a hardcoded copy here is one more place +# that can disagree. +never_adopt_list() { + cat <<'NALIST' +/tmp +/var/tmp +/root +/home +/sources +/build +/usr/src +NALIST + pkgusr_roots + # The wrapper directory. It lives under /usr/lib, which IS an install + # directory, so without this it inherits root:install + g+w and any package + # user can rewrite the wrappers that stop it changing ownership. + printf '%s\n' "$WRAPPERS" +} + +# The REAL chown/chgrp/chmod/install, never the wrapper. +# +# make_wrappers ships a `chgrp` and a `chown` that print a note and `exit 0`. +# They exist so a PACKAGE USER cannot change ownership -- but they are ordinary +# files on $PATH, and if that directory is ever ahead of /usr/bin for root, then +# root's own privileged calls resolve to them too. They report success and do +# nothing. +# +# That failure is invisible by construction, and it is the worst kind this +# project has: `init_pkgusr` then says +# # 46 install directories now belong to group 'install' +# with an empty failure list, while every one of them is still root:root -- so +# no package user can write to /usr/bin, and nothing says why until a build +# fails twenty packages later. `verify --fix` cannot repair it either, because +# its chown goes through the same wrapper. +# +# Ownership and mode are root's business. Call the binaries by absolute path so +# no PATH can come between the decision and the filesystem. +_REAL_CHOWN=""; _REAL_CHGRP=""; _REAL_CHMOD="" +_real_tool() { + # _real_tool <name> -- absolute path to the genuine binary + local n="$1" c + for c in "/usr/bin/$n" "/bin/$n" "/usr/sbin/$n" "/sbin/$n"; do + [ -x "$c" ] && { printf '%s' "$c"; return 0; } + done + # Nothing found in the usual places: fall back to the name and let PATH + # decide, rather than failing outright on an unusual layout. + printf '%s' "$n" +} +real_chown() { [ -n "$_REAL_CHOWN" ] || _REAL_CHOWN="$(_real_tool chown)"; "$_REAL_CHOWN" "$@"; } +real_chgrp() { [ -n "$_REAL_CHGRP" ] || _REAL_CHGRP="$(_real_tool chgrp)"; "$_REAL_CHGRP" "$@"; } +real_chmod() { [ -n "$_REAL_CHMOD" ] || _REAL_CHMOD="$(_real_tool chmod)"; "$_REAL_CHMOD" "$@"; } + +is_never_adopt() { + local d="${1%/}" p + [ -n "$d" ] || return 1 + while IFS= read -r p; do + [ -n "$p" ] || continue + [ "$d" = "${p%/}" ] && return 0 + done < <(never_adopt_list) + return 1 +} + +# What an install directory looks like, applied by NUMBER, in one place. +# +# `chown root:install` needs `root` to RESOLVE. On a chapter 5-6 tree restored +# before book 7.6 there is no /etc/passwd at all -- the prompt literally says +# "I have no name!" -- so every call failed and init-pkgusr reported: +# # 0 install directories are now root:install (group-writable) +# # could not set the group on: /usr /usr/bin /usr/lib /usr/share ... +# while ensure_install_dirs_writable, three hundred lines away, used `chgrp` +# and worked fine. Two rules for one directory, differing in whether they need +# an account to exist -- and only one of them ran early. +# +# Ownership is stored as a NUMBER. Use the number: uid 0 is root whether or not +# anything says so, and INSTALL_GID is ours. `install` may exist as a group +# before any passwd entry does, because init-pkgusr writes /etc/group by hand. +set_install_dir_owner() { + real_chown -h "0:$INSTALL_GID" "$1" 2>/dev/null +} + +# ...and the matching test, also by number, for the same reason: `stat -c %U:%G` +# reports UNKNOWN when nothing resolves, so a directory that was already correct +# would be reported as wrong forever. +install_dir_owner_ok() { + [ "$(stat -c '%u:%g' "$1" 2>/dev/null)" = "0:$INSTALL_GID" ] +} + install_dirs_list() { + # NOTE: /sources is deliberately NOT here. It is in never_adopt_list + # above, and the same conflation that made /root root:install had it in + # BOTH lists: `verify --fix` then made it root:install 775 and the seal + # made it 1775, dropping the o+w that book 3.1 asks for -- which is + # precisely what lets the unprivileged build user unpack into it. + # /sources is root:root 1777 with no group at all; `lfs` (host) applies + # that in get-sources and restart. if [ -f "$INSTALLDIRS" ]; then - grep -v '^[[:space:]]*\(#\|$\)' "$INSTALLDIRS" - return + # An older installdirs.lst may still carry the /sources line. Drop it + # here rather than rewriting the file: this list is consulted by nine + # callers, and one of them fixing the file behind the others' back is + # how the two rules diverged in the first place. + local _ovr + _ovr="$(grep -v '^[[:space:]]*\(#\|$\)' "$INSTALLDIRS" \ + | grep -vx '[[:space:]]*/sources[[:space:]]*')" + # An override that yields NOTHING must not silently disable the whole + # install-directory scheme. Every caller walks this list -- init-pkgusr, + # is_install_dir, verify, the seal -- so an empty return means no + # directory is ever given to the install group, no directory is ever + # protected from adoption, and `verify` reports a clean tree because it + # checked nothing. Fall back to the built-in list and say so. + if [ -n "$_ovr" ]; then + printf '%s\n' "$_ovr" + return + fi + warn "# $INSTALLDIRS lists no directories -- using the built-in list" >&2 fi cat <<'IDLIST' /bin @@ -208,12 +443,7 @@ install_dirs_list() { /boot /boot/efi /efi -/sources /usr/src -/tmp -/var/tmp -/root -/home /etc /etc/opt /etc/sysconfig @@ -271,7 +501,7 @@ IDLIST # write there got the `install` group applied automatically instead of being # asked which collector group should own it -- silently handing gcc's headers # to everyone. Adoption of that tree is prevented by the contents check in -# cmd_adopt_dirs, which is the right mechanism for it. +# `lfs-helper verify --fix`, which is the right mechanism for it. # # /tmp and /var/tmp are world-writable scratch space with the sticky bit # already set -- a package that happens to write there must not come to OWN @@ -340,6 +570,10 @@ matches_install_pattern() { # is this path one of the shared install dirs (root:install), rather than # something the package created for itself? is_install_dir() { + # protected from adoption, though not shared install dirs -- see + # never_adopt_list + is_never_adopt "$1" && return 0 + local raw="${1%/}" p d p="${raw#${SNAP_ROOT%/}}" # make it absolute-in-target [ -n "$p" ] || return 0 @@ -387,31 +621,68 @@ cmd_init_pkgusr() { fi # Directories packages install into. Anything missing is skipped quietly. - local d n=0 failed="" + local d n=0 failed="" didnt_take="" while IFS= read -r d; do d="${SNAP_ROOT%/}$d" [ -d "$d" ] || continue - if ! chgrp install "$d" 2>/dev/null; then + # root:install, OWNER AND GROUP -- not chgrp alone. + # + # This set only the group, while _vfy_install_dirs demands root:install + # and `verify --fix` applies both. Two rules for one directory, and + # they disagreed: chapters 5-6 build as the `lfs` user, book 7.2's + # handover only reaches one level below $LFS, so /usr/share stayed + # lfs-owned and came out `lfs:install`. The group looked right, and + # nothing said the owner was not. + if ! set_install_dir_owner "$d"; then # `|| continue` here reported "0 install directories" with no hint # that anything went wrong -- and the build then failed much later # with permission errors nobody could trace back to this. failed="$failed $d" continue fi - # the hint's definition: chgrp install <dir> && chmod g+w,o+t <dir>. + # CONFIRM it took, rather than trusting the exit code. + # + # A chgrp that returns 0 and changes nothing is not hypothetical: the + # wrapper in $WRAPPERS does exactly that, by design. If it is ever + # ahead of /usr/bin on root's PATH, every directory here is reported + # as done while staying root:root -- and no package user can write to + # /usr/bin for the rest of the build. + if ! install_dir_owner_ok "$d"; then + didnt_take="$didnt_take $d" + continue + fi + # the hint's definition: chown root:install <dir> && chmod g+w,o+t <dir>. # o+t (sticky) is deliberately deferred to seal-install-dirs, because # it would stop package users replacing files the temporary system # installed as root. - chmod g+w "$d" 2>/dev/null || true + real_chmod g+w "$d" 2>/dev/null || true n=$((n+1)) done < <(install_dirs_list) - ok "# $n install directories now belong to group 'install' (group-writable)" + ok "# $n install directories are now root:install (group-writable)" + local f if [ -n "$failed" ]; then warn "# could not set the group on:" - local f for f in $failed; do warn "# $f"; done warn "# The build will hit permission errors in these later." fi + if [ -n "$didnt_take" ]; then + warn "!! the group did NOT change, though the command reported success:" + for f in $didnt_take; do + warn "# $f (still $(stat -c %U:%G "$f" 2>/dev/null))" + done + warn " Package users cannot write into these, so builds WILL fail." + fi + # Zero is not a quiet success. install_dirs_list is overridable through + # $INSTALLDIRS, and an override that yields nothing turns the whole scheme + # into a no-op -- silently, because every later pass, `verify` included, + # walks the same empty list and finds nothing wrong. + if [ "$n" = 0 ]; then + warn "!! NO install directories were set up." + warn " Every package user will be unable to write to /usr/bin and" + warn " the rest of the shared tree." + [ -f "$INSTALLDIRS" ] && \ + warn " An override is in effect and yielded nothing: $INSTALLDIRS" + fi mark_done "pkgusr-init" say "" say "Next: lfs-helper next" @@ -435,13 +706,19 @@ export LC_ALL=POSIX # /usr/local/bin comes FIRST after the wrappers: locally installed # overrides belong ahead of /usr/bin, and a temporary wrapper there # (the no-verify wget, say) is useless if nothing ever finds it. -PATH=/usr/lib/pkgusr:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin +PATH=@WRAPPERS@:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin export PATH export PS1='\u:\w\$ ' export PKG_HOME="$HOME" cd "$HOME" 2>/dev/null || true [ -f "$HOME/build.conf" ] && . "$HOME/build.conf" EOF + # The profile is written with a QUOTED heredoc -- $HOME and $PS1 must + # reach the file untouched -- so the one value that is ours goes in as a + # placeholder and is substituted after. Hardcoding /usr/lib/pkgusr here + # is what let the profile and make_wrappers name two different + # directories for years. + sed -i "s|@WRAPPERS@|$WRAPPERS|" "$PKGUSR_ETC/bash_profile" fi if [ ! -f "$PKGUSR_ETC/bashrc" ]; then cat > "$PKGUSR_ETC/bashrc" <<'EOF' @@ -467,17 +744,18 @@ cd "$SRC_DIR" make $MAKE_OPTS make install EOF - chmod 755 "$PKGUSR_ETC/build" + real_chmod 755 "$PKGUSR_ETC/build" fi return 0 } init_package_user_home() { - local name="$1" home="${2:-$SRCROOT/$1}" + local name="$1" home="${2:-$(pkgusr_home_for "$1")}" ensure_pkgusr_etc || return 1 + ensure_pkgusr_roots mkdir -p "$home" local link target - for link in .bash_profile:bash_profile .bashrc:bashrc build:build; do + for link in $(pkgusr_skel_links); do target="${link#*:}"; link="${link%%:*}" ln -sfn "$PKGUSR_ETC/$target" "$home/$link" done @@ -490,22 +768,74 @@ MAKE_OPTS="" EOF fi [ -f "$home/.project" ] || printf '%s\n\nPackage user for %s.\n' "$name" "$name" > "$home/.project" - chown -R -h "$name:$name" "$home" 2>/dev/null || true + # Give the home to its user, and SAY SO if that fails. + # + # This used to be `2>/dev/null || true`, and a whole tree came out with + # every home root:root 755 -- so no package user could write its own home + # -- with not one line of output saying why. Whatever the cause, the + # reason it took a filesystem check, an su test and three rounds of + # guessing to find is that the failure was suppressed. + # + # It is not fatal: the build can still proceed, and `verify --fix` can + # repair ownership afterwards. But it must be visible. + if ! real_chown -R -h "$name:$name" "$home" 2>/tmp/.pkgusr-chown.$$; then + warn "!! could not give $home to '$name':" + sed 's/^/ /' "/tmp/.pkgusr-chown.$$" >&2 2>/dev/null || true + warn " the package user cannot write its own home." + warn " Repair with: lfs-helper verify --fix" + fi + rm -f "/tmp/.pkgusr-chown.$$" + # and confirm it actually took -- chown can return 0 and still leave the + # directory owned by root if the name resolves to nothing useful + local _own + _own="$(stat -c %U "$home" 2>/dev/null)" + if [ -n "$_own" ] && [ "$_own" != "$name" ]; then + warn "!! $home is still owned by '$_own', not '$name'" + warn " (chown reported success -- the account may not resolve yet)" + fi return 0 } # Steps that are not packages, and must never get a package user. # -# last-step, init-dirs, cfg-* and refind are actions, not software: they own no -# files of their own and installing "as" them makes no sense. A `last-step` -# user appeared because a manifest existed under that name, and everything the -# step touched -- including wget's binary, installed as the wget user -- was -# then handed to it. +# last-step, init-* and refind are actions, not software: they own no files of +# their own and installing "as" them makes no sense. A `last-step` user +# appeared because a manifest existed under that name, and everything the step +# touched -- including wget's binary, installed as the wget user -- was then +# handed to it. +# +# cfg_* is NOT decidable from the name. Most chapter-9 "configuration" +# sections write a file (/etc/hostname, /etc/hosts) and are root's work, but +# some are real packages: book 9.2 is titled LFS-Bootscripts-20250827 -- a +# versioned tarball that unpacks and installs programs into /etc/rc.d/init.d +# and /lib/services. Those files want an owner like any other package's. +# +# `lfs` already draws that line when it generates the scripts: a versioned +# section gets a full phased build (with pkg_glob and unpack_pkg), prose gets +# a plain root script. Read the script instead of guessing, so there is one +# decision and not two that can disagree. They did disagree, and the build +# stopped at 88/99 with +# 'cfg_bootscripts' is a build step, not a package -- no user created. +# install: invalid user 'cfg_bootscripts' +# !! could not stage the install script +# -- lfs had deliberately kept it OUT of rootsteps so it would be built as a +# package user, and then this refused to create one. _is_not_a_package() { - case "$1" in - last-step|init-*|cfg-*|refind) return 0 ;; + # init-ownership is this tool's own step: it owns no files and gets no + # account, exactly like init-dirs and init-files. + case "$1" in init-ownership|init-accounts) return 0 ;; esac + # Callers pass either a step name (`cfg_bootscripts`) or an owner name + # that has already been through pkg_owner_name (`p_cfg_bootscripts`). + # Match on the bare name, or the prefix silently disables every rule here. + local name; name="$(unprefix_pkg_user "$1")" + case "$name" in + last-step|init-*|refind) return 0 ;; + cfg_*) ;; # decided by the script, below + *) return 1 ;; esac - return 1 + local s="$SCRIPTS/$name.sh" + [ -f "$s" ] && grep -q '^pkg_glob=' "$s" && return 1 # a real package + return 0 } # Keep /etc/passwd and /etc/group in id order as users are created. @@ -513,17 +843,47 @@ _is_not_a_package() { # They are appended to, so without this the files drift into build order and # have to be tidied up afterwards -- which only happens if someone remembers. # The files are a few kilobytes; sorting them on each creation costs nothing. +# passwd/group are appended to as users are created; keep them in id order. +# One rule, shared with `verify` -- see _vfy_user_order. _sort_users_quietly() { - cmd_sort_users --run >/dev/null 2>&1 || true + _vfy_user_order 1 >/dev/null 2>&1 || true +} + +# An option is not a package name. +# +# A sanity run found /usr/src/pkgusr/p_--version: a home for an account that +# could never exist, left by something that reached the build with "--version" +# in the name position. Whatever mis-dispatched it, a leading dash here is +# always wrong, and refusing it costs nothing. +_refuse_option_as_name() { + case "${1:-}" in + -*) die "'$1' looks like an option, not a package name" ;; + esac } cmd_add_user() { need_root local name="${1:-}"; [ -n "$name" ] || die "usage: lfs-helper add-user <name>" + _refuse_option_as_name "$name" if _is_not_a_package "$name"; then warn "'$name' is a build step, not a package -- no user created." return 0 fi + # The ACCOUNT name, through the same chokepoint as the home. + # + # This used the argument raw for the account and pkg_owner_name only for + # the home, which is fine while every caller passes an already-prefixed + # owner -- cmd_build does. last_build_step.sh does not: it calls + # lfs-helper add-user wget + # and got an account called `wget` living in /usr/src/pkgusr/p_wget. The + # sanity report found both halves and could not connect them: + # !! accounts without a known prefix: wget urllib3 requests ... + # ? /usr/src/pkgusr/p_wget/ has no matching account + # + # pkg_owner_name is idempotent, so callers that already prefix are + # unaffected -- which is exactly why every name in this file should go + # through it rather than only the ones that look like they need it. + name="$(pkg_owner_name "$name")" pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" if user_exists "$name"; then @@ -536,14 +896,14 @@ cmd_add_user() { # -K UID_MIN/GID_MIN keeps package users in their own range without # us having to track which numbers are already used. if group_exists "$name"; then - useradd -c "package $name" -d "$SRCROOT/$name" -g "$name" \ + useradd -c "package $name" -d "$(pkgusr_home_for "$name")" -g "$name" \ -G install -s /bin/bash -K "UID_MIN=$PKG_UID_MIN" "$name" \ || die "useradd $name failed" else # -U creates the private group together with the user, so the # uid and gid match -- creating the group separately lets the # two counters drift apart (uid 10000 / gid 10001). - useradd -c "package $name" -d "$SRCROOT/$name" -U \ + useradd -c "package $name" -d "$(pkgusr_home_for "$name")" -U \ -G install -s /bin/bash \ -K "UID_MIN=$PKG_UID_MIN" -K "GID_MIN=$PKG_UID_MIN" \ "$name" \ @@ -558,13 +918,14 @@ cmd_add_user() { local uid=$PKG_UID_MIN while id_taken "$uid"; do uid=$((uid+1)); done group_exists "$name" || raw_groupadd "$name" "$uid" - raw_useradd "$name" "$uid" "$SRCROOT/$name" + raw_useradd "$name" "$uid" "$(pkgusr_home_for "$name")" ok "# created package user $name (uid/gid $uid, +install) via /etc/passwd" _sort_users_quietly fi fi - init_package_user_home "$name" "$SRCROOT/$name" - chmod 755 "$SRCROOT/$name" + _h="$(pkgusr_home_for "$name")" + init_package_user_home "$name" "$_h" + real_chmod 755 "$_h" } # --------------------------------------------------------------------------- # @@ -573,7 +934,7 @@ cmd_add_user() { # Build variables (LFS_TGT, MAKEFLAGS) as configured by `lfs config`. The # chroot resets the environment, so they're read from a file inside the tree # rather than inherited -- otherwise a custom LFS_TGT silently reverts. -load_env() { [ -f "$STATE/env" ] && . "$STATE/env"; return 0; } +load_env() { [ -f "$STATE_CONF/env" ] && . "$STATE_CONF/env"; return 0; } load_env # Read a stored snapshot, translating paths recorded OUTSIDE the chroot. @@ -582,12 +943,21 @@ load_env # no paths with the "after" set and the very first in-chroot build looks like it # installed the entire system. load_snapshot() { + # Re-sorted, in the same collation snapshot() used. + # + # The result is fed straight to `comm`, which compares line by line and + # trusts both sides to be sorted the same way. Two things here can break + # that: the file may have been written under a different locale, and the + # sed below rewrites only the lines carrying the host mount prefix -- so a + # file holding both forms comes out in an order that is sorted under + # neither. comm cannot detect either case; it just reports the wrong + # answer, and a package's whole install silently becomes "no new files". local f="$1" m="${LFS_HOST_MOUNT:-}" r="${SNAP_ROOT%/}" if [ -n "$m" ] && grep -q "^$m/" "$f" 2>/dev/null; then sed "s|^$m/|$r/|" "$f" else cat "$f" - fi + fi | LC_ALL=C sort } have_su() { command -v su >/dev/null 2>&1; } @@ -595,23 +965,32 @@ have_su() { command -v su >/dev/null 2>&1; } snapshot_dirs() { local r="${SNAP_ROOT%/}" find "${SNAP_ROOT:-/}" -xdev -type d \ - -not -path "$r/.lfs-pkgusr/*" -not -path "$SRCROOT/*" \ + -not -path "$SRCROOT/*" \ -not -path "$r/dev/*" -not -path "$r/proc/*" \ -not -path "$r/sys/*" -not -path "$r/run/*" \ -not -path "$r/tmp/*" -not -path "$r/sources/*" \ - 2>/dev/null | sort + -not -path "$r/build/*" \ + 2>/dev/null | LC_ALL=C sort } snapshot() { # Exclusions are anchored to the tracked root -- a bare "*/tmp/*" would also # throw away legitimate files like /usr/share/foo/tmp/bar. + # + # LC_ALL=C on the sort, because the result is fed to `comm`, which compares + # line by line and assumes BOTH sides used the same collation. One + # snapshot sorted under a UTF-8 locale and the next under POSIX would make + # comm report files as new that are not, and miss files that are -- silently, + # since comm cannot tell. A package user's profile sets LC_ALL=POSIX and + # root's shell may not, so the two sides really can differ. local r="${SNAP_ROOT%/}" find "${SNAP_ROOT:-/}" -xdev \( -type f -o -type l \) \ - -not -path "$r/.lfs-pkgusr/*" -not -path "$SRCROOT/*" \ + -not -path "$SRCROOT/*" \ -not -path "$r/dev/*" -not -path "$r/proc/*" \ -not -path "$r/sys/*" -not -path "$r/run/*" \ -not -path "$r/tmp/*" -not -path "$r/sources/*" \ - 2>/dev/null | sort + -not -path "$r/build/*" \ + 2>/dev/null | LC_ALL=C sort } # --------------------------------------------------------------------------- # @@ -638,13 +1017,32 @@ pkg_owner_name() { # Perl module names ("XML::Parser") are not valid user names n="${n//:://}"; n="${n//\//-}" case "$n" in - libstdcpp|libstdc++) echo gcc; return ;; + libstdcpp|libstdc++) n=gcc ;; esac # strip a trailing -pass<N> or -tmp n="${n%-tmp}" case "$n" in *-pass[0-9]) n="${n%-pass[0-9]}" ;; esac + # Apply the prefix THIS KIND carries -- `p_` for a package, `cfg_` for a + # config step -- rather than the package prefix regardless. Applying `p_` + # to everything is what produced `p_cfg_bootscripts`: an account wearing + # both prefixes, living in the config root and claiming to be a package. + # + # A legacy stacked name is repaired on the way through: pkgusr_kind strips + # the package prefix before it looks, so `p_cfg_bootscripts` is recognised + # as a config step, and the line below drops the `p_` it should never have + # had. + local pfx; pfx="$(pkgusr_prefix_for "$n")" + if [ -n "${PKGUSR_PREFIX:-}" ]; then n="${n#"$PKGUSR_PREFIX"}"; fi + # Idempotent: this is called on step names AND on values that already went + # through it, so a second pass must not produce p_p_gcc. + if [ -n "$pfx" ]; then + case "$n" in + "$pfx"*) ;; + *) n="${pfx}${n}" ;; + esac + fi echo "$n" } @@ -652,8 +1050,7 @@ pkg_owner_name() { # legitimately run one phase at a time (retrying just `install` after a failed # test suite, say), so remember which phases finished and let `next` pick up # from there instead of suggesting the whole package again. -PHASES="$STATE/phases" -LOGS="$STATE/logs" +PHASES="$STATE_PROGRESS/phases" # Pull the actual error out of a build log. With a parallel make the failing # job's message can be thousands of lines above the end, so showing the tail @@ -662,13 +1059,13 @@ show_build_errors() { local log="$1" n=0 [ -f "$log" ] || return 0 say "" - say "${C_ERR}--- first errors in the log ---${C_OFF}" + fail "--- first errors in the log ---" grep -nE "(^|[^-])\\berror\\b:|\\*\\*\\* \\[|Error [0-9]+|No such file or directory|command not found|undefined reference" \ "$log" 2>/dev/null | grep -vE "Error 1$|Error 2$" | head -12 | while IFS= read -r l; do say " $l" done n=$(grep -cE "\\berror\\b:|\\*\\*\\* \\[" "$log" 2>/dev/null || echo 0) - say "${C_ERR}-------------------------------${C_OFF}" + fail "-------------------------------" say " full log: $log" } @@ -754,11 +1151,11 @@ count_stdin_lines() { printf '%s' "${n:-0}" } -is_root_step() { [ -f "$STATE/rootsteps" ] && grep -qxF "$1" "$STATE/rootsteps"; } +is_root_step() { [ -f "$STATE_PROGRESS/rootsteps" ] && grep -qxF "$1" "$STATE_PROGRESS/rootsteps"; } # Remove a stale unpacked source tree BEFORE the build, as root. # -# /sources is sticky (1777), so only the owner may delete a directory in it. +# $BUILD_ROOT is sticky (1777), so only the owner may delete a directory in it. # Chapters 5-6 unpacked their sources as the `lfs` user, and chapter 8 builds # the same packages as their own package user -- which therefore cannot remove # the leftover tree. The script's own `rm -rf` fails, tar extracts on top of @@ -766,8 +1163,61 @@ is_root_step() { [ -f "$STATE/rootsteps" ] && grep -qxF "$1" "$STATE/rootsteps"; # tar: gcc-15.2.0/README: Cannot open: File exists # tar: gcc-15.2.0: Cannot utime: Operation not permitted # Doing it here, as root, keeps the package user out of the problem entirely. +# +# Two directories now: the tarball is looked up in SOURCES_DIR (read-only to +# builds) and the tree and markers are removed from BUILD_ROOT (scratch). +# What init_package_user_home symlinks into every package user's home. +# +# Named once, because build_root_for has to avoid these names and a second list +# would drift from this one. `build` is the hint's build helper script, linked +# as ~/build -- which is exactly the name the per-package build tree was first +# given, so `mkdir -p ~/build` hit the symlink: +# mkdir: cannot create directory '/usr/src/pkgusr/p_gettext/build': File exists +# cd: /usr/src/pkgusr/p_gettext/build: Not a directory +pkgusr_skel_links() { + printf '%s\n' .bash_profile:bash_profile .bashrc:bashrc build:build +} + +# The subdirectory of a package user's home that its sources unpack into. +# Must not collide with any name in pkgusr_skel_links -- assert_skel_safe. +PKGUSR_BUILD_SUBDIR="${LFS_PKGUSR_BUILD_SUBDIR:-src}" + +# Where a package unpacks and builds. +# +# Its OWN package user's home -- /usr/src/pkgusr/p_gcc/src -- not a shared +# /build that every package writes into at once. +# +# `src`, not `build`: ~/build is already the hint's build helper script. +# +# The shared scratch was a permanent ownership fight. Every build's +# touched-file scan swept the whole tree in, so each package's manifest claimed +# every other package's sources and chowned them; the next package chowned them +# back. A real `verify` reported 3363 wrong-owner and 3348 of them were /build: +# /build/bash-5.3.tar.gz is p_shadow want p_acl +# /build/bash-5.3.tar.gz is p_zstd want p_acl (after "repairing" it) +# No repair could ever settle that, because the question itself was wrong: the +# scratch does not belong to a package, and asking which one owns it has no +# answer. +# +# Under the package user's home it is settled by construction. The user creates +# the tree, so the user owns it; $SRCROOT is already excluded from every tree +# scan, so it can never enter a manifest at all; and removing a package removes +# its build tree with it. That is also how the package-users hint does it. +# +# Steps with no account -- init-*, last-step, refind -- keep the shared scratch. +# They run as root, which owns everything there anyway. +build_root_for() { + if _is_not_a_package "$1"; then + printf '%s' "${BUILD_ROOT:-${SNAP_ROOT%/}/build}" + else + printf '%s/%s' "$(pkgusr_home_for "$1")" "$PKGUSR_BUILD_SUBDIR" + fi +} + clean_stale_source() { - local name="$1" script="$2" srcdir="${SNAP_ROOT%/}/sources" + local name="$1" script="$2" + local srcdir="${SOURCES_DIR:-${SNAP_ROOT%/}/sources}" + local blddir; blddir="$(build_root_for "$name")" [ -d "$srcdir" ] || return 0 local glob; glob="$(sed -n 's/^pkg_glob="\(.*\)"$/\1/p' "$script" | head -n1)" [ -n "$glob" ] || return 0 @@ -786,19 +1236,19 @@ clean_stale_source() { pkg="$(cat /tmp/.lfs-pkg.$$ 2>/dev/null)"; rm -f /tmp/.lfs-pkg.$$ [ -n "$pkg" ] || return 0 - # The build markers are written by whoever built last. /sources is sticky, - # so a marker left by the `lfs` user (chapters 5-6) cannot be replaced by a - # chapter-8 package user, and the build dies with - # /sources/.cc-build-ncurses: Permission denied + # The build markers are written by whoever built last, and BUILD_ROOT is + # sticky, so a marker left by the `lfs` user (chapters 5-6) cannot be + # replaced by a chapter-8 package user, and the build dies with + # .../.cc-build-ncurses: Permission denied # Clear them here, as root; the build recreates them. - rm -f "$srcdir/.cc-build-$name" "$srcdir/.cc-dir-$name" 2>/dev/null + rm -f "$blddir/.cc-build-$name" "$blddir/.cc-dir-$name" 2>/dev/null local dir dir="$(tar tf "$srcdir/$pkg" 2>/dev/null | head -n1 | cut -d/ -f1)" [ -n "$dir" ] || return 0 - [ -d "$srcdir/$dir" ] || return 0 - say "${C_DIM}# [root] removing stale source tree $srcdir/$dir${C_OFF}" - rm -rf "${srcdir:?}/${dir:?}" || warn "could not remove $srcdir/$dir" + [ -d "$blddir/$dir" ] || return 0 + detail "# [root] removing stale source tree $blddir/$dir" + rm -rf "${blddir:?}/${dir:?}" || warn "could not remove $blddir/$dir" } # --------------------------------------------------------------------------- @@ -839,10 +1289,219 @@ _read_shared_prefix() { } COLLECTOR_PREFIX="${LFS_COLLECTOR_PREFIX:-$(_read_shared_prefix || true)}" -GRANTED="$STATE/granted-groups" + +# Package users carry a prefix of their own: `p_gcc`, not `gcc`. +# +# Without it a package user and a system account can collide on one name, and +# the collision is invisible until it bites. LFS chapter 7.6 creates `mail`, +# `news`, `uucp`, `man` and friends, and BLFS daemons bring their own -- a +# `sshd`, `nginx` or `postfix` package would want exactly the name its daemon +# already uses. Two things then share one uid, and since ownership is stored +# as a NUMBER, every file either of them owns reads as belonging to the other. +# +# The prefix also makes `ls -l` self-explanatory: `p_` is a package, `u_` an +# application user, everything else is a real account. +# +# Resolved in the same order and for the same reason as COLLECTOR_PREFIX -- +# two tools disagreeing about it would build two parallel sets of accounts. +_read_pkgusr_prefix() { + local v="" + for f in /usr/share/lfs/config.json "$HOME/.local/share/lfs/config.json"; do + [ -r "$f" ] || continue + v="$(sed -n 's/.*"pkgusr_prefix"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' \ + "$f" | head -n1)" + [ -n "$v" ] && { printf '%s' "$v"; return 0; } + done + for f in /etc/pkgusr/packagemanager.conf; do + [ -r "$f" ] || continue + v="$(sed -n 's/^pkgusr_prefix=\(.*\)$/\1/p' "$f" | head -n1)" + [ -n "$v" ] && { printf '%s' "$v"; return 0; } + done + return 1 +} + +# `${VAR-...}` not `${VAR:-...}`: an explicitly EMPTY value means "no prefix", +# which is how a tree built before prefixes existed keeps working. With `:-` +# an empty value would silently fall through to the default and rename every +# package user out from under that tree. +PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-$(_read_pkgusr_prefix || echo p)}" +PKGUSR_PREFIX="${PKGUSR_PREFIX%_}" +[ -n "$PKGUSR_PREFIX" ] && PKGUSR_PREFIX="${PKGUSR_PREFIX}_" + +# Config steps carry their own prefix, and only their own. +# +# They used to be `p_cfg_bootscripts`: the package prefix stacked on top of the +# step name's own `cfg_`. Two prefixes on one account means neither one tells +# you what the account is -- `p_` claimed it was a package while it lived in the +# config root -- and every rule about prefixes had to special-case it. One +# account, one prefix: `p_gcc` is a package, `cfg_bootscripts` is a config step, +# `u_firefox` is an application user. +CFGUSR_PREFIX="${LFS_CFGUSR_PREFIX-cfg}" +CFGUSR_PREFIX="${CFGUSR_PREFIX%_}" +[ -n "$CFGUSR_PREFIX" ] && CFGUSR_PREFIX="${CFGUSR_PREFIX}_" + +# What KIND of account is this -- `pkg` or `cfg`? +# +# This is the one place that decides, and everything about a kind hangs off it: +# pkgusr_prefix_for and pkgusr_root_for answer from it, pkg_owner_name builds +# the account name with it, pkgusr_home_for places the home with it. Before, +# the kind was re-derived by pattern-matching `cfg_*` at each site, which is how +# a name could be routed to the config root while being given the package +# prefix. +# +# Takes any of the three: a step name (`cfg_bootscripts`), an account name +# (`cfg_bootscripts`, `p_gcc`), or a legacy stacked name (`p_cfg_bootscripts`). +# +# Note there is no `tmp` kind and no `init` kind. A temporary step is the same +# package chapter 8 rebuilds, so it is built as that package's user from the +# start -- see pkg_owner_name's -tmp fold. An init step owns no files at all +# and gets no account -- see step_stage_dir. +pkgusr_kind() { + local n="$1" + [ -n "$PKGUSR_PREFIX" ] && n="${n#"$PKGUSR_PREFIX"}" + case "$n" in + "$CFGUSR_PREFIX"*) printf 'cfg' ;; + *) printf 'pkg' ;; + esac +} + +pkgusr_prefix_for() { + case "$(pkgusr_kind "$1")" in + cfg) printf '%s' "$CFGUSR_PREFIX" ;; + *) printf '%s' "$PKGUSR_PREFIX" ;; + esac +} + +pkgusr_root_for() { + case "$(pkgusr_kind "$1")" in + cfg) printf '%s' "$CFGUSR_ROOT" ;; + *) printf '%s' "$PKGUSR_ROOT" ;; + esac +} + +# Strip the prefix this account's kind carries: account name -> step name. +# +# A config step's own prefix is NOT stripped: `cfg_bootscripts` is what the book +# step is called, so for that kind the account name and the step name are the +# same string. Stripping it would leave `bootscripts`, and every rule that asks +# "is this a config step?" reads the name. +unprefix_pkg_user() { + local n="$1" + [ -n "$PKGUSR_PREFIX" ] || { printf '%s' "$n"; return; } + printf '%s' "${n#"$PKGUSR_PREFIX"}" +} + +# Home of one account, by kind. Takes a step name or an account name. +pkgusr_home_for() { + # Resolve to the ACCOUNT's home, whichever kind of name we are handed. + # + # Callers pass a mixture and always have: build_pkg has the STEP name + # ("man-pages", "util-linux-tmp"), the staging code has the OWNER + # ("p_man-pages", "p_util-linux"). pkg_owner_name is what maps one to the + # other -- it applies the prefix, lowercases, and folds -tmp/-passN back + # onto the real package -- and it is idempotent, so an owner passed in + # comes back unchanged. + # + # Routing by kind WITHOUT that mapping is the bug this replaced: the + # account home was /usr/src/pkgusr/p_man-pages and the build cd'd to + # /usr/src/pkgusr/man-pages, which does not exist: + # line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory + # It only bit the package-user branch -- the root-step branch mkdir -p's + # its own directory first, so chapter 7 built fine and it surfaced at the + # first real package user. + local n; n="$(pkg_owner_name "$1")" + printf '%s/%s' "$(pkgusr_root_for "$n")" "$n" +} + +# Where a step's install script is staged, and what it is run from. +# +# A package step runs from its package user's home. A step that is NOT a +# package -- init-*, last-step, refind, a prose cfg_* section -- never gets an +# account, so asking pkgusr_home_for for its home manufactures a directory +# under the package root for a user that will never exist. That is exactly the +# stray `p_init-dirs` and `p_init-files` a sanity run found: directories with no +# matching account, which every ownership pass then has to report as strays. +# +# Those steps run as root out of the state directory, beside the scripts they +# came from. Nothing there is owned by a package, and nothing pretends to be. +step_stage_dir() { + if _is_not_a_package "$1"; then + printf '%s/steps/%s' "$STATE" "$(unprefix_pkg_user "$1")" + else + pkgusr_home_for "$1" + fi +} + +# Expose the chokepoint to shell scripts. Without this, anything outside +# lfs-helper that needs an account's home re-derives the rule -- prefix, kind, +# root -- and that third copy is exactly how the two roots drift apart. +cmd_pkgusr_home() { + [ -n "${1:-}" ] || die "usage: lfs-helper pkgusr-home <name>" + pkgusr_home_for "$1" + echo +} + +# The other half of the same chokepoint: the ACCOUNT NAME for a package. +# +# `pkgusr-home` was exposed and this was not, so a shell script outside this +# tool could ask where a package lives but not what its user is called -- and +# last_build_step.sh did the only thing left, which was assume: +# +# lfs-helper add-user wget +# chown -R "wget:wget" ... -> chown: invalid user: 'wget:wget' +# +# It had been right by accident while add-user created the account from the raw +# name; the moment add-user started applying the prefix (as everything else +# already did), the assumption broke. Half a chokepoint is not a chokepoint. +cmd_owner_name() { + [ -n "${1:-}" ] || die "usage: lfs-helper owner-name <package>" + pkg_owner_name "$1" +} + +# Every account root, for the passes that walk all of them. +pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; } + +# Create the account roots with the ownership and mode they are meant to have. +# +# They used to appear as a side effect of `mkdir -p` on the first home, so they +# got whatever root's umask gave them. That happened to be right -- +# root:root 0755 -- but nothing said so, and nothing would have noticed it +# becoming something else. Now it is stated, and applied every time. +# +# root:root 0755 is the decision: only root creates account homes, so the +# install group has no business here, and without a sticky bit a group-writable +# root would let any package user delete another package's home. +ensure_pkgusr_roots() { + local r + while IFS= read -r r; do + [ -n "$r" ] || continue + mkdir -p "$r" 2>/dev/null || { + warn "!! could not create the account root $r"; continue; } + # root:install 1775, the same as /usr/src above them. + # + # They were root:root 0755 -- defensible, since only root creates an + # account home, but it made /usr/src three directories with two rules: + # + # drwxrwxr-t root install /usr/src + # drwxr-xr-x root root /usr/src/pkgusr + # drwxr-xr-x root root /usr/src/cfg + # + # and a special case is a thing to remember. One rule for the subtree. + # + # The sticky bit is what makes group-writable safe here, and it is not + # optional: without it any member of `install` could delete or rename + # another package's home. With it, only the owner can. Group write then + # grants only the ability to CREATE, which nothing unprivileged does -- + # `add-user` is root-only -- so this is consistency, not capability. + set_install_dir_owner "$r" 2>/dev/null || true + real_chmod 1775 "$r" 2>/dev/null || true + done < <(pkgusr_roots) +} + +GRANTED="$STATE_GROUPS/granted-groups" # dir|group, one per line: the decisions made so far. Exported so a rebuild # can reuse them instead of asking the same questions again. -COLLECTOR_MAP="$STATE/collector-map" +COLLECTOR_MAP="$STATE_GROUPS/collector-map" require_collector_prefix() { [ -n "$COLLECTOR_PREFIX" ] && return 0 @@ -858,7 +1517,16 @@ require_collector_prefix() { lfs build-system gen-chroot-scripts --run --overwrite" } -collector_group_for() { echo "${COLLECTOR_PREFIX}_$1"; } +# A collector group is named for the PACKAGE, not for the account. +# +# It took the account name straight, so the owner `p_perl` produced +# `nimgnu_p_perl` -- two prefixes stacked, and the wrong two: `nimgnu_` says +# "collector group" and `p_` says "package user", which it is not. The prompt +# offered it as a choice: +# 1) nimgnu_p_perl (everything owned by 'p_perl' -- fewer groups) +# Strip the account's prefix first; the collector prefix is the only one a +# collector group carries. Same rule as everywhere else: one name, one prefix. +collector_group_for() { echo "${COLLECTOR_PREFIX}_$(unprefix_pkg_user "$1")"; } ensure_group() { local grp="$1" @@ -959,11 +1627,11 @@ choose_collector_group() { # An export copied into the tree is a decision already made, even if # `import-groups` has not been run. Consulting it here means the build # never stops to ask a question whose answer is sitting in the tree. - if [ -f "$STATE/collector-groups.import" ]; then + if [ -f "$STATE_GROUPS/collector-groups.import" ]; then local _pre _pre="$(awk -F'|' -v d="$dir" \ '$1 == "dir" && $2 == d { print $3; exit }' \ - "$STATE/collector-groups.import" 2>/dev/null)" + "$STATE_GROUPS/collector-groups.import" 2>/dev/null)" if [ -n "$_pre" ]; then echo "$_pre" return 0 @@ -1073,7 +1741,7 @@ grant_dir_access() { if [ -n "$curgrp" ] && [ "$curgrp" != root ] \ && case "$curgrp" in "$COLLECTOR_PREFIX"*) true ;; *) false ;; esac \ && id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx "$curgrp"; then - chmod g+rwx "$dir" 2>/dev/null || true + real_chmod g+rwx "$dir" 2>/dev/null || true say " $dir -> already group $curgrp, and '$user' is a member" return 0 fi @@ -1094,8 +1762,8 @@ grant_dir_access() { say " $dir is owned by '$_cur_owner' -- asking rather than" say " opening it to every package." else - chgrp install "$dir" 2>/dev/null - chmod g+w "$dir" 2>/dev/null + real_chgrp install "$dir" 2>/dev/null + real_chmod g+w "$dir" 2>/dev/null say " $dir -> group install (shared install directory)" return 0 fi @@ -1118,7 +1786,7 @@ grant_dir_access() { # directories with it. Needing to write into a directory says nothing # about the directories below it, and now that packages record the # directories they create, those already have the right owner. - chown "$user:$user" "$dir" 2>/dev/null + real_chown "$user:$user" "$dir" 2>/dev/null say " $dir -> owned by '$user' (was root, from the temporary system)" return 0 fi @@ -1152,8 +1820,8 @@ grant_dir_access() { fi add_to_group "$owner" "$grp" add_to_group "$user" "$grp" - chgrp "$grp" "$dir" 2>/dev/null - chmod g+rwx "$dir" 2>/dev/null + real_chgrp "$grp" "$dir" 2>/dev/null + real_chmod g+rwx "$dir" 2>/dev/null # Share exactly the directory that was asked for -- nothing below it. # @@ -1167,8 +1835,8 @@ grant_dir_access() { # Installing into a second directory asks again -- and because the package # is already a member of the group by then, that grant is automatic (see # the fast path at the top of this function). - chgrp "$grp" "$dir" 2>/dev/null || return 1 - chmod g+rwx "$dir" 2>/dev/null || true + real_chgrp "$grp" "$dir" 2>/dev/null || return 1 + real_chmod g+rwx "$dir" 2>/dev/null || true say " $dir -> group $grp (owner $owner, + $user)" mkdir -p "$(dirname "$GRANTED")" grep -qxF "$grp|$user|$owner" "$GRANTED" 2>/dev/null \ @@ -1204,7 +1872,7 @@ unwritable_dirs_from_log() { case "$p" in "${SNAP_ROOT%/}"/*) ;; /usr/*|/etc/*|/var/*|/opt/*|/lib/*|/lib64/*|/bin/*|/sbin/*|\ - /home/*|/root/*|/srv/*|/boot/*|/tools/*|/sources/*) ;; + /home/*|/root/*|/srv/*|/boot/*|/tools/*|/sources/*|/build/*) ;; *) continue ;; esac if [ -d "$p" ]; then echo "$p"; else echo "${p%/*}"; fi @@ -1224,8 +1892,8 @@ auto_grant_from_log() { [ -n "$need" ] || return 1 require_collector_prefix say "" - say "${C_WARN}# '$user' could not write into some directories -- granting" - say "# access via collector groups and retrying:${C_OFF}" + note "# '$user' could not write into some directories -- granting" + note "# access via collector groups and retrying:" # Many failing paths walk up to the SAME tree top, so without this the same # directory is granted once per path -- nine identical grants in one round. local seen="" @@ -1322,7 +1990,7 @@ _dedupe_collector_export() { } cmd_export_groups() { - local out="${1:-$STATE/collector-groups.export}" + local out="${1:-$STATE_GROUPS/collector-groups.export}" : > "$out" { echo "# lfs collector groups -- import with: lfs-helper import-groups <file>" @@ -1335,6 +2003,7 @@ cmd_export_groups() { -not -path "${r}/proc/*" -not -path "${r}/sys/*" \ -not -path "${r}/dev/*" -not -path "${r}/run/*" \ -not -path "${r}/tmp/*" -not -path "${r}/sources/*" \ + -not -path "${r}/build/*" \ -printf '%g|%p\n' 2>/dev/null \ | awk -F'|' -v p="${COLLECTOR_PREFIX}_" -v r="$r" ' index($1, p) == 1 { @@ -1378,10 +2047,10 @@ cmd_import_groups() { done if [ -z "$in" ]; then # prefer a file placed here by `lfs build-system gen-chroot-scripts` - if [ -f "$STATE/collector-groups.import" ]; then - in="$STATE/collector-groups.import" + if [ -f "$STATE_GROUPS/collector-groups.import" ]; then + in="$STATE_GROUPS/collector-groups.import" else - in="$STATE/collector-groups.export" + in="$STATE_GROUPS/collector-groups.export" fi fi [ -f "$in" ] || die "no such file: $in" @@ -1417,91 +2086,17 @@ cmd_import_groups() { fi } -# Repair package-user homes and VERSION files after the fact. -# -# Users created before the skeleton existed have no .bash_profile/.bashrc/build -# links, or links copied from the wrong place; and packages built before -# VERSION was recorded have no record of what is installed. This rebuilds both -# from what is already on disk. -cmd_fix_users() { - need_root - local run=0 skel="${LFS_PKGUSR_SKEL:-${SNAP_ROOT%/}/etc/pkgusr/skel-package}" - while [ $# -gt 0 ]; do - case "$1" in - --run) run=1; shift ;; - --skel) skel="${2:?--skel needs a directory}"; shift 2 ;; - *) die "usage: lfs-helper fix-users [--run] [--skel <dir>]" ;; - esac - done - - say "Repair every package user's home:" - say " * .bash_profile, .bashrc and build -> $PKGUSR_ETC" - say " * VERSION, from the install script's name_version" - [ -d "$skel" ] && say " * other files from $skel" - say "" - - local n_link=0 n_ver=0 name home nv script - while IFS= read -r name; do - [ -n "$name" ] || continue - home="$SRCROOT/$name" - [ -d "$home" ] || continue - - # links - local need_links=0 l t - for l in .bash_profile:bash_profile .bashrc:bashrc build:build; do - t="${l#*:}"; l="${l%%:*}" - [ "$(readlink "$home/$l" 2>/dev/null)" = "$PKGUSR_ETC/$t" ] || need_links=1 - done - if [ "$need_links" = 1 ]; then - n_link=$((n_link+1)) - printf ' %-24s links\n' "$name" - if [ "$run" = 1 ]; then - init_package_user_home "$name" "$home" - if [ -d "$skel" ]; then - # skeleton files the host uses, without clobbering the links - cp -a "$skel"/. "$home"/ 2>/dev/null || true - init_package_user_home "$name" "$home" - fi - fi - fi - - # VERSION - if [ ! -f "$home/VERSION" ]; then - script="$SCRIPTS/$name.sh" - nv="" - [ -f "$script" ] && \ - nv="$(sed -n 's/^name_version="\(.*\)"$/\1/p' "$script" | head -n1)" - if [ -n "$nv" ]; then - n_ver=$((n_ver+1)) - printf ' %-24s VERSION -> %s\n' "$name" "$nv" - if [ "$run" = 1 ]; then - { - echo "$nv" - echo "# recorded $(date '+%Y-%m-%d %H:%M:%S') by fix-users" - } > "$home/VERSION" - chown "$name:$name" "$home/VERSION" 2>/dev/null || true - fi - fi - fi - done < <(pkg_users_list) - - say "" - if [ "$run" = 1 ]; then - ok " fixed $n_link home(s) and wrote $n_ver VERSION file(s)" - else - say " $n_link home(s) need links, $n_ver VERSION file(s) missing" - warn "(dry run -- nothing was changed. Add --run to apply.)" - fi -} # every package user we know about pkg_users_list() { - local d name - for d in "$SRCROOT"/*; do - [ -d "$d" ] || continue - name="$(basename "$d")" - user_exists "$name" && echo "$name" - done + local d name root + while IFS= read -r root; do + for d in "$root"/*; do + [ -d "$d" ] || continue + name="$(basename "$d")" + user_exists "$name" && echo "$name" + done + done < <(pkgusr_roots) } cmd_find_user_site() { @@ -1517,7 +2112,7 @@ cmd_find_user_site() { printf " %s\n" "$d" n=$((n+1)) [ "$run" = 1 ] && rm -rf "$d" - done < <(find "$SRCROOT" -maxdepth 4 -type d -path '*/.local/lib' 2>/dev/null | sort) + done < <(find "$SRCROOT" -maxdepth 5 -type d -path '*/.local/lib' 2>/dev/null | sort) say "" if [ "$n" = 0 ]; then ok " none -- every package installed system-wide" @@ -1534,22 +2129,35 @@ cmd_find_user_site() { cmd_clean_sources() { need_root local run=0; [ "${1:-}" = "--run" ] && run=1 - local srcdir="${SNAP_ROOT%/}/sources" + local srcdir="${SOURCES_DIR:-${SNAP_ROOT%/}/sources}" + local blddir="${BUILD_ROOT:-${SNAP_ROOT%/}/build}" [ -d "$srcdir" ] || die "no $srcdir" - say "Tidy $srcdir:" - say " * files owned by a UID that does not exist here -> root" - say " * leftover build markers (.cc-build-*, .cc-dir-*) -> removed" + say "Two directories, two rules:" + say " $srcdir -- downloaded tarballs only" + say " * the directory itself -> root:root 1777 (book 3.1)" + say " * every file in it -> root:root (book 3.1: a host uid means" + say " nothing here, and one above 10000" + say " collides with a real package user)" + say " $blddir -- build scratch" + say " * leftover build markers (.cc-build-*, .cc-dir-*) -> removed" + say " (unpacked trees are left alone: a build may be using one. To" + say " drop them all, delete $blddir -- it is rebuilt on demand.)" say "" local orphans markers - orphans="$(find "$srcdir" -maxdepth 1 \( -nouser -o -nogroup \) 2>/dev/null | sort)" - markers="$(find "$srcdir" -maxdepth 1 -name '.cc-*' 2>/dev/null | sort)" + # Not just -nouser: a uid that DOES resolve here can still be the wrong + # one -- the host's `lfs` is created without a pinned uid, so it may land + # on top of a package user and report a plausible, wrong name. + orphans="$(find "$srcdir" -maxdepth 1 -mindepth 1 -not -type d \ + \( -not -user root -o -not -group root \) 2>/dev/null | sort)" + markers="$(find "$srcdir" "$blddir" -maxdepth 1 -name '.cc-*' \ + 2>/dev/null | sort)" local n_o n_m n_o=$(printf '%s\n' "$orphans" | count_stdin_lines) n_m=$(printf '%s\n' "$markers" | count_stdin_lines) - say " $n_o path(s) with an unknown owner" + say " $n_o file(s) not owned by root" say " $n_m build marker(s)" if [ "$run" = 0 ]; then [ "$n_o" -gt 0 ] && printf '%s\n' "$orphans" | head -6 | sed 's/^/ /' @@ -1557,7 +2165,11 @@ cmd_clean_sources() { warn "(dry run -- nothing was changed. Add --run to apply.)" return 0 fi - [ "$n_o" -gt 0 ] && chown -R root:root $orphans 2>/dev/null + real_chown root:root "$srcdir" 2>/dev/null + real_chmod 1777 "$srcdir" 2>/dev/null + [ -d "$blddir" ] && { chown root:root "$blddir" 2>/dev/null + real_chmod 1777 "$blddir" 2>/dev/null; } + [ "$n_o" -gt 0 ] && real_chown -h root:root $orphans 2>/dev/null [ "$n_m" -gt 0 ] && rm -f $markers say "" ok " tidied: $n_o ownership(s) reset, $n_m marker(s) removed" @@ -1634,8 +2246,6 @@ cmd_fix_perms() { # commands (mkdir, chgrp, chown, chmod, install) first in the package user's # PATH. They neutralise the impossible parts, report what they skipped, and # otherwise behave normally. -WRAPPERS="$STATE/wrappers" - make_wrappers() { mkdir -p "$WRAPPERS" @@ -1783,8 +2393,18 @@ echo "*** mkdir $* (directory already exists -- continuing)" >&2 exit 0 EOF + # plain chmod here on purpose: this function creates the wrappers, and the + # chmod wrapper execs the real binary for everything except setuid. Using + # real_chmod would tie make_wrappers to a helper defined far above it, for + # no gain -- ownership is not being decided here, only the mode of files we + # just wrote ourselves. chmod 755 "$WRAPPERS"/* - chmod a+rx "$WRAPPERS" 2>/dev/null || true + # 0755 root:root, explicitly. This directory sits under /usr/lib, which IS + # an install directory: root:install and group-writable during the build. + # Inheriting that would let any package user rewrite the wrappers -- that + # is, rewrite the rule that stops it changing ownership. + chmod 0755 "$WRAPPERS" 2>/dev/null || true + chown root:root "$WRAPPERS" "$WRAPPERS"/* 2>/dev/null || true } # Re-apply the install-directory permissions. @@ -1806,12 +2426,12 @@ ensure_install_dirs_writable() { local _grp _mode _grp="$(stat -c '%G' "$d" 2>/dev/null)" || continue _mode="$(stat -c '%a' "$d" 2>/dev/null)" || continue - if [ "$_grp" = "install" ] && [ $(( (10#$_mode / 10) % 10 & 2 )) -ne 0 ] - then - continue # already install + group-writable + if install_dir_owner_ok "$d" \ + && [ $(( (10#$_mode / 10) % 10 & 2 )) -ne 0 ]; then + continue # already root:install, g+w fi - chgrp install "$d" 2>/dev/null || continue - chmod g+w "$d" 2>/dev/null || continue + set_install_dir_owner "$d" || continue + real_chmod g+w "$d" 2>/dev/null || continue fixed=$((fixed+1)) done < <(install_dirs_list) @@ -1824,9 +2444,10 @@ ensure_install_dirs_writable() { local _g _m _g="$(stat -c '%G' "$d" 2>/dev/null)" || continue _m="$(stat -c '%a' "$d" 2>/dev/null)" || continue - [ "$_g" = "install" ] && [ $(( (10#$_m / 10) % 10 & 2 )) -ne 0 ] && continue - chgrp install "$d" 2>/dev/null || continue - chmod g+w "$d" 2>/dev/null || continue + install_dir_owner_ok "$d" \ + && [ $(( (10#$_m / 10) % 10 & 2 )) -ne 0 ] && continue + set_install_dir_owner "$d" || continue + real_chmod g+w "$d" 2>/dev/null || continue fixed=$((fixed+1)) done < <( find "$r/usr/share/man" "$r/usr/share/locale" "$r/usr/share/terminfo" \ @@ -1834,7 +2455,7 @@ ensure_install_dirs_writable() { ) [ "$fixed" -gt 0 ] && \ - say "${C_DIM}# re-applied group-writability to $fixed install dir(s)${C_OFF}" + detail "# re-applied group-writability to $fixed install dir(s)" return 0 } @@ -1870,14 +2491,14 @@ diagnose_paths_from_log() { case "$p" in "${SNAP_ROOT%/}"/*) echo "$p" ;; /usr/*|/etc/*|/var/*|/opt/*|/lib/*|/lib64/*|/bin/*|\ - /sbin/*|/home/*|/root/*|/srv/*|/boot/*|/tools/*|/sources/*) + /sbin/*|/home/*|/root/*|/srv/*|/boot/*|/tools/*|/sources/*|/build/*) echo "$p" ;; esac done | head -8)" [ -n "$errpaths" ] || return 0 say "" - say "${C_WARN}--- what those paths actually are ---${C_OFF}" + note "--- what those paths actually are ---" while IFS= read -r p; do [ -n "$p" ] || continue if [ -L "$p" ] && [ ! -e "$p" ]; then @@ -1905,7 +2526,7 @@ diagnose_paths_from_log() { fi shown=$((shown+1)) done <<< "$errpaths" - [ "$shown" -gt 0 ] && say "${C_WARN}-------------------------------------${C_OFF}" + [ "$shown" -gt 0 ] && note "-------------------------------------" return 0 } @@ -1938,17 +2559,17 @@ pregrant_python_sitedirs() { # on every later run, so it never installs system-wide. Only that one user # can import it; every other package fails with # BackendUnavailable: Cannot import 'flit_core.buildapi' - local ulocal="$SRCROOT/$user/.local" + local ulocal="$(pkgusr_home_for "$user")/.local" if [ -d "$ulocal/lib" ]; then - say "${C_WARN}# removing a stale per-user install in $ulocal${C_OFF}" - say "${C_DIM}# (pip put it there because site-packages was not writable;" - say "# nothing outside '$user' can import from it)${C_OFF}" + note "# removing a stale per-user install in $ulocal" + detail "# (pip put it there because site-packages was not writable;" + detail "# nothing outside '$user' can import from it)" rm -rf "$ulocal/lib" fi for d in "$r"/usr/lib/python*/site-packages "$r"/usr/lib/python*; do [ -d "$d" ] || continue su -s /bin/bash "$user" -c "test -w '$d'" 2>/dev/null && continue - say "${C_DIM}# pip package: making sure '$user' can write $d${C_OFF}" + detail "# pip package: making sure '$user' can write $d" grant_dir_access "$d" "$user" break done @@ -1968,13 +2589,13 @@ _note_interrupted() { # Packages recorded as mid-install and never finished. _interrupted_packages() { - [ -f "$STATE/interrupted" ] || return 1 - sort -u "$STATE/interrupted" 2>/dev/null | grep -v '^$' + [ -f "$STATE_PROGRESS/interrupted" ] || return 1 + sort -u "$STATE_PROGRESS/interrupted" 2>/dev/null | grep -v '^$' } _clear_interrupted() { local name="$1" f tmp - f="$STATE/interrupted" + f="$STATE_PROGRESS/interrupted" [ -f "$f" ] || return 0 tmp="$f.$$" grep -vx "$name" "$f" > "$tmp" 2>/dev/null @@ -1992,10 +2613,93 @@ _clear_interrupted() { # # A manifest should list what this step owns, not everything that happened # while it ran. +# Files that belong to NO package, however many packages write to them. +# +# These are shared indexes and generated caches: every package that installs an +# info page rewrites /usr/share/info/dir, and ldconfig rewrites /etc/ld.so.cache +# and /var/cache/ldconfig/aux-cache after every install. They are not "owned" +# by whoever touched them last, so recording them in a manifest makes `verify` +# report a conflict that no repair can settle -- the next package will just +# take them again: +# /usr/share/info/dir is p_gcc want p_binutils +# /usr/share/info/dir is p_gcc want p_glibc +# /etc/ld.so.cache is p_libcap want p_glibc +# /var/cache/ldconfig/aux-cache is p_libcap want p_zlib +# Root keeps them, and no manifest claims them. +never_claim_list() { + cat <<'NCLIST' +/usr/share/info/dir +/etc/ld.so.cache +/etc/ld.so.conf +/var/cache/ldconfig/aux-cache +/etc/mtab +/usr/lib/pkgusr +NCLIST + # The user database, and everything Shadow writes beside it. + # + # These are root's, permanently and by definition -- the whole scheme is + # stored in them. They are not "installed" by anything, but every + # cmd_add_user during a build touches them, so whichever package happened to + # be building at the time swept them into its manifest: + # /etc/.pwd.lock is root want p_ncurses + # /etc/group- is root want p_ncurses + # /etc/passwd- is root want p_psmisc + # ncurses did not install /etc/group-; useradd did, while ncurses was + # building. A package owning the file that says which packages exist is + # also the one ownership mistake with no way back. + printf '%s\n' \ + "$ETC/passwd" "$ETC/passwd-" "$ETC/shadow" "$ETC/shadow-" \ + "$ETC/group" "$ETC/group-" "$ETC/gshadow" "$ETC/gshadow-" \ + "$ETC/.pwd.lock" "$ETC/subuid" "$ETC/subgid" \ + "$ETC/subuid-" "$ETC/subgid-" + + # Mail spools. Shadow creates /var/mail/<name> for every account it makes + # (CREATE_MAIL_SPOOL), so they arrive as a side effect of `useradd` and no + # package ever installs one. Worse, two passes disagreed about this one: + # + # lfs-helper verify --fix builduser /var/mail/lfs lfs -> root + # lfs-helper verify 1 root-owned file(s) that NO manifest claims: + # /var/mail/lfs + # + # The first gave it to root because the build user held it; the second then + # complained that root holds a file no manifest claims. Repairing it made + # the next run report it. A spool belongs to the account it is named for -- + # neither pass should touch it. + printf '%s\n' "${SNAP_ROOT%/}/var/mail" + + # The state directory itself. + # + # It moved under /usr/src in 1.9.0, and /usr/src IS an install directory -- + # so `verify` began walking its own manifests, scripts and logs and + # reporting every one of them as a root-owned file no package claims. A + # clean 104/104 build came back with + # 448 root-owned file(s) that NO manifest claims + # and all 448 were this directory, including the temp files the scan had + # just written for itself. + # + # Nothing in here is a package's file. It is what the build knows ABOUT + # the packages. + printf '%s\n' "$STATE" +} + +is_never_claimed() { + local f="${1%/}" p + [ -n "$f" ] || return 1 + while IFS= read -r p; do + [ -n "$p" ] || continue + p="${SNAP_ROOT%/}${p}" + [ "$f" = "${p%/}" ] && return 0 + case "$f" in "${p%/}"/*) return 0 ;; esac + done < <(never_claim_list) + return 1 +} + _drop_other_packages_files() { local owner="$1" p cur while IFS= read -r p; do [ -n "$p" ] || continue + # a shared index or a generated cache is nobody's file + is_never_claimed "$p" && continue cur="$(stat -c %U "$p" 2>/dev/null)" if [ -n "$cur" ] && [ "$cur" != "$owner" ] && [ "$cur" != root ] \ && [ "$cur" != UNKNOWN ] && user_exists "$cur"; then @@ -2005,49 +2709,550 @@ _drop_other_packages_files() { done } -cmd_build() { - need_root - local name="" phase="all" force=0 run_tests=0 jobs="" auto_fix=1 - while [ $# -gt 0 ]; do - case "$1" in - --phase) phase="$2"; shift 2 ;; - --force) force=1; shift ;; - --tests) run_tests=1; shift ;; - --jobs|-j) jobs="${2:?--jobs needs a number}"; shift 2 ;; - --no-auto-fix) auto_fix=0; shift ;; - -*) die "unknown option: $1" ;; - *) name="$1"; shift ;; - esac - done - [ -n "$name" ] || die "usage: lfs-helper build <package> [--phase P] [--force]" - local s; s="$(script_for "$name")" - [ -f "$s" ] || die "no script for '$name' at $s - Generate them from outside the chroot: - lfs build-system gen-chroot-scripts --run" +# --------------------------------------------------------------------------- # +# Staged installs +# --------------------------------------------------------------------------- # +# `make install` writes straight into the live system, one file at a time, in +# whatever order the Makefile happens to use. Interrupt it and the system is +# neither the old version nor the new one -- and for a toolchain package that +# is unrecoverable: chapter 8's gcc installs its driver early and its C++ +# headers late, so a cancel in between leaves a compiler that cannot compile +# C++, which is exactly what is needed to rebuild it. +# +# So install into a staging directory first and move the result into place: +# +# make install DESTDIR=<stage> the long part; a cancel here costs time +# check the staging tree did anything actually land? +# move it into place short, and moves whole files +# +# DESTDIR is honoured by every autotools, meson and cmake install, which is +# what the books use. A script that ignores it writes to the live system +# anyway -- detected below, and reported rather than silently trusted. - # Clear a stale source tree first, as root -- independent of everything - # below, and it must happen even if staging later has trouble. - case "$phase" in - all|unpack) clean_stale_source "$name" "$s" ;; - esac +_stage_dir_for() { echo "$STATE/stage/$1"; } - if is_done "$name" && [ "$force" = 0 ]; then - if [ -t 0 ]; then - printf "'%s' is already built. Rebuild it? [y/N]: " "$name" - local ans; read -e -r ans - case "$ans" in [Yy]*) ;; *) say "# leaving '$name' as it is"; return 0 ;; esac - else - say "# '$name' is already built (skipping; --force to rebuild)" - return 0 - fi - fi +# Packages where an interrupted install cannot be recovered from. +# +# Staging costs roughly an extra copy of everything installed, so it is not +# worth paying for every package: if man-pages is interrupted you rebuild +# man-pages. If GCC or Glibc is interrupted you have no compiler and no libc, +# and nothing left to rebuild them WITH -- the system is finished. +# +# These are the packages the rest of the build depends on to build at all. +# Override with `lfs config stage_packages` (a space-separated list), or per +# build with --stage / --no-stage. +STAGE_PACKAGES_DEFAULT="gcc glibc binutils bash coreutils" + +_should_stage() { + local name="$1" list item base + list="${LFS_STAGE_PACKAGES:-$STAGE_PACKAGES_DEFAULT}" + case "$list" in + all) return 0 ;; + none) return 1 ;; + esac + # -tmp variants are the same package at an earlier stage + base="${name%-tmp}" + base="${base%-pass1}" + base="${base%-pass2}" + for item in $list; do + [ "$item" = "$name" ] && return 0 + [ "$item" = "$base" ] && return 0 + done + return 1 +} + +# Does this phase look like something DESTDIR can stage? +_phase_can_stage() { + local script="$1" phase="$2" name="$3" + # ONLY a lone install phase. + # + # `--phase all` runs build, install AND the book's post-install commands in + # one script, and those operate on the LIVE system: + # mv -v /usr/bin/chroot /usr/sbin + # With DESTDIR set, chroot is still in the staging tree at that point and + # the move fails. Staging cannot span a phase boundary that only exists + # inside the script, so it is limited to the phase where it is safe. + case "$phase" in install) ;; *) return 1 ;; esac + _should_stage "$name" || return 1 + grep -qE '^[[:space:]]*(make|ninja|meson)[^|;&]*install' "$script" 2>/dev/null +} + +# Move a staging tree into the live system. +# +# cp -a then remove: a rename would fail across filesystems, and merging into +# directories that already exist is the normal case. +# Fold staged directories onto the symlinks the live tree uses. +# +# Where the live tree has a symlink (bin -> usr/bin) and the staging tree has +# a real directory of the same name, move the contents to the symlink's target +# and drop the directory. Without this every package installing into /bin, +# /sbin, /lib or /lib64 fails to merge -- which is most of chapter 7. +# +# Driven by the live tree rather than a hardcoded list: whatever is a symlink +# there is what gets folded. +_resolve_staged_symlink_dirs() { + local stage="$1" d name live target rel dest + for d in "$stage"/*; do + [ -d "$d" ] || continue + [ -L "$d" ] && continue # already a symlink: leave it + name="$(basename "$d")" + live="${SNAP_ROOT%/}/$name" + [ -L "$live" ] || continue # not a symlink here: normal case + target="$(readlink -f "$live" 2>/dev/null)" || continue + [ -d "$target" ] || continue + rel="${target#${SNAP_ROOT%/}/}" + dest="$stage/$rel" + mkdir -p "$dest" || return 1 + say "# folding $name/ into $rel/ (the tree has $name as a symlink)" + cp -a "$d"/. "$dest"/ || return 1 + rm -rf "$d" || return 1 + done + return 0 +} + +# Put a staging tree into the live system, one file at a time, atomically. +# +# `cp -a` overwrites in place: it opens the destination with O_TRUNC and +# writes. For an ordinary file that is fine. For a shared library that is +# CURRENTLY MAPPED it is fatal -- every running process sees the file shrink +# under it. Copying glibc's libc.so.6 that way killed the shell mid-merge: +# +# /usr/bin/env: error while loading shared libraries: +# /usr/lib/libc.so.6: file too short +# +# and the chroot could no longer start any program at all. +# +# So write each file beside its destination and rename() over it. A rename is +# atomic: processes holding the old file keep the old inode until they exit, +# and nothing ever observes a partial file. This is what package managers do, +# and it is the only safe way to replace a library a running system is using. +_install_staged_tree() { + local stage="$1" root="${SNAP_ROOT:-/}" rel dest tmp rc=0 d f + + # directories first, so every destination exists before anything is moved + while IFS= read -r d; do + rel="${d#$stage}" + dest="${root%/}$rel" + if [ -d "$dest" ]; then + # ALREADY THERE -- leave its ownership and mode alone. + # + # The staging tree is created by the package user, so $stage/usr, + # $stage/usr/bin and so on are owned by that user with mode 755. + # Stamping that onto the live tree handed /usr and /usr/bin to + # whichever package was building -- losing the install group and + # g+w, i.e. the entire install-directory scheme, undone by one + # package. Only stamp what this call actually creates. + continue + fi + mkdir -p "$dest" || { rc=1; break; } + real_chown --reference="$d" "$dest" 2>/dev/null || true + real_chmod --reference="$d" "$dest" 2>/dev/null || true + done < <(find "$stage" -mindepth 1 -type d 2>/dev/null | sort) + [ "$rc" = 0 ] || return 1 + + # then files and symlinks, each renamed into place + while IFS= read -r f; do + rel="${f#$stage}" + dest="${root%/}$rel" + tmp="$dest.pkgusr-new.$$" + rm -f "$tmp" + if ! cp -a "$f" "$tmp" 2>/dev/null; then + warn "# could not stage $rel" + rc=1 + continue + fi + # same directory as the destination, so this is a rename, not a copy + if ! mv -f "$tmp" "$dest" 2>/dev/null; then + warn "# could not replace $rel" + rm -f "$tmp" + rc=1 + fi + done < <(find "$stage" -mindepth 1 \( -type f -o -type l \) 2>/dev/null) + + return $rc +} + +_merge_stage() { + local stage="$1" n=0 + [ -d "$stage" ] || return 1 + # nothing staged means DESTDIR was ignored -- say so rather than reporting + # a successful install of nothing + n="$(find "$stage" -mindepth 1 \( -type f -o -type l \) 2>/dev/null | wc -l)" + if [ "${n:-0}" = 0 ]; then + return 2 + fi + # The live tree has /bin -> usr/bin, /sbin -> usr/sbin and /lib -> usr/lib + # (book 4.2). A package that installs into /bin gets a REAL bin/ directory + # in the staging tree, and copying that over the symlink fails: + # cp: cannot overwrite non-directory '/bin' with directory '.../bin' + # So resolve those first: move the contents to where the symlink points. + _resolve_staged_symlink_dirs "$stage" || return 1 + + say "# moving $n staged file(s) into place ..." + # into the tracked root, not a hardcoded "/" -- inside the chroot they are + # the same thing, but hardcoding it makes this untestable and would write + # to the host if anything ever ran it from outside + _install_staged_tree "$stage" || return 1 + rm -rf "$stage" + return 0 +} + +# Hand this package its own files from earlier stages of the build. +# +# A package is built more than once: ncurses in chapter 6 (as the lfs user), +# perl-tmp in chapter 7 (as root), then the real one in chapter 8 as its +# package user. The earlier files are still owned by whoever built them, and +# the package user cannot overwrite or re-mode them: +# +# cp: cannot create regular file '/usr/bin/tic': Permission denied +# Couldn't chmod 644 /usr/lib/perl5/.../perl.pod: Operation not permitted +# +# Same cause, different chapter. So before building X as user X, give X every +# path recorded under X, X-tmp, X-pass1 and X-pass2 -- but only paths still +# owned by root or the build user, never one belonging to another package. +_claim_earlier_stages() { + local name="$1" owner="$2" man base n=0 p cur + [ -n "$owner" ] || return 0 + user_exists "$owner" || return 0 + + # Book titles are capitalised ("Python", "Util-linux") but the earlier + # stage is named from the tarball ("python-tmp"). Matching only the + # step's own spelling meant Python never claimed python-tmp's files, and + # chapter 7's root-owned tree blocked the chapter-8 install: + # install: cannot remove '/usr/lib/python3.13/idlelib/Icons/README.txt' + # Try the step name, the owner name, and both lowercased. + local lname owname + lname="$(printf '%s' "$name" | tr 'A-Z' 'a-z')" + owname="$(printf '%s' "$owner" | tr 'A-Z' 'a-z')" + local -a bases=() + local stem + for stem in "$name" "$lname" "$owner" "$owname"; do + [ -n "$stem" ] || continue + bases+=("$stem" "$stem-tmp" "$stem-pass1" "$stem-pass2") + done + # de-duplicate, so a manifest is not walked twice + local seen="" b + for b in "${bases[@]}"; do + case " $seen " in *" $b "*) continue ;; esac + seen="$seen $b" + done + for base in $seen; do + man="$MANIFESTS/$base.files" + [ -s "$man" ] || continue + while IFS= read -r p; do + [ -n "$p" ] || continue + p="$(strip_host_prefix "$p")" + [ -e "$p" ] || [ -L "$p" ] || continue + cur="$(stat -c %U "$p" 2>/dev/null)" || continue + [ "$cur" = "$owner" ] && continue + # only what the build itself left behind -- never another package's + case "$cur" in + root|lfs|UNKNOWN) ;; + *) continue ;; + esac + real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) + done < "$man" + # the directories too, or the package cannot create files in them + man="$MANIFESTS/$base.dirs" + [ -s "$man" ] || continue + while IFS= read -r p; do + [ -n "$p" ] || continue + p="$(strip_host_prefix "$p")" + [ -d "$p" ] || continue + cur="$(stat -c %U "$p" 2>/dev/null)" || continue + [ "$cur" = "$owner" ] && continue + is_install_dir "$p" && continue # shared, leave it alone + case "$cur" in + root|lfs|UNKNOWN) ;; + *) continue ;; + esac + real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) + done < "$man" + done + [ "$n" -gt 0 ] && say "# gave '$owner' $n path(s) from its earlier build stages" + return 0 +} + +# Book 7.6 rewrites /etc/passwd and /etc/group with a TRUNCATING `>`. +# +# That is correct the first time -- it creates the base accounts in a tree that +# has none. Run it again once package users exist and it wipes every package +# user, every private group, the install group, every collector group and every +# membership. The files on disk then all read as UNKNOWN, and the collector +# map no longer matches reality. +# +# So save the entries this build created, let the step do exactly what the book +# says, and put them back. +_PKGUSR_UID_MIN=10000 + +_protect_user_db() { + local f keep + for f in passwd group; do + [ -f "$ETC/$f" ] || continue + keep="$STATE/tmp/$f.keep.$$" + mkdir -p "$STATE/tmp" 2>/dev/null || true + # package users, collector groups, and the install group with members + awk -F: -v umin="$_PKGUSR_UID_MIN" -v gmin="$COLLECTOR_GID_MIN" \ + '$3 >= umin || $3 >= gmin || $1 == "install" || $1 == "lfs"' \ + "$ETC/$f" > "$keep" 2>/dev/null || true + done +} + +_restore_user_db() { + local f keep n line name + for f in passwd group; do + keep="$STATE/tmp/$f.keep.$$" + [ -s "$keep" ] || { rm -f "$keep"; continue; } + n=0 + while IFS= read -r line; do + name="${line%%:*}" + [ -n "$name" ] || continue + grep -q "^$name:" "$ETC/$f" 2>/dev/null && continue + # never restore an entry whose id has since been handed to someone + # else -- that is exactly how duplicate uids appear + local rid; rid="$(printf '%s' "$line" | cut -d: -f3)" + if [ -n "$rid" ] && awk -F: -v i="$rid" '$3 == i { found = 1 } + END { exit !found }' "$ETC/$f" 2>/dev/null; then + warn "# not restoring '$name': id $rid is now used by "\ + "$(awk -F: -v i="$rid" '$3 == i { print $1; exit }' "$ETC/$f")" + continue + fi + printf '%s\n' "$line" >> "$ETC/$f" + n=$((n+1)) + done < "$keep" + [ "$n" -gt 0 ] && \ + say "# put back $n $f entry(ies) the book's 7.6 block would have wiped" + rm -f "$keep" + done + _vfy_user_order 1 >/dev/null 2>&1 || true +} + +# Does this step rewrite the user database? +_step_rewrites_user_db() { + # any truncating redirect into a passwd or group file, wherever $ETC points + grep -qE '>[[:space:]]*[^>|&]*/(passwd|group)([[:space:]]|$)' "$1" 2>/dev/null +} + +# The generated scripts carry "# package: <name>-<version>" in their header. +_version_from_script() { + [ -r "${1:-}" ] || return 0 + sed -n 's/^# *package *: *//p' "$1" 2>/dev/null | head -1 \ + | sed 's/^.*-\([0-9][0-9.]*\)$/\1/' +} + +_write_pkgusr_info() { + local name="$1" owner="$2" source="${3:-lfs}" version="${4:-}" + local home="$(pkgusr_home_for "$name")" + [ -d "$home" ] || return 0 + { + printf 'source=%s\n' "$source" + printf 'package=%s\n' "$name" + [ -n "$version" ] && printf 'version=%s\n' "$version" + printf 'installed=%s\n' "$(date '+%Y-%m-%d %H:%M:%S' 2>/dev/null)" + printf 'by=lfs-helper %s (build %s)\n' "$LFS_HELPER_VERSION" "$(_build_id)" + } > "$home/.pkgusr" 2>/dev/null || return 0 + [ -n "$owner" ] && real_chown "$owner:$owner" "$home/.pkgusr" 2>/dev/null + real_chmod 644 "$home/.pkgusr" 2>/dev/null + return 0 +} + +# Does this package already own files in the tree? +# +# The counters answer "what did this RUN write?", which is the wrong question +# when a package is built a second time: an installer that skips up-to-date +# files writes nothing, and nothing is indistinguishable from a silent failure +# by counting alone. The tree knows. +_pkg_owned_n=0 +# --------------------------------------------------------------------------- # +# accounts you can actually log in with +# --------------------------------------------------------------------------- # +# +# A built-in step, like init-ownership -- not a user-editable script. +# +# It lived in last_build_step.sh, which is created once on the host and then +# belongs to the person. Wrong home for it twice over: a fix to it reaches +# nobody who already has a copy (that is how `chown: invalid user: 'wget:wget'` +# survived three releases), and this is not a matter of taste anyway. Book 8.5 +# says to run `passwd root`. +# +# It cannot move to `packagemanager setup` either: setup runs on the BOOTED +# system, and you need to log in to run it. A fresh LFS has no root password, +# which is either "anyone can log in as root" or "nobody can log in at all" -- +# and you find out after a reboot, with no way in. So it happens here, inside +# the chroot, while there is still a working shell. +cmd_init_accounts() { + need_root + local force="${1:-}" + if is_done init-accounts && [ "$force" != "--force" ]; then + ok "# init-accounts: already done" + return 0 + fi + + # No terminal (a scripted or resumed run): say what is missing rather than + # blocking, and leave the system unbootable-but-KNOWN instead of hanging. + if [ ! -t 0 ]; then + warn "" + warn "!! Not running interactively, so no passwords were set." + warn " Before rebooting, set at least the root password:" + warn " lfs-helper init-accounts (from a terminal, in the chroot)" + warn " or from outside:" + warn " chroot /mnt/lfs /usr/bin/passwd root" + warn "" + return 0 + fi + + say "" + say "--- accounts ------------------------------------------------" + + if grep -qE '^root:[^:]*:' "$ETC/shadow" 2>/dev/null && + ! grep -qE '^root:(\*|!|)?:' "$ETC/shadow" 2>/dev/null; then + ok "root already has a password." + else + note "root has NO password yet. Without one you may not be able to" + note "log in after rebooting." + passwd root || warn "!! setting the root password failed -- do it before rebooting" + fi + + # The login account for a person. The name is configured up front, so this + # only has to create it and set a password. + local main + main="${LFS_MAIN_USER:-}" + [ -n "$main" ] || main="$(sed -n 's/^main_user=\(.*\)$/\1/p' \ + "$PKGUSR_ETC/packagemanager.conf" 2>/dev/null | head -n1)" + if [ -z "$main" ]; then + say "" + say "No login account is configured. You can add one now, or later with:" + hint " useradd -m -G users <name> && passwd <name>" + hint " (set it up front next time: lfs build-system session --reconfigure)" + printf "Create a login account now? [name, or blank to skip]: " + read -e -r main + fi + if [ -z "$main" ]; then + say "" + mark_done init-accounts + return 0 + fi + + if id "$main" >/dev/null 2>&1; then + ok "Account '$main' already exists." + else + say "Creating login account '$main' ..." + useradd -m -k /etc/skel -G users -s /bin/bash "$main" \ + || { warn "!! could not create $main"; mark_done init-accounts; return 0; } + fi + if grep -qE "^$main:(\*|!|)?:" "$ETC/shadow" 2>/dev/null; then + say "Set a password for '$main':" + passwd "$main" || warn "!! setting the password for $main failed" + fi + + say "" + hint " '$main' can be given access to application users with:" + hint " packagemanager user create <app> --shared --launcher" + say "" + mark_done init-accounts +} + +_package_already_owns_files() { + local owner="$1" d + _pkg_owned_n=0 + [ -n "$owner" ] || return 1 + user_exists "$owner" || return 1 + while IFS= read -r d; do + d="${SNAP_ROOT%/}$d" + [ -d "$d" ] || continue + _pkg_owned_n=$(( _pkg_owned_n + $(find "$d" -xdev -user "$owner" \ + \( -type f -o -type l \) 2>/dev/null | head -200 | wc -l) )) + [ "$_pkg_owned_n" -gt 0 ] && return 0 + done < <(install_dirs_list) + return 1 +} + +cmd_build() { + need_root + local name="" phase="all" force=0 run_tests=0 jobs="" auto_fix=1 + local _force_stage=auto + while [ $# -gt 0 ]; do + case "$1" in + --phase) phase="$2"; shift 2 ;; + --force) force=1; shift ;; + --stage) _force_stage=yes; shift ;; + --no-stage) _force_stage=no; shift ;; + --tests) run_tests=1; shift ;; + --jobs|-j) jobs="${2:?--jobs needs a number}"; shift 2 ;; + --no-auto-fix) auto_fix=0; shift ;; + -*) die "unknown option: $1" ;; + *) name="$1"; shift ;; + esac + done + [ -n "$name" ] || die "usage: lfs-helper build <package> [--phase P] [--force]" + + # A step this tool performs itself, with no script in the book. + # + # The ownership epoch used to be an invisible checkpoint inside build-all. + # It was in the right place and did the right thing, but it never appeared + # in `lfs-helper list`, so the one moment the whole scheme turns on was the + # one moment you could not point at, resume from, or re-run on its own. + if [ "$name" = "init-accounts" ]; then + cmd_init_accounts "$([ "$force" = 1 ] && echo --force)" + mark_done "$name" + return 0 + fi + if [ "$name" = "init-ownership" ]; then + if is_done "$name" && [ "$force" = 0 ]; then + ok "# init-ownership: already done" + return 0 + fi + if ! ownership_possible explain; then + die "init-ownership cannot run yet: book 7.6 (init-files) creates + the user database, and until it exists no name resolves -- not even root's." + fi + establish_ownership 1 || die "could not establish ownership" + mark_done "$name" + return 0 + fi + + local s; s="$(script_for "$name")" + if [ ! -f "$s" ]; then + # A step in the order with no script and no built-in handler almost + # always means the ORDER is older than the tools -- the step was + # removed and the tree still lists it. Say that, rather than only + # offering to generate scripts that may already be there. + _warn_if_scripts_are_stale || true + die "no script for '$name' at $s + Either the step order is older than these tools, or the scripts were never + generated. From outside the chroot: + lfs build-system gen-chroot-scripts --run --overwrite + If '$name' no longer exists in this version, clear it from the order: + lfs-helper undone $name" + fi + + # Clear a stale source tree first, as root -- independent of everything + # below, and it must happen even if staging later has trouble. + case "$phase" in + all|unpack) clean_stale_source "$name" "$s" ;; + esac + + if is_done "$name" && [ "$force" = 0 ]; then + if [ -t 0 ]; then + printf "'%s' is already built. Rebuild it? [y/N]: " "$name" + local ans; read -e -r ans + case "$ans" in [Yy]*) ;; *) say "# leaving '$name' as it is"; return 0 ;; esac + else + say "# '$name' is already built (skipping; --force to rebuild)" + return 0 + fi + fi local chown_after=0 - local as_root=0 staged owner="$name" + # One derivation of the staging directory, held in a variable. The script + # is installed there, the build cd's there, and the failure banner names + # it -- five call sites that used to re-derive it, and so could disagree. + local as_root=0 staged stagedir owner="$name" if is_root_step "$name"; then as_root=1 - staged="$SRCROOT/$name/install_$name" - mkdir -p "$SRCROOT/$name" + stagedir="$(step_stage_dir "$name")" + staged="$stagedir/install_$name" + mkdir -p "$stagedir" install -m 0755 "$s" "$staged" || die "could not stage the install script" # A *-tmp temporary tool is the SAME package chapter 8 rebuilds, so its # files should belong to that package user from the outset -- otherwise @@ -2069,7 +3274,8 @@ cmd_build() { pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" owner="$(pkg_owner_name "$name")" cmd_add_user "$owner" - staged="$SRCROOT/$owner/install_$name" + stagedir="$(pkgusr_home_for "$owner")" + staged="$stagedir/install_$name" install -m 0755 -o "$owner" -g "$owner" "$s" "$staged" \ || die "could not stage the install script" fi @@ -2083,13 +3289,13 @@ cmd_build() { local _book_of_script _book_of_script="$(sed -n 's/^### book *: *//p' "$s" 2>/dev/null | head -n1)" - say "${C_DIM}# script : $s${C_OFF}" + detail "# script : $s" [ -n "$_book_of_script" ] && \ - say "${C_DIM}# book : $_book_of_script${C_OFF}" + detail "# book : $_book_of_script" if [ "$as_root" = 1 ]; then - say "${C_DIM}# user : root (temporary chapter-7 tool) phase: $phase${C_OFF}" + detail "# user : root (temporary chapter-7 tool) phase: $phase" else - say "${C_DIM}# user : $owner (package user) phase: $phase${C_OFF}" + detail "# user : $owner (package user) phase: $phase" fi [ "$phase" = "all" ] && clear_phases "$name" # Record that this package is MID-INSTALL. Cancelling here (Ctrl-C during @@ -2098,7 +3304,7 @@ cmd_build() { # installed the C++ headers -- so the whole toolchain stops working and # nothing on disk says why. The marker is removed when the build finishes, # whether it succeeded or failed cleanly. - local _interrupted="$STATE/interrupted" + local _interrupted="$STATE_PROGRESS/interrupted" mkdir -p "$STATE" 2>/dev/null || true echo "$name" >> "$_interrupted" 2>/dev/null || true # Interrupting during an install leaves the package half on disk, and for @@ -2107,12 +3313,80 @@ cmd_build() { # see the auto-finish in cmd_build_all. trap '_note_interrupted "$name"; exit 130' INT TERM + # An earlier stage of this same package may still own files it is about + # to replace. Claim them first -- see _claim_earlier_stages. + [ "$as_root" = 1 ] || _claim_earlier_stages "$name" "$owner" + + # Book 7.6 truncates /etc/passwd and /etc/group. Keep what this build + # created -- see _protect_user_db. + local _guard_userdb=0 + if _step_rewrites_user_db "$staged"; then + _guard_userdb=1 + _protect_user_db + fi + say "# building $name ..." local rc=0 make_wrappers ensure_install_dirs_writable [ "$as_root" = 1 ] || pregrant_python_sitedirs "$owner" "$s" - local envpass="LFS_CC_PHASE='$phase'" + # Stage the install where the package supports it, so a cancel cannot + # leave the live system half-updated. + local _stage="" + if [ "${_force_stage:-auto}" != no ] \ + && { [ "${_force_stage:-auto}" = yes ] \ + || _phase_can_stage "$staged" "$phase" "$name"; }; then + _stage="$(_stage_dir_for "$name")" + rm -rf "$_stage" + if mkdir -p "$_stage" 2>/dev/null; then + # The build runs as the PACKAGE USER, but this directory was just + # created by root -- so `make install DESTDIR=...` could not write + # into it: + # mkdir: cannot create directory '.../stage/gcc': Permission denied + # Hand it to whoever will be building. + if [ "$as_root" != 1 ] && user_exists "$owner"; then + real_chown "$owner:$owner" "$_stage" 2>/dev/null \ + || warn "# could not give $_stage to '$owner'" + fi + real_chmod 755 "$_stage" 2>/dev/null || true + # the parent too, or the package user cannot traverse into it + real_chmod 755 "$(dirname "$_stage")" 2>/dev/null || true + else + _stage="" + fi + fi + + # The scratch this step unpacks and builds in -- its own package user's + # home, not a shared /build. See build_root_for. + local _bldroot; _bldroot="$(build_root_for "$name")" + # A build tree that is not a directory is fatal, and must say so HERE. + # + # `mkdir -p` on an existing symlink fails with "File exists" and the build + # carried on to fail two lines later with a message about the wrong thing: + # mkdir: cannot create directory '.../p_gettext/build': File exists + # cd: /usr/src/pkgusr/p_gettext/build: Not a directory + # which the recovery code then read as a permissions problem and retried, + # twice, before giving up with advice about collector groups. The actual + # cause was a name collision with ~/build, the hint's build helper. + if [ -e "$_bldroot" ] && [ ! -d "$_bldroot" ]; then + die "the build tree for '$name' is not a directory: + $_bldroot ($(stat -c %F "$_bldroot" 2>/dev/null)) + Something else already owns that name in the package user's home. + Rename it, or set a different one: + LFS_PKGUSR_BUILD_SUBDIR=<name> lfs-helper build $name" + fi + if ! mkdir -p "$_bldroot" 2>/dev/null; then + die "could not create the build tree for '$name': $_bldroot" + fi + if [ "$as_root" != 1 ] || [ "$owner" != "$name" ]; then + # give it to whoever will be building, so the tree it creates is + # already owned correctly and nothing has to be chowned afterwards + user_exists "$owner" && real_chown "$owner:$owner" "$_bldroot" 2>/dev/null + fi + real_chmod 755 "$_bldroot" 2>/dev/null || true + + local envpass="LFS_CC_PHASE='$phase' BUILD_ROOT='$_bldroot'" + [ -n "$_stage" ] && envpass="$envpass DESTDIR='$_stage'" # The wrappers exist to stop a PACKAGE USER doing things it may not: chown, # setuid modes, re-moding shared directories. A ROOT step must not get # them. Root legitimately runs `chown -R wget:wget /sources/wget-1.25.0` @@ -2174,24 +3448,57 @@ cmd_build() { warn "# cannot create a build stamp -- file tracking is off for this step" fi if [ "$as_root" = 1 ]; then - ( cd "$SRCROOT/$name" && eval "env $envpass bash '$staged' '$phase'" ) \ + ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} elif have_su; then # `su -` resets the environment, so the vars go on the command line su - "$owner" -c "env $envpass bash '$staged' '$phase'" \ 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} else - say "${C_DIM}# no 'su' yet (Shadow not built) -- building as root, then" - say "# giving the installed files to '$owner'${C_OFF}" + detail "# no 'su' yet (Shadow not built) -- building as root, then" + detail "# giving the installed files to '$owner'" chown_after=1 - ( cd "$SRCROOT/$name" && eval "env $envpass bash '$staged' '$phase'" ) \ + ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} fi + # Move the staged tree into place BEFORE looking at what changed, so the + # file tracking sees the finished system. Only on success: a failed + # install must leave the live system untouched, which is the whole point. + if [ -n "${_stage:-}" ]; then + if [ "${rc:-0}" = 0 ]; then + _merge_stage "$_stage" + case $? in + 0) ;; + 2) warn "# nothing was staged -- this package ignores DESTDIR" + warn "# it installed straight into the system, so an" + warn "# interrupted run here would leave it half-updated" + rm -rf "$_stage" ;; + *) warn "# could not move the staged files into place" + warn "# they are still in $_stage" + rc=1 ;; + esac + else + say "# install failed -- the system was not touched" + say "# (what it managed to build is in $_stage)" + fi + fi + + [ "${_guard_userdb:-0}" = 1 ] && _restore_user_db + after="$(snapshot)" local newfiles newdirs - newfiles="$(comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after"))" - newdirs="$(comm -13 <(printf '%s\n' "$before_dirs") <(snapshot_dirs))" + # LC_ALL=C on comm as well as on the sort that fed it. + # + # 1.9.6 pinned the sort and stopped there, which is half a fix: comm + # validates the ordering using ITS OWN locale, so C-sorted input compared + # under en_GB.UTF-8 is "not sorted" as far as comm is concerned -- + # comm: file 1 is not in sorted order + # comm: file 2 is not in sorted order + # and it then reports a diff that is simply wrong. The sort and the + # comparison have to agree; pinning either one alone guarantees they do not. + newfiles="$(LC_ALL=C comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after"))" + newdirs="$(LC_ALL=C comm -13 <(printf '%s\n' "$before_dirs") <(snapshot_dirs))" printf '%s\n' "$after" > "$SNAPSHOT" printf '%s\n' "$(snapshot_dirs)" > "$SNAPDIRS" @@ -2216,15 +3523,30 @@ cmd_build() { # became the package's, so a package owned its files but not the # directory holding them. Shared install dirs and directories another # package owns are filtered out below, exactly as for files. + # $BUILD_ROOT is excluded, exactly like $LFS/sources. + # + # It was not, and it is the busiest directory on the system during a + # build: unpacked source trees, object files, the tarballs themselves. + # Every one of them is newer than the stamp, so every package's manifest + # swallowed the whole scratch tree and then chowned it -- and the next + # package chowned it right back. A real `verify` came back with + # 3363 wrong-owner + # of which 3348 were /build, all reading + # /build/bash-5.3.tar.gz is p_shadow want p_acl + # Scratch belongs to no package. The two snapshot scans above already + # knew that; this one did not. Same rule, one more place. + local _bld="${BUILD_ROOT:-$r/build}" touched_dirs="$(find "${r:-/}" -xdev -type d -newer "$stamp" \ - -not -path "$r/.lfs-pkgusr*" -not -path "$SRCROOT/*" \ - -not -path "$r/sources/*" -not -path "$r/dev/*" \ + -not -path "$SRCROOT/*" \ + -not -path "$r/sources/*" -not -path "$_bld/*" \ + -not -path "$_bld" -not -path "$r/dev/*" \ -not -path "$r/proc/*" -not -path "$r/sys/*" \ -not -path "$r/run/*" -not -path "$r/tmp/*" \ 2>/dev/null | sort -u)" touched="$(find "${r:-/}" -xdev \( -type f -o -type l \) -newer "$stamp" \ - -not -path "$r/.lfs-pkgusr/*" -not -path "$SRCROOT/*" \ - -not -path "$r/sources/*" -not -path "$r/dev/*" \ + -not -path "$SRCROOT/*" \ + -not -path "$r/sources/*" -not -path "$_bld/*" \ + -not -path "$r/dev/*" \ -not -path "$r/proc/*" -not -path "$r/sys/*" \ -not -path "$r/run/*" -not -path "$r/tmp/*" \ 2>/dev/null | sort -u)" @@ -2282,7 +3604,7 @@ cmd_build() { skipped_foreign=$((skipped_foreign+1)) continue fi - chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) + real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) done <<< "$(printf '%s\n%s\n' "$newfiles" "$touched" | grep -v '^$' | sort -u)" while IFS= read -r p; do [ -n "$p" ] || continue @@ -2296,7 +3618,7 @@ cmd_build() { skipped_foreign=$((skipped_foreign+1)) continue fi - chown "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) + real_chown "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) done <<< "$(printf '%s\n%s\n' "$newdirs" "$touched_dirs" \ | grep -v '^$' | sort -u)" say "# gave $n newly installed path(s) to '$owner'" @@ -2343,13 +3665,13 @@ cmd_build() { say "" say "# retrying $name (phase $phase), round $_round ..." if [ "$as_root" = 1 ]; then - ( cd "$SRCROOT/$name" && eval "env $envpass bash '$staged' '$phase'" ) \ + ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} elif have_su; then su - "$owner" -c "env $envpass bash '$staged' '$phase'" \ 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} else - ( cd "$SRCROOT/$name" && eval "env $envpass bash '$staged' '$phase'" ) \ + ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} fi after="$(snapshot)" @@ -2383,7 +3705,7 @@ cmd_build() { # Report it plainly: no failure banner, no permission fixing, no retrying. if [ "$rc" = 3 ]; then say "" - say "${C_WARN}$name is waiting on you -- see the message above.${C_OFF}" + note "$name is waiting on you -- see the message above." say " When it's ready: lfs-helper build $name --force" trap - INT TERM _clear_interrupted "$name" @@ -2395,17 +3717,17 @@ cmd_build() { fi if [ "$rc" != 0 ]; then - echo "${C_ERR}" - echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" - echo "! $name FAILED (phase '$phase', exit $rc)" - echo "! script: $s" - echo "! staged: $SRCROOT/$name/install_$name" + fail "" + fail "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" + fail "! $name FAILED (phase '$phase', exit $rc)" + fail "! script: $s" + fail "! staged: $staged" if grep -qiE "permission denied|couldn't copy|cannot create|cannot remove" \ "$log" 2>/dev/null; then - echo "! The package could not write into a directory owned by another" - echo "! package. Grant access (collector groups) and retry:" - echo "! lfs-helper fix-perms $name --phase $phase --run" - echo "! lfs-helper build $name --phase $phase --force" + fail "! The package could not write into a directory owned by another" + fail "! package. Grant access (collector groups) and retry:" + fail "! lfs-helper fix-perms $name --phase $phase --run" + fail "! lfs-helper build $name --phase $phase --force" fi # A later phase often fails because an EARLIER one never finished -- # gcc's `mv /usr/lib/*gdb.py` finds nothing when `make install` aborted @@ -2417,25 +3739,25 @@ cmd_build() { phase_done "$name" "$_prev" || _missing="$_missing $_prev" done if [ -n "$_missing" ]; then - echo "! NOTE: these earlier phases have not completed successfully for" - echo "! this package:$_missing" - echo "! A later phase can fail simply because an earlier one" - echo "! stopped partway and never installed everything. Run:" + fail "! NOTE: these earlier phases have not completed successfully for" + fail "! this package:$_missing" + fail "! A later phase can fail simply because an earlier one" + fail "! stopped partway and never installed everything. Run:" for _prev in $_missing; do - echo "! lfs-helper build $name --phase $_prev --force" + fail "! lfs-helper build $name --phase $_prev --force" done fi fi - echo "! Edit the script, then retry just this phase:" - echo "! lfs-helper build $name --phase $phase --force" + fail "! Edit the script, then retry just this phase:" + fail "! lfs-helper build $name --phase $phase --force" if [ -z "$jobs" ] && [ "$phase" != "install" ]; then - echo "! If the output ends with no obvious error, a parallel build may" - echo "! have raced -- the real message is far above the tail. Retry" - echo "! serially (slower, but the error appears where it happens):" - echo "! lfs-helper build $name --phase $phase --force --jobs 1" + fail "! If the output ends with no obvious error, a parallel build may" + fail "! have raced -- the real message is far above the tail. Retry" + fail "! serially (slower, but the error appears where it happens):" + fail "! lfs-helper build $name --phase $phase --force --jobs 1" fi - echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" - echo "${C_OFF}" + fail "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" + fail "" show_build_errors "$log" diagnose_paths_from_log "$log" "$owner" exit "$rc" @@ -2451,14 +3773,68 @@ cmd_build() { local _tracked; _tracked=$(count_lines "$man") if [ "$_tracked" -eq 0 ] && [ "$_nothing_to_do" = 0 ] \ && [ "$phase" != "unpack" ] && [ "$phase" != "test" ]; then - warn "" - warn "!! $name reported success but installed NO files." - warn " That normally means the install silently did nothing." - warn " Check the log before trusting this step:" - warn " $log" - warn " The step is NOT marked as built." - warn "" - return 1 + # A CONFIGURATION step legitimately installs no new files. cfg_ld + # rewrites /etc/ld.so.conf -- which glibc already created, and still + # owns -- and makes one directory; nothing NEW appears, so the count + # is 0 and the step looked like a silent failure: + # mkdir: created directory '/etc/ld.so.conf.d' + # !! cfg_ld reported success but installed NO files. + # That is the normal shape of a config step, not a fault. Say what + # happened and move on; keep the hard failure for packages, where + # installing nothing really does mean the build did nothing. + if _is_not_a_package "$name"; then + detail "# $name installed no new files -- normal for a" + detail "# configuration step: it writes files another package" + detail "# already owns. Log: $log" + elif _package_already_owns_files "$owner"; then + # A RE-RUN over its own work is not a silent failure. + # + # Many installers skip a file that is already up to date -- perl's + # ExtUtils::Install prints "Installing <path>" for every target and + # then copies only the stale ones. So a second run of a package + # that already installed correctly writes nothing, the snapshot + # diff sees nothing new, no mtime is newer than the stamp, and the + # count is 0 for the best possible reason. + # + # xml-parser hit exactly this: forty files on disk, every one owned + # by p_xml-parser, and the step failed as a no-op. Ask the tree + # rather than the counters -- if this package already owns files, + # the install worked, whenever it happened. + ok "# $name installed no NEW files, but already owns $_pkg_owned_n" + ok "# file(s) on disk -- a re-run over its own work, not a failure." + detail "# Log: $log" + else + warn "" + warn "!! $name reported success but installed NO files." + warn " That normally means the install silently did nothing." + warn "" + # Show the working. There are exactly three ways to reach zero, + # and they need different fixes -- saying only "check the log" + # meant reading a thousand lines of make output to find out which. + local _nf _tc _kept + _nf="$(printf '%s\n' "$newfiles" | grep -cv '^$')" + _tc="$(printf '%s\n' "$touched" | grep -cv '^$')" + _kept="$(count_lines "$man")" + warn " new files (snapshot diff) : $_nf" + warn " files touched since stamp : $_tc" + warn " kept after ownership filter: $_kept" + if [ "$_nf" = 0 ] && [ "$_tc" = 0 ]; then + warn " -> the build wrote nothing this tool could see." + warn " Either the install really did nothing, or it wrote" + warn " somewhere excluded from tracking: $SRCROOT, /sources," + warn " ${BUILD_ROOT:-/build}, /tmp." + else + warn " -> $((_nf + _tc)) path(s) were seen and then dropped," + warn " because each already belongs to another package." + warn " That is a re-install over someone else's files." + warn " Which package: lfs-helper which-package <path>" + fi + warn "" + warn " Full log: $log" + warn " The step is NOT marked as built." + warn "" + return 1 + fi fi # Record what is installed, in the package user's own home. @@ -2468,17 +3844,19 @@ cmd_build() { # record of the latter, "is this up to date?" is unanswerable. local _nv _home _nv="$(sed -n 's/^name_version="\(.*\)"$/\1/p' "$s" 2>/dev/null | head -n1)" - _home="$SRCROOT/$owner" + _home="$(pkgusr_home_for "$owner")" if [ -n "$_nv" ] && [ -d "$_home" ]; then { echo "$_nv" echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper" [ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script" } > "$_home/VERSION" - chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true + real_chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true fi if [ "$phase" = "all" ]; then + _write_pkgusr_info "$name" "$owner" "${LFS_PKG_SOURCE:-lfs}" \ + "$(_version_from_script "$staged")" mark_done "$name" ok "# $name: done" report_granted_groups @@ -2492,6 +3870,8 @@ cmd_build() { say " still to do: $nxt -> lfs-helper build $name --phase $nxt" else # every phase the script defines has now run + _write_pkgusr_info "$name" "$owner" "${LFS_PKG_SOURCE:-lfs}" \ + "$(_version_from_script "$staged")" mark_done "$name" # This build already gave its files to the package user, so there # is nothing for the adoption pass to do. Recording it here is @@ -2515,7 +3895,7 @@ report_granted_groups() { local n; n=$(count_lines "$GRANTED") [ "$n" -gt 0 ] || return 0 say "" - say "${C_WARN}Collector groups used during this run:${C_OFF}" + note "Collector groups used during this run:" local grp user owner while IFS='|' read -r grp user owner; do [ -n "$grp" ] || continue @@ -2546,7 +3926,7 @@ _has_unadopted_packages() { # changed constantly and adoption ran every time. Recording "name size" per # package means a package is re-adopted only when ITS OWN manifest grows -- # which is exactly when it has new files to hand over. -_ADOPTED_LIST() { echo "$STATE/adopted.list"; } +_ADOPTED_LIST() { echo "$STATE_PROGRESS/adopted.list"; } _manifest_size() { wc -c < "$1" 2>/dev/null | tr -d ' ' || echo 0 @@ -2563,16 +3943,56 @@ _adopted_size() { _mark_adopted() { local name="$1" size="$2" f tmp f="$(_ADOPTED_LIST)" + mkdir -p "$(dirname "$f")" 2>/dev/null || true tmp="$f.$$" - { [ -f "$f" ] && grep -v "^$name " "$f"; echo "$name $size"; } > "$tmp" 2>/dev/null - mv -f "$tmp" "$f" 2>/dev/null || rm -f "$tmp" + # Errors here used to be swallowed twice over -- 2>/dev/null on the write + # AND `|| rm -f` on the move -- so a state directory that was missing or + # read-only looked exactly like a successful record, and the adoption pass + # announced itself on every build-all with nothing on disk to say why. + if ! { [ -f "$f" ] && grep -v "^$name " "$f"; echo "$name $size"; } > "$tmp" 2>/dev/null + then + rm -f "$tmp"; warn "could not write $f (adoption will run again)"; return 1 + fi + mv -f "$tmp" "$f" 2>/dev/null || { + rm -f "$tmp"; warn "could not update $f (adoption will run again)"; return 1; } + return 0 +} + +# A package needs adopting if the pass has not run for its manifest at this +# size. +# +# Record that every package has been through the pass. +# +# It used to record only packages whose user EXISTS -- while _needs_adoption +# reported "needs adopting" for precisely the packages whose user does NOT. +# The two conditions were opposites, so a single manifest without a user (a +# name that is not a package, a user creation that failed) could never be +# recorded and kept _has_unadopted_packages at yes forever. That is the +# "re-announces on every build-all" symptom. +# +# The gate asks "has the pass run for this?", not "did it succeed?". A +# package whose user could not be created answers that the same way as one +# whose user exists: yes, and running it again changes nothing. The same +# principle _needs_adoption already applies to files that cannot be +# reassigned -- tried is tried. +_mark_all_adopted() { + local man name + for man in "$MANIFESTS"/*.files; do + [ -s "$man" ] || continue + name="$(basename "${man%.files}")" + _is_not_a_package "$name" && continue + _mark_adopted "$name" "$(_manifest_size "$man")" + done } -# A package needs adopting if it has no user yet, or its manifest has grown -# since the last time it was adopted. _needs_adoption() { local man="$1" name size [ -s "$man" ] || return 1 + # A package that has been TRIED and whose user exists is done, even if some + # of its files could not be reassigned (they belong to another package, or + # no longer exist). Without this the pass reported "the user database + # exists now ..." on every single build-all forever, because one + # unfixable path kept the whole set looking unadopted. name="$(basename "${man%.files}")" _is_not_a_package "$name" && return 1 @@ -2580,66 +4000,34 @@ _needs_adoption() { # files were recorded but whose ownership could not be assigned at the time # because no package users existed yet. # - # Anything in this chroot's step list is built by lfs-helper, which gives - # each file to its package user as part of the build -- there is nothing - # left to adopt, and walking its manifest is pure waste. Checking that - # first is also much cheaper than statting thousands of files. - if steps | grep -qxF "$name"; then + # The test is whether lfs-helper has BUILT it here -- not whether it + # appears in the step list. Several packages appear in BOTH chapter 6 and + # chapter 8 (ncurses, bash, coreutils, binutils, gcc ...). Skipping + # everything in the step list left their chapter-6 files owned by 'lfs', + # and the chapter-8 build could then not overwrite its own files: + # cp: cannot create regular file '/usr/bin/tic': Permission denied + # /usr/bin/tic -rwxr-xr-x lfs:lfs + # Built here => ownership is already right. Not built here => the manifest + # came from chapters 5-6 and still needs adopting. + if is_done "$name"; then return 1 fi - user_exists "$(pkg_owner_name "$name")" || return 0 + # ONE rule, and _mark_all_adopted uses the same one: the pass is needed if + # it has not run for this manifest at this size. + # + # There used to be an extra `user_exists || return 0` here, which is the + # exact case the recorder skipped -- so the two could never agree about a + # package without a user, and the gate stayed open forever. It is also + # redundant: a package that has never been through the pass has no recorded + # size either, so the comparison below already answers "yes". size="$(_manifest_size "$man")" [ "$(_adopted_size "$name")" = "$size" ] && return 1 return 0 } -# Is the compiler in this chroot actually usable? -# -# The book runs a sanity check after gcc-pass2 and again after entering the -# chroot, and it is worth doing before every chapter-8 build: a toolchain that -# cannot link a trivial program produces failures that look like anything but -# a broken compiler -- -# configure: error: cannot compute sizeof (long long) -# fatal error: bits/c++config.h: No such file or directory -# Repair a C++ header tree installed under a different target triplet. -# -# The chroot compiler reports one triplet (say x86_64-pc-linux-gnu) while -# chapter 6 installed its headers under another (x86_64-lfs-linux-gnu), so -# g++ never looks where c++config.h actually is: -# fatal error: bits/c++config.h: No such file or directory -# -# The two trees are the same gcc version on the same machine, so the headers -# are interchangeable -- a symlink from the triplet the compiler expects to -# the one that exists is enough to build the real chapter-8 gcc, which then -# installs a matching set and makes the link redundant. -# Which package installed this path? -# -# The manifests are the record of who installed what, so a file that should -# not exist can be traced to the step that created it. Answers questions like -# "where did this native x86_64-pc-linux-gnu compiler come from?" without -# guessing. -# Sort /etc/passwd and /etc/group by id. -# -# Package users are appended as they are created, so the files end up in build -# order rather than id order -- awkward to read when you are trying to see -# which package owns which id. Sorting is cosmetic and safe: nothing depends -# on the order of these files. System entries (id < 1000) are kept at the top -# in their existing order, because the first few lines of /etc/passwd are -# conventional (root first) and worth not disturbing. -# Files whose owner has no passwd entry in this chroot. -# -# The lfs user exists on the HOST, not in here, so anything it created shows -# as a raw uid. Recreate the account and hand those files to it, rather than -# leaving them owned by nobody in particular. + # Give one user access to one directory, using the normal rules. -# -# lfs-helper grant-dir <dir> <user> [--run] -# -# Exactly what a failed install triggers automatically: if the directory -# already carries a collector group, join it; if not, ask which group should -# own it. Exposed as a command so a script can ask for access BEFORE trying, -# instead of failing first and repairing afterwards. cmd_grant_dir() { need_root local dir="${1:?usage: lfs-helper grant-dir <dir> <user> [--run]}" @@ -2664,139 +4052,15 @@ cmd_grant_dir() { grant_dir_access "$dir" "$user" } -# Is there anything with no passwd entry? Stops at the first hit, so it costs -# nothing on a tree that is already in order. -_has_orphaned_files() { - local r="${SNAP_ROOT%/}" hit - hit="$(find "${r:-/}" -xdev -nouser \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - -print -quit 2>/dev/null)" - [ -n "$hit" ] -} - -cmd_fix_orphans() { - need_root - local run=0 - [ "${1:-}" = "--run" ] && run=1 - - # 1. make sure there IS an lfs user to own them - if ! user_exists lfs; then - say "No 'lfs' user in this chroot -- the build user from outside." - say "Creating it with uid $LFS_BUILD_UID so its files resolve to a name:" - if [ "$run" = 1 ]; then - if have_shadow_tools; then - groupadd -g "$LFS_BUILD_UID" lfs 2>/dev/null || true - useradd -M -u "$LFS_BUILD_UID" -g lfs -s /bin/bash \ - -c "build user from outside the chroot" lfs 2>/dev/null \ - || warn " could not create the lfs user" - else - printf 'lfs:x:%s:\n' "$LFS_BUILD_UID" >> "$ETC/group" - printf 'lfs:x:%s:%s:build user from outside the chroot:/:/bin/bash\n' \ - "$LFS_BUILD_UID" "$LFS_BUILD_UID" >> "$ETC/passwd" - fi - user_exists lfs && ok " created lfs ($LFS_BUILD_UID)" - else - say " would create lfs ($LFS_BUILD_UID)" - fi - say "" - fi - - # 2. hand the orphaned files over - local r="${SNAP_ROOT%/}" - say "Files owned by a uid with no passwd entry:" - # Write the full list to a file rather than capping it. An earlier - # version kept only the first 2000 paths and fixed just those, so /lib and - # /.lfs-pkgusr were still owned by a uid with no name afterwards -- and it - # reported success. - local list="$STATE/tmp/orphans.$$" - mkdir -p "$STATE/tmp" 2>/dev/null || true - find "${r:-/}" -xdev -nouser \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - 2>/dev/null > "$list" - local n - n="$(count_lines "$list")" - if [ "${n:-0}" = 0 ]; then - rm -f "$list" - ok " none -- every file has a real owner" - return 0 - fi - head -n 8 "$list" | sed 's/^/ /' - [ "$n" -gt 8 ] && say " ... and $((n - 8)) more" - say "" - if [ "$run" != 1 ]; then - say " they would be given to 'lfs'" - rm -f "$list" - warn "(dry run -- nothing was changed. Add --run to apply.)" - return 0 - fi - local fixed=0 - while IFS= read -r p; do - [ -n "$p" ] || continue - chown -h lfs:lfs "$p" 2>/dev/null && fixed=$((fixed+1)) - # a few hundred thousand chowns with no output looks like a hang, and - # people cancel things that look hung - [ $((fixed % 5000)) = 0 ] && printf '\r %d of %d ...' "$fixed" "$n" >&2 - done < "$list" - printf '\r%*s\r' 40 '' >&2 - rm -f "$list" - ok " gave $fixed of $n path(s) to 'lfs'" - [ "$fixed" = "$n" ] || warn " ($((n - fixed)) could not be changed)" -} - -cmd_sort_users() { - need_root - local run=0 - [ "${1:-}" = "--run" ] && run=1 - - local f - for f in "$ETC/passwd" "$ETC/group"; do - [ -f "$f" ] || continue - local tmp="$f.sorted.$$" - # field 3 is the uid/gid in both files - { - awk -F: '$3 < 1000' "$f" - awk -F: '$3 >= 1000' "$f" | sort -t: -k3,3n - } > "$tmp" 2>/dev/null || { rm -f "$tmp"; continue; } - - # never write a file with fewer lines than we started with - local a b - a="$(wc -l < "$f")"; b="$(wc -l < "$tmp")" - if [ "$a" != "$b" ]; then - warn " $f: refusing to sort ($a lines in, $b out)" - rm -f "$tmp" - continue - fi - if cmp -s "$f" "$tmp"; then - say " $f is already in id order" - rm -f "$tmp" - continue - fi - if [ "$run" = 1 ]; then - cp -p "$f" "$f.bak" 2>/dev/null || true - cat "$tmp" > "$f" # preserve the inode, mode and owner - rm -f "$tmp" - ok " $f sorted by id (previous kept as $f.bak)" - else - say " $f would be sorted by id" - rm -f "$tmp" - fi - done - [ "$run" = 1 ] || warn "(dry run -- nothing was changed. Add --run to apply.)" -} - -cmd_which_package() { - local target="${1:?usage: lfs-helper which-package <path>}" - local man name hits=0 - - # match the path as recorded (manifests store chroot-absolute paths) - # Manifests store chroot-absolute paths. strip_host_prefix maps a real - # path INTO that form; a path the user typed is usually already in it, so - # try both rather than silently finding nothing. - local needle needle2 - needle="$target" - needle2="$(strip_host_prefix "$target")" +# Which package installed this path? +# +# The manifests are the record of who installed what, so a file that should +# not exist can be traced to the step that created it. +cmd_which_package() { + local target="${1:?usage: lfs-helper which-package <path>}" + local man name hits=0 needle needle2 + needle="$target" + needle2="$(strip_host_prefix "$target")" say "Searching the manifests for: $needle" say "" @@ -2811,14 +4075,12 @@ cmd_which_package() { done if [ "$hits" = 0 ]; then - # not an exact entry -- show packages that installed anything beneath it say " No package recorded that exact path." say "" say " Packages that installed something under it:" - local found=0 + local found=0 n for man in "$MANIFESTS"/*.files; do [ -e "$man" ] || continue - local n n="$(grep -c -e "^${needle%/}/" -e "^${needle2%/}/" "$man" 2>/dev/null)" if [ "${n:-0}" -gt 0 ]; then name="$(basename "${man%.files}")" @@ -2834,78 +4096,71 @@ cmd_which_package() { fi fi - # who owns it now, which may differ from who installed it if [ -e "$target" ]; then say "" say " On disk now: $(stat -c '%A %U:%G' "$target" 2>/dev/null)" fi } -cmd_fix_toolchain() { - need_root - local run=0 - [ "${1:-}" = "--run" ] && run=1 - local mine have hdr base - mine="$(c++ -dumpmachine 2>/dev/null)" \ - || die "no working c++ -- this cannot be repaired automatically" - [ -n "$mine" ] || die "c++ -dumpmachine said nothing" +# Is there anything with no passwd entry? Stops at the first hit, so it costs +# nothing on a tree that is already in order. +_has_orphaned_files() { + local r="${SNAP_ROOT%/}" hit + # see _vfy_orphans: before 7.6 nothing resolves, so everything looks orphaned + _user_db_ready || return 1 + hit="$(find "${r:-/}" -xdev -nouser \ + -not -path "$r/dev/*" -not -path "$r/proc/*" \ + -not -path "$r/sys/*" -not -path "$r/run/*" \ + -not -path "$r/sources/*" \ + -print -quit 2>/dev/null)" + [ -n "$hit" ] +} - hdr="" - for h in /usr/include/c++/*/*/bits/c++config.h; do - [ -e "$h" ] || continue - hdr="$h"; break +# Name the cause of a compiler/header mismatch. +# +# Three different things produce the same "bits/c++config.h: No such file" +# message, and they need completely different fixes: +# +# 1. a chapter-8 gcc installed on top, half-finished. Chapter 6 always +# configures with --host/--target; a compiler whose configure line has +# NEITHER is a native build, which only chapter 8 does. +# 2. LFS_TGT changed between chapter 6 and now. Both names are cross +# triplets, they just differ. +# 3. chapter 6 never finished. There are no headers at all. +_explain_triplet_mismatch() { + local cfg="$1" mine have hdr + mine="$(c++ -dumpmachine 2>/dev/null)" + for hdr in /usr/include/c++/*/*/bits/c++config.h; do + [ -e "$hdr" ] || continue + have="$(basename "$(dirname "$(dirname "$hdr")")")" + break done - [ -n "$hdr" ] || die "no c++config.h anywhere under /usr/include/c++ -- - chapter 6's gcc-pass2 did not install the C++ headers. - Finish it outside the chroot: lfs build-system crosschain --run" - - have="$(basename "$(dirname "$(dirname "$hdr")")")" # the triplet dir - base="$(dirname "$(dirname "$(dirname "$hdr")")")" # .../c++/<version> - - if [ "$have" = "$mine" ]; then - ok "Nothing to repair: the headers are already under '$mine'." - say " If C++ still fails, it is not a triplet mismatch --" - say " run: lfs-helper check-toolchain" - return 0 - fi + [ -n "${have:-}" ] || return 0 + [ "$have" = "$mine" ] && return 0 - say "The compiler and its headers disagree about the target triplet:" - say "" - say " compiler reports : $mine" - say " headers are under: $have" - say " in : $base" - say "" - say "Same gcc version, same machine, so the headers are interchangeable." - say "Link the name the compiler expects to the one that exists:" say "" - say " $base/$mine -> $have" - say "" - say "That is enough to build chapter 8's gcc, which installs a matching" - say "set of its own and makes this link unnecessary." - - if [ -e "$base/$mine" ]; then - warn "" - warn "$base/$mine already exists -- not touching it." - return 1 - fi - if [ "$run" != 1 ]; then - say "" - warn "(dry run -- nothing was changed. Add --run to apply.)" - return 0 - fi - - ln -sv "$have" "$base/$mine" || die "could not create the link" + case "$cfg" in + *--host=*|*--target=*) + say " Both are cross-compiler names, so LFS_TGT changed between" + say " chapter 6 and now. The headers under '$have' are the real" + say " ones; the compiler was built for '$mine'." + say "" + say " Set LFS_TGT back on the host and re-enter:" + say " lfs config lfs_tgt $have" + ;; + *) + warn " This compiler was configured with no --host and no --target," + warn " so it is a NATIVE build -- which only chapter 8 does." + warn " Chapter 8's gcc has been installed on top of chapter 6's," + warn " but did not get as far as installing its C++ headers." + say "" + say " Rebuild it. Installs are staged now, so this cannot leave" + say " the system half-updated again:" + say " lfs-helper build gcc --phase all --force" + ;; + esac say "" - if cmd_check_toolchain >/dev/null 2>&1; then - ok "C++ works now." - say "" - say "Next: lfs-helper build-all" - else - warn "The link is in place but C++ still fails." - say "Run 'lfs-helper check-toolchain' to see what else is wrong." - return 1 - fi } cmd_check_toolchain() { @@ -3009,53 +4264,1281 @@ cmd_check_toolchain() { fi done - # The decisive fact: which directories the compiler actually searches. - # A header can exist, be world-readable, and sit on a fully traversable - # path, and still not be found -- because that directory is not in the - # search list at all. Permissions and existence cannot tell you that; - # only the compiler can. - say "" - say " Where this compiler actually looks for C++ headers:" - printf '' | c++ -x c++ -E -v - 2>&1 \ - | sed -n '/#include <...> search starts here/,/End of search list/p' \ - | grep -v 'search starts here\|End of search list' \ - | sed 's/^/ /' - say "" - say " If the directory holding c++config.h is not in that list, the" - say " compiler was built for a different target than the headers were" - say " installed for -- that is a mismatch, not a missing file." + # The decisive fact: which directories the compiler actually searches. + # A header can exist, be world-readable, and sit on a fully traversable + # path, and still not be found -- because that directory is not in the + # search list at all. Permissions and existence cannot tell you that; + # only the compiler can. + say "" + say " Where this compiler actually looks for C++ headers:" + printf '' | c++ -x c++ -E -v - 2>&1 \ + | sed -n '/#include <...> search starts here/,/End of search list/p' \ + | grep -v 'search starts here\|End of search list' \ + | sed 's/^/ /' + say "" + say " If the directory holding c++config.h is not in that list, the" + say " compiler was built for a different target than the headers were" + say " installed for -- that is a mismatch, not a missing file." + say "" + + # Which of the three causes is it? The compiler's own configure line + # tells them apart, and they need completely different fixes. + _explain_triplet_mismatch "$cfg" + + say " The compiler looks for them under its OWN target triplet:" + say " $(cc -dumpmachine 2>/dev/null || echo '(cc not working)')" + say " If the headers above are under a different triplet, chapter 6's" + say " gcc-pass2 was built for a different target than this compiler" + say " reports -- rebuild it with the LFS_TGT this tree was made with." + fi + + rm -rf "$t" + say "" + if [ "$rc" = 0 ]; then + ok "Toolchain is usable." + else + warn "Do not start chapter 8 until this is fixed." + fi + return $rc +} + +# Has the copied-in export already been applied? +_groups_already_imported() { + local f="$STATE_GROUPS/collector-groups.import" first + [ -f "$f" ] || return 0 + [ -f "$COLLECTOR_MAP" ] || return 1 + # every dir decision in the file must be in the map + while IFS='|' read -r kind a b; do + [ "$kind" = dir ] || continue + grep -qxF "$a|$b" "$COLLECTOR_MAP" 2>/dev/null || return 1 + done < <(grep -vE '^\s*(#|$)' "$f") + return 0 +} + +# The compiler's own target vs where the C++ headers actually are. +# +# A build has MORE THAN ONE of these directories: chapter 6 installs libstdc++ +# under $LFS_TGT (x86_64-lfs-linux-gnu, or whatever vendor string is +# configured) and chapter 8's native gcc installs its own under the triplet it +# reports, usually x86_64-pc-linux-gnu. Both are present afterwards and that +# is correct -- the chapter-6 one is simply left over. +# +# Taking the FIRST glob match and stopping therefore reported a mismatch +# whenever the stale directory happened to sort first, which is alphabetical +# and so pure luck: "nimgnu" < "pc", so a perfectly good toolchain warned on +# every package. What matters is only whether a directory matching the +# compiler exists at all -- if it does, the compiler will find its headers. +_warn_on_triplet_mismatch() { + command -v c++ >/dev/null 2>&1 || return 0 + local mine hdr have found="" + mine="$(c++ -dumpmachine 2>/dev/null)" || return 0 + [ -n "$mine" ] || return 0 + for hdr in /usr/include/c++/*/*/bits/c++config.h; do + [ -e "$hdr" ] || continue + have="$(basename "$(dirname "$(dirname "$hdr")")")" + [ "$have" = "$mine" ] && return 0 # the compiler's own -- fine + found="$found $have" + done + [ -n "$found" ] || return 0 # no C++ headers at all yet + warn "" + warn "!! the compiler and its C++ headers disagree about the target:" + warn " compiler reports : $mine" + warn " headers are under:$found" + warn " Chapter 8's gcc will fail with" + warn " fatal error: bits/c++config.h: No such file or directory" + warn " which looks like a missing file but is a mismatch. See:" + warn " lfs-helper check-toolchain" + warn "" +} + +# Two accounts sharing one uid. +# +# Ownership is stored as a NUMBER. Give two names the same uid and every tool +# resolves that number to whichever name it finds first -- so ncurses' files +# report as owned by 'xz', and the ncurses build cannot overwrite its own +# binaries. It reads exactly like a permissions bug and is not one. +# +# It happens when /etc/passwd is truncated (book 7.6) while package users +# already exist: the next user created starts from PKG_UID_MIN again, and any +# entry restored afterwards collides with it. +_duplicate_ids() { + local f="$1" + awk -F: 'NF >= 3 && $3 != "" { c[$3] = c[$3] " " $1; n[$3]++ } + END { for (id in n) if (n[id] > 1) printf " %s -> %s\n", id, c[id] }' \ + "$f" 2>/dev/null +} + +_check_duplicate_ids() { + local dp dg + dp="$(_duplicate_ids "$ETC/passwd")" + dg="$(_duplicate_ids "$ETC/group")" + [ -z "$dp" ] && [ -z "$dg" ] && return 0 + warn "" + warn "!! two accounts share one id. Ownership is stored as a NUMBER, so" + warn " files will report as owned by the wrong name, and a package will" + warn " not be able to overwrite its own files:" + [ -n "$dp" ] && { warn " in $ETC/passwd:"; printf '%s\n' "$dp" >&2; } + [ -n "$dg" ] && { warn " in $ETC/group:"; printf '%s\n' "$dg" >&2; } + warn "" + warn " Decide which name owns which files, then give one of them a free" + warn " id and chown its files to it. Until then any 'Permission denied'" + warn " on a package's own files is this, not a permissions problem." + warn "" + return 1 +} + +# --------------------------------------------------------------------------- # +# verify -- reconcile the tree against what it should be +# --------------------------------------------------------------------------- # +# Ownership is DERIVED state. The manifests say which package installed which +# path; the install-dir list says which directories are shared; the collector +# map says which shared directories carry which group. Together those three +# fully determine what the tree should look like. +# +# This replaces fix-ownership, adopt-dirs, adopt-existing, fix-orphans, +# fix-users, seal-install-dirs, sort-users and claim-earlier-stages. Each of +# those was added to fix one symptom, they overlapped, and none of them could +# see a problem outside its own narrow remit -- so a staged install quietly +# handing /usr to a package user went unnoticed through four later failures. +# +# One pass, one model, every rule checked every time. + +_vfy_n_checked=0 +_vfy_n_wrong=0 +_vfy_n_fixed=0 + +# Every finding, in full, in a file. +# +# The console prints the first eight of each kind and "... and 44265 more", +# which tells you a number and nothing you can act on -- you cannot grep it, +# diff it against the last run, or feed it to which-package. The log holds +# every line; the console stays short. +VERIFY_LOG="${VERIFY_LOG:-$STATE/logs/verify.log}" + +_vfy_log_open() { + mkdir -p "$(dirname "$VERIFY_LOG")" 2>/dev/null || true + : > "$VERIFY_LOG" 2>/dev/null || { VERIFY_LOG=""; return 0; } + printf '# lfs-helper verify -- %s\n# root: %s\n\n' \ + "$(date '+%Y-%m-%d %H:%M:%S')" "${SNAP_ROOT:-/}" >> "$VERIFY_LOG" +} + +# _vfy_log <section> <line...> +_vfy_log() { + [ -n "$VERIFY_LOG" ] || return 0 + local sec="$1"; shift + printf '%s\t%s\n' "$sec" "$*" >> "$VERIFY_LOG" 2>/dev/null || true +} + +_vfy_report() { + # <what> <path> <found> <expected> + _vfy_n_wrong=$((_vfy_n_wrong + 1)) + printf ' %-9s %-52s %s -> %s\n' "$1" "$2" "$3" "$4" +} + +# --- rule 1: a package owns what its manifests record ---------------------- # +# +# Including every earlier stage of the same package: ncurses is built in +# chapter 6 as the lfs user and again in chapter 8 as its package user, and +# perl-tmp in chapter 7 as root. All of it belongs to the final package user. +# Sort /etc/passwd and /etc/group by id. +# +# Package users are appended as they are created, so the files end up in build +# order rather than id order. Sorting is cosmetic and safe: nothing depends on +# the order of these files. System entries (id < 1000) keep their place. +cmd_sort_users() { + need_root + local run=0 + [ "${1:-}" = "--run" ] && run=1 + + local f + for f in "$ETC/passwd" "$ETC/group"; do + [ -f "$f" ] || continue + local tmp="$f.sorted.$$" + { + awk -F: '$3 < 1000' "$f" + awk -F: '$3 >= 1000' "$f" | sort -t: -k3,3n + } > "$tmp" 2>/dev/null || { rm -f "$tmp"; continue; } + + local a b + a="$(wc -l < "$f")"; b="$(wc -l < "$tmp")" + if [ "$a" != "$b" ]; then + warn " $f: refusing to sort ($a lines in, $b out)" + rm -f "$tmp" + continue + fi + if cmp -s "$f" "$tmp"; then + say " $f is already in id order" + rm -f "$tmp" + continue + fi + if [ "$run" = 1 ]; then + cp -p "$f" "$f.bak" 2>/dev/null || true + cat "$tmp" > "$f" + rm -f "$tmp" + ok " $f sorted by id (previous kept as $f.bak)" + else + say " $f would be sorted by id" + rm -f "$tmp" + fi + done + [ "$run" = 1 ] || warn "(dry run -- nothing was changed. Add --run to apply.)" +} + +# The LAST step of the chroot stage: make the install directories sticky. +# +# During the build they must be group-writable but NOT sticky, so package users +# can replace files the temporary system installed as root. Once everything is +# built that has to be undone, or any package can delete another's files. +# +# Only the install directories. Collector-group directories are shared on +# purpose and are left alone. +cmd_seal_install_dirs() { + need_root + local run=0 + [ "${1:-}" = "--run" ] && run=1 + + say "Sticky bit (o+t) on the shared install directories." + say "" + say "During the build these are group-writable so package users can replace" + say "each other's. Once sticky, a package user can still create files here" + say "but can no longer modify or delete another package's." + say "" + say "Run this when every package is built (end of chapter 8) -- doing it" + say "earlier will break installs that legitimately overwrite older files." + say "" + + local d n=0 missing=0 + while IFS= read -r d; do + [ -n "$d" ] || continue + [ -d "$d" ] || { missing=$((missing+1)); continue; } + [ -k "$d" ] && continue + if [ "$run" = 1 ]; then + real_chmod o+t "$d" 2>/dev/null && n=$((n+1)) + else + n=$((n+1)) + fi + done < <( + { while IFS= read -r d; do + printf '%s\n' "${SNAP_ROOT%/}$d" + done < <(install_dirs_list) + find "${SNAP_ROOT:-/}" -xdev -type d -group install \ + -not -path "$SRCROOT/*" \ + -not -path "${SNAP_ROOT%/}/sources/*" \ + -not -path "${SNAP_ROOT%/}/dev/*" \ + -not -path "${SNAP_ROOT%/}/proc/*" \ + -not -path "${SNAP_ROOT%/}/sys/*" \ + -not -path "${SNAP_ROOT%/}/run/*" 2>/dev/null + } | sort -u + ) + if [ "$run" = 1 ]; then + ok " sticky bit set on $n install directory(ies)" + else + say " $n install directory(ies) would get the sticky bit" + warn "(dry run -- nothing was changed. Add --run to apply.)" + fi + [ "$missing" -gt 0 ] && say " ($missing listed dirs don't exist here -- skipped)" + return 0 +} + +# Give the packages built before the chroot their own users. +# +# The same work verify does, under the name it has always had -- the messages +# elsewhere tell people to run this, and a command that has been removed is +# worse than one that is redundant. +# Is there a user database to add package users to? +# +# A crosschain snapshot is taken BEFORE chapter 7, so /etc/passwd does not +# exist yet -- book 7.6 creates it. Running the adoption pass then produces +# "could not create" for every single package, because there is nothing to +# create them in. The give-away is the prompt: "I have no name!" means even +# root has no passwd entry. +# +# Adoption is not wrong here, just early. It has to wait until 7.6 has run. +_user_db_ready() { + [ -s "$ETC/passwd" ] || return 1 + grep -q '^root:' "$ETC/passwd" 2>/dev/null || return 1 + return 0 +} + +_explain_no_user_db() { + warn "" + warn "There is no user database in this tree yet: $ETC/passwd" + warn "has no root entry, which is why the prompt says \"I have no name!\"." + warn "" + warn "Book 7.6 creates it. This is a chapter 5-6 tree, so nothing can be" + warn "given a package user until chapter 7 has run:" + warn " lfs-helper build-all" + warn "" + warn "build-all does this in the right order by itself -- the chapter-7" + warn "steps first, then the package users. Nothing is wrong with the tree." + warn "" +} + +cmd_adopt_existing() { + need_root + if ! _user_db_ready; then + _explain_no_user_db + return 1 + fi + local run=0 + [ "${1:-}" = "--run" ] && run=1 + say "Give the CHAPTER 5-6 packages their own package users." + say "" + say "Those were built outside the chroot, before any package user existed," + say "so their files are still root-owned -- only their recorded file lists" + say "say who installed what. Packages built in here already got their" + say "owner as part of the build and are not touched." + say "" + if [ "$run" != 1 ]; then + _vfy_manifest_ownership 0 || true + warn "(dry run -- nothing was changed. Add --run to apply.)" + return 0 + fi + _vfy_manifest_ownership 1 || true + cmd_sort_users --run >/dev/null 2>&1 || true +} + +_vfy_manifest_ownership() { + local fix="$1" man base name owner p cur kind + # In BUILD order, not alphabetical. + # + # This pass creates the package users it finds missing, and a user's uid is + # simply the order it was created in. Walking the manifests alphabetically + # gave man-pages (the first chapter-8 package) uid 10000 and binutils -- + # built first, in chapter 5 -- a much higher one, which makes the uids + # useless for reading the build order off the passwd file. + for man in $(_manifests_in_build_order) $(_manifests_in_build_order dirs); do + [ -e "$man" ] || continue + [ -s "$man" ] || continue + case "$man" in *.dirs) kind=dir ;; *) kind=file ;; esac + name="$(basename "$man")"; name="${name%.files}"; name="${name%.dirs}" + _is_not_a_package "$name" && continue + base="$(_stage_base_name "$name")" + owner="$(pkg_owner_name "$base")" + + # A package with a manifest but no user has never been given one -- + # every chapter 5-6 package is in that state, because they were built + # before any package user existed. Creating it here is the job the + # old `adopt-existing` did; without it verify would skip them silently + # and their files would stay owned by 'lfs' forever. + if ! user_exists "$owner"; then + [ "$fix" = 1 ] || { _vfy_report user "$owner" "(missing)" \ + "package user"; continue; } + # stdout suppressed (one line per package is noise here), but NOT + # stderr: a failure to create the account is the whole reason the + # adoption is happening, and hiding it is how a tree came out with + # every adopted home root-owned and nothing saying why. + ( cmd_add_user "$owner" >/dev/null ) || true + # The account may already have existed with an uninitialised home: + # adopted packages came out with 86-byte homes owned by root while + # packages BUILT in the same run had full ones owned by their user. + # init_package_user_home is idempotent, so just run it -- it fills + # in the profile links and hands the home to its user. + user_exists "$owner" && init_package_user_home "$owner" >/dev/null + user_exists "$owner" || { + _vfy_report user "$owner" "(could not create)" "package user" + continue + } + _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + + while IFS= read -r p; do + [ -n "$p" ] || continue + p="$(strip_host_prefix "$p")" + [ -e "$p" ] || [ -L "$p" ] || continue + # scratch, and files no package can own. Manifests written before + # these rules existed still carry them, and re-reporting a conflict + # nothing can settle is worse than not reporting it: the next + # package takes the file straight back. + is_never_claimed "$p" && continue + case "$p" in "${BUILD_ROOT:-${SNAP_ROOT%/}/build}"/*) continue ;; esac + _vfy_n_checked=$((_vfy_n_checked + 1)) + cur="$(stat -c %U "$p" 2>/dev/null)" || continue + [ "$cur" = "$owner" ] && continue + # A shared install directory belongs to root:install, never to a + # package -- even when the package's manifest records creating it. + [ "$kind" = dir ] && is_install_dir "$p" && continue + # A directory holding another package's files is shared in fact, + # whatever the manifest says: taking it would lock the other one + # out. This is what let one package claim gcc's header tree. + if [ "$kind" = dir ] && _dir_contents_foreign "$p" "$owner"; then + continue + fi + # never take a path another package legitimately owns + case "$cur" in + root|lfs|UNKNOWN) ;; + *) user_exists "$cur" && continue ;; + esac + _vfy_report owner "$p" "$cur" "$owner" + if [ "$fix" = 1 ]; then + real_chown -h "$owner:$owner" "$p" 2>/dev/null \ + && _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + done < "$man" + done +} + +# --- rule 2: shared install directories belong to root:install ------------- # +# +# This is the rule a staged install broke: /usr and /usr/bin became +# gcc:gcc 755, losing the install group and g+w -- the entire scheme, undone +# by one package. Checked every run now. +_vfy_install_dirs() { + local fix="$1" d live owner grp mode want_mode sealed + # Before the ownership epoch there is no user database, so `install` is not + # a name and neither is root. Reporting 27 wrong install directories then + # is not a finding -- it is the tree being younger than the question. + if ! ownership_established && ! ownership_possible; then + say " (ownership not established yet -- book 7.6 comes first)" + return 0 + fi + sealed=0 + _install_dirs_are_sealed && sealed=1 + while IFS= read -r d; do + live="${SNAP_ROOT%/}$d" + [ -d "$live" ] || continue + # a symlinked install dir (/bin -> usr/bin) is checked at its target, + # which is in this list too -- see the note in _vfy_install_dirs + [ -L "$live" ] && continue + _vfy_n_checked=$((_vfy_n_checked + 1)) + # by number: `stat -c %U` reports UNKNOWN before 7.6 writes /etc/passwd, + # which would report every correct directory as wrong + owner="$(stat -c %u "$live" 2>/dev/null)" + grp="$(stat -c %g "$live" 2>/dev/null)" + mode="$(stat -c %a "$live" 2>/dev/null)" + # sticky only after the build is finished -- during it, package users + # must be able to replace the temporary system's root-owned files + if [ "$sealed" = 1 ]; then want_mode=1775; else want_mode=775; fi + if [ "$owner" != 0 ] || [ "$grp" != "$INSTALL_GID" ] \ + || [ "$mode" != "$want_mode" ]; then + _vfy_report installdir "$live" \ + "$(stat -c '%U:%G' "$live" 2>/dev/null) $mode" \ + "root:install $want_mode" + if [ "$fix" = 1 ]; then + set_install_dir_owner "$live" || true + real_chmod "$want_mode" "$live" 2>/dev/null || true + _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + fi + done < <(install_dirs_list) +} + +# --- rule 3: every file has an owner that exists here ---------------------- # +_vfy_orphans() { + local fix="$1" r="${SNAP_ROOT%/}" p n=0 + # NOT before book 7.6 has written /etc/passwd. + # + # `find -nouser` asks whether a uid resolves to a name. With no passwd file + # NOTHING resolves, so every file in the tree is an "orphan" -- on a + # chapter 5-6 tree restored before 7.6 this reported + # noowner / (no passwd entry) -> lfs + # noowner /dev (no passwd entry) -> lfs + # ... and 11797 more with no owner + # and `--fix` would have handed the ENTIRE tree, root's own directories + # included, to the build user in one pass. The finding is not wrong so much + # as meaningless: there is nothing yet to resolve against. + if ! _user_db_ready; then + say " (no user database yet -- book 7.6 creates it; not checking owners)" + return 0 + fi + while IFS= read -r p; do + [ -n "$p" ] || continue + _vfy_n_checked=$((_vfy_n_checked + 1)) + n=$((n + 1)) + [ "$n" -le 5 ] && _vfy_report noowner "$p" "(no passwd entry)" "lfs" + if [ "$fix" = 1 ]; then + user_exists lfs || _vfy_make_lfs_user + real_chown -h lfs:lfs "$p" 2>/dev/null \ + && _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + done < <(find "${r:-/}" -xdev -nouser \ + -not -path "$r/dev/*" -not -path "$r/proc/*" \ + -not -path "$r/sys/*" -not -path "$r/run/*" \ + -not -path "$r/sources/*" 2>/dev/null) + [ "$n" -gt 5 ] && say " ... and $((n - 5)) more with no owner" + [ -n "$VERIFY_LOG" ] && say " all $n: grep ^no-owner $VERIFY_LOG" +} + +# --- the build user's leftovers -------------------------------------------- # +# +# Chapters 5-6 run as the `lfs` user OUTSIDE the chroot. Book 7.2 hands the +# tree to root with `chown -R`, but the pass that does it was guarded by a +# single inode -- if $LFS/usr was already root-owned the whole recursive +# handover was skipped, and everything below kept the build uid: +# +# drwxr-xr-x 1 lfs lfs /usr/share/gcc-15.2.0/python/libstdcxx +# +# Those are temporary-system files. They belong to root until a package adopts +# them, so hand them back and let adoption do the rest. Anything a manifest +# claims is left alone: _vfy_manifest_ownership settles those, and it knows +# which package they belong to. +_vfy_build_user_leftovers() { + local fix="$1" r="${SNAP_ROOT%/}" p n=0 + _user_db_ready || return 0 + user_exists lfs || return 0 + while IFS= read -r p; do + [ -n "$p" ] || continue + is_never_claimed "$p" && continue + _vfy_n_checked=$((_vfy_n_checked + 1)) + n=$((n + 1)) + [ "$n" -le 5 ] && _vfy_report builduser "$p" "lfs" "root" + if [ "$fix" = 1 ]; then + real_chown -h root:root "$p" 2>/dev/null \ + && _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + # -xdev, and NOT -type f: the symlinks book 4.2 creates at the tree root -- + # /bin -> usr/bin, /lib, /sbin -- are created by the build user too, and + # came out + # lrwxrwxrwx 1 lfs lfs bin -> usr/bin + # A chown without -h follows the link and retargets /usr/bin instead, which + # is why every chown in this file uses -h. + # + # /build is included deliberately: it is created by the build user outside + # the chroot and should be root's. Its CONTENTS are not -- scratch has no + # owner worth arguing about. The state directory is under $SRCROOT, which + # is excluded wholesale. + done < <(find "${r:-/}" -xdev -user lfs \ + -not -path "$r/dev/*" -not -path "$r/proc/*" \ + -not -path "$r/sys/*" -not -path "$r/run/*" \ + -not -path "$r/tmp/*" \ + -not -path "$r/sources" -not -path "$r/sources/*" \ + -not -path "${BUILD_ROOT:-$r/build}/*" \ + -not -path "$STATE/*" \ + -not -path "$SRCROOT/*" 2>/dev/null) + [ "$n" -gt 5 ] && say " ... and $((n - 5)) more still owned by the build user" + [ "$n" -gt 0 ] && [ -n "$VERIFY_LOG" ] \ + && say " all $n: grep ^builduser $VERIFY_LOG" + return 0 +} + +_vfy_make_lfs_user() { + if have_shadow_tools; then + groupadd -g "$LFS_BUILD_UID" lfs 2>/dev/null || true + useradd -M -u "$LFS_BUILD_UID" -g lfs -s /bin/bash \ + -c "build user from outside the chroot" lfs 2>/dev/null || true + else + printf 'lfs:x:%s:\n' "$LFS_BUILD_UID" >> "$ETC/group" + printf 'lfs:x:%s:%s:build user from outside the chroot:/:/bin/bash\n' \ + "$LFS_BUILD_UID" "$LFS_BUILD_UID" >> "$ETC/passwd" + fi +} + +# --- rule 4: no two accounts share an id ----------------------------------- # +# +# Ownership is stored as a number, so a collision makes files report as owned +# by the wrong name and a package unable to write its own files. Reported +# only: which name should keep the id is a decision, not something to guess. +_vfy_duplicate_ids() { + local f out + for f in passwd group; do + out="$(_duplicate_ids "$ETC/$f")" + [ -n "$out" ] || continue + _vfy_n_wrong=$((_vfy_n_wrong + 1)) + warn " duplicate ids in $ETC/$f -- files will report the wrong owner:" + printf '%s\n' "$out" >&2 + warn " Not repaired: which name keeps the id is your decision." + done +} + +# --- rule 5: passwd and group in id order ---------------------------------- # +_vfy_user_order() { + local fix="$1" f tmp + for f in "$ETC/passwd" "$ETC/group"; do + [ -f "$f" ] || continue + _vfy_n_checked=$((_vfy_n_checked + 1)) + tmp="$f.vfy.$$" + { awk -F: '$3 < 1000' "$f" + awk -F: '$3 >= 1000' "$f" | sort -t: -k3,3n; } > "$tmp" 2>/dev/null + if [ "$(count_lines "$tmp")" != "$(count_lines "$f")" ]; then + rm -f "$tmp"; continue + fi + if ! cmp -s "$f" "$tmp"; then + _vfy_report order "$f" "build order" "id order" + if [ "$fix" = 1 ]; then + cp -p "$f" "$f.bak" 2>/dev/null || true + cat "$tmp" > "$f" + _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + fi + rm -f "$tmp" + done +} + +# The package a manifest name belongs to: gcc-pass2 -> gcc, perl-tmp -> perl. +_stage_base_name() { + local n="$1" + n="${n%-tmp}"; n="${n%-pass1}"; n="${n%-pass2}" + printf '%s' "$n" +} + +# Seal the install directories: sticky, so a package user can still create +# files there but no longer replace another package's. Done once, at the end +# of the build -- during it, package users must be able to replace the +# temporary system's root-owned files. After this, verify expects 1775. +# --- rule 6: every package user has a usable home -------------------------- # +# +# A package user's home is its build directory. If it is missing, or owned by +# someone else, the build cannot even unpack. This is what `fix-users` did. +_vfy_user_homes() { + local fix="$1" man name base owner home cur + for man in "$MANIFESTS"/*.files; do + [ -e "$man" ] || continue + name="$(basename "${man%.files}")" + _is_not_a_package "$name" && continue + base="$(_stage_base_name "$name")" + owner="$(pkg_owner_name "$base")" + user_exists "$owner" || continue + home="$(pkgusr_home_for "$owner")" + _vfy_n_checked=$((_vfy_n_checked + 1)) + if [ ! -d "$home" ]; then + _vfy_report home "$home" "(missing)" "$owner" + if [ "$fix" = 1 ]; then + mkdir -p "$home" 2>/dev/null \ + && chown "$owner:$owner" "$home" 2>/dev/null \ + && _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + continue + fi + cur="$(stat -c %U "$home" 2>/dev/null)" + [ "$cur" = "$owner" ] && continue + _vfy_report home "$home" "$cur" "$owner" + if [ "$fix" = 1 ]; then + chown "$owner:$owner" "$home" 2>/dev/null \ + && _vfy_n_fixed=$((_vfy_n_fixed + 1)) + fi + done +} + +_vfy_seal_install_dirs() { + local d live n=0 + while IFS= read -r d; do + live="${SNAP_ROOT%/}$d" + [ -d "$live" ] || continue + [ -k "$live" ] && continue + chmod o+t "$live" 2>/dev/null && n=$((n+1)) + done < <(install_dirs_list) + # directories that carry the install group but are not on the list + while IFS= read -r live; do + [ -n "$live" ] || continue + [ -k "$live" ] && continue + chmod o+t "$live" 2>/dev/null && n=$((n+1)) + done < <(find "${SNAP_ROOT:-/}" -xdev -type d -group install \ + -not -path "$SRCROOT/*" \ + -not -path "${SNAP_ROOT%/}/sources/*" 2>/dev/null) + ok "# sticky bit set on $n install directory(ies)" +} + +# What in $STATE must survive, and what is scratch. +# +# The state directory mixes two kinds of thing: records that ARE the system's +# ownership (manifests, the collector map, progress) and pure scratch (staging +# trees, logs, temp files). A snapshot, a restore or a restart had to guess +# which was which. Naming them means each can be treated correctly, and the +# format can be versioned so a future tool refuses state it cannot read rather +# than misreading it. +STATE_FORMAT=1 +STATE_DURABLE="manifests collector-map collector-groups.import adopted.list + chroot-progress steporder rootsteps interrupted" +STATE_SCRATCH="stage tmp logs" + +_write_state_version() { + local f="$STATE_PROGRESS/format" + [ -d "$STATE" ] || return 0 + [ -f "$f" ] || printf '%s\n' "$STATE_FORMAT" > "$f" 2>/dev/null || true +} + +_check_state_version() { + local f="$STATE_PROGRESS/format" have + [ -f "$f" ] || { _write_state_version; return 0; } + have="$(head -n1 "$f" 2>/dev/null)" + [ "$have" = "$STATE_FORMAT" ] && return 0 + warn "" + warn "!! this build state is format $have, these tools write format $STATE_FORMAT." + warn " Reading it could misinterpret the manifests, which decide who owns" + warn " every file. Check the tool versions match before continuing:" + warn " lfs-helper --version (inside)" + warn " lfs --version (on the host)" + warn "" + return 1 +} + +# Scratch that can always be thrown away. +cmd_clean_state() { + need_root + local run=0; [ "${1:-}" = "--run" ] && run=1 + local d n=0 sz + say "Scratch in $STATE (never part of the system):" + for d in $STATE_SCRATCH; do + [ -d "$STATE/$d" ] || continue + sz="$(du -sh "$STATE/$d" 2>/dev/null | cut -f1)" + say " $d ${sz:-?}" + n=$((n+1)) + [ "$run" = 1 ] && rm -rf "${STATE:?}/$d" + done + [ "$n" = 0 ] && { ok " nothing to clean"; return 0; } + say "" + say "Kept (these ARE the ownership records):" + for d in $STATE_DURABLE; do + [ -e "$STATE/$d" ] && say " $d" + done + if [ "$run" = 1 ]; then + ok "removed $n scratch directory(ies)" + else + warn "(dry run -- nothing was changed. Add --run to remove them.)" + fi +} + +# --------------------------------------------------------------------------- # +# install-as -- the package-user model, for anything +# --------------------------------------------------------------------------- # +# Everything `build` does, for a command that is not a book step: create the +# user, claim what earlier stages of it own, run under the wrappers, stage the +# install where that is wanted, track what lands on disk, and grant directory +# access when the install needs it. +# +# lfs-helper install-as <name> -- <command ...> +# lfs-helper install-as <name> --script <file> +# +# This is the entry point for a BLFS package, a local script, or anything else +# that should be owned like every other package. Without it a third party has +# to reimplement all of the above, which is how the two halves of this +# toolchain drifted apart in the first place. +# Record what a package user IS, in its own home. +# +# A package user's home tells you nothing about where the package came from. +# On a finished system that matters: an LFS base package, a BLFS package, a +# pip module and a local script are maintained differently and updated from +# different places, and the only way to tell them apart today is to remember. +# +# One file, read by eye or by script: +# /usr/src/gcc/.pkgusr +# source=lfs +# package=gcc +# version=15.2.0 +# installed=2026-08-25 03:40:11 +# by=lfs-helper 1.7.0 (build 0a4b882) +# What kind of package is this? Reads the file above. +cmd_pkgusr_info() { + local name="${1:-}" + if [ -n "$name" ]; then + local f="$(pkgusr_home_for "$name")/.pkgusr" + [ -f "$f" ] || die "no record for '$name' (installed before this was recorded?)" + cat "$f" + return 0 + fi + printf '%-24s %-8s %-14s %s\n' PACKAGE SOURCE VERSION INSTALLED + local d n src ver ins root + while IFS= read -r root; do + for d in "$root"/*; do + [ -d "$d" ] || continue + n="$(basename "$d")" + if [ -f "$d/.pkgusr" ]; then + src="$(sed -n 's/^source=//p' "$d/.pkgusr" | head -1)" + ver="$(sed -n 's/^version=//p' "$d/.pkgusr" | head -1)" + ins="$(sed -n 's/^installed=//p' "$d/.pkgusr" | head -1)" + else + src="?"; ver=""; ins="" + fi + printf '%-24s %-8s %-14s %s\n' "$n" "${src:-?}" "${ver:--}" "${ins:--}" + done + done < <(pkgusr_roots) +} + +cmd_install_as() { + need_root + local name="" script="" as_root=0 + local -a cmdv=() + name="${1:-}"; shift || true + [ -n "$name" ] || die "usage: lfs-helper install-as <name> [--script <f>] [-- <command ...>]" + + while [ $# -gt 0 ]; do + case "$1" in + --script) script="${2:?--script needs a file}"; shift 2 ;; + --as-root) as_root=1; shift ;; + --) shift; cmdv=("$@"); break ;; + *) die "unknown option: $1" ;; + esac + done + + if [ -z "$script" ] && [ "${#cmdv[@]}" = 0 ]; then + die "give either --script <file> or -- <command ...>" + fi + + _is_not_a_package "$name" && die "'$name' is a reserved step name" + + # A generated step script is what `build` consumes, so write one and reuse + # the whole path rather than duplicating it here. + mkdir -p "$SCRIPTS" || die "cannot write $SCRIPTS" + local gen="$SCRIPTS/$name.sh" + if [ -n "$script" ]; then + [ -r "$script" ] || die "no such script: $script" + cp "$script" "$gen" || die "cannot copy $script" + else + { + printf '#!/bin/bash\n' + printf '# generated by: lfs-helper install-as %s\n' "$name" + printf 'set -e\n' + printf '%s\n' "$(_quote_cmd "${cmdv[@]}")" + } > "$gen" || die "cannot write $gen" + fi + chmod 755 "$gen" + + # make it a known step, so status/list/manifests see it like any other + grep -qxF "$name" "$STATE_PROGRESS/steporder" 2>/dev/null || \ + printf '%s\n' "$name" >> "$STATE_PROGRESS/steporder" + if [ "$as_root" = 1 ]; then + grep -qxF "$name" "$STATE_PROGRESS/rootsteps" 2>/dev/null || \ + printf '%s\n' "$name" >> "$STATE_PROGRESS/rootsteps" + fi + + say "# installing '$name' as a package user" + LFS_PKG_SOURCE="${LFS_PKG_SOURCE:-local}" cmd_build "$name" --force +} + +_quote_cmd() { + local a out="" + for a in "$@"; do out="$out $(printf '%q' "$a")"; done + printf '%s' "${out# }" +} + +# Files no manifest claims. +# +# The ownership pass can only act on what a manifest lists. A file installed +# before tracking existed, or by a step whose manifest was lost, stays +# root-owned forever and nothing says why -- it simply never appears in the +# report. "It did not change owner" and "no package claims it" look identical +# from the outside, and they need completely different fixes. +_vfy_unclaimed() { + # LC_ALL=C on every sort AND every comm below. comm validates ordering in + # its own locale, so a C-sorted file compared under a UTF-8 one is "not + # sorted" and the diff that comes back is wrong. Pinning the sort alone is + # half a fix -- the two have to agree. + local list="$STATE/tmp/claimed.$$" found="$STATE/tmp/found.$$" n + mkdir -p "$STATE/tmp" 2>/dev/null || return 0 + + # everything any manifest claims, with the host prefix stripped + : > "$list" + local man p + for man in "$MANIFESTS"/*.files "$MANIFESTS"/*.dirs; do + [ -s "$man" ] || continue + while IFS= read -r p; do + [ -n "$p" ] || continue + strip_host_prefix "$p" >> "$list" + done < "$man" + done + LC_ALL=C sort -u -o "$list" "$list" 2>/dev/null || true + + # root-owned files under the shared install directories + : > "$found" + local d + while IFS= read -r d; do + [ -n "$d" ] || continue + d="${SNAP_ROOT%/}$d" + [ -d "$d" ] || continue + # /etc is configuration: root owns it by right, and the book writes + # most of it directly. Reporting those as unclaimed would bury the + # real cases in noise. + case "$d" in "${SNAP_ROOT%/}/etc"|"${SNAP_ROOT%/}/etc/"*) continue ;; esac + find "$d" -xdev -maxdepth 3 -type f -user root 2>/dev/null >> "$found" + done < <(install_dirs_list) + LC_ALL=C sort -u -o "$found" "$found" 2>/dev/null || true + + # Drop the ones root owns ON PURPOSE. never_claim_list is the same rule + # the manifests use, and it has to apply here too or the report asks a + # question with no right answer -- the six command wrappers under + # /usr/lib/pkgusr are root:root BY DESIGN, and were listed as + # 14 root-owned file(s) that NO manifest claims + # under a heading that says to go and find out which package should own + # them. None should. + local _keep="$STATE/tmp/keep.$$" + : > "$_keep" + while IFS= read -r p; do + [ -n "$p" ] || continue + is_never_claimed "$p" || printf '%s\n' "$p" >> "$_keep" + done < "$found" + mv -f "$_keep" "$found" 2>/dev/null || true + + n="$(LC_ALL=C comm -23 "$found" "$list" 2>/dev/null | wc -l | tr -d ' ')" + if [ "${n:-0}" = 0 ]; then + rm -f "$list" "$found" + ok "every root-owned file is claimed by some manifest" + return 0 + fi + warn "$n root-owned file(s) that NO manifest claims:" + LC_ALL=C comm -23 "$found" "$list" 2>/dev/null | head -8 | sed 's/^/ /' >&2 + [ "$n" -gt 8 ] && warn " ... and $((n - 8)) more" + if [ -n "$VERIFY_LOG" ]; then + LC_ALL=C comm -23 "$found" "$list" 2>/dev/null \ + | sed 's/^/unclaimed\t/' >> "$VERIFY_LOG" 2>/dev/null || true + warn " all $n: grep ^unclaimed $VERIFY_LOG" + fi + warn " These were installed before tracking, or their manifest was lost." + warn " Nothing can give them an owner: no package says it installed them." + warn " Find out which package should have:" + warn " lfs-helper which-package <path>" + rm -f "$list" "$found" + return 1 +} + +# NOTE: there were two cmd_verify definitions in this file. The second won -- +# bash keeps the last -- so this one, the tidy one that called the _vfy_* +# helpers, had never run. Everything it did is in the surviving one below. +# +# That is why a finished build reported three things the _vfy_* guards already +# knew to leave alone: /usr/bin/hostname (installed by coreutils, then by +# inetutils, which owns it), /usr/share/info/dir and /etc/group- (owned by no +# package at all). The guards existed and were unreachable. + +cmd_verify() { + need_root + local fix=0 + [ "${1:-}" = "--fix" ] && fix=1 + [ "${1:-}" = "--run" ] && fix=1 + + _vfy_log_open + + local problems=0 fixed=0 + + # ---- 1. two accounts sharing one id ----------------------------------- # + # Ownership is a number: a collision makes files report as the wrong name + # and a package unable to write its own files. + if ! _check_duplicate_ids; then + problems=$((problems+1)) + warn " (verify cannot repair this: which name owns what is yours to decide)" + fi + + # ---- 2. the shared install directories -------------------------------- # + # root:install, group-writable. A staged install used to stamp the package + # user's ownership onto these, handing /usr and /usr/bin to one package and + # losing the scheme entirely -- silently, until some later build failed. + # _vfy_install_dirs is the implementation; it knows that /bin, /sbin, /lib + # and /lib64 are symlinks into /usr and must be checked at their target, + # and it sets ownership by NUMBER so it works before book 7.6. This used + # to be a second copy that did neither. + _vfy_install_dirs "$fix" || problems=$((problems+1)) + + # ---- 3. every manifest's files belong to its package ------------------- # + # + # A SECOND copy of _vfy_manifest_ownership used to live here, inline, and + # it had none of that function's guards. So it reported -- and with --fix + # would have ACTED ON -- three things the real pass knows to leave alone: + # + # /usr/bin/hostname is p_inetutils want p_coreutils + # /usr/share/info/dir is p_e2fsprogs want p_glibc + # /etc/group- is root want p_ncurses + # + # hostname is installed by coreutils and then again by inetutils, which is + # built later and legitimately owns it; taking it back for coreutils breaks + # inetutils. info/dir and the passwd backups belong to no package at all. + # The guards for every one of those already existed -- three hundred lines + # away, in the function this duplicated. + # + # One implementation. This calls it. + local n_wrong=0 checked=0 + _vfy_n_checked=0; _vfy_n_wrong=0 + _vfy_manifest_ownership "$fix" + checked="$_vfy_n_checked"; n_wrong="$_vfy_n_wrong" + if [ "$n_wrong" -gt 0 ]; then + [ "$n_wrong" -ge 8 ] && warn " ... and $((n_wrong - 8)) more" + [ -n "$VERIFY_LOG" ] && warn " all $n_wrong: grep ^wrong-owner $VERIFY_LOG" + problems=$((problems+1)) + else + ok "package files: $checked path(s) owned by the right package" + fi + + # ---- 3b. files no manifest claims ------------------------------------- # + _vfy_unclaimed || problems=$((problems+1)) + + # ---- 4. files with no owner at all ------------------------------------ # + if _has_orphaned_files; then + warn "some files are owned by a uid with no passwd entry" + problems=$((problems+1)) + if [ "$fix" = 1 ]; then + # _vfy_orphans, NOT cmd_fix_orphans. + # + # There is no cmd_fix_orphans -- `fix-orphans` is dispatched as an + # alias for `verify --fix`, so calling it here was both undefined + # and circular. It died mid-repair with + # line 4793: cmd_fix_orphans: command not found + # after chowning 3473 paths, leaving the pass half-done and the + # exit status wrong. A missing function in the repair path is the + # worst place for one: it fires only when something is already + # broken. + _vfy_orphans 1 || true + else + warn " repair with: lfs-helper fix-orphans --run" + fi + else + ok "every file has a real owner" + fi + + # ---- 4b. the passes the dead copy had and this one did not ------------- # + # homes owned by their own user, the build user's leftovers, and the uid + # order. They were written, and reachable only from the definition bash + # discarded. + [ "$fix" = 1 ] && ensure_pkgusr_roots 2>/dev/null + _vfy_user_homes "$fix" || true + _vfy_build_user_leftovers "$fix" || true + _vfy_user_order "$fix" || true + + # ---- 5. staging trees left behind ------------------------------------- # + local st n_stage=0 + for st in "$STATE"/stage/*; do + [ -d "$st" ] || continue + n_stage=$((n_stage+1)) + warn "left-over staging tree: $st" + warn " that build did not finish; the system was not touched" + done + [ "$n_stage" -gt 0 ] && problems=$((problems+1)) + + say "" + if [ "$problems" = 0 ]; then + ok "the tree matches what the manifests say it should be." + return 0 + fi + [ -n "$VERIFY_LOG" ] && say "full report: $VERIFY_LOG" + if [ "$fix" = 1 ]; then + ok "repaired $fixed path(s); $problems area(s) had problems" + say "Run verify again to confirm." + else + warn "$problems area(s) need attention. Repair with:" + say " lfs-helper verify --fix" + fi + return 1 +} + +# Our own scripts are software installed into /usr/bin, so they get a package +# user like anything else. Without one they are reported forever as +# 1516 root-owned file(s) that NO manifest claims: +# /usr/bin/lfs-helper +# which is true, unhelpful, and buries the cases that matter. +# +# `lfs` copies them in and chowns them when it can, but on a fresh tree it +# cannot: /etc/passwd does not exist until book 7.6. So do it here, once the +# user database is real, and write a manifest so verify and which-package can +# answer for them. +# Everything this toolchain installs into the tree and should own. +# +# The four tools plus packagemanager_install -- and the package-users hint's +# helper scripts, which _install_pkgusr_helpers copies in beside them. Those +# were missing, so a finished build reported eight of them as unclaimed: +# /usr/bin/list_package, uninstall_package, forall_direntries_from, +# grep_all_regular_files_for, both list_suspicious_files, +# /usr/sbin/add_package_user, install_package +# They are ours, they arrived with the tools, and they belong to the same user. +# +# The command WRAPPERS are deliberately not here: they stay root:root so a +# package user cannot rewrite the rule that constrains it. never_claim_list +# covers them, so verify stops asking. +PKGUSR_TOOLS="lfs-helper packagemanager packagemanager_install blfs lfs +add_package_user install_package list_package uninstall_package +forall_direntries_from grep_all_regular_files_for +list_suspicious_files list_suspicious_files_from" + +_adopt_pkgusr_tools() { + pkgusr_ready || return 0 + local owner; owner="$(pkg_owner_name pkgusr)" + user_exists "$owner" || ( cmd_add_user "$owner" ) >/dev/null 2>&1 || return 0 + user_exists "$owner" || return 0 + + local man="$MANIFESTS/pkgusr.files" n=0 t p d + : > "$man.new" 2>/dev/null || return 0 + for t in $PKGUSR_TOOLS; do + # /usr/bin AND /usr/sbin: the hint's add_package_user and + # install_package go to sbin, and looking only in bin is why a finished + # build still reported them as claimed by nobody. + for d in /usr/bin /usr/sbin; do + p="${SNAP_ROOT%/}$d/$t" + [ -e "$p" ] || continue + echo "$p" >> "$man.new" + # never take a file another package installed + local cur; cur="$(stat -c %U "$p" 2>/dev/null)" + case "$cur" in + "$owner") continue ;; + root|lfs|UNKNOWN|"") ;; + *) user_exists "$cur" && continue ;; + esac + real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) + done + done + mv -f "$man.new" "$man" 2>/dev/null || rm -f "$man.new" + [ "$n" -gt 0 ] && detail "# gave $n tool(s) to '$owner'" + return 0 +} + +# --------------------------------------------------------------------------- # +# the ownership epoch +# --------------------------------------------------------------------------- # +# +# ONE step where ownership becomes real, and one marker that says it has. +# +# Before it, ownership is not merely wrong, it is MEANINGLESS. A chapter 5-6 +# tree has no /etc/passwd -- book 7.6 creates it -- so no name resolves, not +# even root's; the prompt in there literally reads "I have no name!". Every +# ownership question asked before that point gets a wrong answer with total +# confidence: +# +# * `chown root:install` fails on every directory, because `root` is not a +# name yet, and init-pkgusr reported +# # 0 install directories are now root:install +# # could not set the group on: /bin /sbin /lib /usr /usr/bin ... +# * `find -nouser` matches EVERY file, because nothing resolves, so a +# verify reported 11802 orphans including / and /proc -- and `--fix` would +# have handed the entire tree to the build user in one pass. +# +# These were three separate bugs with one cause: work that needs a user database +# was being done before there was one. So there is now a single gate. Nothing +# touches ownership before it; everything maintains ownership after it. +# +# It is idempotent and it is resumable -- a snapshot restored from before the +# epoch simply crosses it again. +OWNERSHIP_MARK="$STATE_PROGRESS/ownership-established" + +ownership_established() { [ -f "$OWNERSHIP_MARK" ]; } + +# Can ownership be established yet? Prints why not, when asked to. +ownership_possible() { + local explain="${1:-}" why="" + _user_db_ready || why="there is no user database yet ($ETC/passwd) -- book 7.6 creates it" + if [ -z "$why" ]; then + local t + for t in chown chgrp chmod; do + command -v "$t" >/dev/null 2>&1 || why="$t is not available yet" + done + fi + [ -z "$why" ] || { [ -n "$explain" ] && say " ($why)"; return 1; } + return 0 +} + +# Cross it. Everything that decides who owns what happens here, in order. +establish_ownership() { + local run="${1:-1}" + if ! ownership_possible explain; then + return 1 + fi + if [ "$run" != 1 ]; then + say "Would establish ownership now: install group, install directories," + say "the build user's leftovers, package users, adoption." + return 0 + fi + + say "" + say "==============================================================" + say " Establishing ownership" + say "" + say " The user database exists now, so who-owns-what finally has an" + say " answer. Everything before this point ran as root and was not" + say " tracked; everything after it is owned by the package that" + say " installed it." + say "==============================================================" + + # 1. the install group and the shared directories + pkgusr_ready || cmd_init_pkgusr --run || true + ensure_install_dirs_writable 2>/dev/null || true + # and the account roots. These were only ever touched when a home was + # created, so a tree built before their mode changed kept the old one and + # nothing repaired it -- ownership work belongs at the ownership step. + ensure_pkgusr_roots + + # 2. anything the build user still owns from chapters 5-6. Book 7.2 hands + # the tree to root OUTSIDE the chroot; a restored snapshot, or a tree + # whose /usr already looked root-owned, can arrive here without it -- + # /bin -> usr/bin was still lfs:lfs on one. + say "" + say "Handing back anything the build user still owns:" + _vfy_build_user_leftovers 1 || true + + # 3. package users for everything already built, in build order, and the + # manifests reconciled against them + if [ -d "$MANIFESTS" ] && _has_unadopted_packages; then say "" + say "Giving the already-built packages their own users:" + _vfy_manifest_ownership 1 || true + _mark_all_adopted + cmd_sort_users --run >/dev/null 2>&1 || true + fi - say " The compiler looks for them under its OWN target triplet:" - say " $(cc -dumpmachine 2>/dev/null || echo '(cc not working)')" - say " If the headers above are under a different triplet, chapter 6's" - say " gcc-pass2 was built for a different target than this compiler" - say " reports -- rebuild it with the LFS_TGT this tree was made with." + # 4. our own tools + _adopt_pkgusr_tools 2>/dev/null || true + + # 5. files with no owner -- meaningful only now, for the same reason + if _has_orphaned_files; then + say "" + say "Files with no owner (created outside this chroot):" + _vfy_orphans 1 || true fi - rm -rf "$t" + mkstate + date '+%Y-%m-%d %H:%M:%S' > "$OWNERSHIP_MARK" 2>/dev/null || true say "" - if [ "$rc" = 0 ]; then - ok "Toolchain is usable." - else - warn "Do not start chapter 8 until this is fixed." - fi - return $rc + ok "# ownership established -- every install from here on is tracked" + say "# check it any time with: lfs-helper verify" + say "" + return 0 } -# Has the copied-in export already been applied? -_groups_already_imported() { - local f="$STATE/collector-groups.import" first - [ -f "$f" ] || return 0 - [ -f "$COLLECTOR_MAP" ] || return 1 - # every dir decision in the file must be in the map - while IFS='|' read -r kind a b; do - [ "$kind" = dir ] || continue - grep -qxF "$a|$b" "$COLLECTOR_MAP" 2>/dev/null || return 1 - done < <(grep -vE '^\s*(#|$)' "$f") +# Called after every step in build-all: cross the epoch the moment it becomes +# possible, and never before. One call site, so it cannot half-happen. +_ownership_checkpoint() { + ownership_established && { mark_done init-ownership; return 0; } + ownership_possible || return 0 + establish_ownership 1 || true + # `init-ownership` is a step now, so `list` and `next` must agree with what + # actually happened -- including on a tree whose steporder predates it. + ownership_established && mark_done init-ownership return 0 } +cmd_establish_ownership() { + need_root + local run=0 + [ "${1:-}" = "--run" ] && run=1 + if ownership_established; then + ok "# ownership was established on $(cat "$OWNERSHIP_MARK" 2>/dev/null)" + say "# re-run it anyway with: lfs-helper verify --fix" + return 0 + fi + if ! ownership_possible explain; then + warn "Not yet. Build the chapter-7 steps first:" + warn " lfs-helper build-all" + return 1 + fi + establish_ownership "$run" +} + +# Were these scripts generated by the version of the tools now installed? +# +# The step order and the scripts live in the TREE; the tools live outside it. +# Nothing connected the two, so a tree generated by an older version went on +# running steps that version had: +# +# ===== [104/105] last-step ===== +# chown: invalid user: 'wget:wget' +# +# under a lfs-helper (1.10.0) that had removed `last-step` entirely and no +# longer knew what it was. The chroot TOOLS already carry a build id for +# exactly this reason -- a stale copy has cost more debugging time here than +# anything else -- and the generated scripts did not. +_warn_if_scripts_are_stale() { + local f="$STATE_PROGRESS/generated-by" gen="" + [ -f "$f" ] && gen="$(sed -n 's/^version=//p' "$f" | head -n1)" + [ "$gen" = "$LFS_HELPER_VERSION" ] && return 0 + warn "" + if [ -z "$gen" ]; then + warn "!! These build scripts carry no version stamp, so they predate" + warn " $LFS_HELPER_VERSION. The step order may contain steps this" + warn " version no longer has." + else + warn "!! These build scripts were generated by $gen, but the tools are" + warn " $LFS_HELPER_VERSION. The step order may not match what this" + warn " version knows how to build." + fi + warn " Regenerate them, from OUTSIDE the chroot:" + warn " lfs build-system gen-chroot-scripts --run --overwrite" + warn "" + return 1 +} + cmd_build_all() { need_root local force=0 from="" @@ -3071,22 +5554,31 @@ cmd_build_all() { || die "no such step: $from (see: lfs-helper list)" fi - # Set the package-user system up NOW, before the first package is built, - # not when we first trip over a package that needs it. The earlier it - # exists, the fewer files are installed as root and have to be given back - # afterwards -- and it is what this whole toolchain is for. - if ! pkgusr_ready; then - say "Setting up the package-user system first (it must exist before" - say "any package is built):" - cmd_init_pkgusr --run || true + # Ownership is NOT set up here. It used to be -- "before the first package + # is built" sounds right and is wrong: on a chapter 5-6 tree /etc/passwd + # does not exist yet, so `chown root:install` failed on all 27 directories + # and the run reported + # # 0 install directories are now root:install (group-writable) + # then carried on building. It happens at the ownership epoch instead, + # the moment book 7.6 has created the user database -- see + # _ownership_checkpoint, called after every step below. + _warn_if_scripts_are_stale || true + if ownership_established; then + pkgusr_ready || cmd_init_pkgusr --run || true + else + say "Ownership will be established once book 7.6 has created the user" + say "database; until then these steps run as root and are not tracked." say "" fi + _check_state_version || true + _check_duplicate_ids || true + # A groups export copied into the tree is a set of decisions already made # -- apply them before anything can ask the same questions again. Leaving # this to be run by hand meant the file sat in the tree while the build # stopped to ask about directories it already had an answer for. - if [ -f "$STATE/collector-groups.import" ] && ! _groups_already_imported; then + if [ -f "$STATE_GROUPS/collector-groups.import" ] && ! _groups_already_imported; then say "Applying the collector-group decisions copied into this tree:" cmd_import_groups --run || true say "" @@ -3100,9 +5592,9 @@ cmd_build_all() { # created first, so cancelling part-way through the chown left the user in # place and every later run skipped the repair entirely, with half a # million files still owned by nobody. - if _has_orphaned_files; then + if ownership_established && _has_orphaned_files; then say "Some files have no owner in this chroot (created outside it)." - cmd_fix_orphans --run || true + _vfy_orphans 1 || true say "" fi @@ -3116,13 +5608,15 @@ cmd_build_all() { # * a chapter-8 package that replaces a chapter-7 file writes into a # directory its own user already owns, instead of hitting root. # It is idempotent -- a package that already has its user is skipped. - if pkgusr_ready && [ -d "$MANIFESTS" ] && _has_unadopted_packages; then - say "Giving the already-built packages their own users first:" - cmd_adopt_existing --run || true - say "" - fi - - local total done_n=0 s _needs_you="" + # Only once there IS a user database. On a chapter 5-6 tree /etc/passwd + # does not exist yet (book 7.6 creates it), and every "create the package + # user" would fail. The chapter-7 steps below run as root and need no + # package users, so this simply happens later -- see _adopt_when_ready. + # A tree that is already past 7.6 -- a resumed build, a restored chapter-8 + # snapshot -- crosses the epoch right here instead of mid-loop. + _ownership_checkpoint + + local total done_n=0 s _needs_you="" _adopted_late=0 total=$(steps | wc -l) [ "$total" -gt 0 ] || die "no scripts found in $SCRIPTS Generate them from outside the chroot: @@ -3187,12 +5681,12 @@ cmd_build_all() { # --from: skip everything before the named step, then build onward if [ "$started" = 0 ]; then if [ "$s" = "$from" ]; then started=1; else - say "${C_DIM}[$i/$total] $s -- before --from $from, skipping${C_OFF}" + detail "[$i/$total] $s -- before --from $from, skipping" continue fi fi if is_done "$s" && [ "$force" = 0 ]; then - say "${C_DIM}[$i/$total] $s -- already built, skipping${C_OFF}" + detail "[$i/$total] $s -- already built, skipping" done_n=$((done_n+1)) continue fi @@ -3211,6 +5705,22 @@ cmd_build_all() { warn "" fi + # The chapter-7 steps run as root and create the user database (7.6). + # Once it exists, the chapter 5-6 packages can finally be given their + # users -- before the first package that needs one, and in build order + # so binutils takes the first uid. + # THE ownership epoch. init-files (book 7.6) is the step that creates + # the user database, so this fires immediately after it and exactly + # once. Everything before was root's and untracked; everything after + # belongs to the package that installed it. + _ownership_checkpoint + + # The C++ headers live under the target the compiler was built for. + # If those two names disagree the headers are present but invisible, + # and gcc fails with "bits/c++config.h: No such file or directory" -- + # which reads like a missing file, not a mismatch. Say which it is. + _warn_on_triplet_mismatch + # The toolchain check belongs HERE, not at the top: chapter 7 builds # the temporary tools and needs no C++ compiler at all, so checking up # front blocked the very steps that would fix a broken toolchain -- @@ -3219,55 +5729,12 @@ cmd_build_all() { if ! is_root_step "$s" && [ "${_tc_checked:-0}" = 0 ]; then _tc_checked=1 if ! cmd_check_toolchain >/dev/null 2>&1; then - # A toolchain package interrupted mid-install is the usual - # cause: `make install` replaced /usr/bin/gcc with the new - # compiler but had not yet installed its headers, so the new - # one shadows the working one and nothing can build. The fix - # is to finish the install that was interrupted -- the tool - # knows which package that was, so do it rather than printing - # a diagnosis and stopping. - local _pkg _fixed=0 _candidates - # The marker only exists if the interruption happened while a - # version of this tool that writes one was running. A package - # that has already installed files but is not marked built is - # in exactly the same state, so treat it the same way -- that - # is precisely what a half-installed gcc looks like. - _candidates="$(_interrupted_packages 2>/dev/null)" - if [ -z "$_candidates" ] && [ -s "$MANIFESTS/$s.files" ] \ - && ! is_done "$s"; then - _candidates="$s" - say "" - say "'$s' has already installed files but is not marked" - say "built -- an earlier attempt stopped part-way." - fi - for _pkg in $_candidates; do - warn "" - warn "The compiler is broken, and '$_pkg' was interrupted" - warn "mid-install -- that is almost certainly why." - say "Finishing its install before going on:" - say "" - if ( cmd_build "$_pkg" --phase install --force ); then - _fixed=1 - fi - if cmd_check_toolchain >/dev/null 2>&1; then - ok "" - ok "# the compiler works again -- continuing" - say "" - _fixed=1 - break - fi - done - if ! cmd_check_toolchain >/dev/null 2>&1; then - warn "" - warn "Stopping before '$s': this is the first package that" - warn "needs a working C++ compiler, and there is not one." - if [ "$_fixed" = 1 ]; then - warn "Finishing the interrupted install did not repair it." - fi - say "" - cmd_check_toolchain || true - return 1 - fi + warn "" + warn "Stopping before '$s': this is the first package that" + warn "needs a working C++ compiler, and there is not one." + say "" + cmd_check_toolchain || true + return 1 fi fi @@ -3304,7 +5771,7 @@ cmd_build_all() { say "install directories now, so packages can no longer overwrite" say "each other's files:" say "" - cmd_seal_install_dirs --run + _vfy_seal_install_dirs fi say "" @@ -3358,14 +5825,14 @@ cmd_build_all() { say "no longer overwrite each other's files. (Collector-group" say "directories are shared deliberately and are left alone.)" say "" - cmd_seal_install_dirs --run + _vfy_seal_install_dirs fi # Package users are appended as they are created, so by now /etc/passwd is # in build order. Sorting by id makes it readable -- and it is purely # cosmetic, so it is safe to do without asking. say "" - cmd_sort_users --run || true + _vfy_user_order 1 || true } # Are the install directories already sticky? Checking a handful of the ones @@ -3623,7 +6090,7 @@ _dir_contents_foreign() { } infer_dirs() { - local tmp="$STATE/.dirinfer.$$" + local tmp="$STATE_PROGRESS/.dirinfer.$$" : > "$tmp" local man name owner p d r="${SNAP_ROOT%/}" for man in "$MANIFESTS"/*.files; do @@ -3651,50 +6118,6 @@ infer_dirs() { rm -f "$tmp" } -cmd_adopt_dirs() { - need_root - local run=0; [ "${1:-}" = "--run" ] && run=1 - pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" - - say "Assign directories that packages created for themselves." - say "Derived from the file lists: a directory goes to a package only when" - say "every tracked file under it belongs to that one package. Shared" - say "directories stay root:install." - say "" - [ "$run" = 1 ] || warn "(dry run -- nothing was changed. Add --run to apply.)" - say "" - - local d owner n=0 skipped=0 - while IFS="$(printf '\t')" read -r d owner; do - [ -n "$d" ] && [ -n "$owner" ] || continue - # Never take a directory whose contents belong to someone else. - # - # infer_dirs walks UP from every installed file, so a package that - # drops one file deep inside another package's tree claims every - # parent on the way -- which is how zstd came to own - # /usr/include/c++/15.2.0/bits, gcc's own C++ headers, and gcc could - # then no longer build. Being the only package with a manifest entry - # under a directory does not make it yours. - _dir_contents_foreign "$d" "$owner" && { skipped=$((skipped+1)); continue; } - [ -d "$d" ] || continue - is_install_dir "$d" && { skipped=$((skipped+1)); continue; } - user_exists "$owner" || continue - if [ "$run" = 1 ]; then - chown "$owner:$owner" "$d" 2>/dev/null && n=$((n+1)) - else - n=$((n+1)) - [ "$n" -le 15 ] && say " $d -> $owner" - fi - done < <(infer_dirs) - - say "" - if [ "$run" = 1 ]; then - ok " $n directory(ies) assigned to their package" - else - say " would assign $n directory(ies) ($skipped shared install dirs left alone)" - fi - return 0 -} cmd_repair_adoption() { need_root @@ -3800,8 +6223,9 @@ cmd_repair_adoption() { # Manifests oldest-first: the order the packages were built in. _manifests_in_build_order() { - local f - for f in "$MANIFESTS"/*.files; do + local f pat="*.files" + [ "${1:-}" = dirs ] && pat="*.dirs" + for f in "$MANIFESTS"/$pat; do [ -e "$f" ] || continue # %.Y is nanosecond precision: whole seconds tie for packages built # in the same second, and the order then falls back to alphabetical @@ -3809,287 +6233,13 @@ _manifests_in_build_order() { done | sort -n -k1,1 | cut -f2- } -cmd_adopt_existing() { - need_root - local run=0 verbose=0 - while [ $# -gt 0 ]; do - case "$1" in - --run) run=1; shift ;; - -v|--verbose) verbose=1; shift ;; - *) die "usage: lfs-helper adopt-existing [--run]" ;; - esac - done - pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" - - say "Give the CHAPTER 5-6 packages their own package users." - say "" - say "Those were built outside the chroot, before any package user existed," - say "so their files are still root-owned -- only their recorded file lists" - say "say who installed what. Packages built in here already got their" - say "owner as part of the build and are not touched." - say "" - [ "$run" = 1 ] || warn "(dry run -- nothing was changed. Add --run to apply.)" - say "" - - local man name n nd p tgt total_f=0 total_p=0 skipped=0 - # In BUILD order, not alphabetical. - # - # The uid a package gets is the order its user is created in, so walking - # the manifests alphabetically gave gcc (built 2nd of 22) a higher uid than - # man-pages (the first chapter-8 package). Sorting by manifest mtime -- - # written when the package was built -- makes the uids a record of the - # order the system was actually built in. - for man in $(_manifests_in_build_order); do - [ -e "$man" ] || continue - name="$(basename "${man%.files}")" - # init-dirs/init-files are book housekeeping -- they create the - # directory tree and /etc files and own nothing, so they get no user. - # - # The *-tmp temporary tools DO get adopted, to the base package user - # (perl-tmp -> perl). Leaving them root-owned was a mistake: chapter 8 - # rebuilds exactly these packages, and its package user then cannot - # write into the root-owned tree chapter 7 left behind -- which is what - # made perl fail in /usr/lib/perl5/5.42/core_perl. Since it is the same - # package either way, the same user should own it throughout. - case "$name" in - init-*) skipped=$((skipped+1)); continue ;; - esac - [ -s "$man" ] || { skipped=$((skipped+1)); continue; } - # already done, and its manifest has not grown since: nothing to do. - # Without this the whole pass ran again on every build-all, re-checking - # tens of thousands of paths that were already correct. - if ! _needs_adoption "$man"; then - skipped=$((skipped+1)); continue - fi - - # gcc-pass1/-pass2/libstdcpp all belong to 'gcc', etc. - # show which package is being worked on WHILE it is worked on: a big - # manifest takes a few seconds and silence looks like a hang - printf '\r checking %-40s' "$name" >&2 - local owner; owner="$(pkg_owner_name "$name")" - if ! user_exists "$owner"; then - if [ "$run" = 1 ]; then - # In a SUBSHELL: cmd_add_user dies on failure, and `die` - # exits the whole script -- so `|| continue` never ran and one - # unlucky package aborted the entire adoption pass. - ( cmd_add_user "$owner" ) >/dev/null 2>&1 \ - || { warn " $name: could not create package user '$owner' -- skipping"; \ - skipped=$((skipped+1)); continue; } - fi - fi - - n=0; nd=0 - while IFS= read -r p; do - [ -n "$p" ] || continue - tgt="$(strip_host_prefix "$p")" - [ -e "$tgt" ] || [ -L "$tgt" ] || continue - if [ "$run" = 1 ]; then - # already correct: don't re-chown tens of thousands of paths - [ "$(stat -c %U "$tgt" 2>/dev/null)" = "$owner" ] && continue - chown -h "$owner:$owner" "$tgt" 2>/dev/null && n=$((n+1)) - else - n=$((n+1)) - [ "$verbose" = 1 ] && say " $tgt" - fi - done < "$man" - - if [ -f "$MANIFESTS/$name.dirs" ]; then - while IFS= read -r p; do - [ -n "$p" ] || continue - tgt="$(strip_host_prefix "$p")" - [ -d "$tgt" ] || continue - is_install_dir "$tgt" && continue - if [ "$run" = 1 ]; then - [ "$(stat -c %U "$tgt" 2>/dev/null)" = "$owner" ] && continue - chown "$owner:$owner" "$tgt" 2>/dev/null && nd=$((nd+1)) - else - nd=$((nd+1)) - fi - done < "$MANIFESTS/$name.dirs" - fi - - [ "$run" = 1 ] && _mark_adopted "$name" "$(_manifest_size "$man")" - printf '\r%*s\r' 60 '' >&2 - if [ "$owner" != "$name" ]; then - printf " %-22s %6d file(s) %4d dir(s) -> user '%s'\n" \ - "$name" "$n" "$nd" "$owner" - else - printf " %-22s %6d file(s) %4d dir(s)\n" "$name" "$n" "$nd" - fi - total_f=$((total_f+n)); total_p=$((total_p+1)) - done - - say "" - if [ "$run" = 1 ]; then - ok " $total_f path(s) across $total_p package(s) now owned by package users" - mark_done "adopt-existing" - say "" - say "Every package on the system now belongs to its own user." - say "Next: lfs-helper next" - else - say " would assign $total_f path(s) across $total_p package(s)" - fi - [ "$skipped" -gt 0 ] && say " ($skipped manifest(s) skipped -- not built yet, or empty)" - return 0 -} - -cmd_seal_install_dirs() { - need_root - local run=0; [ "${1:-}" = "--run" ] && run=1 - local d n=0 missing=0 - say "Add the sticky bit to the install directories -- the last step of the" - say "chroot stage. Until now they were group-writable but NOT sticky, so" - say "package users could replace the temporary system's root-owned files and" - say "each other's. Once sticky, a package user can still create files here" - say "but can no longer modify or delete another package's." - say "" - say "Run this when every package is built (end of chapter 8) -- doing it" - say "earlier will break installs that legitimately overwrite older files." - say "" - # Two sources: the known install directories, and ANY directory that - # actually carries the install group -- packages create install dirs the - # static list does not know about (/usr/lib/<pkg>, /usr/share/<pkg>), and - # those need sealing just as much. Collector-group directories are shared - # deliberately and are never touched. - while IFS= read -r d; do - [ -n "$d" ] || continue - [ -d "$d" ] || { missing=$((missing+1)); continue; } - [ -k "$d" ] && continue # already sticky - if [ "$run" = 1 ]; then - chmod o+t "$d" 2>/dev/null && n=$((n+1)) - else - n=$((n+1)) - fi - done < <( - { while IFS= read -r d; do - printf '%s\n' "${SNAP_ROOT%/}$d" - done < <(install_dirs_list) - find "${SNAP_ROOT:-/}" -xdev -type d -group install \ - -not -path "${SNAP_ROOT%/}/.lfs-pkgusr/*" \ - -not -path "$SRCROOT/*" \ - -not -path "${SNAP_ROOT%/}/sources/*" \ - -not -path "${SNAP_ROOT%/}/dev/*" \ - -not -path "${SNAP_ROOT%/}/proc/*" \ - -not -path "${SNAP_ROOT%/}/sys/*" \ - -not -path "${SNAP_ROOT%/}/run/*" 2>/dev/null - } | sort -u - ) - if [ "$run" = 1 ]; then - ok " sticky bit set on $n install directory(ies)" - else - say " would set the sticky bit on $n directory(ies)" - warn "(dry run -- nothing was changed. Add --run to apply.)" - fi - [ "$missing" -gt 0 ] && say " ($missing listed dirs don't exist here -- skipped)" - return 0 -} - -cmd_fix_ownership() { - need_root - local run=0 only="" - while [ $# -gt 0 ]; do - case "$1" in - --run) run=1; shift ;; - -*) die "usage: lfs-helper fix-ownership [<package>] [--run]" ;; - *) only="$1"; shift ;; - esac - done - say "Give each package's tracked files to its package user." - say "Use this once if packages were installed as root before the" - say "package-user system was in place." - say "" - [ "$run" = 1 ] || warn "(dry run -- nothing was changed. Add --run to apply.)" - say "" - local total_f=0 total_p=0 man name n - for man in "$MANIFESTS"/*.files; do - [ -e "$man" ] || continue - name="$(basename "${man%.files}")" - [ -n "$only" ] && [ "$name" != "$only" ] && continue - # chapter-7 temporaries are root-owned on purpose - if is_root_step "$name"; then - say " ${C_DIM}$name -- temporary chapter-7 tool, leaving as root${C_OFF}" - continue - fi - if ! user_exists "$name"; then - if [ "$run" = 1 ]; then - # subshell: cmd_add_user dies on failure, which would exit - # the whole script instead of skipping this one package - ( cmd_add_user "$name" ) >/dev/null \ - || { warn " $name: cannot create user -- skipping"; continue; } - else - say " $name -- would create the package user first" - fi - fi - n=0 - while IFS= read -r p; do - [ -n "$p" ] || continue - [ -e "$p" ] || [ -L "$p" ] || continue - # Only count files that actually need a change: root-owned or - # mis-owned. And chown to $name -- an earlier version used a - # variable that was never set here, so this command reported - # success while changing nothing, which is exactly how a system - # ends up with thousands of root-owned package files. - local cur - cur="$(stat -c %U "$p" 2>/dev/null)" - [ "$cur" = "$name" ] && continue - if [ "$run" = 1 ]; then - chown -h "$name:$name" "$p" 2>/dev/null && n=$((n+1)) - else - n=$((n+1)) - fi - done < "$man" - # dirs the package created itself (never the shared install dirs) - local nd=0 mand="$MANIFESTS/$name.dirs" - if [ -f "$mand" ]; then - while IFS= read -r p; do - [ -n "$p" ] || continue - [ -d "$p" ] || continue - is_install_dir "$p" && continue - if [ "$run" = 1 ]; then - chown "$name:$name" "$p" 2>/dev/null && nd=$((nd+1)) - else - nd=$((nd+1)) - fi - done < "$mand" - fi - printf " %-22s %6d file(s) %4d dir(s)\n" "$name" "$n" "$nd" - total_f=$((total_f+n)); total_p=$((total_p+1)) - done - say "" - if [ "$run" = 1 ]; then - ok " $total_f path(s) across $total_p package(s) now owned by their package user" - say "" - say " Re-apply the shared install directories too (root:install):" - say " lfs-helper init-pkgusr --run" - else - say " would touch $total_f path(s) across $total_p package(s)" - fi -} # Per-command help. `lfs-helper fix-users -h` should say what the command # does, not print the whole usage page again. cmd_help_for() { case "$1" in - fix-users) - cat <<'EOF' -lfs-helper fix-users [--run] [--skel <dir>] - -Repairs package users that were created before the environment existed, or -whose files came from the wrong place. For every package user it: - - * points .bash_profile, .bashrc and build at the shared files in - /etc/pkgusr, replacing any wrong or missing link; - * copies the skeleton files (build.conf, .project) from - /etc/pkgusr/skel-package when that exists; - * writes VERSION from the install script's name_version, so `lfs update` - can tell what is actually installed. - -Nothing is deleted and no package files are touched -- only the user's home. -Without --run it reports what it would change. -EOF - ;; export-groups) cat <<'EOF' lfs-helper export-groups [<file>] @@ -4108,18 +6258,6 @@ Loads a file written by export-groups: recreates the groups with their members, and records the directory-to-group decisions so those directories are never asked about again. With no file it looks for one placed in the tree by `lfs build-system gen-chroot-scripts`. -EOF - ;; - seal-install-dirs) - cat <<'EOF' -lfs-helper seal-install-dirs [--run] - -The LAST step of the chroot stage. Adds the sticky bit to every shared install -directory, so a package user can still create files there but can no longer -modify or delete another package's. - -Run it only when every package is built: doing it earlier breaks installs that -legitimately overwrite an older version's files. EOF ;; build) @@ -4167,34 +6305,39 @@ Follow the build: lfs-helper list every step and its state ('next --all' does the same) lfs-helper check-toolchain can this compiler build anything? - lfs-helper fix-toolchain --run repair a triplet mismatch in the headers Build: lfs-helper build <name> build one step as its package user lfs-helper build-all build everything remaining, in order lfs-helper done|undone <name> mark a step built / clear that mark build options: --phase <p> --force --tests --jobs N --no-auto-fix + --stage / --no-stage (stage the install, or do not) Ownership and permissions (the package-user model): lfs-helper init-pkgusr --run FIRST: install group + install dirs - lfs-helper add-user <name> create one package user (with its links) - lfs-helper fix-ownership --run give every tracked file to its package - lfs-helper adopt-dirs --run same for the directories packages made - lfs-helper fix-users --run repair homes: bash links, VERSION files + lfs-helper add-user <name> create one package user + lfs-helper verify check the tree against what the + manifests say it should be + lfs-helper verify --fix and repair what it can lfs-helper fix-perms <name> --run grant one package the access it needs lfs-helper grant-dir <dir> <user> --run give one user access to one directory - lfs-helper seal-install-dirs --run - LAST: sticky bits on shared dirs Collector groups (shared directories): lfs-helper export-groups [<file>] save the decisions for the next build lfs-helper import-groups --run load them again Look around / tidy up: + lfs-helper install-as <name> -- <cmd ...> + install anything as its own package user + lfs-helper verify [--fix] does the tree match the manifests? + lfs-helper pkgusr-info [<name>] where each package came from + lfs-helper pkgusr-home <name> print an account's home directory + lfs-helper owner-name <package> print the account name for a package lfs-helper manifests [<name>] what each package installed lfs-helper which-package <path> which package installed this file lfs-helper clean-sources --run tidy /sources + lfs-helper clean-state --run remove build scratch (keeps the records) lfs-helper prune-locales --run drop unwanted translations lfs-helper find-user-site --run find pip installs hidden in ~/.local @@ -4225,23 +6368,45 @@ case "${1:-}" in build) shift; cmd_build "$@" ;; build-all) shift; cmd_build_all "$@" ;; manifests) shift; cmd_manifests "$@" ;; - fix-ownership) shift; cmd_fix_ownership "$@" ;; fix-perms) shift; cmd_fix_perms "$@" ;; clean-sources) shift; cmd_clean_sources "$@" ;; find-user-site) shift; cmd_find_user_site "$@" ;; check-toolchain) shift; cmd_check_toolchain "$@" ;; - fix-toolchain) shift; cmd_fix_toolchain "$@" ;; + verify) shift; cmd_verify "$@" ;; + install-as) shift; cmd_install_as "$@" ;; + pkgusr-info) shift; cmd_pkgusr_info "$@" ;; + pkgusr-home) shift; cmd_pkgusr_home "$@" ;; + owner-name) shift; cmd_owner_name "$@" ;; + clean-state) shift; cmd_clean_state "$@" ;; which-package) shift; cmd_which_package "$@" ;; - sort-users) shift; cmd_sort_users "$@" ;; - fix-orphans) shift; cmd_fix_orphans "$@" ;; grant-dir) shift; cmd_grant_dir "$@" ;; - fix-users) shift; cmd_fix_users "$@" ;; + verify) shift; cmd_verify "$@" ;; + install-as) shift; cmd_install_as "$@" ;; + pkgusr-info) shift; cmd_pkgusr_info "$@" ;; + pkgusr-home) shift; cmd_pkgusr_home "$@" ;; + owner-name) shift; cmd_owner_name "$@" ;; + clean-state) shift; cmd_clean_state "$@" ;; export-groups) shift; cmd_export_groups "$@" ;; import-groups) shift; cmd_import_groups "$@" ;; prune-locales) shift; cmd_prune_locales "$@" ;; - adopt-existing) shift; cmd_adopt_existing "$@" ;; - adopt-dirs) shift; cmd_adopt_dirs "$@" ;; repair-adoption) shift; cmd_repair_adoption "$@" ;; + adopt-existing|adopt-dirs) + shift; cmd_adopt_existing "$@" ;; + sort-users) shift; cmd_sort_users "$@" ;; + # The one step where ownership becomes real. build-all crosses it by + # itself the moment book 7.6 has created the user database; this is for + # doing it by hand, or checking whether it has happened. + establish-ownership) shift; cmd_establish_ownership "$@" ;; + init-accounts) shift; cmd_init_accounts "$@" ;; + # The repair commands people were told to run for years. verify is the + # ONE implementation; these are aliases, not second copies -- a command + # that has been deleted is worse than one that is redundant. + fix-ownership|fix-orphans|fix-users|repair-perms) + shift + case "${1:-}" in + --run) set -- --fix ;; + esac + cmd_verify "$@" ;; seal-install-dirs) shift; cmd_seal_install_dirs "$@" ;; done) shift; mark_done "${1:?usage: lfs-helper done <name>}"; ok "marked '$1' built" ;; undone) shift; unmark "${1:?usage: lfs-helper undone <name>}"; ok "cleared '$1'" ;; diff --git a/lfs-sanity.sh b/lfs-sanity.sh new file mode 100644 index 0000000..5fe120e --- /dev/null +++ b/lfs-sanity.sh @@ -0,0 +1,269 @@ +#!/bin/bash +# lfs-sanity.sh -- read-only survey of an lfs-pkgusr tree. +# +# Run it INSIDE the chroot, as root. It changes nothing: no chown, no chmod, +# no files created outside the report. Everything lands in ONE file so it can +# be handed over whole. +# +# ./lfs-sanity.sh -> /tmp/lfs-sanity.txt +# ./lfs-sanity.sh /path/report.txt +# +# Each section says what it checks and what a hit MEANS, so the report is +# readable on its own. + +OUT="${1:-/tmp/lfs-sanity.txt}" +: > "$OUT" || { echo "cannot write $OUT" >&2; exit 1; } +exec 3>&1 1>>"$OUT" 2>&1 + +PKGROOT="${LFS_PKGUSR_ROOT:-/usr/src/pkgusr}" +CFGROOT="${LFS_CFGUSR_ROOT:-/usr/src/cfg}" +STATE="${LFS_PKGUSR_DIR:-${LFS_SRC_ROOT:-/usr/src}/lfs-pkgusr}" +hits=0 +# the package-user prefix, as the tools compute it +PFX="$(sed -n 's/^pkgusr_prefix=\(.*\)$/\1/p' /etc/pkgusr/packagemanager.conf 2>/dev/null | head -1)" +PFX="${PFX:-p}"; PFX="${PFX%_}_" + +sec() { echo; echo "=============================================================="; \ + echo "== $*"; echo "=============================================================="; } +note(){ echo " $*"; } +hit() { echo "!! $*"; hits=$((hits+1)); } + +echo "lfs-pkgusr sanity report -- $(date '+%Y-%m-%d %H:%M:%S')" +echo "host: $(uname -srm)" +echo "roots: PKGROOT=$PKGROOT CFGROOT=$CFGROOT STATE=$STATE" + +sec "0. tool versions (host and chroot copies must MATCH)" +# A stale chroot copy running old code has cost more debugging time in this +# project than anything else, which is why each tool prints a build id. +for t in lfs-helper lfs packagemanager blfs packagemanager_install; do + if command -v "$t" >/dev/null 2>&1; then + printf ' %-22s %s\n' "$t" \ + "$("$t" --version 2>/dev/null | head -1 || echo '(no --version)')" + else + note "$(printf '%-22s NOT INSTALLED' "$t")" + fi +done + +# Is the build finished? +# +# The sticky bit means opposite things on either side of that line. DURING the +# build it is a bug -- package users could not replace the temporary system's +# files. AFTER it, seal-install-dirs is supposed to have set it, and its +# ABSENCE is the bug. Reporting one rule for both flagged six directories on a +# completed 105/105 tree where every one of them was correct. +BUILD_FINISHED=0 +if [ -f "$STATE/progress/steporder" ] && [ -f "$STATE/progress/steps-built" ]; then + _todo="$(grep -c . "$STATE/progress/steporder" 2>/dev/null)" + _done="$(grep -c . "$STATE/progress/steps-built" 2>/dev/null)" + [ -n "$_todo" ] && [ "$_todo" -gt 0 ] && [ "${_done:-0}" -ge "$_todo" ] \ + && BUILD_FINISHED=1 +fi + +sec "1. shared directories: owner, group and mode" +[ "$BUILD_FINISHED" = 1 ] \ + && note "the build is finished, so install directories SHOULD be sticky" \ + || note "the build is not finished, so install directories should NOT be sticky yet" +# Install dirs are root:install and group-writable DURING the build, and only +# become sticky (o+t) at the very end. Sticky early stops package users +# writing into shared directories for the rest of chapter 8. +# /sources and /build are different: book 3.1 makes them root:root 1777. +for d in / /usr /usr/bin /usr/lib /usr/share /usr/include /etc \ + /usr/src "$PKGROOT" "$CFGROOT" /sources /build /bin /sbin /lib; do + [ -e "$d" ] || continue + printf ' %-22s %-16s %s%s\n' "$d" \ + "$(stat -c '%U:%G' "$d" 2>/dev/null)" \ + "$(stat -c '%A' "$d" 2>/dev/null)" \ + "$([ -L "$d" ] && echo ' (symlink)')" +done +echo +for d in /usr /usr/bin /usr/lib /usr/share /usr/include; do + [ -d "$d" ] || continue + [ -L "$d" ] && continue + m="$(stat -c %A "$d" 2>/dev/null)" + [ "${m:5:1}" = w ] || hit "$d is NOT group-writable ($m) -- package users cannot install" + case "$m" in + *t) [ "$BUILD_FINISHED" = 1 ] \ + || hit "$d is already STICKY ($m) -- sealed before the build finished" ;; + *) [ "$BUILD_FINISHED" = 1 ] \ + && hit "$d is NOT sticky ($m) -- the build finished but the seal never fired" ;; + esac + # THE GROUP, not just the mode. + # + # This section printed owner:group from the start and only ever checked the + # mode. A tree ran 34 packages with every install directory at root:root + # 775 -- group-writable, but group `root`, and package users are in + # `install`. The scheme was inert and the report said nothing, because the + # one column that would have shown it was decoration. + g="$(stat -c %G "$d" 2>/dev/null)" + o="$(stat -c %U "$d" 2>/dev/null)" + [ "$g" = install ] || hit "$d is group '$g', want 'install' -- package users cannot write here" + [ "$o" = root ] || hit "$d is owned by '$o', want 'root' -- a package has taken a shared directory" +done +for d in /usr/src "$PKGROOT" "$CFGROOT"; do + [ -d "$d" ] || continue + case "$(stat -c %A "$d" 2>/dev/null)" in + *t) [ "$BUILD_FINISHED" = 1 ] \ + || hit "$d is STICKY -- the seal fired early (it belongs at the end)" ;; + esac +done +for d in /sources /build; do + [ -d "$d" ] || continue + o="$(stat -c '%U:%G' "$d")"; m="$(stat -c %A "$d")" + [ "$o" = "root:root" ] || hit "$d is $o -- book 3.1 says root:root" + [ "$m" = "drwxrwxrwt" ] || hit "$d is $m -- book 3.1 says 1777" +done + +sec "2. ids: uid must equal gid, and nothing may share an id" +# Ownership is stored as a NUMBER. Two names on one id makes files report as +# the wrong package and a package unable to write its own files. +awk -F: '$3>=1000 && $3!=$4 {printf "!! %s has uid %s but gid %s\n", $1,$3,$4}' /etc/passwd +awk -F: '$3>=1000 && $3!=$4' /etc/passwd | grep -q . \ + && hits=$((hits+1)) || note "every account has uid == gid" +d="$(awk -F: '{print $3}' /etc/passwd | sort | uniq -d)" +if [ -n "$d" ]; then + hit "duplicate UIDs: $(echo $d)" + for u in $d; do awk -F: -v u="$u" '$3==u{print " "$1" uid "$3}' /etc/passwd; done +else + note "no duplicate uids" +fi +d="$(awk -F: '{print $3}' /etc/group | sort | uniq -d)" +[ -n "$d" ] && hit "duplicate GIDs: $(echo $d)" || note "no duplicate gids" + +sec "3. account names: every one should carry its prefix" +# Three kinds, three prefixes: p_ packages, cfg_ config steps, u_ application +# users. Anything else escaped the naming scheme -- created before it landed, +# or by a path that bypasses pkg_owner_name. +# +# There is no tmp_ and no init_. A temporary step is the same package chapter 8 +# rebuilds and is built as that package's user, so it is a p_. An init step +# owns no files and gets no account at all. Either prefix appearing here means +# a tool created an account it should not have. +# nobody is 65534 and belongs to the system, not to us -- cap the range +bad="$(awk -F: '$3>=10000 && $3<65000 {print $1}' /etc/passwd \ + | grep -v '^p_\|^u_\|^cfg_')" +if [ -n "$bad" ]; then + hit "accounts without a known prefix:"; printf ' %s\n' $bad +else + note "every account >= 10000 carries a prefix" +fi + +# One account, ONE prefix. `p_cfg_bootscripts` wore both: the package prefix +# stacked on the config step's own. Nothing creates those any more, so finding +# one means a tool is still applying the package prefix regardless of kind. +stacked="$(awk -F: '$3>=10000 && $3<65000 {print $1}' /etc/passwd \ + | grep '^p_cfg_\|^p_u_\|^p_p_')" +if [ -n "$stacked" ]; then + hit "accounts wearing two prefixes:"; printf ' %s\n' $stacked +else + note "no account carries more than one prefix" +fi +note "counts: $(awk -F: '$3>=10000 && $3<65000' /etc/passwd | wc -l) build accounts" + +sec "4. homes: each must be owned by its own user" +# A home owned by root cannot be written by the package user that lives in it. +n=0 +for d in "$PKGROOT"/*/ "$CFGROOT"/*/; do + [ -d "$d" ] || continue + b="$(basename "$d")"; o="$(stat -c %U "$d" 2>/dev/null)" + if ! id "$b" >/dev/null 2>&1; then + echo " ? $d has no matching account (stray directory)" + elif [ "$o" != "$b" ]; then + echo "!! $d owned by '$o', want '$b'"; n=$((n+1)) + fi +done +[ "$n" -gt 0 ] && { hit "$n home(s) not owned by their user"; \ + note "repair: lfs-helper verify --fix"; } || note "every home is owned by its user" + +sec "5. files owned by a uid with NO name" +# The sharp version of the /sources problem: a uid that resolves to nothing is +# unambiguous, whereas a uid that resolves to the WRONG name looks fine. +# /tmp and /build are scratch: nothing there has to have an owner, and both +# are full of half-unpacked trees mid-build. +o="$(find / -xdev \( -path /tmp -o -path /build -o -path /sources \) -prune \ + -o -xdev \( -nouser -o -nogroup \) -print 2>/dev/null | head -40)" +if [ -n "$o" ]; then + hit "files with no owner (first 40):"; printf ' %s\n' $o +else + note "every file resolves to a real user and group" +fi + +sec "6. groups: install and the collectors" +getent group install 2>/dev/null | sed 's/^/ /' \ + || hit "no 'install' group -- shared directories cannot work" +getent group install 2>/dev/null | awk -F: '$3!=9999{print "!! install is gid "$3", want 9999"}' +c="$(getent group 2>/dev/null | awk -F: '$3>=90000' | wc -l)" +note "$c collector group(s) in the 90000+ range" +# same cap as the accounts: nogroup is 65534 and belongs to the system +getent group 2>/dev/null | awk -F: '$3>=10000 && $3<65000 && $1 !~ /^p_|^u_|^cfg_/ \ + {print "!! group "$1" (gid "$3") is outside every convention"}' + +sec "7. how much of the tree still belongs to root" +# Root-owned files under the install dirs are normally the temporary system's, +# waiting to be adopted. A large number late in the build means adoption is +# not running. +for d in /usr/bin /usr/lib /usr/share /usr/include /etc; do + [ -d "$d" ] || continue + printf ' %-16s %6s root-owned of %6s total\n' "$d" \ + "$(find "$d" -xdev -type f -user root 2>/dev/null | wc -l)" \ + "$(find "$d" -xdev -type f 2>/dev/null | wc -l)" +done + +sec "8. build state" +# The state directory is sorted: progress/ holds what the build has done. +# Reading the old flat paths reported `adopted.list: MISSING (the adoption gate +# will never close)` on a tree where adoption had completed perfectly. +PROG="$STATE/progress" +[ -f "$PROG/steporder" ] && note "steps in order: $(grep -c . "$PROG/steporder")" +[ -f "$PROG/steps-built" ] && note "steps built: $(grep -c . "$PROG/steps-built")" +[ -d "$STATE/manifests" ] && note "manifests: $(ls -1 "$STATE/manifests"/*.files 2>/dev/null | wc -l)" +[ -f "$PROG/adopted.list" ] \ + && note "adopted.list: $(grep -c . "$PROG/adopted.list") entries" \ + || note "adopted.list: MISSING (the adoption gate will never close)" +# A manifest whose owner does not exist can never be adopted. +# +# An EMPTY manifest is not that. A prose configuration step -- cfg_clock, +# cfg_hosts, cfg_locale -- writes into files another package already owns, so +# its manifest legitimately records nothing and it legitimately has no account. +# Reporting those listed fourteen findings on a perfect tree: +# ? manifest 'cfg_clock' -> no account 'cfg_clock' +# which is noise, and noise is what makes a real finding easy to miss. +for m in "$STATE"/manifests/*.files; do + [ -e "$m" ] || continue + b="$(basename "${m%.files}")" + case "$b" in last-step|init-*|refind|pkgusr) continue ;; esac + # nothing installed -> nothing to own + grep -q . "$m" 2>/dev/null || continue + # Manifests are named by STEP ("bash", "gcc-pass1", "python-tmp"); accounts + # are named by OWNER ("p_bash"). Comparing the two directly reported every + # manifest in the tree as orphaned. Apply the same mapping the tools do: + # prefix, lowercase, fold -tmp/-passN. + a="$(printf '%s' "$b" | tr 'A-Z' 'a-z' | sed 's/-tmp$//; s/-pass[0-9]$//')" + case "$a" in cfg_*|p_*) ;; *) a="${PFX}${a}" ;; esac + id "$a" >/dev/null 2>&1 && continue + # Last resort: ask the tool who owns the files. A step can install under a + # name that is not its own -- libstdcpp's files belong to p_gcc, because the + # book builds it out of the gcc tree -- and only lfs-helper knows that. + _f="$(grep -m1 . "$m")" + _own="" + command -v lfs-helper >/dev/null 2>&1 && [ -e "$_f" ] \ + && _own="$(stat -c %U "$_f" 2>/dev/null)" + case "$_own" in + p_*|cfg_*) note "manifest '$b' -> no account '$a', but its files belong to '$_own'" ;; + *) echo " ? manifest '$b' -> no account '$a'" ;; + esac +done +command -v lfs-helper >/dev/null 2>&1 && { echo; echo "--- lfs-helper list ---"; \ + lfs-helper list 2>&1 | tail -25; } + +sec "SUMMARY" +if [ "$hits" -eq 0 ]; then + echo " no problems found by these checks" +else + echo " $hits problem area(s) flagged above -- search this file for '!!'" +fi +echo +echo "Nothing was changed. This report is read-only." + +exec 1>&3 +echo "wrote $OUT ($hits problem area(s) flagged)" +echo "hand over with: cat $OUT" diff --git a/packagemanager b/packagemanager index 81a0fee..e7f2844 100644 --- a/packagemanager +++ b/packagemanager @@ -75,7 +75,31 @@ except ImportError: # non-unix: degrade gracefully # /usr/src holds one directory (== home) per package-user. Overridable so the # tool can be exercised against a fixture tree in testing. -BASE_DIR = os.environ.get("PKGUSR_BASE", "/usr/src") +# Where package users live. Config wins over the built-in default, so a +# system built with a different layout keeps it; the environment still wins +# over both, for tests and one-off runs. +def _base_dir_default(): + # CONFIG_PATH is defined further down, so read the path the same way it + # does rather than depending on definition order. + path = os.environ.get("PKGUSR_CONFIG") or "/etc/pkgusr/packagemanager.conf" + try: + with open(path) as f: + for line in f: + k, _, v = line.partition("=") + if k.strip() == "pkgusr_home" and v.strip(): + return v.strip() + except OSError: + pass + return "/usr/src" + + +BASE_DIR = os.environ.get("PKGUSR_BASE") or _base_dir_default() + + +def appuser_home(): + """Where application users live -- separate from package users, because + they are accounts a person uses rather than build trees.""" + return load_config().get("appuser_home") or BASE_DIR INSTALL_SCRIPTS_DIR = os.environ.get("PKGUSR_INSTALL_SCRIPTS", "/etc/pkgusr/install_scripts") TMP_DIR = "/tmp/packagemanager" @@ -85,7 +109,7 @@ TMP_DIR = "/tmp/packagemanager" CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf") # Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure # changes (so stale plans are ignored); TOOL_VERSION is the program version. -TOOL_VERSION = "1.7.0" +TOOL_VERSION = "1.10.1" def _build_id(): @@ -105,6 +129,7 @@ def _build_id(): PLAN_VERSION = 1 _DEFAULT_CONFIG = {"collector_prefix": "sysgroup", "user_prefix": "u", + "pkgusr_prefix": "p", "cfguser_prefix": "cfg", "main_user": ""} _config = None @@ -189,6 +214,138 @@ def user_prefix(): return load_config()["user_prefix"].rstrip("_") + "_" +def pkgusr_prefix(): + """Prefix on PACKAGE users: `p_` gives p_gcc, p_zlib. + + Three kinds of account share one passwd file and one namespace, and + without prefixes they are indistinguishable from each other and from the + system's own accounts -- a package called `man` or `news` collides with a + real one. `p_` for packages, `u_` for application users, `<collector>_` + for the shared-directory groups. + + An explicitly EMPTY value means "no prefix", which is how a tree built + before prefixes keeps working; it is not the same as unset. + """ + v = load_config().get("pkgusr_prefix", _DEFAULT_CONFIG["pkgusr_prefix"]) + v = (v or "").rstrip("_") + return (v + "_") if v else "" + + +def cfguser_prefix(): + """Prefix on CONFIG-STEP users: `cfg_` gives cfg_bootscripts. + + A config step used to be `p_cfg_bootscripts` -- the package prefix stacked + on top of the step name's own `cfg_`. Two prefixes on one account means + neither one identifies it: `p_` said "package" while the account lived in + the config root. One account, one prefix. + """ + v = load_config().get("cfguser_prefix", _DEFAULT_CONFIG["cfguser_prefix"]) + v = (v or "").rstrip("_") + return (v + "_") if v else "" + + +def pkgusr_kind(name): + """What KIND of account is this -- 'pkg', 'cfg' or 'app'? + + The one place that decides. pkgusr_name and pkgusr_root both answer from + it, so a name cannot be given one kind's prefix and placed in another + kind's root -- which is exactly what produced `p_cfg_bootscripts` under + /usr/src/cfg. + + Takes a bare name (`gcc`, `cfg_bootscripts`), an account name (`p_gcc`), + or a legacy stacked name (`p_cfg_bootscripts`). + """ + n = name or "" + ppfx = pkgusr_prefix() + if ppfx and n.startswith(ppfx): + n = n[len(ppfx):] + upfx = user_prefix() + if upfx and n.startswith(upfx): + return "app" + cpfx = cfguser_prefix() + if cpfx and n.startswith(cpfx): + return "cfg" + return "pkg" + + +def pkgusr_name(name): + """Package name -> account name, with the prefix ITS KIND carries. + + IDEMPOTENT, and that is the whole point: this is called on names typed by + a person (`gcc`), on directory names read back from disk (`p_gcc`), and on + values that already went through it. A second pass must not give p_p_gcc. + Because it is idempotent, every caller can apply it without first knowing + which kind of name it is holding. + + A legacy stacked name is repaired on the way through: the package prefix is + stripped before the kind's own prefix is applied, so `p_cfg_bootscripts` + comes back as `cfg_bootscripts`. + """ + if not name: + return name + kind = pkgusr_kind(name) + # an application user or a collector group is not a package and is never + # renamed here + if kind == "app" or name.startswith(collector_prefix()): + return name + pfx = cfguser_prefix() if kind == "cfg" else pkgusr_prefix() + n = name + ppfx = pkgusr_prefix() + if ppfx and n.startswith(ppfx): + n = n[len(ppfx):] + if not pfx or n.startswith(pfx): + return n + return pfx + n + + +def unprefix_pkgusr(name): + """Package-user name -> package name, for display.""" + pfx = pkgusr_prefix() + if pfx and name.startswith(pfx): + return name[len(pfx):] + return name + + +# Accounts live in subdirectories of BASE_DIR by kind, so `ls /usr/src` says +# what a thing IS rather than listing a hundred packages and a few config +# steps as one flat set: +# +# /usr/src/pkgusr/p_gcc a package +# /usr/src/cfg/cfg_bootscripts a config step that installs files +# /usr/src/u_firefox an application user +PKGUSR_SUBDIR = os.environ.get("PKGUSR_SUBDIR", "pkgusr") +CFGUSR_SUBDIR = os.environ.get("CFGUSR_SUBDIR", "cfg") + + +def pkgusr_root(name): + """Which root an account belongs under. Takes any kind of name. + + Answers from pkgusr_kind, the same function pkgusr_name uses, so the + prefix an account is given and the root it is placed in cannot disagree. + """ + kind = pkgusr_kind(name) + if kind == "app": + return BASE_DIR + sub = CFGUSR_SUBDIR if kind == "cfg" else PKGUSR_SUBDIR + return os.path.join(BASE_DIR, sub) if sub else BASE_DIR + + +def pkgusr_roots(): + """Every account root, for the passes that walk all of them.""" + seen, out = set(), [] + for sub in (PKGUSR_SUBDIR, CFGUSR_SUBDIR): + d = os.path.join(BASE_DIR, sub) if sub else BASE_DIR + if d not in seen: + seen.add(d) + out.append(d) + return out + + +def pkgusr_home(name): + """Where a package user lives. Takes either kind of name.""" + return os.path.join(pkgusr_root(name), pkgusr_name(name)) + + def main_user(): return load_config().get("main_user", "").strip() @@ -255,12 +412,28 @@ def ask_yes_no(question): def list_all_users(): """Every package-user, i.e. every directory directly under BASE_DIR.""" - try: - return sorted(d for d in os.listdir(BASE_DIR) - if os.path.isdir(os.path.join(BASE_DIR, d))) - except FileNotFoundError: - printError(f"{BASE_DIR} does not exist.") + # Directory names ARE user names, so they carry the prefix. Hand back + # package names: everything user-facing speaks in package names, and + # pkgusr_name() puts the prefix back wherever it is needed. + # + # A root that does not exist is normal -- a system with no config-step + # packages has no cfg/ directory. ALL of them missing is not, and must + # still be reported: swallowing it turns "nothing is installed here" into + # a silent empty list, which reads as "no packages". + out, found = [], False + for root in pkgusr_roots(): + try: + names = os.listdir(root) + except OSError: + continue + found = True + out += [unprefix_pkgusr(d) for d in names + if os.path.isdir(os.path.join(root, d))] + if not found: + printError(f"{BASE_DIR} does not exist (looked in " + f"{', '.join(pkgusr_roots())}).") return [] + return sorted(set(out)) # --------------------------------------------------------------------------- # @@ -541,7 +714,7 @@ def gather_user(name): u.name, u.uid, u.gid = pw.pw_name, str(pw.pw_uid), str(pw.pw_gid) except KeyError: pass - home = os.path.join(BASE_DIR, name) + home = pkgusr_home(name) if os.path.isdir(home): u.home = home pkg_lst = os.path.join(home, "pkg.lst") @@ -667,7 +840,7 @@ def _warn_if_wrong_file_edited(user, chosen): Editing the second is an easy mistake, and the symptom is silent: the install runs the unedited script and fails in exactly the same way as before, so it looks as though the edit was thrown away.""" - runtime = os.path.join(BASE_DIR, user, f"install_{user}") + runtime = os.path.join(pkgusr_home(user), f"install_{user}") if not os.path.isfile(runtime): return try: @@ -693,7 +866,7 @@ def resolve_script(user, name_version, script_dir, assume_yes=False, we say so. 'install_last' (a read-only record) and 'install_<user>' (the engine's runtime copy) are never treated as sources. --regenerate forces a fresh build from the book.""" - local = os.path.join(BASE_DIR, user, f"install_{name_version}") + local = os.path.join(pkgusr_home(user), f"install_{name_version}") if (not force_regen and os.path.isfile(local) and _script_name_version(local) == name_version): printInfo(f" using LOCAL script (your edits, not the book): {local}") @@ -810,7 +983,7 @@ def assess(u, expected=None, run_cmd=True): # 1) strongest signal: a validate command that prints the installed version. # A per-package 'validate' file in the home persists across versions and # wins over any validate_cmd inside the (regenerated) install script. - vfile = os.path.join(u.home or os.path.join(BASE_DIR, u.name or u.search), + vfile = os.path.join(u.home or pkgusr_home(u.name or u.search), "validate") vcmd = None if os.path.isfile(vfile): @@ -876,7 +1049,7 @@ def cmd_verify(args): printError("verify --set-validate-command needs a package name.") return for name in args.packages: - home = os.path.join(BASE_DIR, name) + home = pkgusr_home(name) if not os.path.isdir(home): printError(f"{name}: no /usr/src/{name} -- add-user it first.") continue @@ -1000,6 +1173,13 @@ def _cmd_path(name): PKGUSR_ETC = os.environ.get("PKGUSR_ETC", "/etc/pkgusr") +# The command wrappers, at the ONE location lfs-helper writes them to. +# +# This used to be spelled out inside the profile heredoc below while lfs-helper +# wrote its wrappers somewhere else entirely, so a package user's PATH led with +# a directory that did not exist. Same name, same variable, both tools. +WRAPPERS = os.environ.get("LFS_WRAPPERS", "/usr/lib/pkgusr") + # The shared environment every package user gets, as the package-users hint # describes it: .bash_profile and .bashrc are SYMLINKS into a central directory # so all package users share one environment, and `build` is the helper each @@ -1016,7 +1196,7 @@ export LC_ALL=POSIX # /usr/local/bin comes FIRST after the wrappers: locally installed # overrides belong ahead of /usr/bin, and a temporary wrapper there # (the no-verify wget, say) is useless if nothing ever finds it. -PATH=/usr/lib/pkgusr:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin +PATH=@WRAPPERS@:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin export PATH # a short prompt that makes it obvious which package user you are @@ -1094,7 +1274,10 @@ def ensure_pkgusr_etc(verbose=True): printError(f"cannot create {PKGUSR_ETC}: {e}") return False for fname, content, mode in ( - ("bash_profile", _PKGUSR_BASH_PROFILE, 0o644), + # @WRAPPERS@ substituted here, not written into the template: + # the template is a plain string so $HOME and \\u reach the file + # untouched, and the wrapper path is the one value that is ours. + ("bash_profile", _PKGUSR_BASH_PROFILE.replace("@WRAPPERS@", WRAPPERS), 0o644), ("bashrc", _PKGUSR_BASHRC, 0o644), ("build", _PKGUSR_BUILD, 0o755)): path = os.path.join(PKGUSR_ETC, fname) @@ -1111,12 +1294,13 @@ def ensure_pkgusr_etc(verbose=True): def init_package_user_home(name, home=None, verbose=True): + name = pkgusr_name(name) # the home belongs to the ACCOUNT, not the package """Give a package user the environment the scheme expects. Without this the account exists but has no .bash_profile, no .bashrc and no build helper -- `su - <pkg>` lands in a bare shell with the wrong umask and no PATH to the wrapper commands.""" - home = home or os.path.join(BASE_DIR, name) + home = home or pkgusr_home(name) if not ensure_pkgusr_etc(verbose=verbose): return False try: @@ -1164,9 +1348,13 @@ def create_package_user(name, home=None): installed as a package user at all. Shadow's own tools do the same job: one user, a private group of the same name, plus the supplementary 'install' group.""" + # Callers hand us a PACKAGE name ("gcc"); the account is p_gcc. Normalise + # once here rather than at each call site -- pkgusr_name is idempotent, so + # a caller that already prefixed is unaffected. + name = pkgusr_name(name) if _user_exists(name): return True - home = home or os.path.join(BASE_DIR, name) + home = home or pkgusr_home(name) if _have_cmd(find_add_package_user()[0]): cmd = find_add_package_user() + [name, name, "10000", "20000", @@ -1233,7 +1421,7 @@ def cmd_nimgnu_list(args): printHeader("collector groups") print() for name, gid, members in rows: - manifest = os.path.join(BASE_DIR, name, "pkg.lst") + manifest = os.path.join(pkgusr_home(name), "pkg.lst") entries = pkglist_entries(manifest) owns = f" owns {len(entries)} dirs/files" if os.path.isfile(manifest) else "" print(f" {name:<26} gid {gid:<8} members: " @@ -1402,7 +1590,7 @@ def _resolve_group_arg(value): def write_group_manifest(group, background=True): """Write /usr/src/<group>/pkg.lst = everything owned by that group, via forall_direntries_from (run as root so the whole tree is scanned).""" - home = os.path.join(BASE_DIR, group) + home = pkgusr_home(group) if not os.path.isdir(home): return False forall = " ".join(shlex.quote(x) for x in find_forall()) @@ -1647,7 +1835,7 @@ def warn_if_nothing_installed(name): useful -- but any build whose install step is a no-op leaves a package user owning nothing, and the problem only surfaces later when something tries to use it.""" - pkglist = os.path.join(BASE_DIR, name, "pkg.lst") + pkglist = os.path.join(pkgusr_home(name), "pkg.lst") n = 0 if os.path.isfile(pkglist): try: @@ -2103,10 +2291,11 @@ def install_local(name, path, run, yes, verb="install", test=False, clean=False) printInfo("\nDry run -- to proceed, re-run with --run, or execute:\n" f" packagemanager_install {name} {path}") return - if not _user_exists(name) and (yes or ask_yes_no( - f"\nPackage-user {name} doesn't exist -- create it first?")): - _run(find_add_package_user() + [name, name, "10000", "20000", - name, "10000", "20000"]) + _acct = pkgusr_name(name) + if not _user_exists(_acct) and (yes or ask_yes_no( + f"\nPackage-user {_acct} doesn't exist -- create it first?")): + _run(find_add_package_user() + [_acct, _acct, "10000", "20000", + _acct, "10000", "20000"]) if not yes and not ask_yes_no(f"Build and {verb} {name} now?"): printWarning("Aborted.") return @@ -2248,7 +2437,7 @@ def cmd_install(args): clear_status() printSuccess(f"Done -- {total} package(s) installed.") printInfo(f" results: packagemanager errors logs: " - f"{os.path.join(BASE_DIR, '<pkg>', 'log')}") + f"{os.path.join(BASE_DIR, pkgusr_name('<pkg>'), 'log')}") # =========================================================================== # @@ -2267,7 +2456,7 @@ _PROTECTED_PREFIXES = ("/etc/", "/var/", "/boot/", "/home/", "/root/", "/srv/", def _pkglist_paths(user): - pl = os.path.join(BASE_DIR, user, "pkg.lst") + pl = os.path.join(pkgusr_home(user), "pkg.lst") return {p for p in pkglist_entries(pl) if p.startswith("/")} @@ -2327,7 +2516,7 @@ def _is_git_package(name): """A package whose install script builds from a git repo (is_git=1, or it clones/pulls) rather than a book tarball.""" p = (_canonical_script_path(name) - or os.path.join(BASE_DIR, name, f"install_{name}")) + or os.path.join(pkgusr_home(name), f"install_{name}")) if not p or not os.path.isfile(p): return False try: @@ -2359,7 +2548,7 @@ def cmd_update(args): and any(_is_git_package(p) for p in args.packages): for name in args.packages: script = _canonical_script_path(name) \ - or os.path.join(BASE_DIR, name, f"install_{name}") + or os.path.join(pkgusr_home(name), f"install_{name}") if not _is_git_package(name): printWarning(f"{name}: not a git package and not in the book -- " f"skipping (check the name, or use --local).") @@ -2614,7 +2803,7 @@ def cmd_update(args): printSuccess(f"Done -- {total} package(s) updated" + (f", {errors} error(s)." if errors else ".")) printInfo(f" results: packagemanager errors logs: " - f"{os.path.join(BASE_DIR, '<pkg>', 'log')}") + f"{os.path.join(BASE_DIR, pkgusr_name('<pkg>'), 'log')}") # =========================================================================== # # command: remove [IMPLEMENTED] # =========================================================================== # @@ -2761,7 +2950,7 @@ def clear_status(): def _pkg_log_dir(user): - return os.path.join(BASE_DIR, user, "log") + return os.path.join(pkgusr_home(user), "log") def _maybe_remove_old_sources(user, keep_name_version, keep_sources=False): @@ -2770,7 +2959,7 @@ def _maybe_remove_old_sources(user, keep_name_version, keep_sources=False): Removes automatically; pass keep_sources=True to skip.""" if keep_sources: return - home = os.path.join(BASE_DIR, user) + home = pkgusr_home(user) keep = keep_name_version victims = [] try: @@ -2988,7 +3177,7 @@ def _run(cmd, show_output=False): def remove_one(name, args): u = gather_user(name) - home = os.path.join(BASE_DIR, name) + home = pkgusr_home(name) has_home = os.path.isdir(home) if not u.has_user and not has_home: printWarning(f"{name}: nothing to remove (no user, no home).") @@ -3112,7 +3301,7 @@ def write_pkg_list(user, background=True): """Regenerate /usr/src/<user>/pkg.lst. Runs list_package AS the package user (su -) when possible, so it only scans directories that user owns -- much faster than scanning the whole filesystem as root.""" - home = os.path.join(BASE_DIR, user) + home = pkgusr_home(user) if not os.path.isdir(home): printError(f"{user}: no package home at {home}") return False @@ -3335,7 +3524,7 @@ def cmd_script(args): fixing permissions (no recompile).""" name = args.package phase = args.phase - home = os.path.join(BASE_DIR, name) + home = pkgusr_home(name) if not _user_exists(name): printError(f"{name}: no such package user.") return @@ -3729,7 +3918,7 @@ def _link_xdg_profile(main, u): printWarning(f" could not write {XDG_SKEL}: {e}") return - home = os.path.join(BASE_DIR, u) + home = pkgusr_home(u) target = os.path.join(home, ".bash_profile") if os.path.islink(target) and os.path.realpath(target) == \ os.path.realpath(XDG_SKEL): @@ -3756,7 +3945,7 @@ def _setup_application_user(u, args): * an optional shared directory both accounts can write * an optional launcher in ~/bin so the program runs as that user """ - home = os.path.join(BASE_DIR, u) + home = os.path.join(appuser_home(), u) main = main_user() if not main: printWarning("\nNo main user is set, so this account cannot be reached " @@ -3960,7 +4149,7 @@ def cmd_user_disallow(args): def cmd_user_delete(args): u = _user_prefixed(args.name) - home = os.path.join(BASE_DIR, u) + home = pkgusr_home(u) has_home = os.path.isdir(home) if not _user_exists(u) and not has_home: printInfo(f"{u}: does not exist.") @@ -4257,7 +4446,7 @@ def _canonical_script_path(name): the file for the currently-installed version; otherwise the versioned file whose name matches its own name_version (not the runtime 'install_<user>' copy or 'install_last').""" - home = os.path.join(BASE_DIR, name) + home = pkgusr_home(name) if not os.path.isdir(home): return None u = gather_user(name) @@ -4429,7 +4618,7 @@ def cmd_regenerate_script(args): os.makedirs(script_dir, exist_ok=True) written, skipped, failed = 0, 0, 0 for name in sorted(set(args.packages)): - home = os.path.join(BASE_DIR, name) + home = pkgusr_home(name) if not os.path.isdir(home): printWarning(f"{name}: no package-user home at {home} -- skipping " f"(create it first with: packagemanager add-user {name}).") @@ -4513,7 +4702,7 @@ def cmd_migrate_scripts(args): pristine, edited, already, cannot = [], [], 0, 0 for name in sorted(set(names)): - if not os.path.isdir(os.path.join(BASE_DIR, name)): + if not os.path.isdir(pkgusr_home(name)): continue st, cand, book_path, book_nv = classify_script(name, script_dir) if st == "no-script": @@ -4526,7 +4715,7 @@ def cmd_migrate_scripts(args): continue # where the migrated script should live: named for the BOOK version, so # it's the default script the next update/reinstall picks up. - out_path = os.path.join(BASE_DIR, name, f"install_{book_nv}") + out_path = os.path.join(pkgusr_home(name), f"install_{book_nv}") if st == "outdated": # book changed, you made no edits -> safe pristine.append((name, cand, book_path, book_nv, out_path)) else: # edited / no-baseline -> hand-merge diff --git a/packagemanager_install b/packagemanager_install index 94fd412..fd513ad 100644 --- a/packagemanager_install +++ b/packagemanager_install @@ -10,17 +10,75 @@ # General Public License <https://www.gnu.org/licenses/> for details. # # -# the package_short_name is the name for te created user +# the package_short_name is the name for the created user # packagemanager_install package_short_name path_to_install_file # +PACKAGEMANAGER_INSTALL_VERSION="1.10.1" +# A fingerprint of this file's own contents, the same as the other four tools. +# +# Without it `lfs-sanity.sh` section 0 -- "host and chroot copies must MATCH" -- +# could not read this one at all, and printed the usage error instead: +# packagemanager_install Please provide an install script with $2 +# A tool you cannot ask "which build are you?" is the one that quietly goes +# stale, and a stale copy in the chroot has cost more debugging time in this +# project than anything else. +_pmi_build_id() { + local f="${BASH_SOURCE[0]:-$0}" + if [ -r "$f" ] && command -v md5sum >/dev/null 2>&1; then + md5sum "$f" 2>/dev/null | cut -c1-7 + else + echo unknown + fi +} + +case "${1:-}" in + --version|-V) + echo "packagemanager_install $PACKAGEMANAGER_INSTALL_VERSION (build $(_pmi_build_id))" + exit 0 ;; + -h|--help) + echo "usage: packagemanager_install <package> <install-script>" + echo " packagemanager_install --version" + exit 0 ;; +esac + + + + +# Where an account's home is. Accounts are grouped by kind under /usr/src, so +# a directory listing says what a thing IS: +# /usr/src/pkgusr/p_gcc a package +# /usr/src/cfg/cfg_bootscripts a config step that installs files +# +# Prefixing is IDEMPOTENT, so this can be handed a package name ("gcc") or an +# account name ("p_gcc") and gives the same answer either way. Building the +# path by hand instead is what once made the build cd into +# /usr/src/pkgusr/man-pages while the account home was p_man-pages. +PKGUSR_ROOT="${LFS_PKGUSR_ROOT:-/usr/src/pkgusr}" +CFGUSR_ROOT="${LFS_CFGUSR_ROOT:-/usr/src/cfg}" +PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-p_}" + +pkgusr_home_for() { + local n="$1" bare + case "$n" in + "") printf '%s' "$PKGUSR_ROOT"; return ;; + esac + if [ -n "$PKGUSR_PREFIX" ]; then + case "$n" in "$PKGUSR_PREFIX"*) ;; *) n="${PKGUSR_PREFIX}${n}" ;; esac + fi + bare="${n#"$PKGUSR_PREFIX"}" + case "$bare" in + cfg_*) printf '%s/%s' "$CFGUSR_ROOT" "$n" ;; + *) printf '%s/%s' "$PKGUSR_ROOT" "$n" ;; + esac +} skel=/etc/pkgusr/skel-package user_name="" install_script="" install_script_basename="" -install_user_dir="/usr/src/$user_name" +install_user_dir="$(pkgusr_home_for "$user_name")" sourced=false date_log="$(date +%Y-%m-%d_%H-%m-%S)" home_user="n76310" @@ -245,7 +303,7 @@ addUserToGroup() { fi checkIfUserInGroup $change_name $change_group && return - echo "$change_group to $change_name" | tee /usr/src/$change_name/log/install_nimgnu_group-$date_log.log + echo "$change_group to $change_name" | tee "$(pkgusr_home_for "$change_name")/log/install_nimgnu_group-$date_log.log" echoBlue "\n\n##########################\n" find $change_dir -group $change_name @@ -397,7 +455,7 @@ EOF installPkg() { user_name="$1" - install_user_dir="/usr/src/$user_name" + install_user_dir="$(pkgusr_home_for "$user_name")" install_dir=/etc/pkgusr/install_scripts install_script="$2" install_script_basename="$(basename $2)" diff --git a/skel-u_xdg/.bash_profile b/skel-u_xdg/.bash_profile new file mode 100644 index 0000000..8f78d9d --- /dev/null +++ b/skel-u_xdg/.bash_profile @@ -0,0 +1,42 @@ +# /etc/pkgusr/skel-u_xdg/.bash_profile +# +# The login profile for SHARED package users -- the ones in the group on your +# XDG runtime directory. It is what lets a program running as its own account +# reach your display, session bus and audio. +# +# Linked into a shared user's home by: +# packagemanager user create <name> --shared +# packagemanager user setup <name> --shared +# +# XDG_RUNTIME_DIR below is written when the file is first installed, from the +# configured main user. Edit it if your session lives somewhere else. + +# The wrappers directory must be the first entry in the PATH. +# The /tools/bin directory must be the last entry in the PATH and can be +# removed at the end of Chapter 6. +export PATH=/usr/lib/pkgusr:/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin + +# Make prompt reflect that we are a package user. +export PROMPT_COMMAND='PS1="package \u:"`pwd`"> "' + +# The session this account is allowed to reach. +export WAYLAND_DISPLAY=wayland-1 +#export DISPLAY=:0 +#export DBUS_SESSION_BUS_ADDRESS=/dev/null # fake dbus so we dont start one... +export XDG_RUNTIME_DIR=@XDG_RUNTIME_DIR@ + +# Go to the home directory whenever we su to a package user. +cd + +if [ -d "$HOME/bin" ] ; then + pathprepend $HOME/bin +fi +if [ -d "$HOME/.local/bin" ] ; then + pathprepend $HOME/.local/bin +fi + +for script in /etc/profile.d/*.sh ; do + if [ -r $script ] ; then + . $script + fi +done diff --git a/test_lfs_crosschain.sh b/test_lfs_crosschain.sh index 1646784..baf9205 100644 --- a/test_lfs_crosschain.sh +++ b/test_lfs_crosschain.sh @@ -13,15 +13,66 @@ LFS_TOOL="${1:-./lfs}" LFS_BOOK="${2:-/mnt/user-data/uploads/LFS-BOOK-12_4-NOCHUNKS.html}" T=$(mktemp -d /tmp/lfstest.XXXXXX) PASS=0; FAIL=0 -ok() { echo " PASS $1"; PASS=$((PASS+1)); } -bad() { echo " FAIL $1"; FAIL=$((FAIL+1)); } +# Count in FILES, not shell variables. +# +# Several checks call ok/bad from inside a subshell -- `( ... ) && ok || bad`, +# a pipeline, a `while read` fed by process substitution. A variable +# incremented there dies with the subshell, so the suite printed 446 PASS +# lines and reported 433. A wrong count is worse than no count: it can hide a +# regression behind a number that still looks healthy. +_COUNTDIR="${TMPDIR:-/tmp}/lfstest-count.$$" +mkdir -p "$_COUNTDIR" +: > "$_COUNTDIR/pass" +: > "$_COUNTDIR/fail" + +ok() { echo " PASS $1"; echo x >> "$_COUNTDIR/pass"; } +bad() { echo " FAIL $1"; echo x >> "$_COUNTDIR/fail"; } + +# Lift ONE function out of a bash source file, to exercise it in isolation. +# +# This exists because `sed -n "/^fn() {/,/^}/p"` cannot do it. A one-line +# function -- +# pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; } +# -- has no line starting with `}`, so sed runs on and swallows whatever +# function comes next. That is not a cosmetic problem: the swallowed body +# referenced $STATE, `set -u` killed the subshell before a single assertion +# ran, the output collected nothing, and the test reported PASS. It had been +# passing that way while its four comparisons were also wrong. +# +# A test that cannot fail is worse than no test, so the slicer is shared and +# there is exactly one of it. +_slice_fn() { # _slice_fn <file> <fn> + awk -v fn="$2" ' + BEGIN { start = "^" fn "\\(\\) \\{" } + !inside && $0 ~ start { + print + if ($0 !~ /\}[[:space:]]*$/) inside = 1 + next + } + inside { print; if ($0 ~ /^\}/) inside = 0 } + ' "$1" +} -cleanup() { rm -rf "$T"; } +# the python-heredoc blocks report their own verdict; fold them in the same way +_count_rc() { + if [ "$1" -eq 0 ]; then echo x >> "$_COUNTDIR/pass" + else echo x >> "$_COUNTDIR/fail"; fi +} + +_pass_total() { wc -l < "$_COUNTDIR/pass" | tr -d ' '; } +_fail_total() { wc -l < "$_COUNTDIR/fail" | tr -d ' '; } + +cleanup() { rm -rf "$T" "$_COUNTDIR"; } trap cleanup EXIT export LFS="$T/lfs" -export BUILD_ROOT="$LFS/sources" -mkdir -p "$LFS/sources" "$LFS/tools/bin" +# Two directories, as the tools use them: SOURCES_DIR holds the downloaded +# tarballs and is never written by a build; BUILD_ROOT is the scratch a build +# unpacks into. They were one directory once, which is what every stale +# marker and foreign-ownership bug in /sources came from. +export SOURCES_DIR="$LFS/sources" +export BUILD_ROOT="$LFS/build" +mkdir -p "$LFS/sources" "$LFS/build" "$LFS/tools/bin" # ---------------------------------------------------------------- fake package # A package that behaves like a real autotools one: out-of-tree build dir, @@ -77,14 +128,14 @@ else bad "full run failed (exit $?)"; sed 's/^/ /' "$out" fi -[ -f "$LFS/sources/demo-1.0/build/Makefile" ] \ +[ -f "$LFS/build/demo-1.0/build/Makefile" ] \ && ok "built out-of-tree in build/ subdir" || bad "build/ subdir missing" [ -f "$LFS/tools/bin/demo" ] \ && ok "package actually installed into \$LFS/tools/bin" \ || bad "install produced no file" [ -f "$LFS/tools/fixup.txt" ] \ && ok "configure (post-install) phase ran" || bad "configure phase did not run" -grep -q "^/.*build$" "$LFS/sources/.cc-build-demo" 2>/dev/null \ +grep -q "^/.*build$" "$LFS/build/.cc-build-demo" 2>/dev/null \ && ok "recorded build dir for resume" || bad "build dir not recorded" # ------------------------------------------------- single-phase re-run works @@ -110,7 +161,8 @@ grep -qi "installed to" "$T/f.log" \ # missing tarball must fail cleanly, not silently rm -rf "$T/lfs2"; mkdir -p "$T/lfs2/sources" -if LFS="$T/lfs2" BUILD_ROOT="$T/lfs2/sources" bash "$T/demo.sh" unpack \ +if LFS="$T/lfs2" SOURCES_DIR="$T/lfs2/sources" \ + BUILD_ROOT="$T/lfs2/build" bash "$T/demo.sh" unpack \ > "$T/m.log" 2>&1; then bad "missing tarball reported success" else @@ -119,7 +171,7 @@ fi echo echo " PASS: $PASS FAIL: $FAIL" -[ "$FAIL" -eq 0 ] +[ "$(_fail_total)" -eq 0 ] # ---- tarball name resolution against a real-world /sources listing --------- # # Regression guard: book titles are messy (Util-linux-2.41.1, XML::Parser-2.47, @@ -150,8 +202,8 @@ for title, want in cases: bad += 0 if hit else 1 sys.exit(1 if bad else 0) PY -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi -echo " PASS: $PASS FAIL: $FAIL (incl. tarball-name cases above)" +_count_rc $? +echo " PASS: $(_pass_total) FAIL: $(_fail_total) (incl. tarball-name cases above)" # ---- source vs documentation tarballs -------------------------------------- # # tcl ships tcl8.6.16-html.tar.gz alongside tcl8.6.16-src.tar.gz, and the docs @@ -245,7 +297,7 @@ if bad: sys.exit(1) print(" PASS commands inside note/tip boxes are ignored") PY2 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- a test's log reader must stay with the test --------------------------- # # GMP writes a log in the test block and reads it back in the next one: @@ -272,20 +324,24 @@ if problems: sys.exit(1) print(" PASS a test's log reader travels with the test suite") PY3 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- a stale source tree from an earlier chapter must be cleared ----------- # -# /sources is sticky (1777), so only the owner may delete a directory there. +# BUILD_ROOT is sticky (1777), so only the owner may delete a directory there. # Chapters 5-6 unpack as the `lfs` user; chapter 8 rebuilds the same packages as # their own package user, which then cannot remove the leftover tree -- the # script's `rm -rf` fails, tar extracts on top, and the build drowns in # tar: gcc-15.2.0/README: Cannot open: File exists # lfs-helper must clear it as root BEFORE handing over to the package user. -stl="$T/stale"; mkdir -p "$stl/sources" "$stl/src/demo-9.9" +# +# The tarball is READ from the sources directory and the stale tree is removed +# from the build scratch: two directories, and the cleanup must use the right +# one for each. +stl="$T/stale"; mkdir -p "$stl/sources" "$stl/build" "$stl/src/demo-9.9" echo fresh > "$stl/src/demo-9.9/NEW" ( cd "$stl/src" && tar czf "$stl/sources/demo-9.9.tar.gz" demo-9.9 ) -mkdir -p "$stl/sources/demo-9.9"; echo old > "$stl/sources/demo-9.9/OLD" -chmod 1777 "$stl/sources" +mkdir -p "$stl/build/demo-9.9"; echo old > "$stl/build/demo-9.9/OLD" +chmod 1777 "$stl/sources" "$stl/build" cat > "$stl/script.sh" <<'EOF' pkg_glob="demo-9.9.tar.* demo-9.9.tgz" EOF @@ -293,11 +349,18 @@ EOF helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then ( STATE="$stl"; SNAP_ROOT="$stl"; C_DIM=; C_OFF= + SOURCES_DIR="$stl/sources"; BUILD_ROOT="$stl/build" say(){ :; }; warn(){ :; } - eval "$(sed -n '/^clean_stale_source() {/,/^}/p' "$helper_src")" + # build_root_for decides where a step's scratch lives; a package builds + # in its own home, so the routine no longer reads $BUILD_ROOT directly. + # Stub it to the shared tree this fixture set up. + eval "$(_slice_fn "$helper_src" clean_stale_source)" + build_root_for() { printf '%s' "$BUILD_ROOT"; } clean_stale_source demo "$stl/script.sh" ) - if [ -d "$stl/sources/demo-9.9" ]; then + if [ -d "$stl/build/demo-9.9" ]; then bad "stale source tree was not cleared before the build" + elif [ ! -f "$stl/sources/demo-9.9.tar.gz" ]; then + bad "the cleanup removed the tarball instead of the unpacked tree" else ok "a stale source tree is cleared before the build" fi @@ -411,7 +474,7 @@ for src, want in cases: if bad: sys.exit(1) print(" PASS symlink steps are made re-runnable (-f added only where missing)") PY4 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- collector groups are configured, automatic, and reported -------------- # helper_src="$(dirname "$LFS_TOOL")/lfs-helper" @@ -436,7 +499,7 @@ if "collector_prefix" in keys and "lfs_mount" in keys: else: print(f" FAIL REQUIRED_CONFIG is missing keys: {keys}"); sys.exit(1) PY5 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- collector group name is a choice, not a fixed rule -------------------- # # Naming after the OWNER (nimgnu_binutils) groups everything that package owns; @@ -487,11 +550,12 @@ if "--no-same-owner" not in body: print(" FAIL unpack does not use tar --no-same-owner"); sys.exit(1) print(" PASS unpack never restores the archive's UIDs") PY6 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q 'rm -f "\$srcdir/.cc-build-\$name"' "$helper_src" \ + # markers live in the build scratch now, not in the tarball store + grep -q 'rm -f "\$blddir/.cc-build-\$name"' "$helper_src" \ && ok "stale build markers are cleared before a build" \ || bad "stale build markers are not cleared" grep -q "^cmd_clean_sources()" "$helper_src" \ @@ -561,7 +625,7 @@ print(" PASS 'locales' is a documented setting" if "locales" in keys else " FAIL no 'locales' setting") sys.exit(0 if "locales" in keys else 1) PY7 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- a package owns the files it OVERWRITES, not just new ones ------------- # # Chapter 8's bison replaces the binaries chapter 7's bison-tmp installed as @@ -598,7 +662,7 @@ for c, want in cases: if bad: sys.exit(1) print(" PASS suffixed test targets (test_harness, check-TESTS) are recognised") PY8 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- failures report what the paths actually are ---------------------------- # # "No such file or directory" on a destination can mean a missing parent OR a @@ -671,10 +735,10 @@ if [ -f "$helper_src" ]; then grep -q 'grep -aoE "/\[A-Za-z0-9._/+-\]{6,}"' "$helper_src" \ && ok "unquoted paths in error messages are recognised" \ || bad "only quoted paths are extracted; bash/perl messages missed" - # *-tmp packages must be adopted to the base package user - grep -q 'init-\*) skipped=' "$helper_src" \ - && ok "only init-* steps are skipped; -tmp tools get the base user" \ - || bad "-tmp steps are skipped, leaving root-owned trees behind" + # -tmp manifests must map to the BASE package user (perl-tmp -> perl) + grep -q "^_stage_base_name()" "$helper_src" \ + && ok "the -tmp and -passN manifests map to the base package user" \ + || bad "-tmp steps leave root-owned trees behind" fi # ---- package user names must be normalised --------------------------------- # @@ -739,18 +803,23 @@ if missing: print(" FAIL lfs is missing: %s" % ", ".join(missing)); sys.exit(1) print(" PASS 'lfs next' and the packagemanager handover exist") PY9 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- bootloader safety ----------------------------------------------------- # # The book runs `grub-install /dev/sda`, which writes a disk's boot sector. On # a machine with an existing boot partition that is destructive, so grub is only -# generated when explicitly chosen; the default is rEFInd, which only ever adds -# files to an already-mounted, already-formatted ESP. +# generated when explicitly chosen. +# +# The default is NONE -- it used to be rEFInd, on the reasoning that rEFInd only +# ever adds files to an already-mounted ESP. True, but it still stopped a +# finished build to ask about a bootloader on a machine that already had one. +# rEFInd remains the safe CHOICE; it is just no longer made for you. When it is +# chosen, the script must still be non-destructive. python3 - "$LFS_TOOL" <<'PY10' import sys, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -if lfs.bootloader() != "refind": - print(" FAIL default bootloader is %r, expected refind" % lfs.bootloader()) +if lfs.bootloader() != "none": + print(" FAIL default bootloader is %r, expected none" % lfs.bootloader()) sys.exit(1) script = lfs._refind_script() bad = [c for c in ("grub-install /dev/", "mkfs", "sgdisk", "parted", "of=/dev/") @@ -764,7 +833,7 @@ for guard in ("nothing is mounted there", "not FAT", print(" FAIL rEFInd script lacks the %r guard" % guard); sys.exit(1) print(" PASS rEFInd is the default and its script cannot touch a raw device") PY10 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then @@ -851,7 +920,11 @@ if [ -f "$helper_src" ]; then grep -q 'K "UID_MIN=\$PKG_UID_MIN"' "$helper_src" \ && ok "Shadow allocates package user ids (no hand-picked uid)" \ || bad "ids are still hand-picked and can collide" - grep -q 'useradd -c "package \$name" -d "\$SRCROOT/\$name" -U' "$helper_src" \ + # -U is the point: it creates the private group WITH the user so the ids + # match. The home now comes from pkgusr_home_for (packages and config + # steps live in different roots), so match on -U, not on the path. + grep -q 'useradd -c "package \$name" -d "\$(pkgusr_home_for "\$name")" -U' \ + "$helper_src" \ && ok "the private group is created with the user, so uid == gid" \ || bad "group and user ids can drift apart" grep -q 'getent passwd "\$1" >/dev/null 2>&1 && return 0' "$helper_src" \ @@ -883,7 +956,7 @@ for c, want in cases: if bad: sys.exit(1) print(" PASS test scaffolding travels with the tests, real steps do not") PY10 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- book placeholders must be filled in ----------------------------------- # # The book writes PAGE=<paper_size> ./configure for a human to edit. A shell @@ -901,7 +974,7 @@ if "your name here" not in left2: print(" FAIL an unknown placeholder was not reported"); sys.exit(1) print(" PASS book placeholders are substituted, unknown ones reported") PY11 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- packages must not re-mode things they do not own ---------------------- # # iproute2 does `install -m 0755 -d /usr/sbin` on a directory that already @@ -979,7 +1052,7 @@ if problems: sys.exit(1) print(" PASS rEFInd guidance survives set -e and offers a skip") PY12 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- an existing rEFInd gets an entry, not a reinstall ---------------------- # # If the machine already boots with rEFInd, reinstalling it is wrong; what is @@ -1000,7 +1073,7 @@ if problems: sys.exit(1) print(" PASS an existing rEFInd gets an LFS entry, with a backup") PY13 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then @@ -1016,14 +1089,14 @@ python3 - "$LFS_TOOL" <<'PY14' import sys, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() names = [n for n, _s in lfs.CHROOT_CONFIG_STEPS_9] -need = ["cfg-hostname", "cfg-hosts", "cfg-clock", "cfg-inputrc", "cfg-network"] +need = ["cfg_hostname", "cfg_hosts", "cfg_clock", "cfg_inputrc", "cfg_network"] missing = [n for n in need if n not in names] if missing: print(" FAIL chapter 9 is missing: %s" % ", ".join(missing)); sys.exit(1) # writing a config file installs nothing to compile: they must be root steps print(" PASS chapter 9 configuration steps are defined") PY14 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # rEFInd writes only to the ESP, so tracking zero files is expected, not a fault python3 - "$LFS_TOOL" <<'PY15' @@ -1035,7 +1108,7 @@ if not sc.rstrip().endswith("exit 4"): sys.exit(1) print(" PASS rEFInd reports success without tracking files in the tree") PY15 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- a "configuration" section that is really a package -------------------- # # 9.2 LFS-Bootscripts-20250827 is a tarball to unpack and install, not a config @@ -1050,7 +1123,7 @@ pv = lfs._title_pkgver("9.2. LFS-Bootscripts-20250827") if not pv or not any(c.isdigit() for c in pv): print(" FAIL a versioned config section is not recognised as a package") sys.exit(1) -glob = lfs._pkg_glob_for(pv, "cfg-bootscripts") +glob = lfs._pkg_glob_for(pv, "cfg_bootscripts") import fnmatch if not any(fnmatch.fnmatch("lfs-bootscripts-20250827.tar.xz", p) for p in glob.split()): print(" FAIL the bootscripts tarball would not be found"); sys.exit(1) @@ -1061,7 +1134,7 @@ for plain in ("9.5.3. Configuring the System Hostname", print(" FAIL %r was mistaken for a package" % plain); sys.exit(1) print(" PASS versioned config sections build as packages, the rest do not") PY16 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- config steps that rewrite existing files ------------------------------- # # /etc/hosts already exists (7.6 created it), so rewriting it adds no NEW file @@ -1079,7 +1152,7 @@ fi python3 - "$LFS_TOOL" <<'PY17' import sys, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -out, left = lfs._fill_placeholders("LC_ALL=<locale name> locale charmap", "cfg-locale") +out, left = lfs._fill_placeholders("LC_ALL=<locale name> locale charmap", "cfg_locale") if "<locale name>" in out or left: print(" FAIL <locale name> not substituted: %r" % out); sys.exit(1) out2, left2 = lfs._fill_placeholders("echo <charmap>", "x") @@ -1095,7 +1168,7 @@ if "needs a decision from you" in sc2: print(" FAIL a filled placeholder still triggered the guard"); sys.exit(1) print(" PASS locale placeholders fill from config; unfillable ones stop the step") PY17 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- 'next' must know where the build actually is --------------------------- # # The crosschain progress file is CLEARED when the chain completes, so an empty @@ -1109,7 +1182,7 @@ for fn in ("_crosschain_looks_done", "_chroot_progress", "_chroot_steporder"): print(" FAIL lfs is missing %s" % fn); sys.exit(1) print(" PASS build state is inferred from the tree, not just a progress file") PY18 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- collector groups can be carried to the next build ---------------------- # helper_src="$(dirname "$LFS_TOOL")/lfs-helper" @@ -1150,7 +1223,7 @@ if bad: sys.exit(1) print(" PASS placeholders are filled; includes and emails are left alone") PY19 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? python3 - "$LFS_TOOL" <<'PY20' import sys, importlib.machinery as m @@ -1161,7 +1234,7 @@ if "collector_import_file" not in keys or not hasattr(lfs, "_install_collector_g sys.exit(1) print(" PASS a collector-group export can be supplied via config") PY20 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- SNAP_ROOT="/" must not break file tracking ----------------------------- # # Inside the chroot SNAP_ROOT is "/", and "${SNAP_ROOT%/}" trims that to the @@ -1186,7 +1259,7 @@ if not hasattr(lfs, "network_mode") or lfs.network_mode() != "dhcp": print(" FAIL network does not default to dhcp"); sys.exit(1) print(" PASS network defaults to dhcp (no static config generated)") PY21 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the tooling's own Python dependencies ---------------------------------- # # base LFS ships neither requests nor beautifulsoup4, so 'lfs' and 'blfs' fail @@ -1203,7 +1276,7 @@ if "bs4" not in mods or "requests" not in mods: print(" FAIL the dependency list is incomplete: %s" % mods); sys.exit(1) print(" PASS the tooling's Python modules are checked before handover") PY22 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the tools must survive missing optional modules ------------------------ # # base LFS ships neither requests nor beautifulsoup4. Failing at IMPORT time @@ -1287,7 +1360,7 @@ if "<IP address" in open(p).read(): print(" FAIL a placeholder resolv.conf was treated as usable"); sys.exit(1) print(" PASS the chroot gets DNS; real configs kept, placeholders replaced") PY23 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- package users need the shared environment ------------------------------ # # The hint's scheme gives every package user symlinks into /etc/pkgusr: @@ -1323,7 +1396,7 @@ if "Run the configure phase too?" not in src: print(" FAIL reinstall does not ask about the configure phase"); sys.exit(1) print(" PASS lfs can install/update, and asks before re-running configure") PY24 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- BLFS is preferred when both books have a package ----------------------- # # BLFS' script carries the dependency information and the configure options a @@ -1346,7 +1419,7 @@ if "DNS" not in msg2: print(" FAIL a DNS failure is not explained"); sys.exit(1) print(" PASS BLFS is preferred, and network failures explain themselves") PY25 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- help is grouped, not one long list ------------------------------------- # bad_help="" @@ -1368,8 +1441,6 @@ lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() problems = [] if not hasattr(lfs, "_install_pkgusr_helpers"): problems.append("the hint's helper scripts are never copied into the chroot") -if "wget" not in lfs.LAST_STEP_DEFAULT: - problems.append("wget is not installed by the final build step") # scripts must say which book they came from body = lfs._crosschain_script_body("d", "sid", "D-1.0", "d.tar.*", ["make"]) if "### book" not in body: @@ -1382,7 +1453,7 @@ if problems: sys.exit(1) print(" PASS helpers copied, wget added from BLFS, scripts record their book") PY26 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then @@ -1394,40 +1465,101 @@ if [ -f "$helper_src" ]; then || bad "the book behind a script is not shown" fi -# ---- the user's own final build step ---------------------------------------- # -# A script on the HOST that runs last in the chroot: its SOURCES array is -# downloaded by get-sources, and the rest runs after every book package. It -# ships with wget because LFS has no download tool at all. +# ---- accounts you can log in with, as a built-in step ----------------------- # +# This was last_build_step.sh: a script created once on the HOST and then owned +# by the person, running last in the chroot. Wrong home twice over -- a fix to +# it reached nobody who already had a copy (that is how +# `chown: invalid user: 'wget:wget'` survived three releases), and setting the +# root password is not a matter of taste: book 8.5 says to do it. +# +# wget and the Python modules moved to `packagemanager setup`, which is where +# bootstrapping the tooling belongs. The root password could NOT move there: +# setup runs on the booted system, and you need to log in to run it. python3 - "$LFS_TOOL" <<'PY27' -import sys, os, tempfile, importlib.machinery as m +import sys, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() problems = [] -for fn in ("last_step_path", "ensure_last_step_script", "last_step_sources"): - if not hasattr(lfs, fn): - problems.append("lfs is missing %s" % fn) +# the old machinery must be gone, not merely unused +for gone in ("last_step_path", "ensure_last_step_script", "last_step_sources", + "LAST_STEP_DEFAULT", "_write_last_step_copy"): + if gone in src: + problems.append("%s is still here" % gone) +# and the built-in step takes its place, last in the order +if 'order.append("init-accounts")' not in src: + problems.append("init-accounts is not appended to the step order") if problems: - for p in problems: print(" FAIL %s" % p) + for p in problems: + print(" FAIL %s" % p) sys.exit(1) -# the SOURCES array must be PARSED, not executed -d = tempfile.mkdtemp() -p = os.path.join(d, "last.sh") -open(p, "w").write( - '#!/bin/bash\n' - 'SOURCES=(\n' - ' "https://example.org/a-1.0.tar.gz"\n' - ' # a comment\n' - ' "https://example.org/b-2.0.tar.xz"\n' - ')\n' - 'rm -rf / # would be catastrophic if this were executed\n') -old = lfs.load_config -lfs.load_config = lambda: {"last_step_script": p} -urls = lfs.last_step_sources() -lfs.load_config = old -if urls != ["https://example.org/a-1.0.tar.gz", "https://example.org/b-2.0.tar.xz"]: - print(" FAIL SOURCES not parsed correctly: %s" % urls); sys.exit(1) -print(" PASS the final step's SOURCES are parsed (never executed) to fetch") +print(" PASS the final step is built in, not a script that can go stale") PY27 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? + +# lfs-helper must actually perform it, and it must own no files +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" cmd_init_accounts | grep -q 'passwd root' \ + || _p="$_p;init-accounts does not set the root password" + # a non-interactive run must SAY what is missing, never block + _slice_fn "$helper_src" cmd_init_accounts | grep -q '\[ ! -t 0 \]' \ + || _p="$_p;init-accounts would hang on a scripted run" + case "$(_slice_fn "$helper_src" cmd_build)" in + *'"$name" = "init-accounts"'*) ;; + *) _p="$_p;lfs-helper would look for a book script for init-accounts" ;; + esac + case "$(_slice_fn "$helper_src" _is_not_a_package)" in + *init-accounts*) ;; + *) _p="$_p;init-accounts would be given a package user" ;; + esac + grep -q 'init-accounts)' "$helper_src" \ + || _p="$_p;init-accounts cannot be run by hand" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the root password is set inside the chroot, before you can reboot" + fi +fi + +# ---- what the built system needs before it can fetch anything --------------- # +# LFS ships no download tool, so a freshly booted system cannot get the sources +# for its own next package. The tarball has to be on disk before the chroot is +# sealed, and get-sources is the last moment there is still a network. +# +# Resolved from the BLFS BOOK, not a hardcoded URL: the book already knows where +# wget comes from and keeps it current, and a second list of links is a second +# thing to go stale. +python3 - "$LFS_TOOL" <<'PYBOOT' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() +problems = [] +for fn in ("bootstrap_packages", "bootstrap_sources"): + if not hasattr(lfs, fn): + problems.append("there is no %s" % fn) +if hasattr(lfs, "bootstrap_packages"): + lfs.load_config = lambda: {} + if lfs.bootstrap_packages() != ["wget"]: + problems.append("wget is not the default bootstrap package") + lfs.load_config = lambda: {"bootstrap_packages": ""} + if lfs.bootstrap_packages(): + problems.append("bootstrap packages cannot be turned off") + lfs.load_config = lambda: {"bootstrap_packages": "wget curl"} + if lfs.bootstrap_packages() != ["wget", "curl"]: + problems.append("more than one bootstrap package is not accepted") +if "blfs" not in src or '"sources"' not in src: + problems.append("the URLs are not resolved from the BLFS book") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the bootstrap tarball is resolved from the book, not hardcoded") +PYBOOT +_count_rc $? + # the dry run must print EVERY url, not a sample grep -q "print the WHOLE list" "$LFS_TOOL" \ @@ -1458,12 +1590,27 @@ if problems: sys.exit(1) print(" PASS build commands redirect to lfs-helper inside the chroot") PY28 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? -# the user's final step must be the LAST entry in the order, after the bootloader -grep -q 'order.append("last-step") # always last' "$LFS_TOOL" \ - && ok "the user's final step is ordered last" \ - || bad "the final step is not last in the build order" +# The accounts step must be the LAST entry in the order, after the bootloader. +# It sets the root password, so everything that could still fail should have +# failed by the time it runs -- and it is the last thing standing between a +# built system and a reboot you cannot log in after. +python3 - "$LFS_TOOL" <<'PYLAST' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r'^def cmd_bs_gen_chroot_scripts\(.*?(?=\n\ndef )', src, re.S | re.M) +body = m.group(0) if m else src +appends = re.findall(r'order\.append\("([^"]+)"\)', body) +if not appends: + print(" FAIL nothing is appended to the step order") + sys.exit(1) +if appends[-1] != "init-accounts": + print(" FAIL the last step in the order is %r, not init-accounts" % appends[-1]) + sys.exit(1) +print(" PASS the accounts step is ordered last") +PYLAST +_count_rc $? # ---- the wrappers are for package users, not for root ----------------------- # # A root step legitimately runs `chown -R wget:wget /sources/wget-1.25.0` to @@ -1478,15 +1625,24 @@ if [ -f "$helper_src" ]; then grep -q "envpass=\"\$envpass PATH='/usr/local/bin" "$helper_src" \ && ok "the root PATH has no wrapper directory" \ || bad "root's PATH still starts with the wrappers" - # /sources and /usr/src are shared build space, never a package's tree + # /usr/src is shared install space; /sources and /build are scratch. Both + # kinds must be beyond a package's reach, but by DIFFERENT mechanisms -- + # conflating them is what once made /root root:install drwxrwx---, and what + # made /sources root:install 775, dropping the o+w a build needs. ( SNAP_ROOT=/; INSTALLDIRS="" eval "$(sed -n '/^install_dirs_list() {/,/^}/p' "$helper_src")" eval "$(sed -n '/^install_dir_patterns() {/,/^}/p' "$helper_src")" eval "$(sed -n '/^matches_install_pattern() {/,/^}/p' "$helper_src")" eval "$(sed -n '/^is_install_dir() {/,/^}/p' "$helper_src")" - is_install_dir /sources && is_install_dir /usr/src ) \ - && ok "/sources and /usr/src are never given to a package" \ - || bad "/sources could be handed to a collector group" + eval "$(sed -n '/^never_adopt_list() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^is_never_adopt() {/,/^}/p' "$helper_src")" + is_install_dir /sources && is_install_dir /build \ + && is_install_dir /usr/src \ + && is_never_adopt /sources && is_never_adopt /build \ + && ! install_dirs_list | grep -qx /sources \ + && ! install_dirs_list | grep -qx /build ) \ + && ok "/sources, /build and /usr/src are never given to a package" \ + || bad "/sources or /build could be handed to a collector group" fi # ---- output that is easy to read and to copy -------------------------------- # @@ -1503,10 +1659,6 @@ import re for m2 in re.finditer(r'sys\.stderr\.write\([^)]*dry run[^)]*\)', src): problems.append("an uncoloured dry-run message remains: %s" % m2.group(0)[:50]) # the self-removing wget wrapper -if "check_certificate = off" not in lfs.LAST_STEP_DEFAULT: - problems.append("certificate checking is not disabled for the first fetch") -if "lfs-temporary-no-verify" not in lfs.LAST_STEP_DEFAULT: - problems.append("no marker to find and undo it later") # the host's skeleton is preferred over any default we invent if not hasattr(lfs, "_install_pkgusr_skel"): problems.append("the host's skel-package is not copied") @@ -1515,24 +1667,39 @@ if problems: sys.exit(1) print(" PASS dry runs are coloured, hints are copyable, wget wrapper self-removes") PY29 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q "^cmd_fix_users()" "$helper_src" \ - && ok "existing package users' homes and VERSION files can be repaired" \ - || bad "no way to repair existing package users" + grep -q "^cmd_verify()" "$helper_src" \ + && ok "the tree can be reconciled against what it should be" \ + || bad "no way to repair a tree that has drifted" fi # ---- /usr/local/bin must precede /usr/bin ----------------------------------- # # The temporary no-verify wget wrapper lives in /usr/local/bin. With that # missing from the package user's PATH the wrapper was never found, and every # HTTPS download failed with "cannot verify ... certificate". +# Assert the ORDER, not the literal line. This used to grep for +# PATH=/usr/lib/pkgusr:/usr/local/bin +# which is the wrapper path spelled out a third time -- so unifying the two +# wrapper directories broke a test that was really about /usr/local/bin. bad_path="" for f in lfs-helper packagemanager; do t="$(dirname "$LFS_TOOL")/$f" [ -f "$t" ] || continue - grep -q "PATH=/usr/lib/pkgusr:/usr/local/bin" "$t" || bad_path="$bad_path $f" + _pl="$(grep -m1 '^PATH=.*:/usr/bin' "$t")" + [ -n "$_pl" ] || { bad_path="$bad_path $f(no-PATH)"; continue; } + # the wrappers first, then /usr/local/bin, then /usr/bin + case "$_pl" in + PATH=*:/usr/local/bin:*) ;; + *) bad_path="$bad_path $f(local-bin-not-after-wrappers)"; continue ;; + esac + _pre="${_pl%%:/usr/bin*}" + case "$_pre" in + *:/usr/local/bin*) ;; + *) bad_path="$bad_path $f(local-bin-after-usr-bin)" ;; + esac done [ -z "$bad_path" ] \ && ok "/usr/local/bin precedes /usr/bin for package users" \ @@ -1552,7 +1719,7 @@ if not hasattr(lfs, "cmd_bs_sync_tools"): sys.exit(1) print(" PASS the package-user environment can be synced into the tree") PY30 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the certificate workaround must undo itself ---------------------------- # # A $PATH wrapper was never seen: builds run through `su -` with the package @@ -1579,7 +1746,7 @@ if not hasattr(lfs, "_sync_report"): print(" FAIL sync output has no single consistent shape"); sys.exit(1) print(" PASS tool versions are stamped and sync output is consistent") PY31 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the certificate workaround must apply itself, not wait to be run ------- # # A system with no CA certificates cannot fetch make-ca -- the very thing that @@ -1690,7 +1857,7 @@ if dt > 1.0: print(" PASS a large log is scanned in %.0f ms and the error is still found" % (dt * 1000)) PY32 - if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi + _count_rc $? fi for f in lfs-helper packagemanager_install; do t="$(dirname "$LFS_TOOL")/$f" @@ -1756,7 +1923,7 @@ if problems: sys.exit(1) print(" PASS no function recurses before doing any work") PY32 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? pmt="$(dirname "$LFS_TOOL")/packagemanager" if [ -f "$pmt" ]; then @@ -1849,7 +2016,7 @@ for want, why in ( print(" FAIL %s" % why); sys.exit(1) print(" PASS one command resumes the build and ends in the chroot") PY33 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- run must WALK you through, not print homework -------------------------- # # `lfs books` only lists books: running it can never complete the "choose a @@ -1872,7 +2039,7 @@ if problems: sys.exit(1) print(" PASS run asks for decisions instead of printing them") PY34 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- nested command blocks, and progress while working ---------------------- # # extract_commands_phased walked only the DIRECT children of the installation @@ -1937,11 +2104,11 @@ if problems: sys.exit(1) print(" PASS /etc/fstab is generated correctly for the configured disk") PY35 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? -grep -q 'root_steps.append("cfg-fstab")' "$LFS_TOOL" \ +grep -q 'root_steps.append("cfg_fstab")' "$LFS_TOOL" \ && ok "writing /etc/fstab runs as root" \ - || bad "cfg-fstab would run as a package user and fail" + || bad "cfg_fstab would run as a package user and fail" # ---- a book that does not exist must be refused ----------------------------- # # set-default warned and then saved anyway, so a typo ("ls") became the @@ -1970,7 +2137,7 @@ if cfg.get("default") == "ls": print(" FAIL the bad book was saved anyway"); sys.exit(1) print(" PASS a book that does not exist is refused, not saved") PY36 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- reset clears settings, never the built system -------------------------- # python3 - "$LFS_TOOL" <<'PY37' @@ -1998,7 +2165,7 @@ if problems: sys.exit(1) print(" PASS reset warns, confirms, and spares the built system") PY37 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the build book and the install book are separate ----------------------- # # You may build the system from one book and then install packages on it from a @@ -2036,7 +2203,7 @@ if problems: sys.exit(1) print(" PASS building and installing can use different books") PY38 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- every command must run to completion ----------------------------------- # # `lfs config` crashed with NameError on a variable removed during an edit -- @@ -2094,7 +2261,7 @@ if problems: sys.exit(1) print(" PASS a fresh tree is given to the lfs user, not mistaken for a built one") PY39 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- sync-tools must hand over what it copied ------------------------------- # # layout chowns the tree to lfs, then sync-tools copies the package-user @@ -2112,7 +2279,7 @@ if "_chown_tree_to_lfs" not in body: sys.exit(1) print(" PASS sync-tools hands over what it copied in") PY40 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- every step announces itself -------------------------------------------- # # Steps can run for many minutes with no output; without a consistent header @@ -2136,7 +2303,7 @@ for need, why in (("NOW:", "does not say what is running"), print(" FAIL the banner %s" % why); sys.exit(1) print(" PASS every step shows what is running, and what is next") PY41 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the package-user system is not optional -------------------------------- # # build-all stopped at the first chapter-8 package and asked the user to run @@ -2178,7 +2345,7 @@ problems = [] def tree(done=None, tools_gcc=True, temp_system=False): t = tempfile.mkdtemp() - os.makedirs(os.path.join(t, ".lfs-pkgusr")) + os.makedirs(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress")) os.makedirs(os.path.join(t, "tools", "bin")) os.makedirs(os.path.join(t, "usr", "bin")) if tools_gcc: @@ -2188,7 +2355,7 @@ def tree(done=None, tools_gcc=True, temp_system=False): open(os.path.join(t, "usr", "bin", b), "w").close() if done is not None: json.dump({"book": "12.4", "done": done}, - open(os.path.join(t, ".lfs-pkgusr", + open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "crosschain-progress.json"), "w")) return t @@ -2218,7 +2385,7 @@ if problems: sys.exit(1) print(" PASS an unfinished toolchain is detected and blocks the chroot") PY42 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- snapshots: save the build, put it back --------------------------------- # # LFS is a long sequence of irreversible steps; without this a bad package @@ -2257,7 +2424,7 @@ if problems: sys.exit(1) print(" PASS snapshots preserve ownership and restore safely") PY43 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- already-built packages are adopted before anything new ----------------- # # Chapters 5-7 run before package users exist, so their files are root-owned @@ -2266,14 +2433,19 @@ if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi # chapter-8 package -- while bash, coreutils and glibc still owned nothing. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *"cmd_adopt_existing --run"*) + # Adoption now lives in establish_ownership, which build-all reaches + # through _ownership_checkpoint. Assert that it still happens and still + # happens early -- not where the code used to sit. + ba="$(_slice_fn "$helper_src" cmd_build_all)" + eo="$(_slice_fn "$helper_src" establish_ownership)" + case "$eo" in + *"_vfy_manifest_ownership 1"*) ok "already-built packages get their users before new builds" ;; *) bad "chapter 5-7 packages are never given their package users" ;; esac - # adoption must come BEFORE the build loop - a="$(printf '%s' "$ba" | grep -n "cmd_adopt_existing --run" | head -1 | cut -d: -f1)" + # the checkpoint must be reachable BEFORE the build loop, so a tree that is + # already past 7.6 adopts before it builds anything new + a="$(printf '%s' "$ba" | grep -n "_ownership_checkpoint" | head -1 | cut -d: -f1)" l="$(printf '%s' "$ba" | grep -n "while read -r s" | head -1 | cut -d: -f1)" if [ -n "$a" ] && [ -n "$l" ] && [ "$a" -lt "$l" ]; then ok "adoption happens before the first new package is built" @@ -2316,15 +2488,13 @@ if [ -f "$helper_src" ]; then && ok "gcc's header subtree is protected by the contents check" \ || bad "nothing stops another package adopting gcc's headers" - # and the check must run before the chown, not after - ad="$(sed -n '/^cmd_adopt_dirs() {/,/^}/p' "$helper_src")" - c="$(printf '%s' "$ad" | grep -n "_dir_contents_foreign" | head -1 | cut -d: -f1)" - w="$(printf '%s' "$ad" | grep -n "chown " | head -1 | cut -d: -f1)" - if [ -n "$c" ] && [ -n "$w" ] && [ "$c" -lt "$w" ]; then - ok "the check runs before anything is chowned" - else - bad "directories are chowned before being checked" - fi + # verify must never take a path another package legitimately owns + mo="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" + case "$mo" in + *'user_exists "$cur" && continue'*) + ok "a path owned by another real package is left alone" ;; + *) bad "verify could take another package's files" ;; + esac fi # ---- "nearly finished" is not finished -------------------------------------- # @@ -2368,7 +2538,7 @@ if problems: sys.exit(1) print(" PASS a toolchain missing only its last step is not called finished") PY44 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- a build may not claim files it did not install ------------------------- # # "touched" is every file with an mtime newer than the build stamp, which is @@ -2471,33 +2641,6 @@ if [ -f "$helper_src" ]; then ok "missing, unreachable and not-searched are told apart" fi -# ---- repairing a target-triplet mismatch ------------------------------------ # -# The chroot compiler reported x86_64-pc-linux-gnu while chapter 6 installed -# its headers under x86_64-lfs-linux-gnu, so g++ never looked where -# c++config.h actually was. Same gcc version on the same machine, so the two -# trees are interchangeable and a compatibility link unblocks the build. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^cmd_fix_toolchain()" "$helper_src" \ - && ok "a triplet mismatch can be repaired" \ - || bad "nothing repairs a triplet mismatch" - fb="$(sed -n '/^cmd_fix_toolchain() {/,/^}/p' "$helper_src")" - case "$fb" in - *'[ "$run" != 1 ]'*) ok "the repair is dry-run by default" ;; - *) bad "the repair changes things without being asked" ;; - esac - case "$fb" in - *'already exists -- not touching it'*) - ok "it refuses rather than clobbering an existing directory" ;; - *) bad "the repair could overwrite a real header directory" ;; - esac - case "$fb" in - *'[ "$have" = "$mine" ]'*) - ok "it does nothing when the triplets already agree" ;; - *) bad "the repair would link a triplet onto itself" ;; - esac -fi - # ---- tracing a file back to the step that installed it ---------------------- # # "which step went wrong" should be answerable from the record, not guessed. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" @@ -2589,7 +2732,7 @@ if problems: sys.exit(1) print(" PASS test scaffolding is kept out of the build phase") PY45 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the groups import is asked for with the other session settings --------- # # It has to be known BEFORE the chroot scripts are generated (that is when the @@ -2605,7 +2748,7 @@ if "no such file: %s -- leaving it unset" not in src: print(" FAIL a nonexistent export path would be accepted"); sys.exit(1) print(" PASS the groups import is part of the session setup, and validated") PY46 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the compiler check must not block chapter 7 ---------------------------- # # Requiring a working C++ compiler before ANY step blocked chapter 7 -- which @@ -2698,7 +2841,7 @@ if 'sub.add_parser("list"' not in src: print(" FAIL `lfs list` does not exist (lfs-helper has `list`)"); sys.exit(1) print(" PASS `lfs list` matches `lfs-helper list`") PY47 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? if [ -f "$helper_src" ]; then grep -q 'if \[ "\${1:-}" = "--all" \]; then shift; cmd_list' "$helper_src" \ && ok "\`lfs-helper next --all\` matches \`lfs next --all\`" \ @@ -2720,40 +2863,6 @@ case "$out" in *) ok "no colour when the output is not a terminal" ;; esac -# ---- adoption must not repeat itself ---------------------------------------- # -# A whole-manifest signature was useless as a "have we done this" marker: every -# build writes a manifest, so it changed constantly and the full pass -- tens -# of thousands of paths -- ran again on every single build-all. Adoption is -# now tracked per package, by manifest size, so a package is re-adopted only -# when ITS OWN manifest grows. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_needs_adoption()" "$helper_src" \ - && ok "adoption is decided per package, not for the whole set" \ - || bad "adoption still runs wholesale" - grep -q "^_mark_adopted()" "$helper_src" \ - && ok "each adopted package is recorded" \ - || bad "nothing records which packages are already adopted" - grep -q "_adoption_is_current" "$helper_src" \ - && bad "the whole-manifest signature is still in use" \ - || ok "the useless whole-manifest signature is gone" - na="$(sed -n '/^_needs_adoption() {/,/^}/p' "$helper_src")" - case "$na" in - *'[ -s "$man" ] || return 1'*) ok "empty manifests never look unadopted" ;; - *) bad "an empty manifest would force adoption forever" ;; - esac - case "$na" in - *'_adopted_size "$name"'*) ok "a package that grew is adopted again" ;; - *) bad "new files in an adopted package would be missed" ;; - esac - ae="$(sed -n '/^cmd_adopt_existing() {/,/^}/p' "$helper_src")" - case "$ae" in - *'= "$owner" ] && continue'*) - ok "paths already owned correctly are skipped" ;; - *) bad "every path is re-chowned even when already correct" ;; - esac -fi - # ---- directories must be recorded, not just files --------------------------- # # _run_tracked computed the directory diff as `after_d - before_d if before_d # else set()`, and took the before-snapshot AFTER the build. On a first run @@ -2764,13 +2873,13 @@ python3 - "$LFS_TOOL" <<'PY48' import sys, os, tempfile, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() root = tempfile.mkdtemp() -os.makedirs(os.path.join(root, ".lfs-pkgusr", "manifests")) -os.makedirs(os.path.join(root, "usr")) +os.makedirs(os.path.join(root, "usr", "src", "lfs-pkgusr", "manifests")) +os.makedirs(os.path.join(root, "usr"), exist_ok=True) script = ("mkdir -p %s/usr/include/c++/15.2.0/experimental\n" "echo hdr > %s/usr/include/c++/15.2.0/experimental/algorithm\n" % (root, root)) lfs._run_tracked("gcc-pass2", script, as_lfs=False, lfs=root) -dirs = os.path.join(root, ".lfs-pkgusr", "manifests", "gcc-pass2.dirs") +dirs = os.path.join(root, "usr", "src", "lfs-pkgusr", "manifests", "gcc-pass2.dirs") problems = [] if not os.path.isfile(dirs) or not open(dirs).read().strip(): problems.append("no directories recorded on a first run") @@ -2791,7 +2900,7 @@ if problems: sys.exit(1) print(" PASS a package records the directories it creates, first run included") PY48 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- the collector-group export must not duplicate entries ------------------ # helper_src="$(dirname "$LFS_TOOL")/lfs-helper" @@ -2868,14 +2977,14 @@ if problems: sys.exit(1) print(" PASS building and using are separated, and the settings are asked for") PY51 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- manifests report directories, not just files --------------------------- # # A package owns the directories it creates as well as the files in them. # Reporting only files hid the tracking bug that left the C++ header tree # root-owned while the headers inside belonged to gcc. -mfd="$T/manifests-report/.lfs-pkgusr/manifests" +mfd="$T/manifests-report/usr/src/lfs-pkgusr/manifests" mkdir -p "$mfd" printf '/usr/bin/a\n/usr/bin/b\n' > "$mfd/good-pkg.files" printf '/usr/lib/thing\n' > "$mfd/good-pkg.dirs" @@ -2953,101 +3062,6 @@ if [ -f "$helper_src" ]; then fi fi -# ---- an interrupted toolchain package is finished, not just reported -------- # -# Cancelling gcc's `make install` leaves the new compiler in /usr/bin without -# its headers, so it shadows the working one and nothing builds. The tool -# knows which package was interrupted, so the right move is to finish that -# install -- printing a diagnosis and stopping leaves the user to work it out. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *'cmd_build "$_pkg" --phase install --force'*) - ok "an interrupted install is finished automatically" ;; - *) bad "a half-installed toolchain package is only reported" ;; - esac - case "$ba" in - *"the compiler works again -- continuing"*) - ok "the build continues once the compiler works" ;; - *) bad "the build stops even after repairing the compiler" ;; - esac - case "$ba" in - *"Finishing the interrupted install did not repair it"*) - ok "it says so when finishing the install was not enough" ;; - *) bad "a failed repair is indistinguishable from no repair" ;; - esac - # the repair must be attempted BEFORE giving up - i_fix="$(printf '%s' "$ba" | grep -n 'phase install --force' | head -1 | cut -d: -f1)" - i_stop="$(printf '%s' "$ba" | grep -n 'Stopping before' | head -1 | cut -d: -f1)" - if [ -n "$i_fix" ] && [ -n "$i_stop" ] && [ "$i_fix" -lt "$i_stop" ]; then - ok "the repair is tried before stopping" - else - bad "it stops before trying to repair" - fi - # and no unverified signal trickery - grep -q "_on_interrupt" "$helper_src" \ - && bad "an unverified interrupt handler is still present" \ - || ok "interrupts are recorded plainly, not intercepted" -fi - -# ---- a partial install is detected without a marker ------------------------- # -# The interrupted-marker only exists if the interruption happened while a -# version of the tool that writes one was running. A package that has already -# installed files but is not marked built is in exactly the same state -- which -# is what a half-installed gcc looks like -- so it must be detected from the -# tree, not only from the marker. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *'[ -s "$MANIFESTS/$s.files" ]'*'! is_done "$s"'*) - ok "a partial install is detected from the tree, not just a marker" ;; - *) bad "an interruption from an older run would go undetected" ;; - esac - case "$ba" in - *"has already installed files but is not marked"*) - ok "it says why it thinks the package is half-installed" ;; - *) bad "the repair happens without explaining itself" ;; - esac -fi - -# ---- the build ends by sealing the install directories ---------------------- # -# Until the build is finished the install dirs must be group-writable but NOT -# sticky, so package users can replace the temporary system's root-owned files. -# Once everything is built that has to be undone, or any package can delete -# another's files. It was implemented but never offered, so a finished build -# left the system permanently unsealed. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - # it used to only SUGGEST this; a finished build now does it - case "$ba" in - *"cmd_seal_install_dirs --run"*) - ok "a finished build seals the install directories" ;; - *) bad "a finished build leaves the system unsealed" ;; - esac - grep -q "^_install_dirs_are_sealed()" "$helper_src" \ - && ok "it only says so while the dirs are still unsealed" \ - || bad "the reminder would repeat after sealing" - # sealing must touch install dirs only -- collector-group dirs are shared - # deliberately and a sticky bit there would defeat the point - sd="$(sed -n '/^cmd_seal_install_dirs() {/,/^}/p' "$helper_src")" - case "$sd" in - *"done < <(install_dirs_list)"*) - ok "sealing walks the install directories only" ;; - *) bad "sealing could reach collector-group directories" ;; - esac - case "$sd" in - *COLLECTOR*) bad "sealing touches collector-group directories" ;; - *) ok "collector-group directories are left shared" ;; - esac - # and o+t is the right bit: create yes, delete-another's no - case "$sd" in - *'chmod o+t'*) ok "the sticky bit is what gets set" ;; - *) bad "sealing sets something other than the sticky bit" ;; - esac -fi - # ---- build-all must not lose steps or leave the system unsealed ------------- # # Three faults in one run: a step that exits 3 ("needs a decision") had its # code flattened to 1 and then discarded, so refind slipped past and the build @@ -3073,7 +3087,7 @@ if [ -f "$helper_src" ]; then *) bad "the build could claim to be complete with steps pending" ;; esac case "$ba" in - *"cmd_seal_install_dirs --run"*) + *"_vfy_seal_install_dirs"*) ok "sealing is done as the final step, not suggested" ;; *) bad "sealing is still left to the user" ;; esac @@ -3128,7 +3142,7 @@ if problems: sys.exit(1) print(" PASS entering the chroot refreshes the tools first") PY52 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- every tool reports its version ----------------------------------------- # # Without one there is no way to tell whether the copy inside the chroot is the @@ -3170,106 +3184,6 @@ after="$(bash "$tmpcopy/lfs-helper" --version 2>&1)" && ok "the build id changes when the file changes" \ || bad "the build id does not track the file contents" -# ---- sealing covers every install-group directory --------------------------- # -# The static list does not know about install dirs packages create themselves -# (/usr/lib/<pkg>, /usr/share/<pkg>), and those need the sticky bit too. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - sd="$(sed -n '/^cmd_seal_install_dirs() {/,/^}/p' "$helper_src")" - case "$sd" in - *"-type d -group install"*) - ok "sealing finds install-group directories, not just a fixed list" ;; - *) ok_or_bad=1; bad "install dirs outside the static list stay unsealed" ;; - esac - case "$sd" in - *'[ -k "$d" ] && continue'*) - ok "an already-sticky directory is skipped" ;; - *) bad "sealing re-chmods directories needlessly" ;; - esac - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *'[ "$s" = "last-step" ] && ! _install_dirs_are_sealed'*) - ok "sealing happens immediately before the final step" ;; - *) bad "sealing is not tied to the step before last-step" ;; - esac -fi - -# ---- adoption is for chapter 5-6 packages ONLY ------------------------------ # -# Adoption exists for one thing: packages built outside the chroot, before any -# package user existed, whose files are still root-owned. Anything in this -# chroot's step list is built by lfs-helper, which assigns ownership as part of -# the build -- walking its manifest is pure waste, and it made the pass appear -# on every single build-all with no explanation of why. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - na="$(sed -n '/^_needs_adoption() {/,/^}/p' "$helper_src")" - case "$na" in - *'steps | grep -qxF "$name"'*) - ok "packages built in the chroot are never adopted" ;; - *) bad "chroot-built packages are still walked by the adoption pass" ;; - esac - # the cheap check must come BEFORE statting thousands of files - i_step="$(printf '%s' "$na" | grep -n 'steps | grep -qxF' | head -1 | cut -d: -f1)" - i_stat="$(printf '%s' "$na" | grep -n '_manifest_size' | head -1 | cut -d: -f1)" - if [ -n "$i_step" ] && [ -n "$i_stat" ] && [ "$i_step" -lt "$i_stat" ]; then - ok "the cheap check runs before reading manifests" - else - bad "manifests are read before the step-list check" - fi - ae="$(sed -n '/^cmd_adopt_existing() {/,/^}/p' "$helper_src")" - case "$ae" in - *"CHAPTER 5-6"*) - ok "the pass says which packages it is for" ;; - *) bad "the pass does not explain what it is adopting" ;; - esac -fi - -# ---- files from outside the chroot need an owner that exists in it ---------- # -# The lfs user exists on the HOST, not inside, so everything it created -- -# /sources, the tree handed over at 7.2 -- shows as a bare uid and every -# ownership check calls it UNKNOWN. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^LFS_BUILD_UID=9998" "$helper_src" \ - && ok "the build user has a fixed id, clear of the package users" \ - || bad "no reserved id for the outside build user" - grep -q "^cmd_fix_orphans()" "$helper_src" \ - && ok "files with no owner can be repaired" \ - || bad "orphaned files stay owned by nobody" - fo="$(sed -n '/^cmd_fix_orphans() {/,/^}/p' "$helper_src")" - case "$fo" in - *"-nouser"*) ok "it finds files by missing passwd entry, not by guessing" ;; - *) bad "orphan detection does not use -nouser" ;; - esac - # the virtual filesystems are the host's -- never touch them - for d in "dev" "proc" "sys" "run"; do - case "$fo" in - *"-not -path \"\$r/$d/*\""*) ;; - *) bad "orphan repair could reach /$d" ;; - esac - done - ok "the virtual filesystems are excluded from orphan repair" - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *"cmd_fix_orphans --run"*) - ok "the build recreates the outside build user by itself" ;; - *) bad "the user has to know to run fix-orphans" ;; - esac - case "$ba" in - *"cmd_sort_users --run"*) - ok "passwd and group end up sorted by id" ;; - *) bad "passwd stays in build order" ;; - esac - # sorting must be the LAST thing, once every package user exists - i_sort="$(printf '%s' "$ba" | grep -n 'cmd_sort_users --run' | head -1 | cut -d: -f1)" - i_seal="$(printf '%s' "$ba" | grep -n 'cmd_seal_install_dirs --run' | tail -1 | cut -d: -f1)" - if [ -n "$i_sort" ] && [ -n "$i_seal" ] && [ "$i_sort" -gt "$i_seal" ]; then - ok "sorting happens after every user has been created" - else - bad "passwd is sorted before all the users exist" - fi -fi - # ---- what 4.3 hands over, 7.2 must hand back -------------------------------- # # The two lists were written separately and drifted: 4.3 included `lib` and 7.2 # did not, so /lib stayed owned by the lfs user for the rest of the build while @@ -3299,7 +3213,7 @@ if problems: sys.exit(1) print(" PASS the tree is handed over and handed back symmetrically") PY53 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- scratch and human directories are never owned by a package ------------- # # man-pages ended up owning /tmp, which would break every other user of it. @@ -3310,22 +3224,19 @@ if [ -f "$helper_src" ]; then eval "$(sed -n '/^install_dir_patterns() {/,/^}/p' "$helper_src")" eval "$(sed -n '/^matches_install_pattern() {/,/^}/p' "$helper_src")" eval "$(sed -n '/^is_install_dir() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^never_adopt_list() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^is_never_adopt() {/,/^}/p' "$helper_src")" for d in /tmp /var/tmp /root /home; do is_install_dir "$d" || exit 1; done exit 0 ) \ && ok "/tmp, /var/tmp, /root and /home are never adopted" \ || bad "a package could come to own /tmp or a home directory" # orphan repair must fix everything it found, not a sample of it - fo="$(sed -n '/^cmd_fix_orphans() {/,/^}/p' "$helper_src")" + fo="$(sed -n '/^_vfy_orphans() {/,/^}/p' "$helper_src")" case "$fo" in *"head -n 2000"*) bad "orphan repair still only fixes the first 2000" ;; *) ok "orphan repair fixes every file it found" ;; esac - case "$fo" in - *'gave $fixed of $n path'*) - ok "it reports how many it actually changed" ;; - *) bad "orphan repair claims success without counting" ;; - esac fi # ---- "not found in book" when there IS no book ------------------------------ # @@ -3369,7 +3280,7 @@ if "BLFS book" not in src: print(" FAIL install-tools does not report the BLFS book"); sys.exit(1) print(" PASS the BLFS book is installed into the chroot with the tools") PY54 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # ---- a cached book must actually be usable ---------------------------------- # # A book imported by hand, or copied in by install-tools, has no metadata @@ -3437,7 +3348,7 @@ print(" FAIL " + problems[0] if problems else " PASS install-tools writes a config packagemanager can read") sys.exit(1 if problems else 0) PY55 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? pmt="$(dirname "$LFS_TOOL")/packagemanager" if [ -f "$pmt" ]; then @@ -3535,65 +3446,29 @@ if [ -f "$helper_src" ]; then fi # ---- build steps are not packages ------------------------------------------- # -# last-step, init-*, cfg-* and refind are actions, not software. A `last-step` -# package user was created because a manifest existed under that name, and +# init-*, cfg_* and refind are actions, not software. A `last-step` package +# user was once created because a manifest existed under that name, and # everything the step touched -- including wget's binary, installed as the wget -# user -- was then handed to it. +# user -- was then handed to it. last-step is gone, but the rule that made it +# harmful is not: any step name could still become a package user. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then grep -q "^_is_not_a_package()" "$helper_src" \ && ok "steps that are not packages are recognised" \ || bad "any step name could become a package user" - au="$(sed -n '/^cmd_add_user() {/,/^}/p' "$helper_src")" + au="$(_slice_fn "$helper_src" cmd_add_user)" case "$au" in *_is_not_a_package*) ok "add-user refuses to create one for a step" ;; - *) bad "add-user would still create a last-step user" ;; + *) bad "add-user would still create a user for a build step" ;; esac - na="$(sed -n '/^_needs_adoption() {/,/^}/p' "$helper_src")" + na="$(_slice_fn "$helper_src" _needs_adoption)" case "$na" in *_is_not_a_package*) ok "adoption skips step names too" ;; *) bad "adoption could recreate the user add-user refused" ;; esac fi -# the last-step script installs each package as its OWN user, from wheels -python3 - "$LFS_TOOL" <<'PY56' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -t = lfs.LAST_STEP_DEFAULT -sources = t.split("SOURCES=(")[1].split(")")[0] -problems = [] -if "install_python_module" not in t: - problems.append("the python modules the tools need are not installed") -if "build_as_package_user wget" not in t: - problems.append("wget is not built as the wget package user") -# A modern sdist needs its build backend, and with no index reachable pip -# cannot fetch one: "BackendUnavailable: Cannot import 'hatchling.build'". -# A wheel is already built, so pip just unpacks it. -for line in sources.splitlines(): - line = line.strip() - if "files.pythonhosted.org" not in line: - continue - if not line.rstrip('"').endswith(".whl"): - problems.append("a python module is fetched as an sdist: %s" - % line.split("/")[-1][:40]) -for mod in ("requests", "beautifulsoup4"): - if mod not in sources: - problems.append("%s is not in SOURCES" % mod) -if "--no-build-isolation" in t: - problems.append("--no-build-isolation is still used") -# pip is told --no-deps, so every runtime dependency must be listed itself -for dep in ("urllib3", "idna", "certifi", "soupsieve"): - if dep not in t: - problems.append("%s (a runtime dependency) is never installed" % dep) -if t.index("install_python_module urllib3") > t.index("install_python_module requests"): - problems.append("requests is installed before its dependencies") -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS the last step installs each package as its own user, from wheels") -PY56 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi + # a hint that installs a package must pull its dependencies too python3 - "$LFS_TOOL" <<'PY58' @@ -3608,36 +3483,8 @@ if bad: sys.exit(1) print(" PASS install hints pull dependencies with --recursive") PY58 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi - -# ---- site-packages is python's, not shared infrastructure ------------------- # -# A module user cannot write into site-packages until it joins the collector -# group that owns it. The fix must be the SAME mechanism every other package -# uses -- join the existing collector group, or ask which group should own the -# directory. Handing site-packages to the `install` group would mean "anyone -# may install here", which is exactly what the collector groups exist to avoid. -python3 - "$LFS_TOOL" <<'PY59' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -t = lfs.LAST_STEP_DEFAULT -problems = [] -if "_grant_site_packages" not in t: - problems.append("nothing gives the module user access to site-packages") -if "lfs-helper grant-dir" not in t: - problems.append("it does not use the normal grant mechanism") -# the install group must never be applied to python's tree -if "grp=install" in t or "chgrp install" in t: - problems.append("site-packages would be handed to the install group") -# and it must happen before pip, not after it fails -if "_grant_site_packages" in t and \ - t.index('_grant_site_packages "$user"') > t.index("python3 -m pip install"): - problems.append("access is granted after pip has already failed") -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS site-packages access uses the normal collector-group rules") -PY59 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? + helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then @@ -3682,7 +3529,7 @@ if bad: sys.exit(1) print(" PASS every helper is defined before it is used") PY60 - if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi + _count_rc $? # ---- one implementation of the collector-group rules --------------------- # # A package installed after boot must behave exactly like one built during @@ -3708,44 +3555,6 @@ PY60 fi fi -# ---- uids should read as the build order ------------------------------------ # -# The uid a package gets is the order its user is created in. Adoption walked -# the manifests alphabetically, so gcc (built 2nd of 22) ended up with a HIGHER -# uid than man-pages (the first chapter-8 package) -- which makes the uids -# useless for deciding what to rebuild first. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_manifests_in_build_order()" "$helper_src" \ - && ok "manifests can be walked in build order" \ - || bad "adoption still walks manifests alphabetically" - ae="$(sed -n '/^cmd_adopt_existing() {/,/^}/p' "$helper_src")" - case "$ae" in - *'for man in $(_manifests_in_build_order)'*) - ok "adoption creates users in build order" ;; - *) bad "adoption does not use the build order" ;; - esac - # whole seconds tie for packages built in the same second - bo="$(sed -n '/^_manifests_in_build_order() {/,/^}/p' "$helper_src")" - case "$bo" in - *"stat -c %.Y"*) ok "sub-second timestamps keep the order exact" ;; - *) bad "packages built in the same second would tie" ;; - esac - - # behaviour: oldest manifest first, regardless of name - bod="$T/buildorder"; mkdir -p "$bod" - for p in zzz-first aaa-second mmm-third; do - echo "/usr/bin/$p" > "$bod/$p.files" - sleep 0.02 - done - got="$(MANIFESTS="$bod"; eval "$bo"; _manifests_in_build_order \ - | xargs -n1 basename | sed 's/.files//' | tr '\n' ' ')" - case "$got" in - "zzz-first aaa-second mmm-third "*) - ok "the order follows when they were written, not their names" ;; - *) bad "build order came out as: $got" ;; - esac -fi - # ---- editing the wrong install script fails silently ------------------------ # # A package user's home holds two files with nearly identical names: # install_<name>-<version> the source -- the one to edit @@ -3766,13 +3575,19 @@ if [ -f "$pmt" ]; then import sys, os, time, tempfile, importlib.machinery as m pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() base = tempfile.mkdtemp() -os.makedirs(os.path.join(base, "p")) -src = os.path.join(base, "p", "install_p-1.0") -rt = os.path.join(base, "p", "install_p") +pm.BASE_DIR = base +# the directory on disk is the USER's, so build the fixture through the same +# helper the code uses -- the package "p" becomes the account p_p, and +# hard-coding "p" here made this test fail the moment prefixes were turned on +# resolve the home the same way the code does -- the account is p_p AND it +# lives under the package root, so neither part can be hard-coded here +_h = pm.pkgusr_home("p") +os.makedirs(_h) +src = os.path.join(_h, "install_p-1.0") +rt = os.path.join(_h, "install_p") open(src, "w").write("real") time.sleep(0.02) open(rt, "w").write("edited") -pm.BASE_DIR = base import io, contextlib buf = io.StringIO() with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): @@ -3789,7 +3604,7 @@ else: print(" FAIL warned=%s quiet=%s" % (warned, quiet)) sys.exit(1) PY61 - if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi + _count_rc $? fi # ---- installing make-ca does not create any certificates -------------------- # @@ -3863,51 +3678,9 @@ if warned and quiet: else: print(" FAIL warned=%s quiet=%s" % (warned, quiet)); sys.exit(1) PY62 - if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi -fi - - -# ---- passwd and group stay sorted, whoever creates the user ----------------- # -# Both halves of the toolchain create package users, and both append. Sorting -# only at the end of build-all left anything installed afterwards -- by -# packagemanager, from BLFS -- appended out of order again. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_sort_users_quietly()" "$helper_src" \ - && ok "lfs-helper sorts after creating a user" \ - || bad "users created by lfs-helper are appended unsorted" -fi -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - au="$(sed -n '/^addUser() {/,/^}/p' "$pmi")" - case "$au" in - *"lfs-helper sort-users --run"*) - ok "packagemanager sorts after creating a user too" ;; - *) bad "users created by packagemanager are appended unsorted" ;; - esac - # it must not fail when lfs-helper is absent - case "$au" in - *"command -v lfs-helper"*) ok "it degrades quietly without lfs-helper" ;; - *) bad "user creation would fail where lfs-helper is missing" ;; - esac + _count_rc $? fi -# behaviour: package users before collector groups, system entries first -srt="$T/sortcheck"; mkdir -p "$srt/etc" "$srt/usr/src" "$srt/.lfs-pkgusr" -printf 'root:x:0:\ninstall:x:9999:\nnimgnu_x:x:90005:aaa\nzzz:x:10099:\naaa:x:10001:\n' \ - > "$srt/etc/group" -printf 'root:x:0:0:root:/root:/bin/bash\n' > "$srt/etc/passwd" -if [ -f "$helper_src" ]; then - LFS_ETC="$srt/etc" LFS_SNAP_ROOT="$srt" LFS_SRC_ROOT="$srt/usr/src" \ - LFS_PKGUSR_DIR="$srt/.lfs-pkgusr" \ - bash "$helper_src" sort-users --run >/dev/null 2>&1 - order="$(cut -d: -f1 "$srt/etc/group" | tr '\n' ' ')" - case "$order" in - "root install aaa zzz nimgnu_x "*) - ok "system entries, then package users, then collector groups" ;; - *) bad "sorted order came out as: $order" ;; - esac -fi # ---- prompts must allow line editing ---------------------------------------- # # Without readline an arrow key inserts an escape sequence instead of moving @@ -3932,7 +3705,7 @@ if "try:" not in window or "ImportError" not in window: sys.exit(1) print(" PASS readline is optional, not required") PY64 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? for f in lfs-helper packagemanager_install; do p="$(dirname "$LFS_TOOL")/$f" @@ -3986,7 +3759,263 @@ if problems: sys.exit(1) print(" PASS restart warns, confirms, keeps the sources and spares the host") PY65 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? + +# ---- sources and build scratch are two directories -------------------------- # +# They used to be one, and every problem in this area came from that. A build +# unpacked into the tarball store, dropped .cc-* markers there, and left both +# owned by whoever built -- so a rebuild had to pick debris out of the thing it +# most needed to keep. Split, and the whole class of problem is structural: +# /sources holds only what get-sources downloaded, and BUILD_ROOT is deleted +# whole. +python3 - "$LFS_TOOL" <<'PY65B' +import sys, os, re, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +body = open(sys.argv[1]).read() + +if 'BUILD_ROOT="${BUILD_ROOT:-$LFS/sources}"' in body: + problems.append("BUILD_ROOT still defaults to the tarball store") +for need in ('SOURCES_DIR="${SOURCES_DIR:-$LFS/sources}"', + 'BUILD_ROOT="${BUILD_ROOT:-$LFS/build}"'): + if need not in body: + problems.append("generated scripts do not set %s" % need.split("=")[0]) + +# generate a real script and read what it does +cmds = ["mkdir -v build\ncd build", "../configure", "make", "make install"] +s = lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-1.0", + "demo-1.0.tar.*", cmds) +# the markers must live in scratch, never in the tarball store +for marker in (".cc-build-demo", ".cc-dir-demo"): + for line in s.splitlines(): + if marker in line and "SOURCES_DIR" in line: + problems.append("%s is written to the tarball store" % marker) +if '$BUILD_ROOT/.cc-build-demo' not in s: + problems.append("the build marker is not written to BUILD_ROOT") +# unpack must extract into scratch +if 'cd "$BUILD_ROOT"' not in s: + problems.append("unpack does not work in BUILD_ROOT") +# and the symlink farm must be built before the tarball is looked for +u = s[s.index("unpack_pkg()"):s.index("#### UNPACK DONE ####")] +if "_link_sources" not in u: + problems.append("unpack does not build the sources symlink farm") + +# the chroot copy must map BOTH directories, not just one +c = lfs._chrootify(s) +if 'SOURCES_DIR="${SOURCES_DIR:-/sources}"' not in c: + problems.append("the chroot script does not point at /sources") +if 'BUILD_ROOT="${BUILD_ROOT:-/build}"' not in c: + problems.append("the chroot script does not point at /build") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS tarball store and build scratch are separate directories") +PY65B +_count_rc $? + +# ---- a book command may reach a sibling tarball through .. ------------------ # +# The book does this, in Tcl: +# +# cd .. +# tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 +# +# `cd ..` leaves the build subdirectory, so ".." is then the package root's +# parent -- which used to be the sources directory and is now BUILD_ROOT. +# Splitting the two would break every command of this shape. +# +# The fix must not be a list of packages known to do it: that would be a second +# thing to keep in sync with the book, and wrong the first time a new release +# adds another. unpack_pkg symlinks EVERY downloaded file into BUILD_ROOT, so +# "../<anything-we-downloaded>" resolves for any package in any future book. +# This test builds a package that does exactly what Tcl does. +_sbx="$T/split" +mkdir -p "$_sbx/lfs/sources" "$_sbx/lfs/out" "$_sbx/mk/demo-2.0" "$_sbx/mk/doc" +cat > "$_sbx/mk/demo-2.0/configure" <<'EOF' +#!/bin/bash +cat > Makefile <<'MK' +all: + @echo compiling +install: + @true +MK +EOF +chmod +x "$_sbx/mk/demo-2.0/configure" +( cd "$_sbx/mk" && tar czf "$_sbx/lfs/sources/demo-2.0.tar.gz" demo-2.0 ) +echo html > "$_sbx/mk/doc/index.html" +( cd "$_sbx/mk" && tar czf "$_sbx/lfs/sources/demo-2.0-html.tar.gz" doc ) +echo patch > "$_sbx/lfs/sources/demo-2.0-fix-1.patch" + +python3 - "$LFS_TOOL" "$_sbx/demo.sh" <<'PYSPLIT' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +cmds = [ + "mkdir -v build\ncd build", + "../configure", + "make", + "make install", + # the book's Tcl shape, verbatim in structure + "cd ..\ntar -xf ../demo-2.0-html.tar.gz -C ../../out\n" + "cp -v ../demo-2.0-fix-1.patch ../../out/patch.seen", +] +open(sys.argv[2], "w").write( + lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-2.0", + "demo-2.0.tar.*", cmds)) +PYSPLIT + +# The suite exports SOURCES_DIR/BUILD_ROOT for other tests; clear them here so +# this exercises the DEFAULTS a real build would get. +( cd "$_sbx" && env -u SOURCES_DIR -u BUILD_ROOT LFS="$_sbx/lfs" \ + bash demo.sh all ) >"$_sbx/log" 2>&1 +_sp="" +[ -d "$_sbx/lfs/out/doc" ] || _sp="$_sp;a sibling tarball reached through .. was not found" +[ -f "$_sbx/lfs/out/patch.seen" ] || _sp="$_sp;a patch reached through .. was not found" +# and the whole point: a full build must not have touched the tarball store +_left="$(ls -A "$_sbx/lfs/sources" | grep -v '^demo-2\.0' | tr '\n' ' ' || true)" +[ -z "$_left" ] || _sp="$_sp;a build wrote into the tarball store: $_left" +[ -d "$_sbx/lfs/build/demo-2.0" ] || _sp="$_sp;the package was not unpacked into BUILD_ROOT" +[ -f "$_sbx/lfs/build/.cc-dir-demo" ] || _sp="$_sp;build markers did not land in BUILD_ROOT" +[ -L "$_sbx/lfs/build/demo-2.0-html.tar.gz" ] || _sp="$_sp;the farm entry is not a symlink" +if [ -n "$_sp" ]; then + printf '%s\n' "${_sp#;}" | tr ';' '\n' | while IFS= read -r _m; do + [ -n "$_m" ] && bad "$_m" + done + sed 's/^/ /' "$_sbx/log" | tail -8 +else + ok "'../<tarball>' resolves for any package, and /sources stays clean" +fi + +# ---- /sources has ONE owner convention, and it is the book's ---------------- # +# Three rules had drifted apart, and the result on disk was a mixture: +# +# drwxrwxrwt 1 9998 install /mnt/lfs/sources/ +# -rw-r--r-- 1 lfs 9998 acl-2.3.2.tar.xz +# +# * lfs's _chown_tree_to_lfs added "sources" to the 4.3 handover set, so it +# chowned /sources to the HOST's lfs user. Book 4.3 does no such thing. +# * lfs-helper carried /sources in install_dirs_list AS WELL AS in +# never_adopt_list -- the exact conflation the comment above +# never_adopt_list warns about, the one that made /root root:install. +# verify --fix then made it root:install 775 and the seal made it 1775, +# dropping the o+w that lets the unprivileged build user unpack at all. +# * cmd_clean_sources reset unknown owners to root:root. +# +# The host lfs account is created by book 4.3's own useradd, which pins no +# uid -- it was 10753 on a Debian host. Package users start at 10000, so that +# is not merely "an unnamed UID" as the book puts it: it COLLIDES with a real +# package user, and tarballs report as owned by some unrelated package. +# +# Book 3.1 settles it and the reasoning is given there in full: +# chmod -v a+wt $LFS/sources -> root:root 1777 +# chown root:root $LFS/sources/* -> the contents too +# World-writable is what grants the build user access, so no group is involved. +python3 - "$LFS_TOOL" <<'PY65C' +import sys, os, stat, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] + +if not hasattr(lfs, "_normalize_sources"): + print(" FAIL no single rule for /sources ownership") + sys.exit(1) + +# 4.3 hands over the book's directories and NOT /sources +if "sources" in lfs.LFS_HANDOVER_DIRS: + problems.append("/sources is in the 4.3 handover set") +body = open(sys.argv[1]).read() +h = body[body.index("def _chown_tree_to_lfs"):body.index("def _ensure_lfs_home")] +if 'LFS_HANDOVER_DIRS + ("sources",)' in h: + problems.append("the 4.3 chown still adds /sources to the handover") + +d = tempfile.mkdtemp() +src = os.path.join(d, "sources"); os.makedirs(src) +tar = os.path.join(src, "bash-5.3.tar.gz"); open(tar, "w").close() +tree = os.path.join(src, "acl-2.3.2"); os.makedirs(tree) +os.chmod(src, 0o775) # what verify --fix used to leave + +try: + os.chown(tar, 10753, 10753) # a host uid inside the package range + os.chown(tree, 10753, 10753) +except OSError: + print(" SKIP cannot chown here (needs root)") +else: + lfs._normalize_sources(d, run=True, quiet=True) + st = os.stat(src) + if stat.S_IMODE(st.st_mode) != (stat.S_ISVTX | 0o777): + problems.append("/sources is not 1777 -- the build user cannot unpack " + "(mode %o)" % stat.S_IMODE(st.st_mode)) + if st.st_uid != 0 or st.st_gid != 0: + problems.append("/sources itself is not root:root") + if os.lstat(tar).st_uid != 0: + problems.append("a downloaded file keeps a host uid (book 3.1)") + # an unpacked tree belongs to whoever is building it; taking it away + # mid-build would stop that build. Book 3.1 runs before any unpack. + if os.lstat(tree).st_uid != 10753: + problems.append("an unpacked tree was chowned out from under its build") + # running it twice must change nothing + if lfs._normalize_sources(d, run=True, quiet=True) != (0, False): + problems.append("the rule is not idempotent") + +# and it must actually be applied where sources appear or survive +g = body[body.index("def cmd_bs_get_sources"):body.index("def _title_pkgver")] +if "_normalize_sources" not in g: + problems.append("get-sources does not apply book 3.1's chown") +r = body[body.index("def cmd_bs_restart"):body.index("def cmd_bs_set_book")] +if "_normalize_sources" not in r: + problems.append("restart does not apply book 3.1's chown") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS /sources is root:root 1777 with root-owned contents (book 3.1)") +PY65C +_count_rc $? + +# ---- /sources is scratch, not a shared install directory -------------------- # +# Being in install_dirs_list meant verify --fix set it root:install 775 and +# seal-install-dirs set 1775 -- both drop the o+w of book 3.1's `a+wt`, and +# o+w is the only thing that lets the unprivileged build user unpack there. +# It belongs in never_adopt_list (no package may own it) and nowhere else. +if [ -r "$(dirname "$LFS_TOOL")/lfs-helper" ]; then + HELPER="$(dirname "$LFS_TOOL")/lfs-helper" +else + HELPER="" +fi +if [ -n "$HELPER" ]; then + _p="" + # the DEFAULT list, i.e. the heredoc, must not carry it + if sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -qx '/sources'; then + _p="$_p;/sources is still a shared install directory" + fi + # but it must still be a directory no package may adopt + if ! sed -n '/^never_adopt_list()/,/^}/p' "$HELPER" | grep -qx '/sources'; then + _p="$_p;/sources may now be adopted by a package" + fi + # an installdirs.lst from an older run must not bring it back + if ! sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -q 'grep -vx.*/sources'; then + _p="$_p;a stale installdirs.lst can still reintroduce /sources" + fi + # the build scratch is the same kind of thing and needs the same rules + if ! sed -n '/^never_adopt_list()/,/^}/p' "$HELPER" | grep -qx '/build'; then + _p="$_p;/build may be adopted by a package" + fi + if sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -qx '/build'; then + _p="$_p;/build is a shared install directory" + fi + # every tree scan must skip it -- otherwise each snapshot, ownership pass + # and collector-group map walks every unpacked source tree + if [ "$(grep -c 'not -path "\${\?r}\?/build/\*"' "$HELPER")" -lt 3 ]; then + _p="$_p;a tree scan still descends into the build scratch" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "/sources and /build are scratch, not shared install directories" + fi +else + bad "lfs-helper not found beside $LFS_TOOL" +fi # ---- snapshots must not read live kernel filesystems ------------------------ # # Excluding "./sys/*" skips the CONTENTS but still archives the ./sys directory @@ -4016,44 +4045,7 @@ if problems: sys.exit(1) print(" PASS snapshots skip live filesystems and verify what they wrote") PY66 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi - -# ---- a cancelled step must be resumed, not skipped -------------------------- # -# The orphan repair was gated on "does the lfs user exist" -- but the user is -# created FIRST. Cancelling part-way left the user in place, so every later -# run skipped the repair entirely with half a million files still ownerless. -# A gate must test the WORK, never a side effect of having started it. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_has_orphaned_files()" "$helper_src" \ - && ok "the orphan repair is gated on the files, not on the user" \ - || bad "a cancelled orphan repair would be skipped forever" - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *"if ! user_exists lfs; then"*) - bad "build-all still gates the repair on the user existing" ;; - *) ok "build-all gates it on there being work to do" ;; - esac - # -print -quit: stop at the first hit, so it costs nothing when clean - ho="$(sed -n '/^_has_orphaned_files() {/,/^}/p' "$helper_src")" - case "$ho" in - *"-print -quit"*) ok "the check stops at the first orphan" ;; - *) bad "the check walks the whole tree every time" ;; - esac - # long loops must show progress, or they get cancelled for looking hung - fo="$(sed -n '/^cmd_fix_orphans() {/,/^}/p' "$helper_src")" - case "$fo" in - *"fixed % 5000"*) ok "a long repair shows progress" ;; - *) bad "half a million chowns run with no output" ;; - esac - # and a silent skip hides real failures - ip="$(sed -n '/^cmd_init_pkgusr() {/,/^}/p' "$helper_src")" - case "$ip" in - *"could not set the group on"*) - ok "directories that could not be regrouped are named" ;; - *) bad "a failed chgrp is silently counted as nothing to do" ;; - esac -fi +_count_rc $? # ---- an application user must be usable, not just exist --------------------- # # `user create` made the account and stopped there: you could not become it, @@ -4110,31 +4102,40 @@ fi # A freshly built system has root with NO password. Depending on the login # manager that is either "anyone can log in as root" or "nobody can log in at # all" -- discovered after a reboot, with no way in. -python3 - "$LFS_TOOL" <<'PY67' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -t = lfs.LAST_STEP_DEFAULT -problems = [] -if "set_up_login_accounts" not in t: - problems.append("nothing asks for a root password") -body = t[t.index("set_up_login_accounts() {"):] if \ - "set_up_login_accounts() {" in t else "" -if "passwd root" not in body: - problems.append("the root password is never set") -# a scripted run has no terminal: it must say what is missing, not hang -if "[ ! -t 0 ]" not in body: - problems.append("a non-interactive run would block on a prompt") -if "chroot /mnt/lfs /usr/bin/passwd root" not in body: - problems.append("it does not say how to set the password later") -# the login account name is already configured -- reuse it, do not re-ask -if "main_user=" not in body: - problems.append("it ignores the configured login account") -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS the build ends by setting up accounts you can log in with") -PY67 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +# +# It lives in lfs-helper now, as the built-in init-accounts step. It could not +# move to `packagemanager setup` with the rest of the old final step: setup runs +# on the BOOTED system, and you need to log in to run it. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _ia="$(_slice_fn "$helper_src" cmd_init_accounts)" + _p="" + case "$_ia" in + *'passwd root'*) ;; + *) _p="$_p;the root password is never set" ;; + esac + case "$_ia" in + *'NO password yet'*) ;; + *) _p="$_p;nothing explains why a root password matters" ;; + esac + # a scripted run has no terminal: it must say what is missing, not hang + case "$_ia" in + *'! -t 0'*) ;; + *) _p="$_p;a non-interactive run would block on a prompt" ;; + esac + case "$_ia" in + *'useradd -m'*) ;; + *) _p="$_p;no login account is ever created" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build ends with a system you can log into" + fi +fi + # password detection must read /etc/shadow correctly shd="$T/shadowcheck"; mkdir -p "$shd" @@ -4223,7 +4224,7 @@ if "_owner_name_of" not in src: print(" FAIL the refusal does not name the actual owner"); sys.exit(1) print(" PASS 7.2 hands the tree to root whoever owned it") PY68 -if [ $? -eq 0 ]; then PASS=$((PASS+1)); else FAIL=$((FAIL+1)); fi +_count_rc $? # behaviour: a tree owned by an unknown uid must still be handed over hod="$T/handover"; mkdir -p "$hod/usr/bin" "$hod/etc" @@ -4237,6 +4238,4227 @@ else ok "(cannot test unknown-uid handover as this user)" fi +# ---- a shared user reaches the session -------------------------------------- # +# A "shared user" is one that belongs to the group on the main user's XDG +# runtime directory: +# drwxrws--- n76310 u_xdg_runtime /tmp/xdg-n76310 +# The display socket, the session bus and audio live there, so without that +# group a graphical program starts and then cannot draw anything. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _join_xdg_runtime_group" "$pmt" \ + && ok "a shared user joins the session group" \ + || bad "a shared user cannot reach the display or audio" + jx="$(sed -n '/^def _join_xdg_runtime_group/,/^def _setup_application_user/p' "$pmt")" + # the group must be READ from the directory, never assumed by name + case "$jx" in + *"getgrgid(os.stat(rt).st_gid)"*) + ok "the group is read from the runtime directory" ;; + *) bad "the session group name is hardcoded" ;; + esac + case "$jx" in + *'u_xdg_runtime"'*) bad "a specific group name is hardcoded" ;; + *) ok "no group name is baked in" ;; + esac + # a personal group is not a shared group + case "$jx" in + *"which is not a shared "*) + ok "it refuses to add the user to a personal group" ;; + *) bad "an application user could join the person's own group" ;; + esac + # both /run/user/<uid> and /tmp/xdg-<user> layouts + xd="$(sed -n '/^def _xdg_runtime_dir_of/,/^def _join_xdg/p' "$pmt")" + for p in "/run/user" "/tmp/xdg-"; do + case "$xd" in + *"$p"*) ;; + *) bad "the runtime directory search misses $p" ;; + esac + done + ok "both runtime-directory layouts are searched" + # --shared and --share-dir are different things + grep -q '"--share-dir"' "$pmt" \ + && ok "the session group and a shared directory are separate options" \ + || bad "--shared conflates the session group with a directory" +fi + +# ---- the setup must be applicable after the fact ---------------------------- # +# An account made earlier -- or by hand -- should be able to reach the same +# state without deleting and recreating it. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def cmd_user_setup" "$pmt" \ + && ok "an existing user can be set up after the fact" \ + || bad "the setup only happens at creation time" + grep -q '"setup"' "$pmt" \ + && ok "`user setup` is registered" \ + || bad "the setup command is not reachable" + # it must refuse on a user that does not exist, rather than half-creating + cs="$(sed -n '/^def cmd_user_setup/,/^def _write_desktop_entry/p' "$pmt")" + case "$cs" in + *"no such user"*) ok "setup refuses on a user that does not exist" ;; + *) bad "setup would act on a missing user" ;; + esac + # the same code path as create, or the two drift + case "$cs" in + *_setup_application_user*) ok "create and setup share one code path" ;; + *) bad "setup reimplements what create does" ;; + esac + + # menu entry + grep -q "^def _write_desktop_entry" "$pmt" \ + && ok "a menu entry is written" \ + || bad "no desktop entry is created" + de="$(sed -n '/^def _write_desktop_entry/,/^def _write_launcher/p' "$pmt")" + for field in "Desktop Entry" "Exec=" "StartupWMClass=" "Icon="; do + case "$de" in + *"$field"*) ;; + *) bad "the desktop entry has no $field" ;; + esac + done + ok "the desktop entry has the fields a menu needs" + # a launcher for a program that is not installed is a broken command + sa="$(sed -n '/# 6. a launcher and a menu entry/,/_write_desktop_entry(main, u, app)/p' "$pmt")" + case "$sa" in + *"no launcher or menu entry"*) + ok "nothing is written when the program is not installed" ;; + *) bad "a launcher could point at a program that does not exist" ;; + esac + # and neither file is clobbered if it is already there + case "$de" in + *"already exists -- left alone"*) ok "an existing menu entry is kept" ;; + *) bad "the desktop entry would be overwritten" ;; + esac +fi + +# ---- changing LFS_TGT invalidates the toolchain ----------------------------- # +# gcc-pass2 installs its C++ headers under $LFS_TGT. Change LFS_TGT afterwards +# and the compiler looks under the new name while the headers sit under the +# old one -- present, readable, and invisible. It surfaces forty packages +# later as "bits/c++config.h: No such file or directory", which reads like a +# missing file rather than a mismatch. +python3 - "$LFS_TOOL" <<'PY69' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() +problems = [] +if not hasattr(lfs, "_crosschain_tgt_mismatch"): + problems.append("nothing detects a changed target triplet") +# the triplet must be RECORDED with the build, or there is nothing to compare +mk = src[src.index("def _mark_crosschain_done"):] +mk = mk[:mk.index("def lfs_tgt")] +if '"lfs_tgt"' not in mk: + problems.append("the target is not recorded with the build progress") +# and checked before the chroot, not after +if "this tree was built for a different target" not in src: + problems.append("a changed target is not reported") +i_check = src.find("this tree was built for a different target") +i_prep = src.find('if action == "prepare":') +if i_check < i_prep: + problems.append("the check is not inside chroot prepare") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a changed target triplet is caught before the chroot") +PY69 +_count_rc $? + +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_warn_on_triplet_mismatch()" "$helper_src" \ + && ok "the mismatch is also reported inside the chroot" \ + || bad "inside the chroot the mismatch is only a missing-file error" + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" + case "$ba" in + *_warn_on_triplet_mismatch*) + ok "it is checked before building, not after failing" ;; + *) bad "the mismatch check never runs" ;; + esac +fi + +# ---- three causes, one error message ---------------------------------------- # +# "bits/c++config.h: No such file or directory" is produced by three different +# situations needing three different fixes. The compiler's own configure line +# tells them apart: +# no --host and no --target -> a NATIVE gcc, which only chapter 8 builds: +# its install is half-finished +# both are cross names -> LFS_TGT changed since chapter 6 +# no headers at all -> chapter 6 never finished +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_explain_triplet_mismatch()" "$helper_src" \ + && ok "the cause of a mismatch is named, not just the symptom" \ + || bad "all three causes give the same message" + ex="$(sed -n '/^_explain_triplet_mismatch() {/,/^}/p' "$helper_src")" + case "$ex" in + *"it is a NATIVE build"*) + ok "a native compiler is recognised as chapter 8's" ;; + *) bad "a half-installed chapter-8 gcc is not identified" ;; + esac + case "$ex" in + *"LFS_TGT changed between"*) + ok "two cross names are recognised as a changed LFS_TGT" ;; + *) bad "a changed LFS_TGT is not identified" ;; + esac + # the two cases must give DIFFERENT fixes -- that is the whole point + # the two causes need different fixes -- that is the whole point + case "$ex" in + *"lfs config lfs_tgt"*"--phase all --force"*|*"--phase all --force"*"lfs config lfs_tgt"*) + ok "each cause gets the fix that applies to it" ;; + *) bad "the same fix is offered for different causes" ;; + esac + # and it must say nothing when the names agree + case "$ex" in + *'[ "$have" = "$mine" ] && return 0'*) + ok "it is silent when there is no mismatch" ;; + *) bad "it would explain a mismatch that does not exist" ;; + esac +fi + +# ---- installs are staged, not written straight into the system -------------- # +# `make install` writes into the live system one file at a time, in Makefile +# order. Interrupt it and the system is neither the old version nor the new +# one -- and for gcc that is unrecoverable: it installs its driver early and +# its C++ headers late, so a cancel in between leaves a compiler that cannot +# compile C++, which is what you need to rebuild it. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_merge_stage()" "$helper_src" \ + && ok "installs go to a staging tree first" \ + || bad "installs still write straight into the live system" + grep -q "DESTDIR='\$_stage'" "$helper_src" \ + && ok "DESTDIR points the install at the staging tree" \ + || bad "the staging tree is created but never used" + + # a FAILED install must leave the system untouched -- the whole point + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *'if [ "${rc:-0}" = 0 ]; then'*'_merge_stage'*) + ok "the staged tree is moved in only on success" ;; + *) bad "a failed install could still be merged" ;; + esac + case "$cb" in + *"the system was not touched"*) + ok "a failed install says the system is unchanged" ;; + *) bad "a failed install does not say what state things are in" ;; + esac + + # a package that ignores DESTDIR writes to the system anyway: say so + ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" + case "$ms" in + *"return 2"*) ok "an empty staging tree is detected" ;; + *) bad "a package ignoring DESTDIR would look like a clean install" ;; + esac + case "$cb" in + *"ignores DESTDIR"*) ok "and reported, not silently trusted" ;; + *) bad "ignoring DESTDIR is not reported" ;; + esac + # merging must target the tracked root, not a hardcoded / + # the installer, not the merge wrapper, is what writes + it2="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" + case "$it2" in + *'root="${SNAP_ROOT:-/}"'*) ok "the merge targets the tracked root" ;; + *) bad "the merge writes to a hardcoded /" ;; + esac + + # the workarounds this replaces must be gone + grep -q "cmd_fix_toolchain" "$helper_src" \ + && bad "the symlink workaround is still present" \ + || ok "the triplet symlink workaround is gone" + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" + case "$ba" in + *"Finishing its install before going on"*) + bad "the auto-finish workaround is still present" ;; + *) ok "the auto-finish workaround is gone" ;; + esac +fi + +# ---- a shared user must know where the session is --------------------------- # +# Being in the session group is necessary but not sufficient: XDG_RUNTIME_DIR, +# WAYLAND_DISPLAY and the package-user PATH all come from +# /etc/pkgusr/skel-u_xdg/.bash_profile. Without it the account can reach the +# socket and still not find it. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _link_xdg_profile" "$pmt" \ + && ok "a shared user gets the XDG profile" \ + || bad "a shared user has no XDG_RUNTIME_DIR" + lp="$(sed -n '/^def _link_xdg_profile/,/^def _setup_application_user/p' "$pmt")" + # a symlink, so editing the file once reaches every shared account + case "$lp" in + *"os.symlink(XDG_SKEL, target)"*) ok "it is linked, not copied" ;; + *) bad "each account gets its own copy, which then drift" ;; + esac + # the runtime directory belongs to whoever is logged in: fill it in, do + # not ship someone else's path + case "$lp" in + *"@XDG_RUNTIME_DIR@"*) ok "the runtime directory is filled in, not shipped" ;; + *) bad "a hardcoded runtime directory would be installed" ;; + esac + case "$lp" in + *"kept the old one as"*) ok "an existing profile is kept, not destroyed" ;; + *) bad "the user's own .bash_profile would be overwritten" ;; + esac + # and it must be wired into the shared path + sa="$(sed -n '/# 4. the session/,/# 5. a shared directory/p' "$pmt")" + case "$sa" in + *_link_xdg_profile*) ok "--shared links the profile as well as the group" ;; + *) bad "the profile is never linked" ;; + esac +fi + +# the file must ship, and the Makefile must install it without clobbering +skel="$(dirname "$LFS_TOOL")/skel-u_xdg/.bash_profile" +[ -f "$skel" ] \ + && ok "the XDG profile ships with the tools" \ + || bad "the XDG profile is not shipped" +mkf="$(dirname "$LFS_TOOL")/Makefile" +if [ -f "$mkf" ]; then + grep -q "skel-u_xdg" "$mkf" \ + && ok "make install installs it" \ + || bad "make install does not install the XDG profile" + grep -q "kept your existing .bash_profile" "$mkf" \ + && ok "reinstalling does not overwrite your edits" \ + || bad "a reinstall would clobber the XDG profile" +fi + +# ---- a snapshot must say what state it captured ----------------------------- # +# Restoring a snapshot named "chrosscompile_done" brought back a tree whose +# toolchain was ALREADY broken -- the snapshot had been taken after chapter 8 +# damaged it. Nothing said so, at save or at restore, so it looked like the +# restore had failed rather than the snapshot being bad. +python3 - "$LFS_TOOL" <<'PY70' +import sys, os, json, tempfile, importlib.machinery as m +store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store +os.makedirs(os.path.join(store, "books"), exist_ok=True) +json.dump({"default": "12.4", "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, + open(os.path.join(store, "config.json"), "w")) +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +if not hasattr(lfs, "_toolchain_state"): + print(" FAIL a snapshot does not record the toolchain state"); sys.exit(1) + +def tree(headers=None, native=False): + t = tempfile.mkdtemp() + if headers: + p = os.path.join(t, "usr", "include", "c++", "15.2.0", headers, "bits") + os.makedirs(p) + open(os.path.join(p, "c++config.h"), "w").close() + if native: + os.makedirs(os.path.join(t, "usr", "lib", "gcc", "x86_64-pc-linux-gnu")) + return t + +problems = [] +if not lfs._toolchain_state(tree("x86_64-nimgnu-linux-gnu")).startswith("ok"): + problems.append("a clean chapter-6 tree is not recognised") +if not lfs._toolchain_state(tree("x86_64-nimgnu-linux-gnu", native=True)) \ + .startswith("BROKEN"): + problems.append("a native gcc over the cross one is not detected") +if "unfinished" not in lfs._toolchain_state(tree()): + problems.append("a tree with no C++ headers is not detected") +# headers under a different target than configured +st = lfs._toolchain_state(tree("x86_64-lfs-linux-gnu")) +if not st.startswith("BROKEN"): + problems.append("headers under the wrong target are not detected") +src = open(sys.argv[1]).read() +if "Restoring it later will restore the breakage too" not in src: + problems.append("saving a broken tree gives no warning") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a snapshot records and reports the state it captured") +PY70 +_count_rc $? + +# ---- a snapshot label must not hide what it captured ------------------------ # +# "chapters 5-6: 22/22 steps" was all a snapshot recorded, so a tree where +# chapter 8 had already run -- and already broken the toolchain -- looked +# pristine. Restoring it restored the damage, and it read as a failed restore +# rather than a bad snapshot. +python3 - "$LFS_TOOL" <<'PY71' +import sys, os, json, tempfile, importlib.machinery as m +store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store +os.makedirs(os.path.join(store, "books"), exist_ok=True) +json.dump({"default": "12.4", "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, + open(os.path.join(store, "config.json"), "w")) +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "_tree_state"): + problems.append("nothing reports the state of a tree") +if not hasattr(lfs, "cmd_bs_verify"): + problems.append("there is no way to ask what state a tree is in") + +t = tempfile.mkdtemp() +os.makedirs(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress")) +os.makedirs(os.path.join(t, "usr", "src", "zlib")) +os.makedirs(os.path.join(t, "usr", "lib", "gcc", "x86_64-pc-linux-gnu")) +p = os.path.join(t, "usr", "include", "c++", "15.2.0", + "x86_64-nimgnu-linux-gnu", "bits") +os.makedirs(p); open(os.path.join(p, "c++config.h"), "w").close() +names = [n for n, _s, _t in lfs.CROSSCHAIN_STEPS] +json.dump({"book": "12.4", "done": names, "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, + open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "crosschain-progress.json"), "w")) +open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "steps-built"), "w").write("zlib\n") +state = dict(lfs._tree_state(t)) +# chapters 5-6 alone is not the whole story +if "chapters 7-9" not in state: + problems.append("chapter 7-9 progress is not reported") +if "WARNING" not in state: + problems.append("a native gcc in the tree is not flagged") +if not state.get("toolchain", "").startswith("BROKEN"): + problems.append("the toolchain state is not reported") +# and the snapshot must carry it +src = open(sys.argv[1]).read() +if '"state":' not in src: + problems.append("the snapshot metadata does not record the tree state") +if problems: + for p2 in problems: print(" FAIL %s" % p2) + sys.exit(1) +print(" PASS a tree's real state is reported and recorded with snapshots") +PY71 +_count_rc $? + +# ---- staged installs must cope with the merged-/usr symlinks ---------------- # +# Book 4.2 makes /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib. A +# package installing into /bin produces a REAL bin/ directory in the staging +# tree, and copying that over the symlink fails: +# cp: cannot overwrite non-directory '/bin' with directory '.../bin' +# That broke every package installing into those paths -- most of chapter 7 -- +# on a completely fresh build. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_resolve_staged_symlink_dirs()" "$helper_src" \ + && ok "staged directories are folded onto the tree's symlinks" \ + || bad "a package installing into /bin cannot be merged" + ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" + i_fold="$(printf '%s' "$ms" | grep -n "_resolve_staged_symlink_dirs" | head -1 | cut -d: -f1)" + i_cp="$(printf '%s' "$ms" | grep -n '_install_staged_tree' | head -1 | cut -d: -f1)" + if [ -n "$i_fold" ] && [ -n "$i_cp" ] && [ "$i_fold" -lt "$i_cp" ]; then + ok "the folding happens before the copy" + else + bad "the copy runs first and still fails on the symlinks" + fi + # driven by the live tree, not a hardcoded list of names + rs="$(sed -n '/^_resolve_staged_symlink_dirs() {/,/^}/p' "$helper_src")" + case "$rs" in + *'[ -L "$live" ] || continue'*) + ok "it folds whatever the tree actually has as a symlink" ;; + *) bad "the symlink names are hardcoded" ;; + esac + + # behaviour: bin/ and sbin/ fold, usr/bin/ is left alone + sd="$T/stagedirs"; mkdir -p "$sd/usr/bin" "$sd/usr/sbin" + ln -sf usr/bin "$sd/bin"; ln -sf usr/sbin "$sd/sbin" + mkdir -p "$sd/stage/bin" "$sd/stage/usr/bin" + echo m > "$sd/stage/bin/mount"; echo w > "$sd/stage/usr/bin/wall" + ( SNAP_ROOT="$sd"; say() { :; } + eval "$rs" + _resolve_staged_symlink_dirs "$sd/stage" ) >/dev/null 2>&1 + if [ -f "$sd/stage/usr/bin/mount" ] && [ ! -d "$sd/stage/bin" ] \ + && [ -f "$sd/stage/usr/bin/wall" ]; then + ok "bin/ is folded into usr/bin/ and nothing else is disturbed" + else + bad "the folding did not produce the expected staging tree" + fi +fi + +# ---- replacing a library a running system is using -------------------------- # +# `cp -a` opens the destination with O_TRUNC and writes. For a shared library +# that is currently mapped that is fatal: every running process sees the file +# shrink under it. Merging glibc that way killed the shell mid-install -- +# /usr/bin/env: error while loading shared libraries: +# /usr/lib/libc.so.6: file too short +# -- and the chroot could not start any program afterwards. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_install_staged_tree()" "$helper_src" \ + && ok "staged files are installed one at a time" \ + || bad "the merge still overwrites files in place" + it="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" + case "$it" in + *'mv -f "$tmp" "$dest"'*) + ok "each file is renamed into place, never truncated" ;; + *) bad "files are still written over in place" ;; + esac + # the temp file must be in the SAME directory, or mv is a copy, not a rename + case "$it" in + *'tmp="$dest.pkgusr-new.$$"'*) + ok "the temporary is beside its destination, so mv is atomic" ;; + *) bad "the temporary is elsewhere and mv would copy" ;; + esac + ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" + case "$ms" in + *'cp -a "$stage"/. '*) bad "the wholesale cp -a is still there" ;; + *) ok "the wholesale copy is gone" ;; + esac + + # behaviour: replacing a file a process holds open must not disturb it + ad="$T/atomic"; mkdir -p "$ad/usr/lib" "$ad/stage/usr/lib" + printf 'OLDOLDOLD' > "$ad/usr/lib/lib.so" + printf 'NEWNEWNEWNEW' > "$ad/stage/usr/lib/lib.so" + before_ino="$(stat -c %i "$ad/usr/lib/lib.so")" + ( SNAP_ROOT="$ad"; say() { :; }; warn() { :; } + eval "$it" + _install_staged_tree "$ad/stage" ) >/dev/null 2>&1 + after_ino="$(stat -c %i "$ad/usr/lib/lib.so")" + if [ "$before_ino" != "$after_ino" ] \ + && [ "$(cat "$ad/usr/lib/lib.so")" = "NEWNEWNEWNEW" ]; then + ok "the replacement is a new inode, so open files keep the old one" + else + bad "the file was overwritten in place (inode unchanged)" + fi +fi + +# ---- staging is not worth paying for on every package ----------------------- # +# Staging costs roughly an extra copy of everything installed -- measurably +# slower on a big package. It is worth it only where an interrupted install +# cannot be recovered from: interrupt man-pages and you rebuild man-pages; +# interrupt gcc or glibc and there is no compiler and no libc left to rebuild +# them WITH. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^STAGE_PACKAGES_DEFAULT=" "$helper_src" \ + && ok "only the packages that cannot be recovered are staged" \ + || bad "every package pays the staging cost" + for p in gcc glibc binutils bash coreutils; do + grep -q "^STAGE_PACKAGES_DEFAULT=.*$p" "$helper_src" \ + || bad "$p is not staged, but an interrupted install would be fatal" + done + ok "the toolchain packages are all in the default list" + + ss="$(sed -n '/^_should_stage() {/,/^}/p' "$helper_src")" + # -tmp and -passN are the same package at an earlier stage + case "$ss" in + *'base="${name%-tmp}"'*) ok "the -tmp and -passN variants count too" ;; + *) bad "gcc-pass1 would not be staged" ;; + esac + case "$ss" in + *"LFS_STAGE_PACKAGES"*) ok "the list is configurable" ;; + *) bad "the list is hardcoded" ;; + esac + case "$ss" in + *"all) return 0"*"none) return 1"*) ok "all / none are accepted" ;; + *) bad "there is no way to turn staging fully on or off" ;; + esac + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *"--stage)"*"--no-stage)"*) ok "a single build can override it" ;; + *) bad "staging cannot be forced or skipped per build" ;; + esac + + # behaviour + ( STAGE_PACKAGES_DEFAULT="$(sed -n 's/^STAGE_PACKAGES_DEFAULT=\"\(.*\)\"$/\1/p' \ + "$helper_src" | head -1)" + eval "$ss" + _should_stage gcc && _should_stage glibc && _should_stage gcc-pass2 \ + && ! _should_stage man-pages && ! _should_stage zlib ) \ + && ok "gcc and glibc stage; man-pages and zlib do not" \ + || bad "the wrong packages are being staged" +fi + +# ---- the staging directory must belong to whoever builds -------------------- # +# It is created by root, but the build runs as the PACKAGE USER, so +# `make install DESTDIR=...` could not write into it: +# mkdir: cannot create directory '/.lfs-pkgusr/stage/gcc': Permission denied +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *'chown "$owner:$owner" "$_stage"'*) + ok "the staging directory is given to the package user" ;; + *) bad "the package user cannot write into its own staging directory" ;; + esac + # and the parent must be traversable, or the chown alone is not enough + case "$cb" in + *'chmod 755 "$(dirname "$_stage")"'*) + ok "the staging parent is traversable" ;; + *) bad "the package user cannot reach into the staging directory" ;; + esac + # a root step needs no chown -- there is no package user to give it to + case "$cb" in + *'[ "$as_root" != 1 ] && user_exists "$owner"'*) + ok "a root step is not given away to a user that may not exist" ;; + *) bad "the chown would fail on a root step" ;; + esac + # the owner must be known BEFORE the stage is made, or it chowns to nothing + i_owner="$(printf '%s' "$cb" | grep -n 'owner="\$(pkg_owner_name' | head -1 | cut -d: -f1)" + i_stage="$(printf '%s' "$cb" | grep -n '_stage="\$(_stage_dir_for' | head -1 | cut -d: -f1)" + if [ -n "$i_owner" ] && [ -n "$i_stage" ] && [ "$i_owner" -lt "$i_stage" ]; then + ok "the owner is resolved before the staging directory is made" + else + bad "the staging directory is created before the owner is known" + fi +fi + +# ---- a package must own the files its earlier stages installed -------------- # +# The same package is built more than once: ncurses in chapter 6 (as the lfs +# user), perl-tmp in chapter 7 (as root), then the real one in chapter 8 as +# its package user. The earlier files keep the earlier owner, and the package +# user cannot overwrite or re-mode them: +# cp: cannot create regular file '/usr/bin/tic': Permission denied +# Couldn't chmod 644 /usr/lib/perl5/.../perl.pod: Operation not permitted +# One cause, two chapters -- so one mechanism, applied to every package. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_claim_earlier_stages()" "$helper_src" \ + && ok "a package claims the files its earlier stages installed" \ + || bad "a rebuild cannot overwrite its own earlier files" + ce="$(sed -n '/^_claim_earlier_stages() {/,/^}/p' "$helper_src")" + # every naming shape the same package takes across the book + # the stages are built from a loop now, so check the suffixes it appends + case "$ce" in + *'"$stem-tmp" "$stem-pass1" "$stem-pass2"'*) + ok "the -tmp and -passN stages are all claimed" ;; + *) bad "an earlier stage suffix is missing" ;; + esac + # ONLY what the build left behind -- never another package's file + case "$ce" in + *"root|lfs|UNKNOWN) ;;"*) + ok "only root- and lfs-owned paths are claimed" ;; + *) bad "it could take a file belonging to another package" ;; + esac + # directories too, or the package cannot create files in them + case "$ce" in + *'.dirs'*) ok "the directories are claimed as well as the files" ;; + *) bad "the package could not create files in its own directories" ;; + esac + # shared install dirs stay shared + case "$ce" in + *"is_install_dir"*) ok "shared install directories are left alone" ;; + *) bad "a shared directory could be taken by one package" ;; + esac + # and it must run BEFORE the build, not after it fails + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + i_claim="$(printf '%s' "$cb" | grep -n '_claim_earlier_stages' | head -1 | cut -d: -f1)" + i_build="$(printf '%s' "$cb" | grep -n 'building \$name' | head -1 | cut -d: -f1)" + if [ -n "$i_claim" ] && [ -n "$i_build" ] && [ "$i_claim" -lt "$i_build" ]; then + ok "the claim happens before the build starts" + else + bad "the claim runs too late to help" + fi +fi + +# ---- checks that silently always passed --------------------------------------# +# From an independent review of `lfs` against the book. Each of these hid a +# real failure rather than reporting it. +python3 - "$LFS_TOOL" <<'PY73' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() +problems = [] + +# md5 verification: `md5sum -c … | grep -v ": OK"` returned 0 on a CORRUPTED +# tree -- grep inverts the verdict and the pipe discards md5sum's status. +i = src.index("verifying md5sums") +seg = src[i:i + 900] +# ignore comment lines: the fix is documented with the broken command quoted +code = "\n".join(l for l in seg.splitlines() if not l.lstrip().startswith("#")) +if 'grep -v ": OK"' in code: + problems.append("md5 verification still inverts its own verdict") +if "md5sum -c md5sums --quiet" not in seg or "sys.exit(1)" not in seg: + problems.append("a bad md5sum does not fail the download") + +# The book's toolchain gate must stay a gate. These five greps MUST produce +# output -- no match means the build is linked against the host -- and the +# blanket `|| true` turned the only real check into a guaranteed pass. +gates = [ + 'grep -E -o "/usr/lib.*/S?crt[1in].*succeeded" dummy.log', + 'grep -B3 "^ /usr/include" dummy.log', + "grep 'SEARCH.*/usr/lib' dummy.log", + 'grep "/lib.*/libc.so.6 " dummy.log', + 'grep found dummy.log', +] +out = lfs._soften_diagnostics("\n".join(gates)) +for line in out.splitlines(): + if "|| true" in line: + problems.append("a toolchain gate is still softened: %s" % line[:40]) +# but an ordinary diagnostic grep must STILL be softened +if "|| true" not in lfs._soften_diagnostics("grep something optional.txt"): + problems.append("ordinary diagnostic greps are no longer softened") + +# 9.9's `cat > /etc/shells` block is <pre class="root">, not "userinput", so +# cfg_shells resolved to zero commands and /etc/shells was never written. +pi = src.index("classes = tag.get(\"class\") or []") +if '"root", "install"' not in src[pi:pi + 700]: + problems.append("only 'userinput' blocks are parsed; /etc/shells is missed") +# the accepted-class tuple itself must not include "screen" (output samples) +accept = src[src.index("any(c in classes for c in ("):][:120] +if "screen" in accept: + problems.append("'screen' blocks (output samples) would be run as commands") + +# a step that produces nothing must say so +if "has no command blocks in this book" not in src: + problems.append("a step resolving to zero blocks is still skipped silently") + +# book 4.4's ~/.bashrc opens with these; without them the build runs with +# bash's command hash on and root's umask +env = lfs._lfs_env_prefix("/mnt/lfs") +if not env.startswith("set +h\numask 022\n"): + problems.append("the build environment is missing set +h / umask 022") + +# ^ anchored to the whole string, so a later line never matched +if not (lfs._TEST_DIAG_RE.flags & 8): + problems.append("_TEST_DIAG_RE has no re.M; ^ only matches the first line") + +# dead duplicate flagged by pyflakes +if src.count("\ndef _pkg_glob_for") != 1: + problems.append("_pkg_glob_for is defined more than once") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the checks that always passed now actually check") +PY73 +_count_rc $? + +# ---- a staged install must not take the shared install directories ---------- # +# The staging tree is created by the package user, so $stage/usr, $stage/usr/bin +# and so on are owned by that user with mode 755. Stamping that onto the live +# tree handed /usr and /usr/bin to whichever package was building, losing the +# install group and g+w -- the whole install-directory scheme, undone by one +# package. Verified: root:install 775 became gcc:gcc 755. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + it="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" + case "$it" in + *"ALREADY THERE -- leave its ownership and mode alone"*) + ok "an existing directory keeps its owner and mode" ;; + *) bad "a staged install still restamps existing directories" ;; + esac + + # behaviour: existing dirs untouched, new ones owned by the package + sd="$T/stageown"; mkdir -p "$sd/usr/bin" "$sd/stage/usr/bin" "$sd/stage/usr/lib/fresh" + chmod 775 "$sd/usr" "$sd/usr/bin" + before_usr="$(stat -c %a "$sd/usr")" + echo p > "$sd/stage/usr/bin/p"; echo l > "$sd/stage/usr/lib/fresh/l" + chmod -R 755 "$sd/stage" + ( SNAP_ROOT="$sd"; say() { :; }; warn() { :; } + eval "$it"; _install_staged_tree "$sd/stage" ) >/dev/null 2>&1 + if [ "$(stat -c %a "$sd/usr")" = "$before_usr" ] \ + && [ -f "$sd/usr/bin/p" ] && [ -f "$sd/usr/lib/fresh/l" ]; then + ok "existing modes survive and the files still install" + else + bad "the staged merge changed an existing directory's mode" + fi +fi + +# ---- book 7.6 truncates the user database ----------------------------------- # +# `cat > /etc/passwd` is right the first time and catastrophic the second: it +# wipes every package user, private group, the install group, every collector +# group and every membership. The files on disk then all read as UNKNOWN. +if [ -f "$helper_src" ]; then + grep -q "^_protect_user_db()" "$helper_src" \ + && ok "the user database is protected across 7.6" \ + || bad "re-running init-files would wipe every package user" + grep -q "^_step_rewrites_user_db()" "$helper_src" \ + && ok "steps that rewrite passwd/group are detected" \ + || bad "the protection is not tied to the steps that need it" + # the detector must not fire on scripts that merely READ the files + dt="$(sed -n '/^_step_rewrites_user_db() {/,/^}/p' "$helper_src")" + rd="$T/dbdetect"; mkdir -p "$rd" + printf 'grep root /etc/passwd\n' > "$rd/read.sh" + printf 'cat > /etc/passwd << "EOF"\nroot:x:0:0::/root:/bin/bash\nEOF\n' > "$rd/write.sh" + ( eval "$dt" + _step_rewrites_user_db "$rd/write.sh" && ! _step_rewrites_user_db "$rd/read.sh" ) \ + && ok "it fires on a rewrite, not on a read" \ + || bad "the user-database detector is wrong" + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *_protect_user_db*_restore_user_db*) + ok "entries are saved before the step and put back after" ;; + *) bad "the protection is never applied around the build" ;; + esac +fi + +# ---- two accounts sharing one uid ------------------------------------------- # +# Ownership is stored as a NUMBER. Give two names the same uid and every tool +# resolves it to whichever name comes first -- ncurses' binaries reported as +# /usr/bin/tic -rwxr-xr-x xz:xz +# and the ncurses build could not overwrite its own files. It reads exactly +# like a permissions bug and is not one. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_check_duplicate_ids()" "$helper_src" \ + && ok "duplicate ids are detected and named" \ + || bad "a uid collision would look like a permissions bug forever" + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" + case "$ba" in + *_check_duplicate_ids*) ok "the check runs before anything is built" ;; + *) bad "the collision check never runs" ;; + esac + # restoring entries after book 7.6 must not CREATE a collision + rd="$(sed -n '/^_restore_user_db() {/,/^}/p' "$helper_src")" + case "$rd" in + *'is now used by'*) + ok "an entry whose id was reassigned is not restored" ;; + *) bad "restoring after 7.6 could create the collision itself" ;; + esac + + # behaviour: the detector finds a real collision and ignores a clean file + dd="$T/dupids"; mkdir -p "$dd" + printf 'root:x:0:0::/root:/bin/bash\nxz:x:10005:10005::/x:/bin/bash\nncurses:x:10005:10005::/n:/bin/bash\n' > "$dd/bad" + printf 'root:x:0:0::/root:/bin/bash\nxz:x:10005:10005::/x:/bin/bash\n' > "$dd/good" + du="$(sed -n '/^_duplicate_ids() {/,/^}/p' "$helper_src")" + ( eval "$du" + [ -n "$(_duplicate_ids "$dd/bad")" ] && [ -z "$(_duplicate_ids "$dd/good")" ] ) \ + && ok "it finds a real collision and stays quiet on a clean file" \ + || bad "the duplicate-id detector is wrong" +fi + +# ---- one reconcile pass instead of eight repair commands -------------------- # +# fix-ownership, adopt-dirs, adopt-existing, fix-orphans, fix-users, +# sort-users, seal-install-dirs and claim-earlier-stages each fixed ONE symptom +# and could not see a problem outside its own remit -- so a staged install +# quietly handing /usr to a package user went unnoticed through four later +# failures. Ownership is derived state: the manifests, the install-dir list +# and the collector map fully determine what the tree should be. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^cmd_verify()" "$helper_src" \ + && ok "one command reconciles the tree" \ + || bad "there is no single check of the whole model" + # The commands it replaces must not be SECOND IMPLEMENTATIONS -- but they + # must still be callable. Deleting them outright broke every message that + # tells people to run them: + # !! unknown command 'adopt-existing' + for c in fix-ownership fix-orphans fix-users adopt-existing; do + grep -qE "^\s+.*\b$c[|)]" "$helper_src" \ + || bad "$c is referenced in the hints but not dispatched" + done + ok "the familiar repair names still work" + # ... and route to verify rather than repeating its logic + disp="$(sed -n '/fix-ownership|fix-orphans/,/cmd_verify/p' "$helper_src")" + case "$disp" in + *cmd_verify*) ok "they are aliases for verify, not second copies" ;; + *) bad "a repair command has its own implementation again" ;; + esac + + # every rule the model needs + for r in _vfy_manifest_ownership _vfy_install_dirs _vfy_orphans \ + _vfy_duplicate_ids _vfy_user_order; do + grep -q "^$r()" "$helper_src" || bad "verify has no $r rule" + done + ok "verify checks ownership, install dirs, orphans, ids and order" + + # reporting by default, repairing only when asked + cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" + case "$cv" in + *'"${1:-}" = "--fix" ] && fix=1'*|*"--fix|--run) fix=1"*) + ok "verify reports unless asked to repair" ;; + *) bad "verify repairs without being asked" ;; + esac + # manifest ownership must settle before orphans, or a file whose uid no + # longer resolves is handed to 'lfs' instead of to its package + i_own="$(printf '%s' "$cv" | grep -n '_vfy_manifest_ownership' | head -1 | cut -d: -f1)" + i_orp="$(printf '%s' "$cv" | grep -n '_vfy_orphans' | head -1 | cut -d: -f1)" + if [ -n "$i_own" ] && [ -n "$i_orp" ] && [ "$i_own" -lt "$i_orp" ]; then + ok "manifest ownership is settled before orphans are claimed" + else + bad "an orphan pass could take a file that belongs to a package" + fi + + # A1 end to end: the install dirs are checked, so a staged install taking + # /usr is caught rather than surfacing four failures later + vd="$T/verifydirs"; mkdir -p "$vd/usr/bin" "$vd/.lfs-pkgusr/manifests" + chmod 755 "$vd/usr" "$vd/usr/bin" + id_out="$( ( SNAP_ROOT="$vd"; ETC="$vd/etc"; INSTALLDIRS="" + MANIFESTS="$vd/.lfs-pkgusr/manifests" + say() { echo "$*"; }; warn() { echo "$*"; } + _vfy_n_checked=0; _vfy_n_wrong=0; _vfy_n_fixed=0 + INSTALL_GID=9999 + for _f in install_dirs_list _install_dirs_are_sealed _vfy_report \ + set_install_dir_owner install_dir_owner_ok _vfy_install_dirs; do + eval "$(_slice_fn "$helper_src" "$_f")" + done + # the pass is gated on the ownership epoch now: before book 7.6 there + # is no user database, so `install` is not a name and reporting 27 + # wrong directories would be noise. This fixture is past that. + ownership_established() { return 0; } + ownership_possible() { return 0; } + _vfy_install_dirs 0 ) 2>&1 )" + case "$id_out" in + *"root:install"*) ok "a wrongly-owned install directory is reported" ;; + *) bad "verify does not notice install dirs losing the install group" ;; + esac +fi + +# ---- ownership is derived state, so it can be checked -------------------------# +# The manifests say who installed what, the install-dir list says what is +# shared, the collector map says which groups govern which directories. +# Together they determine what the tree SHOULD look like -- so a discrepancy +# can be found before it becomes a failed build, instead of being chased one +# symptom at a time through fix-ownership, adopt-dirs, fix-perms, fix-orphans +# and claim-earlier-stages. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^cmd_verify()" "$helper_src" \ + && ok "the tree can be checked against the manifests" \ + || bad "there is no way to ask whether ownership is correct" + cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" + # the A1 case: an install directory taken by a package user + # The wording lives in the _vfy_* helpers now -- cmd_verify calls them + # rather than carrying its own copy of each rule. Assert the check runs. + case "$cv" in + *_vfy_install_dirs*) + ok "it catches an install directory taken by a package" ;; + *) bad "the install-directory scheme is not checked" ;; + esac + _vfy_install_dirs_body="$(_slice_fn "$helper_src" _vfy_install_dirs)" + case "$_vfy_install_dirs_body" in + *'"root:install $want_mode"'*) ;; + *) bad "the install-dir check no longer says what it wants" ;; + esac + case "$cv" in + *_vfy_manifest_ownership*) + ok "it catches files owned by the wrong package" ;; + *) bad "mis-owned files are not detected" ;; + esac + case "$cv" in + *_check_duplicate_ids*) ok "it checks for id collisions too" ;; + *) bad "a uid collision would not be reported" ;; + esac + # report by default, repair only when asked + case "$cv" in + *'[ "${1:-}" = "--fix" ] && fix=1'*) + ok "it reports by default and repairs only with --fix" ;; + *) bad "verify changes things without being asked" ;; + esac + # a collision is NOT auto-repairable -- which name owns what is a decision + case "$cv" in + *"verify cannot repair this"*) + ok "it does not guess at an id collision" ;; + *) bad "verify would silently reassign ids" ;; + esac +fi + +# ---- durable records and scratch must be distinguishable ---------------------# +# $STATE mixes records that ARE the ownership (manifests, collector map) with +# pure scratch (staging trees, logs). A snapshot or a restart had to guess. +if [ -f "$helper_src" ]; then + grep -q "^STATE_DURABLE=" "$helper_src" \ + && ok "the durable records are named" \ + || bad "nothing says which state must survive" + grep -q "^STATE_SCRATCH=" "$helper_src" \ + && ok "the scratch is named" \ + || bad "nothing says which state is disposable" + grep -q "manifests" <<< "$(sed -n '/^STATE_DURABLE=/,/^STATE_SCRATCH=/p' "$helper_src")" \ + && ok "the manifests count as durable" \ + || bad "the manifests could be cleaned away" + grep -q "^STATE_FORMAT=" "$helper_src" \ + && ok "the state format is versioned" \ + || bad "an incompatible state directory would be misread" + cs="$(sed -n '/^cmd_clean_state() {/,/^}/p' "$helper_src")" + case "$cs" in + *'rm -rf "${STATE:?}/$d"'*) + ok "cleaning scratch cannot expand to rm -rf /" ;; + *) bad "an unset STATE would delete the root" ;; + esac +fi + +# ---- the suite's own count must be trustworthy ------------------------------ # +# ok/bad used shell variables, and several checks call them from inside a +# subshell -- `( ... ) && ok || bad`, a pipeline, a `while read` fed by process +# substitution. An increment there dies with the subshell, so the suite +# printed 446 PASS lines and reported 433. A wrong count is worse than none: +# it hides a regression behind a number that still looks healthy. +cnt="$T/countcheck"; mkdir -p "$cnt" +( _COUNTDIR="$cnt"; : > "$cnt/pass"; : > "$cnt/fail" + ok() { echo x >> "$_COUNTDIR/pass"; } + # the case that used to be lost: incremented inside a subshell + ( ok ) ; ( true && ok ) ; echo | while read -r _; do ok; done ) +n="$(wc -l < "$cnt/pass" | tr -d ' ')" +[ "$n" = 3 ] \ + && ok "counts survive subshells, pipelines and process substitution" \ + || bad "the counter loses $((3 - n)) of 3 increments from subshells" + +# and every assertion must be counted exactly once +_asserts="$(grep -cE '^\s*(ok|bad) "' "$0" 2>/dev/null || echo 0)" +[ "$_asserts" -gt 0 ] \ + && ok "the suite counts assertions, not printed lines" \ + || bad "the suite has no assertions to count" + +# ---- host layout and built-system layout are separate ------------------------ # +# Configuring where package users live on THIS machine must not change where +# they live on every system built from it. The whole point of the build is +# that the new system is not this one. +python3 - "$LFS_TOOL" <<'PY74' +import sys, os, json, tempfile, importlib.machinery as m +store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store +os.makedirs(os.path.join(store, "books"), exist_ok=True) +json.dump({"default": "12.4", + "pkgusr_home": "/usr/src", + "target_pkgusr_home": "/usr/src/package_users", + "target_appuser_home": "/home/shared_users"}, + open(os.path.join(store, "config.json"), "w")) +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "layout"): + print(" FAIL the layout is not configurable"); sys.exit(1) +if lfs.layout("pkgusr_home") != "/usr/src": + problems.append("the host layout was changed by a target setting") +if lfs.layout("pkgusr_home", target=True) != "/usr/src/package_users": + problems.append("the built system does not use its own layout") +if lfs.layout("appuser_home", target=True) != "/home/shared_users": + problems.append("application users do not get their own base") +# The default moved deliberately: accounts are grouped by kind, so a +# directory listing says what a thing IS. Existing trees are NOT migrated -- +# a fresh build is required, and migrate-layout is still unwritten. Pin the +# new values so the next move is also a decision and not an accident. +d = lfs.LAYOUT_DEFAULTS +if d["target_pkgusr_home"] != "/usr/src/pkgusr" \ + or d["pkgusr_home"] != "/usr/src/pkgusr": + problems.append("packages no longer default to /usr/src/pkgusr") +if d["target_cfguser_home"] != "/usr/src/cfg" \ + or d["cfguser_home"] != "/usr/src/cfg": + problems.append("config steps no longer default to /usr/src/cfg") +# and the target value must reach the chroot and the built system's config +src = open(sys.argv[1]).read() +if 'LFS_SRC_ROOT="{layout("pkgusr_home", target=True)}"' not in src: + problems.append("the chroot build environment does not carry the layout") +if 'pkgusr_home=%s' not in src: + problems.append("the built system's config does not record the layout") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS host and built-system layouts are configured independently") +PY74 +_count_rc $? + +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def appuser_home" "$pmt" \ + && ok "application users have their own base directory" \ + || bad "application users still share the package-user base" + # an unconfigured system must keep the old layout + lyd="$T/layout"; mkdir -p "$lyd" + printf 'collector_prefix=n\n' > "$lyd/pm.conf" + got="$(PKGUSR_CONFIG="$lyd/pm.conf" python3 -c " +import importlib.machinery as m, os +pm = m.SourceFileLoader('pm', '$pmt').load_module() +print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" + [ "$got" = "/usr/src /usr/src" ] \ + && ok "an unconfigured system keeps the existing layout" \ + || bad "the layout changed without being asked (got '$got')" + printf 'collector_prefix=n\npkgusr_home=/usr/src/package_users\nappuser_home=/home/shared_users\n' \ + > "$lyd/pm2.conf" + got2="$(PKGUSR_CONFIG="$lyd/pm2.conf" python3 -c " +import importlib.machinery as m, os +pm = m.SourceFileLoader('pm', '$pmt').load_module() +print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" + [ "$got2" = "/usr/src/package_users /home/shared_users" ] \ + && ok "a configured system uses the layout it was built with" \ + || bad "the configured layout is ignored (got '$got2')" +fi + +# ---- /root must never carry the install group -------------------------------- # +# Two different ideas got conflated. An install directory is SHARED: it gets +# the install group and g+w so any package user can add files. /root, /home +# and the scratch dirs are something else -- they must never be adopted by a +# package, but they are not shared. Listing them as install dirs produced +# drwxrwx--- 1 root install /root +# handing every package user access to root's home. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^never_adopt_list()" "$helper_src" \ + && ok "protected-from-adoption is separate from shared" \ + || bad "the two ideas are still conflated" + ( SNAP_ROOT=/; INSTALLDIRS="" + eval "$(sed -n '/^never_adopt_list() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^is_never_adopt() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^install_dirs_list() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^install_dir_patterns() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^matches_install_pattern() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^is_install_dir() {/,/^}/p' "$helper_src")" + # never group-writable + for d in /root /home /tmp /var/tmp; do + install_dirs_list | grep -qx "$d" && exit 1 + done + # but still never adopted + for d in /root /home /tmp /var/tmp; do is_install_dir "$d" || exit 1; done + # and the real install dirs are untouched + install_dirs_list | grep -qx /usr/bin || exit 1 + exit 0 ) \ + && ok "/root and /home are protected but not group-writable" \ + || bad "/root or /home would get the install group" +fi + +# ---- nothing owned by the build user survives into the system --------------- # +# Book 7.2 hands over the directories 4.2 created, but the tree ROOT and the +# mount points are not in that list, so /, /dev, /proc, /sys, /run, /media, +# /opt and /tools stayed owned by a user that does not exist in the chroot: +# drwxr-xr-x 1 lfs lfs 178 / +python3 - "$LFS_TOOL" <<'PY75' +import sys +src = open(sys.argv[1]).read() +if "give the top level to root" not in src: + print(" FAIL the tree root and mount points keep the build user") + sys.exit(1) +i = src.index("give the top level to root") +seg = src[i:i + 500] +problems = [] +if "/sources" not in seg: + problems.append("/sources is not excluded; the book keeps it with lfs") +if "-R " in seg: + problems.append("the handover is recursive; it must only touch the top level") +if 'chown -h root:root' not in seg: + problems.append("symlinks at the top level are not handed over") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the tree root and mount points are handed back to root") +PY75 +_count_rc $? + +# ---- the package-user model must be usable from outside ---------------------- # +# `build` only works for steps that already exist in the step list with a +# generated script, so a BLFS package or a local script had no way to say +# "install this as its own package user and track it" -- it would have to +# reimplement user creation, the wrappers, staging, tracking and the permission +# grants. That is exactly how the two halves of this toolchain drifted apart. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^cmd_install_as()" "$helper_src" \ + && ok "anything can be installed as its own package user" \ + || bad "third-party installs cannot use the model" + ia="$(sed -n '/^cmd_install_as() {/,/^}/p' "$helper_src")" + # both shapes a caller might have + case "$ia" in + *"--script)"*) ok "an existing install script can be used" ;; + *) bad "only inline commands are accepted" ;; + esac + case "$ia" in + *"--) shift; cmdv=("*) ok "a bare command can be used" ;; + *) bad "only a script file is accepted" ;; + esac + # it must REUSE cmd_build, not reimplement it -- one code path or they drift + case "$ia" in + *'cmd_build "$name" --force'*) + ok "it reuses the build path rather than duplicating it" ;; + *) bad "install-as reimplements what build already does" ;; + esac + # the result must be a normal package, visible to every query + case "$ia" in + *'"$STATE_PROGRESS/steporder"'*) + ok "the package becomes visible to status, list and manifests" ;; + *) bad "an install-as package would be invisible to the other commands" ;; + esac + # reserved names must be refused + case "$ia" in + *"_is_not_a_package"*) ok "reserved step names are refused" ;; + *) bad "install-as could create a user for last-step or cfg_*" ;; + esac + # a command with spaces or quotes must survive being written to a script + grep -q "^_quote_cmd()" "$helper_src" \ + && ok "arguments are quoted, so paths with spaces survive" \ + || bad "a command containing spaces would be mangled" +fi + +# ---- a capitalised step must find its lowercase earlier stage --------------- # +# Book titles are capitalised ("Python", "Util-linux") but the chapter-7 stage +# is named from the tarball ("python-tmp"). Matching only the step's own +# spelling meant Python never claimed python-tmp's files, and chapter 7's +# root-owned tree blocked the chapter-8 install: +# install: cannot remove '/usr/lib/python3.13/idlelib/Icons/README.txt' +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + ce="$(sed -n '/^_claim_earlier_stages() {/,/^}/p' "$helper_src")" + case "$ce" in + *"tr 'A-Z' 'a-z'"*) ok "a capitalised step finds its lowercase stage" ;; + *) bad "Python would never claim python-tmp's files" ;; + esac + case "$ce" in + *'for stem in "$name" "$lname" "$owner" "$owname"'*) + ok "the owner's spelling is tried as well as the step's" ;; + *) bad "only the step name is tried" ;; + esac + case "$ce" in + *"de-duplicate"*) ok "a manifest is not walked twice" ;; + *) bad "the same manifest could be walked four times" ;; + esac +fi + +# ---- a package user should record where it came from ------------------------ # +# An LFS base package, a BLFS package, a pip module and a local script are +# maintained differently and updated from different places. Nothing in a +# package user's home said which it was. +if [ -f "$helper_src" ]; then + grep -q "^_write_pkgusr_info()" "$helper_src" \ + && ok "each package user records its source" \ + || bad "there is no way to tell an LFS package from a BLFS one" + grep -q "^cmd_pkgusr_info()" "$helper_src" \ + && ok "it can be read back" \ + || bad "the record is written but never readable" + # both completion paths must write it, not just one + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + n="$(printf '%s' "$cb" | grep -c '_write_pkgusr_info')" + [ "${n:-0}" -ge 2 ] \ + && ok "every path that completes a build records it" \ + || bad "only one of the two done paths records the source" + # the helper must be defined before cmd_build uses it + i_def="$(grep -n '^_write_pkgusr_info()' "$helper_src" | head -1 | cut -d: -f1)" + i_use="$(grep -n '^cmd_build() {' "$helper_src" | head -1 | cut -d: -f1)" + [ -n "$i_def" ] && [ -n "$i_use" ] && [ "$i_def" -lt "$i_use" ] \ + && ok "it is defined before the build uses it" \ + || bad "the branding helper is defined after its first use" + ia="$(sed -n '/^cmd_install_as() {/,/^}/p' "$helper_src")" + case "$ia" in + *"LFS_PKG_SOURCE"*) ok "install-as records what kind of install it was" ;; + *) bad "a BLFS or local install would be recorded as lfs" ;; + esac +fi + +# ---- adopt-existing must still exist ----------------------------------------- # +# The pass moved into verify, but the command was removed while messages +# elsewhere still told people to run it: +# !! unknown command 'adopt-existing' +# A command that has been deleted is worse than one that is redundant. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^cmd_adopt_existing()" "$helper_src" \ + && ok "adopt-existing is still a command" \ + || bad "a command the messages tell people to run does not exist" + grep -qE "^\s+adopt-existing\|adopt-dirs\)" "$helper_src" \ + && ok "it is reachable from the command line" \ + || bad "adopt-existing is defined but not dispatched" + # every command named in a hint must actually dispatch + for c in fix-ownership fix-orphans grant-dir verify which-package \ + check-toolchain seal-install-dirs sort-users install-as; do + # a command may be dispatched on its own or as part of an alternation + grep -qE "^\s+[a-z|-]*\b$c[|)]" "$helper_src" \ + || bad "$c is referenced but not dispatched" + done + ok "every command referenced in the hints is dispatched" + + # ---- and the users must be created in BUILD order --------------------- # + # This pass creates the package users it finds missing, and a uid is just + # the order of creation. Alphabetical order gave man-pages (first + # chapter-8 package) 10000 and binutils -- built first, in chapter 5 -- a + # much higher one, so the passwd file no longer reads as the build order. + vm="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" + case "$vm" in + *'$(_manifests_in_build_order)'*) + ok "the users are created in build order" ;; + *) bad "manifests are still walked alphabetically" ;; + esac + mo="$(sed -n '/^_manifests_in_build_order() {/,/^}/p' "$helper_src")" + case "$mo" in + *'[ "${1:-}" = dirs ]'*) + ok "the directory manifests are ordered the same way" ;; + *) bad "the .dirs manifests are still alphabetical" ;; + esac +fi + +# ---- package users cannot exist before book 7.6 ----------------------------- # +# A crosschain snapshot is taken BEFORE chapter 7, so /etc/passwd does not +# exist -- 7.6 creates it. Running the adoption pass then printed +# user binutils (could not create) -> package user +# forty times, once per package, with no hint why. The give-away is the +# prompt: "I have no name!" means even root has no passwd entry. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_user_db_ready()" "$helper_src" \ + && ok "the user database is checked before creating users" \ + || bad "adoption would fail once per package with no explanation" + ae="$(sed -n '/^cmd_adopt_existing() {/,/^}/p' "$helper_src")" + case "$ae" in + *_explain_no_user_db*) + ok "it explains that the tree is simply too early" ;; + *) bad "the error does not say what is actually wrong" ;; + esac + # and the work must still happen, later, without being asked again + # The late pass is the ownership epoch now: build-all calls + # _ownership_checkpoint after every step, and the marker makes it fire once. + ba="$(_slice_fn "$helper_src" cmd_build_all)" + case "$ba" in + *_ownership_checkpoint*) + ok "the packages are adopted once chapter 7 has created it" ;; + *) bad "a chapter 5-6 tree would never adopt its packages" ;; + esac + case "$(_slice_fn "$helper_src" _ownership_checkpoint)" in + *ownership_established*) ok "the late pass runs once, not per step" ;; + *) bad "the late adoption would repeat on every step" ;; + esac + ud="$(sed -n '/^_user_db_ready() {/,/^}/p' "$helper_src")" + case "$ud" in + *"grep -q '^root:'"*) + ok "an empty or rootless passwd counts as not ready" ;; + *) bad "a truncated passwd would look usable" ;; + esac +fi + +# ---- "did not change owner" vs "no package claims it" ----------------------- # +# The ownership pass can only act on what a manifest lists. A file installed +# before tracking existed, or whose manifest was lost, stays root-owned and +# never appears in the report at all -- so a skipped file and an unclaimed one +# look identical from the outside, and they need different fixes. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_vfy_unclaimed()" "$helper_src" \ + && ok "files no manifest claims are reported" \ + || bad "an unclaimed file is indistinguishable from a skipped one" + vu="$(sed -n '/^_vfy_unclaimed() {/,/^}/p' "$helper_src")" + # /etc is root's by right; reporting it would bury the real cases + case "$vu" in + *'"${SNAP_ROOT%/}/etc"'*) ok "configuration under /etc is not reported" ;; + *) bad "every file in /etc would be listed as unclaimed" ;; + esac + # the manifest paths carry the host prefix; comparison must strip it + case "$vu" in + *strip_host_prefix*) ok "manifest paths are compared without the prefix" ;; + *) bad "every claimed file would look unclaimed inside the chroot" ;; + esac + case "$vu" in + *"lfs-helper which-package"*) + ok "it says how to find out which package should own them" ;; + *) bad "the report gives no way forward" ;; + esac + cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" + case "$cv" in + *_vfy_unclaimed*) ok "verify runs the check" ;; + *) bad "the unclaimed check is never called" ;; + esac +fi + + +# ---- a versioned chapter-9 "configuration" section is a PACKAGE ------------- # +# Book 9.2 is titled LFS-Bootscripts-20250827: a versioned tarball that unpacks +# and installs programs into /etc/rc.d/init.d and /lib/services. `lfs` gets +# this right -- _title_pkgver finds a version, so it generates a full phased +# build and deliberately keeps the step OUT of rootsteps, meaning "build this +# as a package user". +# +# lfs-helper then blanket-matched cfg_* as "not a package", refused to create +# the user, and the non-root path staged the script as that user anyway: +# 'cfg_bootscripts' is a build step, not a package -- no user created. +# install: invalid user 'cfg_bootscripts' +# !! could not stage the install script +# The build stopped at 88/99. Two tools, two answers, no shared authority. +# +# The script itself is the authority: a package has pkg_glob, prose does not. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cd="$T/cfgpkg"; mkdir -p "$_cd" + printf 'pkg_glob="lfs-bootscripts-*.tar.*"\n' > "$_cd/cfg_bootscripts.sh" + printf '#!/bin/bash\necho x > /etc/hostname\n' > "$_cd/cfg_hostname.sh" + _r="$( SCRIPTS="$_cd"; PKGUSR_PREFIX="p_" + # _is_not_a_package strips the package-user prefix first, so the + # helper it calls has to be here too -- without it every name came + # back empty and matched nothing, which reads as "is a package". + eval "$(sed -n '/^unprefix_pkg_user() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^_is_not_a_package() {/,/^}/p' "$helper_src")" + # a versioned cfg section installs files and needs an owner + _is_not_a_package cfg_bootscripts && echo "bootscripts-has-no-user" + # a prose one writes root's config and must NOT get a user + _is_not_a_package cfg_hostname || echo "hostname-got-a-user" + # a cfg step with no script at all is prose until proven otherwise + _is_not_a_package cfg_nothing || echo "unknown-cfg-step-got-a-user" + # the genuinely-not-packages must stay that way + _is_not_a_package last-step || echo "last-step-got-a-user" + _is_not_a_package init-dirs || echo "init-dirs-got-a-user" + _is_not_a_package refind || echo "refind-got-a-user" + # and a normal package is still a package + _is_not_a_package gcc && echo "gcc-lost-its-user" + # callers pass the OWNER name too, already prefixed + _is_not_a_package p_cfg_bootscripts && echo "prefixed-form-misread" )" + if [ -z "$_r" ]; then + ok "a versioned cfg_* section is built as its own package user" + else + printf '%s\n' "$_r" | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + fi + # and lfs must agree: a versioned cfg section is kept out of rootsteps + grep -q 'pkg_like_cfg.append(cname)' "$LFS_TOOL" \ + && ok "lfs keeps a versioned cfg section out of the root steps" \ + || bad "lfs no longer treats a versioned cfg section as a package" +fi + +# ---- the triplet warning must not fire on a healthy toolchain --------------- # +# A finished build has TWO C++ header directories: chapter 6 installs libstdc++ +# under $LFS_TGT (x86_64-lfs-linux-gnu, or whatever vendor string is set) and +# chapter 8's native gcc installs its own under x86_64-pc-linux-gnu. Both are +# present and that is correct -- the chapter-6 one is simply left over. +# +# The check took the FIRST glob match and stopped, so it reported a mismatch +# whenever the stale directory sorted first -- which is alphabetical, i.e. luck: +# "nimgnu" < "pc", so a perfectly good toolchain warned on every single package. +# What matters is only whether a directory matching the compiler exists. +if [ -f "$helper_src" ]; then + _tp="$T/triplet"; mkdir -p "$_tp/15.2.0/x86_64-nimgnu-linux-gnu/bits" \ + "$_tp/15.2.0/x86_64-pc-linux-gnu/bits" + touch "$_tp/15.2.0/x86_64-nimgnu-linux-gnu/bits/c++config.h" \ + "$_tp/15.2.0/x86_64-pc-linux-gnu/bits/c++config.h" + # the fixed shape: scan every match, stay silent if one is the compiler's + _fn="$(sed -n '/^_warn_on_triplet_mismatch() {/,/^}/p' "$helper_src")" + _p="" + case "$_fn" in + *'[ "$have" = "$mine" ] && return 0'*) ;; + *) _p="$_p;the check does not stop at the compiler's own headers" ;; + esac + case "$_fn" in + *"break"*) _p="$_p;the check still stops at the first directory found" ;; + esac + case "$_fn" in + *'[ -n "$found" ] || return 0'*) ;; + *) _p="$_p;a tree with no C++ headers yet would warn" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the triplet warning ignores leftover chapter-6 headers" + fi +fi + +# ---- the adoption gate must actually close --------------------------------- # +# Chapters 5-6 build before /etc/passwd exists, so their files are recorded in +# manifests with no owner. Once book 7.6 creates the user database, the +# adoption pass hands each package its files, and adopted.list records +# "name size" so it does not run again. +# +# It ran again on EVERY build-all, and the cause was that the gate and the +# recorder tested opposite conditions: +# _needs_adoption: user_exists "$owner" || return 0 # no user -> NEEDS +# _mark_all_adopted: user_exists "$owner" || continue # no user -> SKIP +# So a single manifest without a user could never be recorded and kept +# _has_unadopted_packages at yes forever. One rule now, used by both. +# +# The write also swallowed every error (2>/dev/null on the redirect AND +# `|| rm -f` on the move), so a missing or read-only state directory looked +# exactly like a successful record. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # the state directory is sorted now: adopted.list lives under progress/ + _ad="$T/adopt"; mkdir -p "$_ad/manifests" "$_ad/progress" + for f in haveuser nouser; do echo "/usr/bin/$f" > "$_ad/manifests/$f.files"; done + _out="$( STATE="$_ad"; MANIFESTS="$_ad/manifests"; SCRIPTS="$_ad/scripts" + STATE_PROGRESS="$_ad/progress" + warn(){ :; } + user_exists(){ [ "$1" = haveuser ]; } # only one package has a user + pkg_owner_name(){ echo "$1"; } + is_done(){ return 1; } # none built by lfs-helper here + PKGUSR_PREFIX="p_" + # _is_not_a_package strips the prefix first, so its helper is needed too + for fn in _ADOPTED_LIST _manifest_size _adopted_size _mark_adopted \ + _mark_all_adopted _needs_adoption _has_unadopted_packages \ + unprefix_pkg_user _is_not_a_package; do + eval "$(sed -n "/^$fn() {/,/^}/p" "$helper_src")" + done + _has_unadopted_packages || echo "gate-was-closed-before-the-pass" + _mark_all_adopted + # the package WITHOUT a user must be recorded too -- that is the fix + grep -q '^nouser ' "$_ad/progress/adopted.list" 2>/dev/null \ + || echo "a package without a user is never recorded" + # and the gate must now be shut + _has_unadopted_packages && echo "the gate stays open after the pass" + # but a manifest that GROWS has new files to hand over -- reopen + echo /usr/bin/extra >> "$_ad/manifests/haveuser.files" + _has_unadopted_packages || echo "a grown manifest is not re-adopted" )" + if [ -z "$_out" ]; then + ok "the adoption gate closes once the pass has run" + else + printf '%s\n' "$_out" | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + fi + # a failed write must be reported, not swallowed: an unrecorded adoption + # means the pass runs forever and nothing on disk says why + _w="$( STATE=/dev/null/nope + warn(){ echo "reported"; } + for fn in _ADOPTED_LIST _manifest_size _mark_adopted; do + eval "$(sed -n "/^$fn() {/,/^}/p" "$helper_src")" + done + _mark_adopted p 12 2>/dev/null && echo "claimed-success" )" + case "$_w" in + *reported*) ok "a failed adoption record is reported, not swallowed" ;; + *) bad "a failed adoption record still looks like success" ;; + esac +fi + +# ---- chapter-8 configuration must be generated, and must not rot ----------- # +# nsswitch.conf, the timezone and the dynamic-loader config were never +# generated: the step table only harvested chapter 9. Without /etc/nsswitch.conf +# glibc falls back to compiled-in defaults; without a timezone the system runs +# UTC forever. +# +# Two of those sections carry ids the BOOK TOOLCHAIN generated -- +# idm139921653990176 8.5.2.1. Adding nsswitch.conf +# -- which change every time the book is rebuilt. Naming one in the table +# works exactly once and then resolves to nothing, silently. Silently is the +# real danger: /etc/shells went missing from finished systems because +# cfg_shells resolved to zero blocks and was skipped without a word. So a step +# may name an id OR a title fragment, and this test fails if any chapter-8 +# entry stops resolving. +python3 - "$LFS_TOOL" "$LFS_BOOK" <<'PYCFG8' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +try: + from bs4 import BeautifulSoup +except ImportError: + print(" SKIP beautifulsoup4 not installed"); sys.exit(0) + +if not hasattr(lfs, "CHROOT_CONFIG_STEPS_8"): + print(" FAIL chapter-8 configuration is not generated at all"); sys.exit(1) + +soup = BeautifulSoup(open(sys.argv[2], encoding='utf-8', errors='replace').read(), + 'html.parser') +secs = {sid: (t, sec) for sid, t, sec in lfs.iter_sections(soup)} + +for name, sid in lfs.CHROOT_CONFIG_STEPS_8: + e = lfs._resolve_section(secs, sid) + if not e: + problems.append("%s: '%s' resolves to no section" % (name, sid)) + elif not e[1]["commands"]: + problems.append("%s: resolved but has no command blocks" % name) + +# the resolver itself: id first, then title, and never a silent guess +if lfs._resolve_section(secs, "ch-config-hostname") is None: + problems.append("a stable section id no longer resolves") +if lfs._resolve_section(secs, "Adding nsswitch.conf") is None: + problems.append("a section cannot be found by its title") +if lfs._resolve_section(secs, "zzz-no-such-section") is not None: + problems.append("a nonexistent section resolves to something") +# an ambiguous fragment must NOT be resolved by picking the first match +if lfs._resolve_section(secs, "Configuring") is not None: + problems.append("an ambiguous title fragment is silently resolved") + +# /tools deletion and stripping are opt-in: neither should ever be a default +if lfs.remove_tools(): + problems.append("/tools is deleted by default") +if lfs.strip_binaries(): + problems.append("binaries are stripped by default") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS chapter-8 configuration resolves by id or title, and cannot rot") +PYCFG8 +_count_rc $? + +# cfg_rm-tools deletes a root-owned tree, so it must be a root step -- never a +# package user's job, and never adoptable +grep -q 'root_steps.append("cfg_rm-tools")' "$LFS_TOOL" \ + && ok "removing /tools is a root step" \ + || bad "removing /tools is not a root step" + +# ---- every build decision is asked UP FRONT --------------------------------- # +# The build used to discover decisions halfway through, or at the very end: +# last_build_step.sh blocked on `passwd root` after a six-hour build, and its +# own comment admits the non-interactive path leaves the system +# "unbootable-but-known". Collecting the answers in `session` is what lets +# `lfs run` and `build-all` finish without a person sitting there. +# +# They must also REACH the chroot: the config file is on the host and the +# chroot environment is reset, so anything not written into the tree's env +# file is simply not there when it is needed. +python3 - "$LFS_TOOL" <<'PYASK' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +keys = [k for k, _, _ in lfs._SESSION_KEYS] +for k in ("timezone", "remove_tools", "strip", "main_user"): + if k not in keys: + problems.append("'%s' is not asked during session" % k) + +# Destructive or slow options must never default to on. Deleting the +# cross-toolchain and rewriting every binary are both things a person should +# have chosen, not discovered. +if lfs.remove_tools(): + problems.append("/tools is deleted by default") +if lfs.strip_binaries(): + problems.append("binaries are stripped by default") + +# each answer has to be written into the tree, or the chroot never sees it +body = open(sys.argv[1]).read() +i = body.index("def write_chroot_env") +j = body.find("\ndef ", body.index("_make_world_readable(path)", i)) +w = body[i:j if j > i else len(body)] +for var in ("LFS_TIMEZONE", "LFS_MAIN_USER", "LFS_STRIP", "LFS_REMOVE_TOOLS"): + if var not in w: + problems.append("%s never reaches the chroot" % var) + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS every build decision is collected before the build starts") +PYASK +_count_rc $? + +# and the accounts step must USE the configured name rather than stopping to +# ask for it -- the whole point of asking early +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + case "$(_slice_fn "$helper_src" cmd_init_accounts)" in + *main_user*) ok "the final step uses the login account chosen up front" ;; + *) bad "the final step stops to ask for a name already configured" ;; + esac +fi + +# ---- a section that ships a FILE instead of commands ----------------------- # +# Book 9.6.8 (The rc.site File) has no <kbd class="command"> at all. It prints +# /etc/sysconfig/rc.site as a <pre class="auto"> block with every line +# commented out, for the reader to copy and uncomment. The parser harvests +# command blocks only, so cfg_rcsite resolved to zero blocks and the file was +# never created -- the same silent-nothing failure that once lost /etc/shells. +# +# An all-commented file is still worth installing: it documents every knob the +# boot scripts have, in the place the boot scripts look for it. +python3 - "$LFS_TOOL" "$LFS_BOOK" <<'PYRCSITE' +import sys, subprocess, tempfile, os, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "_auto_file_block"): + print(" FAIL a section that ships a file instead of commands is ignored") + sys.exit(1) + +raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() +blk = lfs._auto_file_block(raw, "ch-config-site", "/etc/sysconfig/rc.site") +if not blk: + problems.append("rc.site is not extracted from the book") +else: + d = tempfile.mkdtemp() + sh = blk.replace("/etc/sysconfig", d + "/etc/sysconfig") + open(d + "/t.sh", "w").write("set -e\n" + sh + "\n") + r = subprocess.run(["bash", d + "/t.sh"], capture_output=True, text=True) + out = d + "/etc/sysconfig/rc.site" + if r.returncode != 0: + problems.append("the generated rc.site block is not valid bash: %s" + % r.stderr.strip()[:80]) + elif not os.path.isfile(out): + problems.append("the block runs but writes no file") + else: + body = open(out).read() + if len(body.splitlines()) < 50: + problems.append("rc.site came out truncated (%d lines)" + % len(body.splitlines())) + # a quoted heredoc: the body is full of # and shell metacharacters and + # must land byte-for-byte as the book prints it, not expanded + if "$" in body and "EOF" not in blk.split("\n")[1]: + problems.append("the heredoc is not quoted -- the body would expand") + +# a section with neither commands nor a file body must still be reported, not +# silently skipped +if lfs._auto_file_block(raw, "ch-config-sysklogd", "/etc/x") is not None: + problems.append("a prose section is treated as if it shipped a file") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a section that ships a file body is installed, not skipped") +PYRCSITE +_count_rc $? + +# ---- the package-user prefix must be one decision, not three --------------- # +# lfs-helper has ALWAYS read a prefix and defaulted to `p`: +# PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-$(_read_pkgusr_prefix || echo p)}" +# but nothing ever wrote the setting -- `lfs` did not put it in the chroot env +# and `packagemanager` did not implement it at all. So the default applied by +# accident, and the built system would look for `gcc` while the build had +# created `p_gcc`. Same species as cfg_bootscripts and adopted.list: one +# decision held in several places that could disagree. +# +# Prefixing MUST be idempotent. It is applied to names typed by a person +# (gcc), to directory names read back from disk (p_gcc), and to values that +# already went through it -- lfs-helper's own comment names p_p_gcc as the +# thing to avoid. +python3 - "$LFS_TOOL" <<'PYPFX' +import sys, os, tempfile, importlib.machinery as m +problems = [] +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +pmp = os.path.join(os.path.dirname(sys.argv[1]) or ".", "packagemanager") +if not os.path.exists(pmp): + print(" SKIP packagemanager not found next to the tool"); sys.exit(0) +pm = m.SourceFileLoader('pm', pmp).load_module() + +if not hasattr(lfs, "pkgusr_prefix"): + problems.append("lfs cannot say what the package-user prefix is") +if not hasattr(pm, "pkgusr_name"): + problems.append("packagemanager does not implement the prefix") + +if not problems: + # the two tools must agree, or the build and the built system diverge + if lfs.pkgusr_prefix() != pm.pkgusr_prefix(): + problems.append("lfs and packagemanager disagree: %r vs %r" + % (lfs.pkgusr_prefix(), pm.pkgusr_prefix())) + if pm.pkgusr_prefix() != "p_": + problems.append("the default prefix is not p_ (got %r)" + % pm.pkgusr_prefix()) + if pm.pkgusr_name("gcc") != "p_gcc": + problems.append("a package name is not prefixed") + # idempotent -- the whole reason every call site can apply it blindly + if pm.pkgusr_name(pm.pkgusr_name("gcc")) != "p_gcc": + problems.append("prefixing twice gives p_p_gcc") + if pm.unprefix_pkgusr("p_gcc") != "gcc": + problems.append("the prefix cannot be stripped for display") + # the OTHER kinds of account must not be touched: an application user and + # a collector group are not packages + if pm.pkgusr_name("u_firefox") != "u_firefox": + problems.append("an application user was given the package prefix") + if pm.pkgusr_name(pm.collector_prefix() + "python") \ + != pm.collector_prefix() + "python": + problems.append("a collector group was given the package prefix") + # the home is the ACCOUNT's, whichever kind of name it is asked with + if pm.pkgusr_home("gcc") != pm.pkgusr_home("p_gcc"): + problems.append("a package name and its user name give different homes") + +# and it has to REACH the chroot, or lfs-helper falls back to a guess +body = open(sys.argv[1]).read() +if "LFS_PKGUSR_PREFIX" not in body: + problems.append("the prefix never reaches the chroot") +if "pkgusr_prefix" not in [k for k, _, _ in lfs._SESSION_KEYS]: + problems.append("the prefix is never asked for") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS lfs, lfs-helper and packagemanager share one package-user prefix") +PYPFX +_count_rc $? + +# lfs-helper's side: it must take the value from the environment lfs wrote, +# and an explicitly EMPTY value must mean "no prefix" -- not fall through to +# the default and rename every user out from under an existing tree +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q 'PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-' "$helper_src" \ + && ok "an empty prefix means none, and is not the same as unset" \ + || bad "an empty prefix falls through to the default" +fi + +# ---- accounts live in subdirectories by kind ------------------------------- # +# Flat under /usr/src, a hundred package users and a handful of config steps +# were one undifferentiated list, and nothing about a directory said what kind +# of thing it was. Now: +# /usr/src/pkgusr/p_gcc a package +# /usr/src/cfg/p_cfg_bootscripts a config step that installs files +# All three tools have to agree, or the build creates a home the built system +# never looks in. +python3 - "$LFS_TOOL" <<'PYSPLITHOME' +import sys, os, tempfile, importlib.machinery as m +problems = [] +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +pmp = os.path.join(os.path.dirname(sys.argv[1]) or ".", "packagemanager") +if not os.path.exists(pmp): + print(" SKIP packagemanager not found next to the tool"); sys.exit(0) +d = tempfile.mkdtemp(); os.environ["PKGUSR_BASE"] = d +pm = m.SourceFileLoader('pm', pmp).load_module() + +for k in ("target_pkgusr_home", "target_cfguser_home", + "pkgusr_home", "cfguser_home"): + if k not in lfs.LAYOUT_DEFAULTS: + problems.append("layout has no '%s'" % k) +if lfs.LAYOUT_DEFAULTS.get("pkgusr_home") == lfs.LAYOUT_DEFAULTS.get("cfguser_home"): + problems.append("packages and config steps share one directory") +if not lfs.LAYOUT_DEFAULTS.get("pkgusr_home", "").endswith("/pkgusr"): + problems.append("packages do not default to /usr/src/pkgusr") +if not lfs.LAYOUT_DEFAULTS.get("cfguser_home", "").endswith("/cfg"): + problems.append("config steps do not default to /usr/src/cfg") + +# a package and a config step must land in DIFFERENT roots, whichever form of +# the name they are asked with +if pm.pkgusr_home("gcc") == pm.pkgusr_home("cfg_bootscripts"): + problems.append("a package and a config step share a root") +if not pm.pkgusr_home("gcc").startswith(os.path.join(d, "pkgusr")): + problems.append("a package is not under the package root") +if not pm.pkgusr_home("cfg_bootscripts").startswith(os.path.join(d, "cfg")): + problems.append("a config step is not under the config root") +# prefixed form must resolve identically -- callers pass both +if pm.pkgusr_home("p_cfg_bootscripts") != pm.pkgusr_home("cfg_bootscripts"): + problems.append("the prefixed name lands somewhere else") + +# listing must walk BOTH roots, or half the accounts vanish +os.makedirs(os.path.join(d, "pkgusr", pm.pkgusr_name("gcc"))) +os.makedirs(os.path.join(d, "cfg", pm.pkgusr_name("cfg_bootscripts"))) +got = pm.list_all_users() +for want in ("gcc", "cfg_bootscripts"): + if want not in got: + problems.append("'%s' is missing from the account list %r" % (want, got)) + +# and both roots must reach the chroot +body = open(sys.argv[1]).read() +for var in ("LFS_PKGUSR_ROOT", "LFS_CFGUSR_ROOT"): + if var not in body: + problems.append("%s never reaches the chroot" % var) + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS packages and config steps live in separate roots") +PYSPLITHOME +_count_rc $? + +# lfs-helper's side: one chokepoint that routes by kind, and the passes that +# walk every account must walk both roots -- not just the package one +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SRCROOT=/tmp/_hr; PKGUSR_ROOT="$SRCROOT/pkgusr"; CFGUSR_ROOT="$SRCROOT/cfg" + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + # pkg_owner_name too: pkgusr_home_for calls it, and without it every + # home came back as the bare root with no account on the end + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ + unprefix_pkg_user pkg_owner_name pkgusr_home_for pkgusr_roots; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + # a package gets the package prefix and the package root + [ "$(pkgusr_home_for gcc)" = "$SRCROOT/pkgusr/p_gcc" ] || echo "package-in-wrong-root" + [ "$(pkgusr_home_for p_gcc)" = "$SRCROOT/pkgusr/p_gcc" ] || echo "prefixed-package-in-wrong-root" + # a config step gets the CONFIG prefix and the config root -- not the + # package prefix stacked on top of its own + [ "$(pkgusr_home_for cfg_bootscripts)" = "$SRCROOT/cfg/cfg_bootscripts" ] || echo "cfg-in-wrong-root" + # and a legacy p_cfg_ name is repaired rather than carried forward + [ "$(pkgusr_home_for p_cfg_bootscripts)" = "$SRCROOT/cfg/cfg_bootscripts" ] || echo "stacked-cfg-not-repaired" + [ "$(pkgusr_roots | wc -l)" = 2 ] || echo "not-both-roots-walked" )" + if [ -z "$_o" ]; then + ok "lfs-helper routes each account to the root for its kind" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + # no home may be built by hand any more -- that is how the two roots drift + grep -q '"\$SRCROOT/\$name"' "$helper_src" \ + && bad "a home is still built from SRCROOT directly" \ + || ok "every home goes through the one chokepoint" +fi + +# ---- the home a build cd's into is the home the account has ---------------- # +# The build creates the account with useradd -d <home>, then cd's into that +# home to run the staged script. Those two must be the same path, and they +# are reached from DIFFERENT names: cmd_add_user has the owner +# ("p_man-pages"), build_pkg has the step name ("man-pages"). +# +# pkgusr_home_for routed by kind but never applied the prefix, so: +# # created package user p_man-pages ... +# line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory +# Chapter 7 did not catch it -- root steps mkdir -p their own directory first, +# so it surfaced at the first real package user, nine steps in. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SRCROOT=/usr/src; PKGUSR_ROOT="$SRCROOT/pkgusr"; CFGUSR_ROOT="$SRCROOT/cfg" + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ + pkg_owner_name unprefix_pkg_user pkgusr_home_for; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + # THE bug: a step name and its owner must give one home + [ "$(pkgusr_home_for man-pages)" = "$(pkgusr_home_for p_man-pages)" ] \ + || echo "step name and owner give different homes" + # and it must be the account's, i.e. prefixed + [ "$(pkgusr_home_for man-pages)" = "/usr/src/pkgusr/p_man-pages" ] \ + || echo "the home is not the account's: $(pkgusr_home_for man-pages)" + # book titles are capitalised, tarballs are not + [ "$(pkgusr_home_for Man-Pages)" = "$(pkgusr_home_for man-pages)" ] \ + || echo "a capitalised book title gives a different home" + # earlier stages fold onto the real package -- util-linux-tmp is + # util-linux's, which is where its files end up + [ "$(pkgusr_home_for util-linux-tmp)" = "$(pkgusr_home_for util-linux)" ] \ + || echo "an earlier stage gets its own home" + [ "$(pkgusr_home_for gcc-pass1)" = "$(pkgusr_home_for gcc)" ] \ + || echo "a pass1 build gets its own home" + # config steps still route to their own root, and wear ONE prefix -- + # their own. This used to read p_cfg_bootscripts: the package prefix + # stacked on the step name's own, so the account claimed to be a package + # while living in the config root. + [ "$(pkgusr_home_for cfg_bootscripts)" = "/usr/src/cfg/cfg_bootscripts" ] \ + || echo "a config step is not under the config root" )" + if [ -z "$_o" ]; then + ok "a step name and its owner resolve to the same account home" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the rule must be reachable from shell, or every other script re-derives it + # -- and a third copy is how the roots drift apart + grep -q 'pkgusr-home)' "$helper_src" \ + && ok "the home rule is available to scripts as 'lfs-helper pkgusr-home'" \ + || bad "shell scripts have to re-derive where an account lives" +fi + +# nothing outside the chokepoints may build an account path by hand +for _f in packagemanager_install lfs-completion.bash last_build_step.sh; do + _p="$(dirname "$LFS_TOOL")/$_f" + [ -f "$_p" ] || continue + if grep -qE '"?/usr/src/\$[a-z_]+' "$_p"; then + bad "$_f builds an account path by hand" + else + ok "$_f asks for the account home instead of building it" + fi +done + +# ---- chapter-8 config steps must not stop the build ------------------------ # +# Two ways they did, both from lifting a sub-section out of its parent package. +# +# cfg_timezone (8.5.2.2) ends with an INTERACTIVE menu whose only output is +# the name to put in the placeholder: +# tzselect +# ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime +# `session` already asks for the timezone, so both are answered. The same +# section also unpacks tzdata with a path relative to GLIBC's build directory +# (`tar -xf ../../tzdata2025b.tar.gz`), which resolves nowhere once the step +# stands on its own. +python3 - "$LFS_TOOL" <<'PYCFG8RUN' +import sys, re, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +body = ("tar -xf ../../tzdata2025b.tar.gz\n" + "zic -d $ZONEINFO -p America/New_York\n" + "tzselect\n" + "ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime\n") +out, left = lfs._fill_placeholders(body, "cfg_timezone") +if left: + problems.append("the timezone step still asks for %s" % left) +if "<xxx>" in out: + problems.append("the timezone placeholder is not filled in") +if re.search(r'^\s*tzselect\s*$', out, flags=re.M): + problems.append("the interactive tzselect menu still runs") +if "../../tzdata" in out: + problems.append("tzdata is still read relative to glibc's build directory") +if "SOURCES_DIR" not in out: + problems.append("tzdata is not taken from the sources directory") +if "/usr/share/zoneinfo//etc/localtime" in out.replace(" ", ""): + problems.append("the localtime symlink lost its zone name") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the timezone step is answered from config, not from a prompt") +PYCFG8RUN +_count_rc $? + +# cfg_ld rewrites /etc/ld.so.conf -- which glibc created and still owns -- and +# makes one directory. Nothing NEW appears, so the manifest count is 0 and the +# step was failed as a silent no-op: +# mkdir: created directory '/etc/ld.so.conf.d' +# !! cfg_ld reported success but installed NO files. +# That is the normal shape of a configuration step. Packages must still fail. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # to the end of the branch, not a fixed window: the message grew a + # breakdown of where the count came from, and a +6 window stopped + # reaching the `return 1` it is asserting + _blk="$(sed -n '/reported success but installed NO files/,/^ fi$/p' "$helper_src")" + # the whole function, not a fixed window above the message: the branch + # gained a third case (a re-run over its own work) and a -B 14 window + # stopped reaching the config-step test it is asserting + _pre="$(_slice_fn "$helper_src" cmd_build)" + case "$_pre" in + *'_is_not_a_package "$name"'*) + ok "a config step may install no new files; a package may not" ;; + *) bad "installing no new files still fails a configuration step" ;; + esac + case "$_blk" in + *'return 1'*) ok "a package that installs nothing still fails" ;; + *) bad "the no-files check no longer fails a package" ;; + esac +fi + +# ---- a symlinked install directory is checked at its target ---------------- # +# Book 7.x makes /bin, /sbin, /lib and /lib64 symlinks into /usr. `test -d` +# FOLLOWS a symlink but `stat` does not, so verify compared the LINK's own +# mode and owner -- always lrwxrwxrwx root:root -- against what a directory +# should look like, and reported all of them on every run: +# /bin root:root lrwxrwxrwx (want root:install, group-writable) +# Chowning a symlink changes nothing that matters, and the target (/usr/bin) +# is in the same list and checked on its own. Both the report and the --fix +# path need the guard: without it, --fix "repairs" a symlink forever. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # There used to be two copies of the install-dir rule and so two guards to + # count. cmd_verify calls _vfy_install_dirs now, so there is one -- and + # the check and the fix share it, which is the property this was after. + _vid="$(_slice_fn "$helper_src" _vfy_install_dirs)" + if printf '%s\n' "$_vid" | grep -q '\[ -L "\$live" \] && continue'; then + ok "a symlinked install dir is skipped by both the check and the fix" + else + bad "a symlinked install dir is still reported" + fi + # and the guard must come BEFORE the stat, or it reports then skips + if sed -n '/install directories not in the expected state/,-12p' \ + "$helper_src" >/dev/null 2>&1; then :; fi + _blk="$(grep -A 4 '\[ -d "\$live" \] || continue' "$helper_src" | head -20)" + case "$_blk" in + *'-L "$live"'*) ok "the symlink guard runs before stat" ;; + *) bad "stat runs before the symlink guard" ;; + esac +fi + +# ---- verify must write every finding to a file ----------------------------- # +# The console prints the first eight of each kind and then +# ... and 44265 more +# which is a number and nothing you can act on: it cannot be grepped, diffed +# against the previous run, or fed to which-package. Every finding goes to a +# log, tagged by kind, while the console stays short. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '^VERIFY_LOG=' "$helper_src" || _p="$_p;verify has no log file" + grep -q '_vfy_log_open' "$helper_src" || _p="$_p;the log is never opened" + # every truncated list must also be written in full + for _k in wrong-owner unclaimed installdir; do + grep -q "$_k" "$helper_src" || _p="$_p;nothing logs '$_k' findings" + done + # and the summary must say where it is, or nobody finds it + grep -q 'full report: \$VERIFY_LOG' "$helper_src" \ + || _p="$_p;the summary never names the log" + # a read-only state dir must not abort verify -- the report still matters + grep -q 'VERIFY_LOG=""; return 0' "$helper_src" \ + || _p="$_p;verify dies instead of running when the log cannot be written" + + _o="$( STATE=/tmp/_vst; SNAP_ROOT=/; VERIFY_LOG=/tmp/_vst/verify.log + rm -rf /tmp/_vst + eval "$(sed -n '/^_vfy_log_open() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^_vfy_log() {/,/^}/p' "$helper_src")" + _vfy_log_open + _vfy_log wrong-owner "/usr/bin/python3 is root want p_python" + _vfy_log unclaimed "/usr/bin/lfs-helper" + [ -s /tmp/_vst/verify.log ] || echo "the log is empty after findings" + grep -q '^wrong-owner' /tmp/_vst/verify.log || echo "wrong-owner not tagged" + grep -q '^unclaimed' /tmp/_vst/verify.log || echo "unclaimed not tagged" + # a fresh run must not append to the last one's findings + _vfy_log_open + grep -q '^wrong-owner' /tmp/_vst/verify.log && echo "the log is not reset per run" + # an unwritable location must be survivable + VERIFY_LOG=/dev/null/nope/verify.log + _vfy_log_open >/dev/null 2>&1 || echo "an unwritable log aborts verify" )" + [ -n "$_o" ] && _p="$_p;$(printf '%s' "$_o" | tr '\n' ';')" + + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "verify writes every finding to a log, tagged by kind" + fi +fi + +# ---- the built system gets the whole toolchain, and it has an owner -------- # +# Only lfs-helper was copied into the tree, because only lfs-helper runs during +# the build. But the built system needs the rest -- packagemanager cannot be +# installed BY packagemanager -- and a system that boots without them has no +# way to install anything at all. They are a few files and they refresh on +# every `lfs run`. +# +# They are also software installed into /usr/bin, so they get a package user +# like anything else. Without one they are reported forever as +# 1516 root-owned file(s) that NO manifest claims: +# /usr/bin/lfs-helper +# which is true, unhelpful, and buries the cases that matter. +python3 - "$LFS_TOOL" <<'PYTOOLS' +import sys, os, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] + +for t in ("lfs-helper", "packagemanager", "packagemanager_install", "blfs"): + if t not in getattr(lfs, "TOOLCHAIN_SCRIPTS", ()): + problems.append("%s is not copied into the tree" % t) + +# the owner is a package user like any other, so it carries the prefix +if lfs.pkgusr_prefix() + lfs.PKGUSR_TOOLS_USER != "p_pkgusr": + problems.append("the tools' owner is %r, not p_pkgusr" + % (lfs.pkgusr_prefix() + lfs.PKGUSR_TOOLS_USER)) + +# the uid must be read from the TREE's passwd, never this machine's -- a host +# uid means nothing inside the chroot, the same mistake book 3.1 warns about +d = tempfile.mkdtemp() +os.makedirs(os.path.join(d, "etc")) +if lfs._uid_in_tree(d, "p_pkgusr") is not None: + problems.append("a uid is invented when the tree has no passwd file") +with open(os.path.join(d, "etc", "passwd"), "w") as f: + f.write("root:x:0:0::/root:/bin/bash\np_pkgusr:x:10042:10042::/usr/src:/bin/bash\n") +if lfs._uid_in_tree(d, "p_pkgusr") != 10042: + problems.append("the tree's own passwd file is not used") +if lfs._uid_in_tree(d, "nosuchuser") is not None: + problems.append("an unknown user resolves to something") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the whole toolchain is installed and owned by p_pkgusr") +PYTOOLS +_count_rc $? + +# lfs-helper claims them once /etc/passwd is real -- lfs cannot, because on a +# fresh tree the account does not exist until book 7.6 +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '_adopt_pkgusr_tools' "$helper_src" || _p="$_p;the tools are never claimed" + grep -q 'PKGUSR_TOOLS=' "$helper_src" || _p="$_p;there is no list of tools to claim" + # it must run in build-all, beside the other adoption pass + case "$(_slice_fn "$helper_src" establish_ownership)" in + *_adopt_pkgusr_tools*) ;; + *) _p="$_p;the tools are not claimed when ownership is established" ;; + esac + # and it must never take another package's file + sed -n '/^_adopt_pkgusr_tools() {/,/^}/p' "$helper_src" \ + | grep -q 'user_exists "\$cur" && continue' \ + || _p="$_p;claiming the tools could steal another package's file" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the tools are claimed once the user database exists" + fi +fi + +# ---- an ADOPTED package gets a real home, like a built one ----------------- # +# Packages built in a run owned their files and had full homes; packages +# ADOPTED from chapter 5-6 manifests in the same run did not: +# drwxr-xr-x 1 p_bison p_bison 148 p_bison <- built +# drwxr-xr-x 1 root root 86 p_gcc <- adopted +# 86 vs 148 bytes: the adopted homes were missing the profile links and had +# never been chowned. cmd_add_user was called with BOTH streams suppressed, +# so whatever went wrong said nothing. Adoption now initialises the home +# explicitly (init_package_user_home is idempotent) and stderr is let through. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _blk="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" + _p="" + case "$_blk" in + *'init_package_user_home "$owner"'*) ;; + *) _p="$_p;an adopted package's home is never initialised" ;; + esac + case "$_blk" in + *'cmd_add_user "$owner" >/dev/null 2>&1'*) + _p="$_p;adoption still hides why a user could not be created" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "an adopted package gets an initialised, owned home" + fi +fi + +# ---- one account, one prefix ----------------------------------------------- # +# A config step used to be `p_cfg_bootscripts`: pkg_owner_name applied the +# PACKAGE prefix to every name regardless of kind, so the account wore its own +# `cfg_` and `p_` on top of it. Neither prefix then identified the account -- +# `p_` said "package" while it lived in /usr/src/cfg -- and every rule that +# reads a prefix had to special-case the stacked form. +# +# The fix is that kind is decided ONCE (pkgusr_kind) and the prefix and the +# root are both answered from it, so a name cannot be given one kind's prefix +# and placed in another kind's root. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ + unprefix_pkg_user pkg_owner_name pkgusr_home_for; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + [ "$(pkg_owner_name cfg_bootscripts)" = "cfg_bootscripts" ] \ + || echo "a config step still gets the package prefix: $(pkg_owner_name cfg_bootscripts)" + [ "$(pkg_owner_name gcc)" = "p_gcc" ] \ + || echo "a package lost its prefix: $(pkg_owner_name gcc)" + # kind decides both answers, so they cannot disagree + [ "$(pkgusr_kind cfg_nsswitch)" = "cfg" ] || echo "a cfg step is not kind cfg" + [ "$(pkgusr_kind gcc)" = "pkg" ] || echo "a package is not kind pkg" + # a tree or a config written before the split holds stacked names; they + # are repaired on the way through rather than carried forward + [ "$(pkg_owner_name p_cfg_bootscripts)" = "cfg_bootscripts" ] \ + || echo "a stacked legacy name survives: $(pkg_owner_name p_cfg_bootscripts)" + # and the whole thing is still idempotent -- that property is what lets + # every call site apply it without knowing which kind of name it holds + for _n in gcc p_gcc cfg_bootscripts p_cfg_bootscripts; do + _a="$(pkg_owner_name "$_n")"; _b="$(pkg_owner_name "$_a")" + [ "$_a" = "$_b" ] || echo "not idempotent: $_n -> $_a -> $_b" + done )" + if [ -z "$_o" ]; then + ok "each kind of account wears exactly one prefix, its own" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# packagemanager must reach the same answer, or the built system looks for +# accounts the build never created +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pm_src" ]; then + python3 - "$pm_src" <<'PYONEPFX' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +ns = {"os": __import__("os"), "BASE_DIR": "/usr/src", + "PKGUSR_SUBDIR": "pkgusr", "CFGUSR_SUBDIR": "cfg", + "_DEFAULT_CONFIG": {"collector_prefix": "sysgroup", "user_prefix": "u", + "pkgusr_prefix": "p", "cfguser_prefix": "cfg", + "main_user": ""}} +ns["load_config"] = lambda: dict(ns["_DEFAULT_CONFIG"]) +for fn in ("collector_prefix", "user_prefix", "pkgusr_prefix", "cfguser_prefix", + "pkgusr_kind", "pkgusr_name", "unprefix_pkgusr", "pkgusr_root", + "pkgusr_home"): + m = re.search(r"^def %s\(.*?(?=\n\ndef |\n\n# |\nPKGUSR_SUBDIR)" % fn, src, re.S | re.M) + if not m: + problems.append("packagemanager has no %s" % fn) + continue + exec(m.group(0), ns) +if not problems: + want = { + # name -> (account name, home) + "gcc": ("p_gcc", "/usr/src/pkgusr/p_gcc"), + "p_gcc": ("p_gcc", "/usr/src/pkgusr/p_gcc"), + "cfg_bootscripts": ("cfg_bootscripts", "/usr/src/cfg/cfg_bootscripts"), + # the stacked legacy form is repaired, not carried forward + "p_cfg_bootscripts": ("cfg_bootscripts", "/usr/src/cfg/cfg_bootscripts"), + "u_firefox": ("u_firefox", "/usr/src/u_firefox"), + } + for given, (name, home) in want.items(): + got = ns["pkgusr_name"](given) + if got != name: + problems.append("packagemanager names %s as %s, want %s" % (given, got, name)) + got = ns["pkgusr_home"](given) + if got != home: + problems.append("packagemanager puts %s at %s, want %s" % (given, got, home)) + again = ns["pkgusr_name"](ns["pkgusr_name"](given)) + if again != ns["pkgusr_name"](given): + problems.append("packagemanager is not idempotent on %s" % given) +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS packagemanager and lfs-helper name each kind identically") +PYONEPFX + _count_rc $? +fi + +# ---- a step that is not a package leaves no home behind --------------------- # +# cmd_build did `mkdir -p "$(pkgusr_home_for "$name")"` for EVERY root step, +# including the ones _is_not_a_package will never give an account. A sanity run +# then found /usr/src/pkgusr/p_init-dirs and p_init-files: directories with no +# matching account, which every ownership pass has to report as strays. +# +# init-*, last-step and refind own no files, so they are staged and run out of +# the state directory instead -- beside the scripts they came from. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( STATE=/.lfs-pkgusr; SCRIPTS="$STATE/scripts" + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for unprefix_pkg_user \ + pkg_owner_name pkgusr_home_for _is_not_a_package step_stage_dir; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + # an empty answer must not read as "not under an account root" + command -v step_stage_dir >/dev/null 2>&1 \ + || echo "there is no step_stage_dir -- every root step still gets a home" + for _s in init-dirs init-files last-step refind; do + _d="$(step_stage_dir "$_s" 2>/dev/null)" + [ -n "$_d" ] || { echo "$_s resolves to no directory at all"; continue; } + case "$_d" in + "$PKGUSR_ROOT"/*|"$CFGUSR_ROOT"/*) + echo "$_s still gets a home under an account root" ;; + esac + done + # a real package is unaffected: it still runs from its user's home + [ "$(step_stage_dir gcc)" = "$PKGUSR_ROOT/p_gcc" ] \ + || echo "a package no longer runs from its own home: $(step_stage_dir gcc)" )" + if [ -z "$_o" ]; then + ok "a step with no account leaves no directory under the account roots" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the staged path, the mkdir, the cd and the failure banner must be ONE + # value. They were four separate calls to pkgusr_home_for, free to disagree. + _cb="$(_slice_fn "$helper_src" cmd_build)" + case "$_cb" in + *'cd "$(pkgusr_home_for'*) + bad "the build still re-derives the directory it cd's into" ;; + *) ok "the staged script, the cd and the banner share one path" ;; + esac +fi + +# ---- the account roots are a decision, not a side effect -------------------- # +# /usr/src/pkgusr and /usr/src/cfg came out root:root 0755 because `mkdir -p` +# on the first home created them with root's umask. Nothing said so, and +# nothing would have noticed it changing. +# +# They carry the same root:install 1775 as /usr/src above them -- one rule for +# the subtree instead of three directories with two rules. The STICKY BIT is +# what makes group-writable safe and is not optional: without it any member of +# `install` could delete or rename another package's home. +# +# They are still NOT install dirs. Being in never_adopt_list is the other +# half: the mode is shared, but no package may ever take one as its own. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _ep="$(_slice_fn "$helper_src" ensure_pkgusr_roots)" + case "$_ep" in + *'chmod 1775'*) ;; + *) _p="$_p;the account roots have no stated mode" ;; + esac + case "$_ep" in + *set_install_dir_owner*) ;; + *) _p="$_p;the account roots do not get the install group" ;; + esac + # sticky, or group-write means any package can delete another's home + case "$_ep" in + *17[0-9][0-9]*) ;; + *) _p="$_p;the account roots are group-writable WITHOUT the sticky bit" ;; + esac + case "$(_slice_fn "$helper_src" init_package_user_home)" in + *ensure_pkgusr_roots*) ;; + *) _p="$_p;the account roots are still created as a side effect" ;; + esac + # never_adopt_list must reach them through pkgusr_roots -- a second literal + # list is one more place to forget when a root is added + case "$(_slice_fn "$helper_src" never_adopt_list)" in + *pkgusr_roots*) ;; + *) _p="$_p;never_adopt_list does not cover the account roots" ;; + esac + # and they must NOT be install dirs + if _slice_fn "$helper_src" install_dirs_list | grep -qx '/usr/src/pkgusr'; then + _p="$_p;the package root is an install dir -- any package user could delete a home" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the account roots are created deliberately and never adopted" + fi +fi + +# ---- /sources is normalised on the way into the chroot ---------------------- # +# Book 3.1 wants $LFS/sources root:root 1777. The host `lfs` account comes from +# book 4.3's own useradd, which pins no uid, so it lands wherever the host had a +# gap -- 10753 on a Debian host. Package users start at 10000, so that uid does +# not merely read as a number inside the chroot, it COLLIDES with a real package +# user and tarballs report as owned by something unrelated. +# +# _normalize_sources was only called from get-sources and restart, so a tree set +# up before it existed kept the host uid forever. Entry is the one path every +# build goes through, whatever it did before. +python3 - "$LFS_TOOL" <<'PYSRCNORM' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +problems = [] +if not m: + problems.append("cmd_bs_chroot is gone") +else: + body = m.group(0) + enter = body.split('if action == "enter"') + if len(enter) < 2: + problems.append("chroot enter is gone") + elif "_normalize_sources" not in enter[1]: + problems.append("nothing normalises /sources on chroot entry") + prep = body.split('if action == "prepare"') + if len(prep) > 1 and "_normalize_sources" not in prep[1].split('if action ==')[0]: + problems.append("nothing normalises /sources after the 7.2 handover") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS /sources is given back to root on the way into the chroot") +PYSRCNORM +_count_rc $? + +# ---- a wrapper must never stand in for a privileged tool -------------------- # +# make_wrappers ships a `chgrp` and a `chown` that print a note and exit 0, so a +# PACKAGE USER cannot change ownership. They are ordinary files on $PATH. If +# that directory is ever ahead of /usr/bin for root, root's own privileged calls +# resolve to them too -- and they report success while doing nothing. +# +# A tree ran 34 packages that way: init-pkgusr said +# # 46 install directories now belong to group 'install' +# with an empty failure list, while every one was still root:root. No package +# user could write to /usr/bin, `verify --fix` could not repair it (its chown +# went through the same wrapper), and nothing said why. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + # the wrappers really do exit 0 without acting -- that is the hazard + _slice_fn "$helper_src" make_wrappers | grep -q 'package users cannot change ownership' \ + || _p="$_p;the chown wrapper no longer explains itself" + # so ownership and mode must go through the binary, by absolute path + for _fn in real_chown real_chgrp real_chmod; do + _slice_fn "$helper_src" "$_fn" | grep -q . \ + || _p="$_p;there is no $_fn -- a wrapper can shadow root's own calls" + done + _slice_fn "$helper_src" _real_tool | grep -q '/usr/bin/' \ + || _p="$_p;_real_tool does not resolve an absolute path" + # and no privileged pass may call the bare command any more + for _fn in cmd_init_pkgusr init_package_user_home _vfy_install_dirs ensure_pkgusr_roots; do + # only where the tool is INVOKED -- start of a command, or after + # &&/||/;/!. Matching it anywhere caught the warning text that explains + # the hazard, which would have made this test impossible to satisfy. + if _slice_fn "$helper_src" "$_fn" \ + | grep -vE '^[[:space:]]*#' \ + | grep -qE '(^[[:space:]]*|&&[[:space:]]*|\|\|[[:space:]]*|;[[:space:]]*|![[:space:]]*)(chown|chgrp|chmod) '; then + _p="$_p;$_fn still calls a bare chown/chgrp/chmod" + fi + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "root's ownership calls cannot resolve to a package-user wrapper" + fi + + # exit 0 is not proof. init-pkgusr must read the group back. + _blk="$(_slice_fn "$helper_src" cmd_init_pkgusr)" + _p="" + # owner AND group: the rule is root:install, and reading back only the group + # is how /usr/share sat at lfs:install without anything noticing + case "$_blk" in + *'stat -c %U:%G'*) ;; + *) _p="$_p;init-pkgusr trusts the exit code instead of reading the result back" ;; + esac + case "$_blk" in + *'NO install directories'*) ;; + *) _p="$_p;init-pkgusr reports zero install directories as success" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "init-pkgusr confirms the install group took, and zero is an error" + fi +fi + +# ---- an empty override must not disable the whole scheme -------------------- # +# install_dirs_list is overridable through $INSTALLDIRS and returned early when +# the file existed. An override yielding nothing meant NO directory was given +# to the install group, NO directory was protected from adoption, and `verify` +# reported a clean tree because it walked the same empty list. Nine callers, +# one silent no-op. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( STATE=/tmp/_idl.$$; INSTALLDIRS="$STATE/installdirs.lst" + mkdir -p "$STATE" + eval "$(_slice_fn "$helper_src" install_dirs_list)" + warn() { :; } + # no override at all -> the built-in list + [ "$(install_dirs_list 2>/dev/null | grep -c .)" -gt 10 ] \ + || echo "the built-in list is empty" + # an override with only comments and blanks -> fall back, never nothing + printf '# just a comment\n\n \n' > "$INSTALLDIRS" + [ "$(install_dirs_list 2>/dev/null | grep -c .)" -gt 10 ] \ + || echo "an empty override silently disables every install directory" + # a real override is still honoured + printf '/usr/bin\n/usr/lib\n' > "$INSTALLDIRS" + [ "$(install_dirs_list 2>/dev/null | grep -c .)" = 2 ] \ + || echo "a real override is no longer honoured" + rm -rf "$STATE" )" + if [ -z "$_o" ]; then + ok "an override that lists nothing falls back to the built-in list" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- an option is not a package name ---------------------------------------- # +# /usr/src/pkgusr/p_--version turned up in a sanity run: a home for an account +# that could never exist, left by something that reached add-user with +# "--version" in the name position. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _slice_fn "$helper_src" _refuse_option_as_name | grep -q 'looks like an option' \ + && case "$(_slice_fn "$helper_src" cmd_add_user)" in + *_refuse_option_as_name*) ok "a name starting with a dash is refused, not made into a user" ;; + *) bad "add-user does not refuse an option-shaped name" ;; + esac \ + || bad "there is no guard against an option being used as a package name" +fi + +# ---- the sanity report checks the group it prints --------------------------- # +# Section 1 printed owner:group from the start and asserted only the mode. A +# tree ran 34 packages with every install directory at root:root 775 -- +# group-writable, but group `root`, while package users are in `install`. The +# column that would have shown it was decoration. +san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$san" ]; then + _p="" + grep -q "want 'install'" "$san" \ + || _p="$_p;the sanity report does not check the install group" + grep -q "a package has taken a shared directory" "$san" \ + || _p="$_p;the sanity report does not check who owns a shared directory" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the sanity report asserts the install-dir owner and group" + fi +fi + +# ---- one wrapper directory, not two ----------------------------------------- # +# make_wrappers wrote to $STATE/wrappers while /etc/pkgusr/bash_profile -- and +# packagemanager's copy of it -- put /usr/lib/pkgusr first on a package user's +# PATH. Nothing created that, so a package user's login shell led with a +# directory that did not exist, and the wrappers only ever applied because +# cmd_build overrides PATH on the command line. Two names for one decision. +# +# /usr/lib/pkgusr is the survivor: $STATE is build state, and packagemanager on +# the finished system hands its users the same profile. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '^WRAPPERS="\${LFS_WRAPPERS:-\${SNAP_ROOT%/}/usr/lib/pkgusr}"' "$helper_src" \ + || _p="$_p;the wrapper directory is not /usr/lib/pkgusr" + grep -q 'WRAPPERS="\$STATE/wrappers"' "$helper_src" \ + && _p="$_p;the old \$STATE/wrappers definition is still there" + # the profile must not name the directory a second time + case "$(_slice_fn "$helper_src" ensure_pkgusr_etc)" in + *'PATH=/usr/lib/pkgusr:'*) + _p="$_p;the profile still hardcodes the wrapper path instead of using \$WRAPPERS" ;; + esac + case "$(_slice_fn "$helper_src" ensure_pkgusr_etc)" in + *'s|@WRAPPERS@|$WRAPPERS|'*) ;; + *) _p="$_p;the profile's PATH is not filled in from \$WRAPPERS" ;; + esac + # and it must be protected: under /usr/lib, which IS an install dir, so + # without this any package user could rewrite the wrappers that constrain it + case "$(_slice_fn "$helper_src" never_adopt_list)" in + *'"$WRAPPERS"'*) ;; + *) _p="$_p;the wrapper directory can be adopted or given to the install group" ;; + esac + case "$(_slice_fn "$helper_src" make_wrappers)" in + *'chown root:root "$WRAPPERS"'*) ;; + *) _p="$_p;the wrappers are not explicitly root-owned" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "there is one wrapper directory, and it outlives the build" + fi +fi + +# packagemanager builds the same profile for the finished system, so it must +# reach the same path -- a second hardcoded copy is how these drifted +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pm_src" ]; then + _p="" + grep -q '^WRAPPERS = os.environ.get("LFS_WRAPPERS", "/usr/lib/pkgusr")' "$pm_src" \ + || _p="$_p;packagemanager has no WRAPPERS setting" + grep -q 'PATH=/usr/lib/pkgusr:' "$pm_src" \ + && _p="$_p;packagemanager still hardcodes the wrapper path in the profile" + grep -q '_PKGUSR_BASH_PROFILE.replace("@WRAPPERS@", WRAPPERS)' "$pm_src" \ + || _p="$_p;packagemanager does not fill the wrapper path into the profile" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both tools write a profile pointing at the same wrapper directory" + fi +fi + +# ---- an install directory is root:install, owner AND group ------------------- # +# init-pkgusr set only the GROUP, while _vfy_install_dirs demands root:install +# and `verify --fix` applies both. Two rules for one directory. +# +# They disagreed on a real build: chapters 5-6 build as the `lfs` user, and book +# 7.2's handover only reaches one level below $LFS, so /usr/share stayed +# lfs-owned and came out +# lfs:install drwxrwxr-x /usr/share +# The group looked right and nothing said the owner was not. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _blk="$(_slice_fn "$helper_src" cmd_init_pkgusr)" + _p="" + # owner AND group -- through the shared rule, which applies both by number. + # This used to look for `real_chown root:install`, which is the by-name form + # that cannot work before book 7.6 writes /etc/passwd. + case "$_blk" in + *'set_install_dir_owner "$d"'*) ;; + *) _p="$_p;init-pkgusr sets the group but not the owner of an install directory" ;; + esac + case "$_blk" in + *'stat -c %U:%G'*) ;; + *) _p="$_p;init-pkgusr reads back only the group, so a wrong owner goes unnoticed" ;; + esac + # the two rules must want the same thing + case "$(_slice_fn "$helper_src" _vfy_install_dirs)" in + *root:install*) ;; + *) _p="$_p;verify no longer wants root:install" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "init-pkgusr and verify agree that an install dir is root:install" + fi +fi + +# ---- every tool we install gets an owner ------------------------------------- # +# install_helper copies lfs/lfs-helper/packagemanager/blfs and calls +# _chown_tools. _install_pkgusr_helpers copied the hint's scripts beside them +# and never did -- so /usr/bin/list_package, uninstall_package, +# forall_direntries_from and the rest came out root:root while the four tools +# next to them belonged to p_pkgusr. Two copy routines, one claiming ownership. +# +# The WRAPPER directory is the deliberate exception: it stays root:root, because +# a package user that could rewrite chown or install could rewrite the rule that +# constrains it. +python3 - "$LFS_TOOL" <<'PYHELPEROWN' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def _install_pkgusr_helpers\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_install_pkgusr_helpers is gone") +else: + body = m.group(0) + if "_chown_tools" not in body: + problems.append("the hint's helpers are copied but never given an owner") + if "wrapper_sub" not in body: + problems.append("the wrapper directory is not excluded from the chown") +if not re.search(r'^WRAPPERS = os\.environ\.get\("LFS_WRAPPERS"', src, re.M): + problems.append("lfs does not read the shared WRAPPERS setting") +# and the helper table must not spell the wrapper path out a fourth time +tbl = re.search(r"PKGUSR_HELPERS = \[(.*?)\n\]", src, re.S) +if tbl and '"usr/lib/pkgusr"' in tbl.group(1): + problems.append("PKGUSR_HELPERS still hardcodes the wrapper path") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS every tool installed into the tree is given an owner") +PYHELPEROWN +_count_rc $? + +# ---- scratch never enters a manifest ---------------------------------------- # +# $BUILD_ROOT is the busiest directory on the system during a build: unpacked +# source trees, object files, the tarballs themselves. Every one is newer than +# the build stamp, so the touched-file scan swallowed the whole scratch tree +# into every package's manifest and then chowned it -- and the next package +# chowned it right back. A real `verify` came back with 3363 wrong-owner, of +# which 3348 were /build: +# /build/bash-5.3.tar.gz is p_shadow want p_acl +# The two snapshot scans already excluded it; the build's own scan did not. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cb="$(_slice_fn "$helper_src" cmd_build)" + _p="" + case "$_cb" in + *'-not -path "$_bld/*"'*) ;; + *) _p="$_p;the build scratch is still scanned into every manifest" ;; + esac + case "$_cb" in + *'BUILD_ROOT:-$r/build'*) ;; + *) _p="$_p;the scratch exclusion does not follow \$BUILD_ROOT" ;; + esac + # and verify must ignore scratch left in manifests written before this + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in + *'BUILD_ROOT'*) ;; + *) _p="$_p;verify still reports scratch recorded by an older build" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build scratch is never recorded as a package's files" + fi +fi + +# ---- some files belong to no package ---------------------------------------- # +# Every package that installs an info page rewrites /usr/share/info/dir, and +# ldconfig rewrites /etc/ld.so.cache after every install. Recording those in a +# manifest makes verify report a conflict no repair can settle -- the next +# package takes them straight back: +# /usr/share/info/dir is p_gcc want p_binutils +# /etc/ld.so.cache is p_libcap want p_glibc +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SNAP_ROOT=/ + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + for f in /usr/share/info/dir /etc/ld.so.cache /var/cache/ldconfig/aux-cache; do + is_never_claimed "$f" || echo "$f can still be claimed by a package" + done + # the wrapper directory too: it is root:root on purpose + is_never_claimed /usr/lib/pkgusr/chown \ + || echo "a wrapper can be claimed by a package" + # but a real package file must NOT be swept up by this + is_never_claimed /usr/bin/bash \ + && echo "an ordinary package file is treated as unclaimable" )" + if [ -z "$_o" ]; then + ok "shared indexes and generated caches are owned by no package" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- the 7.2 handover is decided by the tree, not one inode ------------------ # +# `_owner_is_root` looks at $LFS/usr. The pass it guarded is +# `chown -h -R root:root` over EIGHT trees. A tree whose /usr was already +# root-owned got no handover at all, and everything chapters 5-6 created below +# it kept the host uid: +# drwxr-xr-x 1 lfs lfs /usr/share/gcc-15.2.0/python/libstdcxx +# lfs:install drwxrwxr-x /usr/share +# It must also refuse to run once package users own files -- chown -R root:root +# over /usr is right before chapter 8 and catastrophic after it. +python3 - "$LFS_TOOL" <<'PYHANDOVER' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +for fn in ("_handover_needed", "_handover_is_safe"): + if not re.search(r"^def %s\(" % fn, src, re.M): + problems.append("there is no %s" % fn) +m = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) +if m and "_find_uid" not in m.group(0): + problems.append("_handover_needed still decides from a single directory") +m = re.search(r"^def _handover_is_safe\(.*?(?=\n\ndef )", src, re.S | re.M) +# It must ask the TREE's passwd which accounts are package users, not infer it +# from a uid range: book 4.3 pins no uid for `lfs`, so on one host the build +# account came out 10753 -- inside the package range -- and a range filter +# excluded the very user it was looking for. +if m and "_pkg_users_in_tree" not in m.group(0): + problems.append("_handover_is_safe infers package users from a uid range") +if not re.search(r"^def _pkg_users_in_tree\(", src, re.M): + problems.append("there is no _pkg_users_in_tree") +m2 = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) +if m2 and "PKG_UID_MIN" in m2.group(0): + problems.append("_handover_needed still filters the build user out by uid range") +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + if "_handover_needed" not in body: + problems.append("chroot prepare still guards 7.2 on _owner_is_root alone") + if "_handover_is_safe" not in body: + problems.append("chroot prepare would run chown -R over a chapter-8 tree") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the 7.2 handover is decided by the tree and refuses a built one") +PYHANDOVER +_count_rc $? + +# and a tree that already has the damage must be repairable +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _vfy_build_user_leftovers | grep -q '\-user lfs' \ + || _p="$_p;verify cannot find files the build user still owns" + _slice_fn "$helper_src" _vfy_build_user_leftovers | grep -q 'real_chown -h root:root' \ + || _p="$_p;verify --fix cannot hand the build user's leftovers back to root" + case "$(_slice_fn "$helper_src" cmd_verify)" in + *_vfy_build_user_leftovers*) ;; + *) _p="$_p;verify does not run the build-user pass" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a tree the handover skipped can be repaired from inside the chroot" + fi +fi + +# ---- every function called is a function that exists ------------------------ # +# `verify --fix` called cmd_fix_orphans, which does not exist -- `fix-orphans` +# is dispatched as an ALIAS for `verify --fix`, so the call was both undefined +# and circular. It died mid-repair: +# line 4793: cmd_fix_orphans: command not found +# after chowning 3473 paths, leaving the pass half-done and the status wrong. +# +# A missing function in the REPAIR path is the worst place for one: it only +# fires when something is already broken, which is when it is least welcome. +for _t in lfs-helper; do + _src="$(dirname "$LFS_TOOL")/$_t" + [ -f "$_src" ] || continue + _missing="" + # comments stripped first: the fix for this bug NAMES the function it + # removed, and matching that text would make the test permanently red + for _fn in $(grep -vE '^[[:space:]]*#' "$_src" \ + | grep -oE '(^|[^_[:alnum:]])cmd_[a-z0-9_]+' \ + | grep -oE 'cmd_[a-z0-9_]+' | sort -u); do + grep -qE "^${_fn}\(\) \{" "$_src" || _missing="$_missing $_fn" + done + if [ -n "$_missing" ]; then + for _m in $_missing; do bad "$_t calls $_m, which is not defined"; done + else + ok "$_t defines every cmd_* function it calls" + fi +done + +# ---- a package builds in its own home, not a shared scratch ----------------- # +# /build was a permanent ownership fight. Every build's touched-file scan swept +# the whole tree in, so each manifest claimed every other package's sources and +# chowned them; the next package chowned them back. Repairing it made it worse, +# because the question had no answer: +# /build/bash-5.3.tar.gz is p_shadow want p_acl +# /build/bash-5.3.tar.gz is p_zstd want p_acl (after the repair) +# +# Under the package user's home it is settled by construction: the user creates +# the tree so the user owns it, and $SRCROOT is already excluded from every tree +# scan, so it can never reach a manifest at all. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SNAP_ROOT=/; SRCROOT=/usr/src + PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + STATE=/.lfs-pkgusr; SCRIPTS="$STATE/scripts" + PKGUSR_BUILD_SUBDIR=src + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for unprefix_pkg_user \ + pkg_owner_name pkgusr_home_for _is_not_a_package build_root_for; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + if ! command -v build_root_for >/dev/null 2>&1; then + echo "there is no build_root_for -- the scratch is still shared" + exit 0 # the rest would be noise, not findings + fi + # a package builds under its own account's home + [ "$(build_root_for gcc)" = "/usr/src/pkgusr/p_gcc/$PKGUSR_BUILD_SUBDIR" ] \ + || echo "gcc does not build in its own home: $(build_root_for gcc)" + # an earlier stage folds onto the same package, so it reuses that tree + [ "$(build_root_for util-linux-tmp)" = "$(build_root_for util-linux)" ] \ + || echo "an earlier stage builds somewhere else than its package" + # and it must be under $SRCROOT, which every tree scan already excludes -- + # that is what keeps it out of manifests without another find exclusion + case "$(build_root_for gcc)" in + "$SRCROOT"/*) ;; + *) echo "the build tree is not under SRCROOT, so scans will sweep it in" ;; + esac + # a step with no account keeps the shared scratch: it runs as root, which + # owns everything there anyway + case "$(build_root_for init-dirs)" in + /usr/src/*) echo "a step with no account was given a home to build in" ;; + esac )" + if [ -z "$_o" ]; then + ok "each package unpacks and builds inside its own account's home" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # and the build must actually be told about it + case "$(_slice_fn "$helper_src" cmd_build)" in + *"BUILD_ROOT='\$_bldroot'"*) + ok "the per-package build root is passed into the build" ;; + *) bad "the build still inherits whatever BUILD_ROOT it was started with" ;; + esac +fi + +# ---- the root symlinks are chowned, not followed ---------------------------- # +# Book 4.2 creates /bin -> usr/bin, /lib and /sbin as the build user. They came +# out lfs:lfs, and a chown WITHOUT -h follows the link and retargets /usr/bin +# instead -- silently doing the wrong thing to the right-looking path. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _blk="$(_slice_fn "$helper_src" _vfy_build_user_leftovers)" + _p="" + case "$_blk" in + *'real_chown -h'*) ;; + *) _p="$_p;the build-user repair follows symlinks instead of chowning them" ;; + esac + case "$(printf '%s\n' "$_blk" | grep -vE '^[[:space:]]*#')" in + *'-type f'*) _p="$_p;the build-user scan skips symlinks and directories" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a symlink owned by the build user is chowned, not followed" + fi +fi + +# ---- the tree can be set up before book 7.6 exists -------------------------- # +# A chapter 5-6 snapshot restored before 7.6 has NO /etc/passwd -- the chroot +# prompt says "I have no name!". `chown root:install` needs `root` to resolve, +# so every call failed and init-pkgusr reported: +# # 0 install directories are now root:install (group-writable) +# # could not set the group on: /usr /usr/bin /usr/lib /usr/share ... +# while ensure_install_dirs_writable used `chgrp` and worked fine. Two rules +# for one directory, and only one of them survived an empty passwd file. +# +# Ownership is stored as a NUMBER, so use the number. +_p="" +_pre="$T/pre76" +mkdir -p "$_pre/etc" "$_pre/usr/share/man/man3" +: > "$_pre/etc/passwd" # exists, but empty: no root entry +printf 'install:x:9999:\n' > "$_pre/etc/group" +# the shape that broke: a name that cannot resolve +if chown root:install "$_pre/usr/share/man/man3" 2>/dev/null; then + : # this host HAS a root and an install group, so the fixture proves nothing +else + # numeric must work where the name did not + if chown -h 0:9999 "$_pre/usr/share/man/man3" 2>/dev/null; then + [ "$(stat -c '%u:%g' "$_pre/usr/share/man/man3")" = "0:9999" ] \ + || _p="$_p;a numeric chown did not take" + else + _p="$_p;even a numeric chown fails here -- fixture is not usable" + fi +fi +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # one helper, used by all three passes that set an install directory + _slice_fn "$helper_src" set_install_dir_owner | grep -q '0:\$INSTALL_GID' \ + || _p="$_p;the install-dir owner is not set by number" + _slice_fn "$helper_src" install_dir_owner_ok | grep -q "%u:%g" \ + || _p="$_p;the install-dir check reads names, which are UNKNOWN before 7.6" + for _fn in cmd_init_pkgusr ensure_install_dirs_writable _vfy_install_dirs; do + _slice_fn "$helper_src" "$_fn" | grep -q 'set_install_dir_owner' \ + || _p="$_p;$_fn does not use the shared install-dir rule" + done + # and no pass may still reach for the names + for _fn in cmd_init_pkgusr ensure_install_dirs_writable _vfy_install_dirs; do + _slice_fn "$helper_src" "$_fn" | grep -vE '^[[:space:]]*#' \ + | grep -qE 'chown root:install|chgrp install ' \ + && _p="$_p;$_fn still sets an install dir by name" + done +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "install directories are set by number, so a tree with no passwd works" +fi + +# ---- nothing is an orphan before there is a passwd file --------------------- # +# `find -nouser` asks whether a uid resolves. With no passwd file NOTHING +# resolves, so every file in the tree is an "orphan". On a restored chapter 5-6 +# tree that reported +# noowner / (no passwd entry) -> lfs +# noowner /dev (no passwd entry) -> lfs +# ... and 11797 more with no owner +# and `--fix` would have handed the ENTIRE tree -- root's own directories +# included -- to the build user in one pass. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do + _slice_fn "$helper_src" "$_fn" | grep -q '_user_db_ready' \ + || _p="$_p;$_fn runs before the user database exists" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the ownership passes wait for book 7.6 to write /etc/passwd" + fi +fi + +# ---- nothing privileged may need a name to resolve -------------------------- # +# init-pkgusr runs BEFORE book 7.6 creates /etc/passwd. The prompt in there +# literally reads "I have no name!" because not even root resolves. So +# `chown root:install` fails on every directory: +# # 0 install directories are now root:install (group-writable) +# # could not set the group on: /bin /sbin /lib /lib64 /etc /usr ... +# and every package user is then locked out of /usr/bin for the whole build. +# +# Ownership is stored as a NUMBER. uid 0 is root whether anything says so or +# not, and the install gid is ours -- init-pkgusr writes /etc/group by hand, so +# the group can exist long before any passwd entry does. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" set_install_dir_owner | grep -q '0:\$INSTALL_GID' \ + || _p="$_p;an install directory is set by name, which fails before 7.6" + _slice_fn "$helper_src" install_dir_owner_ok | grep -q "%u:%g" \ + || _p="$_p;the readback uses names, so a correct directory reads as wrong" + # No privileged pass may reach for the NAMES. Match the INVOCATION, not the + # string: these functions legitimately print "now root:install" and carry + # trailing comments saying the same, and matching those made the test + # impossible to satisfy while changing nothing about the behaviour. + for _fn in cmd_init_pkgusr _vfy_install_dirs ensure_install_dirs_writable; do + _slice_fn "$helper_src" "$_fn" | grep -vE '^[[:space:]]*#' \ + | grep -qE '(real_)?(chown|chgrp)( +-[a-zA-Z]+)* +(root:install|install) ' \ + && _p="$_p;$_fn still sets the group by name, which fails before 7.6" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "install directories are set by number, so 7.6 is not a prerequisite" + fi + + # and the orphan pass must not fire before there is a database to be + # orphaned FROM. With no passwd file `find -nouser` matches everything: a + # real run reported 11802 orphans including / and /proc, and --fix would have + # handed the entire tree to the build user in one pass. + _p="" + case "$(_slice_fn "$helper_src" _vfy_orphans)" in + *_user_db_ready*) ;; + *) _p="$_p;the orphan pass would hand the whole tree to the build user before 7.6" ;; + esac + case "$(_slice_fn "$helper_src" _has_orphaned_files)" in + *_user_db_ready*) ;; + *) _p="$_p;a tree with no passwd file is reported as entirely orphaned" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "an orphan needs a user database to be missing from" + fi +fi + +# ---- the build user's uid is not a range ------------------------------------ # +# Book 4.3's useradd pins no uid, so the host's `lfs` account lands wherever the +# host had a gap -- 10753 on a Debian host, INSIDE the package-user range. Any +# rule that identifies the build user by "uid outside 10000+" therefore excludes +# the very account it is looking for. That is the same mistake _normalize_sources +# exists to correct, and _handover_needed walked straight into it: it reported +# # 7.2 already done (tree owned by root) +# while /bin -> usr/bin was still lfs:lfs. +python3 - "$LFS_TOOL" <<'PYUIDRANGE' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_handover_needed is gone") +else: + body = re.sub(r"^\s*#.*$", "", m.group(0), flags=re.M) + if "PKG_UID_MIN" in body: + problems.append("_handover_needed still identifies the build user by uid range") +m = re.search(r"^def _handover_is_safe\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_handover_is_safe is gone") +else: + body = re.sub(r"^\s*#.*$", "", m.group(0), flags=re.M) + if "_pkg_users_in_tree" not in body: + problems.append("_handover_is_safe does not read the tree's own passwd") +if not re.search(r"^def _pkg_users_in_tree\(", src, re.M): + problems.append("there is no _pkg_users_in_tree") +else: + m = re.search(r"^def _pkg_users_in_tree\(.*?(?=\n\ndef |\Z)", src, re.S | re.M) + if m and "pkgusr_prefix" not in m.group(0): + problems.append("_pkg_users_in_tree does not identify accounts by prefix") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the build user is identified by the tree, never by a uid range") +PYUIDRANGE +_count_rc $? + +# ---- one step where ownership becomes real ---------------------------------- # +# Before book 7.6 there is no /etc/passwd, so no name resolves -- not even +# root's; the prompt in the chroot literally reads "I have no name!". Every +# ownership question asked before that point gets a confident wrong answer: +# +# chown root:install fails on every directory -> "0 install directories" +# find -nouser matches every file in the tree -> 11802 "orphans" +# +# Three separate bugs, one cause: work that needs a user database was being done +# before there was one. So there is one gate now. Nothing touches ownership +# before it; everything maintains ownership after it. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in ownership_established ownership_possible establish_ownership \ + _ownership_checkpoint cmd_establish_ownership; do + _slice_fn "$helper_src" "$_fn" | grep -q . \ + || _p="$_p;there is no $_fn" + done + # the gate must test the user database and the tools it needs + _op="$(_slice_fn "$helper_src" ownership_possible)" + case "$_op" in + *_user_db_ready*) ;; + *) _p="$_p;the gate does not wait for the user database" ;; + esac + case "$_op" in + *chown*) ;; + *) _p="$_p;the gate does not check that chown is available" ;; + esac + # it must be durable, so a resumed or restored build does not repeat it + grep -q '^OWNERSHIP_MARK=' "$helper_src" \ + || _p="$_p;crossing the epoch is not recorded, so it repeats" + case "$(_slice_fn "$helper_src" establish_ownership)" in + *'> "$OWNERSHIP_MARK"'*) ;; + *) _p="$_p;establish_ownership never writes its marker" ;; + esac + # and everything ownership-related must be inside it, not scattered + _eo="$(_slice_fn "$helper_src" establish_ownership)" + for _need in cmd_init_pkgusr _vfy_build_user_leftovers _vfy_manifest_ownership \ + _adopt_pkgusr_tools _vfy_orphans; do + case "$_eo" in + *"$_need"*) ;; + *) _p="$_p;$_need does not happen at the ownership epoch" ;; + esac + done + # reachable by hand as well as automatically + grep -q 'establish-ownership)' "$helper_src" \ + || _p="$_p;there is no way to run the ownership step by hand" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "ownership is established once, at one step, and recorded" + fi + + # the passes that need a user database must refuse to run without one + _p="" + for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do + _slice_fn "$helper_src" "$_fn" | grep -q '_user_db_ready' \ + || _p="$_p;$_fn would run before there is a user database" + done + case "$(_slice_fn "$helper_src" _vfy_install_dirs)" in + *ownership_*) ;; + *) _p="$_p;verify would report 27 wrong install dirs on a chapter 5-6 tree" ;; + esac + # build-all must NOT set the package-user system up before the epoch -- + # that is what printed "0 install directories are now root:install" + case "$(_slice_fn "$helper_src" cmd_build_all)" in + *ownership_established*) ;; + *) _p="$_p;build-all still sets up ownership before 7.6 has run" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "nothing asks who owns what before there is a user database" + fi +fi + +# ---- the build tree must not collide with the skeleton ---------------------- # +# init_package_user_home symlinks .bash_profile, .bashrc and `build` into every +# package user's home -- `build` being the hint's build helper script. The +# per-package build tree was first called `build` too, so `mkdir -p ~/build` hit +# the symlink and the build died two lines later complaining about the wrong +# thing: +# mkdir: cannot create directory '.../p_gettext/build': File exists +# cd: /usr/src/pkgusr/p_gettext/build: Not a directory +# The recovery code then read that as a permissions problem and retried twice +# before advising collector groups. None of it was true. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" pkgusr_skel_links | grep -q . \ + || _p="$_p;the skeleton links are not named in one place" + # the home skeleton must be built FROM that list, not from a second copy + case "$(_slice_fn "$helper_src" init_package_user_home)" in + *pkgusr_skel_links*) ;; + *) _p="$_p;init_package_user_home keeps its own copy of the link list" ;; + esac + # and the build subdir must not be one of them + _sub="$(grep -m1 '^PKGUSR_BUILD_SUBDIR=' "$helper_src" \ + | sed 's/.*:-\([a-z_]*\)}.*/\1/')" + if [ -z "$_sub" ]; then + _p="$_p;there is no PKGUSR_BUILD_SUBDIR" + else + for _l in $(_slice_fn "$helper_src" pkgusr_skel_links \ + | grep -oE "'[^']+'|[a-z._:]+" | tr ' ' '\n'); do + case "$_l" in + "$_sub":*|"$_sub") _p="$_p;the build tree is called '$_sub', which the home skeleton already links" ;; + esac + done + fi + # and a name that is taken must be reported HERE, not as a cd failure later + case "$(_slice_fn "$helper_src" cmd_build)" in + *'is not a directory'*) ;; + *) _p="$_p;a build tree that is not a directory fails somewhere else" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build tree cannot collide with the package user's home skeleton" + fi +fi + +# ---- no function is defined twice ------------------------------------------ # +# `cmd_verify` was defined TWICE. bash keeps the last, so the tidy definition +# that called the _vfy_* helpers had never run -- and the surviving one carried +# its own inline copies of the manifest and install-dir rules, without any of +# the guards the helpers had grown. +# +# A finished 104/104 build therefore reported three things the real pass knows +# to leave alone, and --fix would have acted on all three: +# /usr/bin/hostname is p_inetutils want p_coreutils (inetutils owns it) +# /usr/share/info/dir is p_e2fsprogs want p_glibc (nobody owns it) +# /etc/group- is root want p_ncurses (useradd wrote it) +# +# A duplicate definition is invisible: both parse, one silently wins. +for _t in lfs-helper; do + _src="$(dirname "$LFS_TOOL")/$_t" + [ -f "$_src" ] || continue + _dups="$(grep -oE '^[a-z_][a-z0-9_]*\(\) \{' "$_src" | sort | uniq -d)" + if [ -n "$_dups" ]; then + printf '%s\n' "$_dups" | while IFS= read -r _d; do + bad "$_t defines ${_d%\(\) \{} more than once -- one of them never runs" + done + else + ok "$_t defines each function exactly once" + fi +done + +# and verify must USE the helpers rather than carry its own copy of the rules +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cv="$(_slice_fn "$helper_src" cmd_verify)" + _p="" + for _need in _vfy_install_dirs _vfy_manifest_ownership _vfy_orphans \ + _vfy_user_homes _vfy_build_user_leftovers _vfy_user_order; do + case "$_cv" in + *"$_need"*) ;; + *) _p="$_p;verify does not run $_need" ;; + esac + done + # no second copy of the ownership rule inline + case "$(printf '%s\n' "$_cv" | grep -vE '^[[:space:]]*#')" in + *'want $owner'*) _p="$_p;verify still carries its own copy of the ownership rule" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "verify runs the one implementation of each check" + fi +fi + +# ---- a collector group is named for the package ----------------------------- # +# It took the ACCOUNT name, so owner `p_perl` produced `nimgnu_p_perl` -- two +# prefixes, and the wrong two: `nimgnu_` says collector group, `p_` says package +# user, which it is not. The prompt offered it as a choice: +# 1) nimgnu_p_perl (everything owned by 'p_perl' -- fewer groups) +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( COLLECTOR_PREFIX=nimgnu; PKGUSR_PREFIX=p_ + eval "$(_slice_fn "$helper_src" unprefix_pkg_user)" + eval "$(_slice_fn "$helper_src" collector_group_for)" + [ "$(collector_group_for p_perl)" = "nimgnu_perl" ] \ + || echo "a collector group still carries the package-user prefix: $(collector_group_for p_perl)" + # a bare package name is unchanged + [ "$(collector_group_for perl)" = "nimgnu_perl" ] \ + || echo "a bare package name does not reach the same group" + # a config step keeps its own prefix -- cfg_ is part of the step's name + [ "$(collector_group_for cfg_bootscripts)" = "nimgnu_cfg_bootscripts" ] \ + || echo "a config step's group name lost its kind: $(collector_group_for cfg_bootscripts)" )" + if [ -z "$_o" ]; then + ok "a collector group carries the collector prefix and no other" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- root's own files are root's ------------------------------------------- # +# Every cmd_add_user during a build writes /etc/passwd, /etc/group and Shadow's +# backups of them. Whichever package happened to be building at the time swept +# them into its manifest: +# /etc/.pwd.lock is root want p_ncurses +# /etc/group- is root want p_ncurses +# /etc/passwd- is root want p_psmisc +# ncurses did not install /etc/group-; useradd did, while ncurses was building. +# A package owning the file that says which packages exist is the one ownership +# mistake with no way back. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( ETC=/etc; SNAP_ROOT=/ + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + for f in /etc/passwd /etc/passwd- /etc/group /etc/group- \ + /etc/shadow /etc/gshadow /etc/.pwd.lock; do + is_never_claimed "$f" || echo "$f can be claimed by a package" + done + # ordinary configuration is still a package's to own + is_never_claimed /etc/fstab \ + && echo "an ordinary config file was swept into the never-claim list" )" + if [ -z "$_o" ]; then + ok "the user database belongs to root and to no package" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the tools we install are ours, in bin AND sbin + _p="" + grep -q 'add_package_user install_package' "$helper_src" \ + || _p="$_p;the hint's helper scripts are not claimed by p_pkgusr" + case "$(_slice_fn "$helper_src" _adopt_pkgusr_tools)" in + *'/usr/bin /usr/sbin'*) ;; + *) _p="$_p;only /usr/bin is searched, so the sbin helpers stay unclaimed" ;; + esac + # and the wrappers must not be reported as needing an owner + case "$(_slice_fn "$helper_src" _vfy_unclaimed)" in + *is_never_claimed*) ;; + *) _p="$_p;verify asks who should own the wrappers, which are root's by design" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "our own tools get an owner; the wrappers stay root's" + fi +fi + +# ---- book 4.3 must not run on a built tree ---------------------------------- # +# `_chown_tree_to_lfs` is `chown -R lfs` over /usr, /etc, /var and the rest. +# Before chapter 5 that is book 4.3 and exactly right. On a BUILT tree it hands +# the finished system to the host's build account in one pass, and every +# package's ownership is gone -- recoverable only from the manifests. +# +# It happened: `lfs build-system run` on a 104/104 tree replayed step 4.2 and +# drwxrwxr-t 1 lfs install /mnt/lfs/usr +# came back with the host uid 10753 written through the tree. Inside the chroot +# that resolves to nothing -- the chroot's own `lfs` is 9998 -- so `ls -al /` +# showed a raw number where every owner should have been. +python3 - "$LFS_TOOL" <<'PY43' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def _chown_tree_to_lfs\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_chown_tree_to_lfs is gone") +else: + body = m.group(0) + if "_handover_is_safe" not in body: + problems.append("book 4.3 would chown a built tree to the build user") + # the guard has to come BEFORE the chown, or it reports and then does it + # the INVOCATION, not the comment above the guard that names the command + g = body.find("if not _handover_is_safe") + c = body.find("_run_bash('chown -R lfs") + if c < 0: + problems.append("the 4.3 chown is no longer recognisable") + elif g < 0 or g > c: + problems.append("the 4.3 guard runs after the chown it is guarding") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS book 4.3 refuses to hand a built tree back to the build user") +PY43 +_count_rc $? + +# ---- the ownership step is visible ------------------------------------------ # +# It ran as an invisible checkpoint inside build-all: right place, right work, +# but it never appeared in `lfs-helper list`. The one moment the whole scheme +# turns on was the one moment you could not point at, resume from, or re-run. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + # lfs puts it in the step order, straight after book 7.6 + grep -q 'order.insert(order.index("init-files") + 1, "init-ownership")' "$LFS_TOOL" \ + || _p="$_p;init-ownership is not in the step order" + # lfs-helper performs it itself -- there is no script for it in the book + case "$(_slice_fn "$helper_src" cmd_build)" in + *'"$name" = "init-ownership"'*) ;; + *) _p="$_p;lfs-helper would look for a book script for init-ownership" ;; + esac + # it owns no files, so it gets no account and no home + case "$(_slice_fn "$helper_src" _is_not_a_package)" in + *init-ownership*) ;; + *) _p="$_p;init-ownership would be given a package user" ;; + esac + # and running it must record itself, so list and next agree with reality + case "$(_slice_fn "$helper_src" cmd_build)" in + *'mark_done "$name"'*) ;; + *) _p="$_p;running init-ownership does not mark it built" ;; + esac + case "$(_slice_fn "$helper_src" _ownership_checkpoint)" in + *mark_done*) ;; + *) _p="$_p;a tree that crossed the epoch still shows the step as pending" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the ownership step appears in the list, and can be run on its own" + fi +fi + +# ---- a chapter-8 tree can still be taken back from the build user ----------- # +# Refusing the recursive 7.2 chown on a built tree is right -- it would strip +# every package user in one pass. But refusing and stopping there left the tree +# with NO way forward: +# +# 4.3 had taken /usr, /etc, /var and the root symlinks, +# 7.2 said "not touching it, repair from the manifests instead", +# chroot enter said "the tree is not owned by root yet: run chroot prepare" +# +# -- pointing at the command that had just declined. The precise repair is to +# reclaim what the BUILD USER holds, by uid: that leaves every package user's +# files alone, so it is safe at any stage. +python3 - "$LFS_TOOL" <<'PYRECLAIM' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +for fn in ("_build_user_uid", "_handover_build_user_only"): + if not re.search(r"^def %s\(" % fn, src, re.M): + problems.append("there is no %s" % fn) +m = re.search(r"^def _handover_build_user_only\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + # selects by uid, so package users are untouched + if "-uid" not in body: + problems.append("the reclaim does not select on the build user's uid") + # user only: 4.3 chowned the user, so the inverse must not take the group + # as well -- that would strip `install` off every shared directory + # the INVOCATION -- the docstring above it explains why the group is left + # alone, and matching that text made this impossible to satisfy + code = "\n".join(l for l in body.splitlines() + if not l.lstrip().startswith("#")).split('"""') + code = "".join(code[::2]) + if "chown -h root:root" in code or "chown -R" in code: + problems.append("the reclaim takes the group too, undoing the install group") + if "chown -h root" not in body: + problems.append("the reclaim follows symlinks instead of chowning them") +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + if "_handover_build_user_only" not in body: + problems.append("a chapter-8 tree is still left with no way forward") + # and entering must repair rather than point back at prepare + enter = body.split('if action == "enter"') + if len(enter) > 1 and "_handover_build_user_only" not in enter[1]: + problems.append("chroot enter still sends you to a command that declined") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a built tree can be taken back from the build user without loss") +PYRECLAIM +_count_rc $? + +# end to end: the build user's paths go back to root, the package user's do not +if [ "$(id -u)" = 0 ] && command -v python3 >/dev/null 2>&1; then + _fx="$T/reclaim/mnt/lfs" + mkdir -p "$_fx/usr/bin" "$_fx/etc" "$_fx/var" "$_fx/tools" + ( cd "$_fx" && ln -sf usr/bin bin ) + echo pkg > "$_fx/usr/bin/owned-by-a-package" + echo cfg > "$_fx/etc/fstab" + # 4242 stands in for a package user, 4243 for the host's build account + chown 4242:4242 "$_fx/usr/bin/owned-by-a-package" + for _d in usr usr/bin etc var tools etc/fstab; do chown 4243 "$_fx/$_d"; done + chown -h 4243 "$_fx/bin" + _out="$(python3 - "$LFS_TOOL" "$_fx" <<'PYFX' +import sys, importlib.util +src = open(sys.argv[1]).read().split('if __name__ ==')[0] +spec = importlib.util.spec_from_loader("t", None) +m = importlib.util.module_from_spec(spec) +exec(compile(src, "lfs", "exec"), m.__dict__) +m._build_user_uid = lambda: 4243 +print(m._handover_build_user_only(sys.argv[2], run=True)) +PYFX +)" + _bad="" + [ "$(stat -c %u "$_fx/usr")" = 0 ] || _bad="$_bad;/usr was not taken back" + [ "$(stat -c %u "$_fx/bin")" = 0 ] || _bad="$_bad;the root symlink was not taken back" + [ "$(stat -c %u "$_fx/usr/bin/owned-by-a-package")" = 4242 ] \ + || _bad="$_bad;a package user's file was taken from it" + [ -L "$_fx/bin" ] || _bad="$_bad;the symlink was followed and replaced" + if [ -n "$_bad" ]; then + printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "reclaiming takes the build user's paths and leaves the packages' alone" + fi + rm -rf "$T/reclaim" +fi + +# ---- state lives under /usr/src, and it is sorted --------------------------- # +# It was /.lfs-pkgusr: a hidden directory at the root of the filesystem, sitting +# next to /boot and /etc as if it were part of the system being built. It is +# not -- it is this toolchain's working notes ABOUT that system -- and it holds +# twenty-two loose files at the top level. +# +# Under /usr/src it sits beside the accounts it describes, and the tree scans +# exclude ONE path instead of two, because everything under $SRCROOT already is. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '^STATE="\${LFS_PKGUSR_DIR:-\${SRCROOT%/}/lfs-pkgusr}"' "$helper_src" \ + || _p="$_p;lfs-helper state is not under \$SRCROOT" + grep -q '^PKGUSR_DIR = "usr/src/lfs-pkgusr"' "$LFS_TOOL" \ + || _p="$_p;lfs state is not under /usr/src" + # both tools must name the SAME directory -- they read each other's manifests + case "$(grep -m1 '^STATE=' "$helper_src")" in + *lfs-pkgusr*) ;; + *) _p="$_p;the two tools no longer agree on where state lives" ;; + esac + # SRCROOT has to be defined before what derives from it: `set -u` turns a + # forward reference into "SRCROOT: unbound variable" and the tool will not + # even print its version + _isrc="$(grep -n '^SRCROOT=' "$helper_src" | head -1 | cut -d: -f1)" + _ista="$(grep -n '^STATE=' "$helper_src" | head -1 | cut -d: -f1)" + if [ -n "$_isrc" ] && [ -n "$_ista" ] && [ "$_isrc" -gt "$_ista" ]; then + _p="$_p;SRCROOT is defined after the paths derived from it" + fi + # nothing loose at the top: every state file belongs to a named group + for _v in PROGRESS SNAPSHOT SNAPDIRS INSTALLDIRS; do + case "$(grep -m1 "^$_v=" "$helper_src")" in + *'$STATE_PROGRESS/'*|*'$STATE_CONF/'*|*'$STATE_GROUPS/'*) ;; + *) _p="$_p;$_v is still a loose file at the top of the state dir" ;; + esac + done + # and the directories are created together, so no path can find one missing + case "$(_slice_fn "$helper_src" mkstate)" in + *STATE_PROGRESS*STATE_GROUPS*STATE_CONF*) ;; + *) _p="$_p;mkstate does not create every state directory" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "state lives under /usr/src, sorted into named directories" + fi +fi + +# ---- colour means one thing --------------------------------------------------# +# Four colours, one meaning each, the same in both tools -- they print into the +# same terminal one after the other, and a log where the same colour means two +# different things is worse than no colour at all. Prose gets none: colour on +# every other line stops carrying information. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in say ok warn note detail hint die fail; do + grep -qE "^$_fn\(\) +\{" "$helper_src" \ + || _p="$_p;lfs-helper has no $_fn" + done + # say() is prose and must not paint anything + case "$(grep -m1 '^say()' "$helper_src")" in + *C_*) _p="$_p;plain prose is coloured" ;; + esac + # the vocabulary is the interface: no ad-hoc colour outside those helpers + # the helper DEFINITIONS are where the colour lives; everything else must go + # through them + _adhoc="$(grep -nE '(say|echo) "\$\{C_(DIM|WARN|ERR|OK)\}' "$helper_src" \ + | grep -vE '^[0-9]+:(ok|warn|note|detail|hint|die|fail)\(\)' | wc -l)" + [ "${_adhoc:-0}" = 0 ] \ + || _p="$_p;$_adhoc line(s) still paint colour by hand instead of using a helper" + # lfs must offer the same set, or the two halves of a build read differently + for _fn in warn note ok fail hint; do + grep -qE "^def $_fn\(" "$LFS_TOOL" || _p="$_p;lfs has no $_fn" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both tools share one colour vocabulary, and prose has none" + fi +fi + +# ---- the collector export must land where the reader looks ------------------ # +# `lfs` copies the export INTO the tree; `lfs-helper` reads it back out. Those +# two paths are the whole mechanism -- nothing else connects them -- and when +# the state directory was sorted, the writer kept using the top level while the +# reader moved into groups/. A rebuild with a perfectly good export then asked +# +# 'p_gcc' needs to install into: /usr/lib/bfd-plugins +# that directory belongs to package 'p_binutils' +# Which group should share it? +# +# for a directory the file already answered. Silent, because a question that +# gets asked looks the same as a question that was never answered. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # what the writer writes + _w="$(python3 - "$LFS_TOOL" <<'PYW' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def _install_collector_groups\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + d = re.search(r'dst_dir = pkgusr_state\(lfs, "([a-z]+)"\)', m.group(0)) + print(d.group(1) if d else "TOPLEVEL") +else: + print("MISSING") +PYW +)" + # what the reader reads + _r="$(grep -o '\$STATE_[A-Z]*/collector-groups\.import' "$helper_src" \ + | head -1 | sed 's|\$STATE_\([A-Z]*\)/.*|\1|' | tr 'A-Z' 'a-z')" + _p="" + [ "$_w" = groups ] || _p="$_p;lfs writes the collector export to '$_w', not groups/" + [ "$_r" = groups ] || _p="$_p;lfs-helper reads the collector export from '$_r', not groups/" + [ "$_w" = "$_r" ] || _p="$_p;the two tools disagree about where the export lives" + # an edit on the host has to reach the build, so entry re-copies it + case "$(python3 - "$LFS_TOOL" <<'PYE' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +body = m.group(0) if m else "" +enter = body.split('if action == "enter"') +print("yes" if len(enter) > 1 and "_install_collector_groups" in enter[1] else "no") +PYE +)" in + yes) ;; + *) _p="$_p;editing the export on the host never reaches the build" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the collector export is written where lfs-helper reads it" + fi + + # end to end: a decision in the file must not be asked about again + _cg="$T/collgroups"; mkdir -p "$_cg/groups" + printf '# prefix|nimgnu\ndir|/usr/lib/bfd-plugins|nimgnu_bfd-plugins\n' \ + > "$_cg/groups/collector-groups.import" + _o="$( STATE="$_cg"; STATE_GROUPS="$_cg/groups"; SNAP_ROOT=/ + COLLECTOR_MAP="$_cg/groups/collector-map"; COLLECTOR_PREFIX=nimgnu + PKGUSR_PREFIX="p_" + eval "$(_slice_fn "$helper_src" unprefix_pkg_user)" + eval "$(_slice_fn "$helper_src" collector_group_for)" + # the lookup runs before any of these matter, but they are called on the + # way to it + group_exists() { return 1; } + user_in_group() { return 1; } + eval "$(_slice_fn "$helper_src" sanitise_group_name)" + eval "$(_slice_fn "$helper_src" choose_collector_group)" + # a missing helper must FAIL, not skip: an empty result reading as a + # pass is the trap this suite has fallen into twice + command -v choose_collector_group >/dev/null 2>&1 \ + || { echo "there is no choose_collector_group"; exit 0; } + # no tty, so it can only answer from what it already knows + # <dir> <owner> <user> -- the user is the package asking for access + _g="$(choose_collector_group /usr/lib/bfd-plugins p_binutils p_gcc \ + </dev/null 2>/dev/null)" + [ "$_g" = nimgnu_bfd-plugins ] \ + || echo "a directory the export already answered is asked about again: got '$_g'" )" + if [ -z "$_o" ]; then + ok "a directory named in the export is never asked about again" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- verify must not report its own state directory ------------------------- # +# The state directory moved under /usr/src in 1.9.0, and /usr/src IS an install +# directory -- so `verify` began walking its own manifests, scripts and logs and +# reporting each one as a root-owned file no package claims. A clean 104/104 +# build came back with +# 448 root-owned file(s) that NO manifest claims +# and all 448 were this directory, including the two temp files the scan had +# just written for itself: +# unclaimed /usr/src/lfs-pkgusr/tmp/found.13431 +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr + WRAPPERS=/usr/lib/pkgusr + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + command -v is_never_claimed >/dev/null 2>&1 \ + || { echo "there is no is_never_claimed"; exit 0; } + for f in "$STATE/manifests/gcc.files" "$STATE/scripts/gcc.sh" \ + "$STATE/logs/gcc-all.log" "$STATE/progress/steps-built" \ + "$STATE/tmp/found.13431"; do + is_never_claimed "$f" || echo "verify still reports $f" + done + # an account home under the SAME parent is a package's, and must not be + # swept up by this + is_never_claimed /usr/src/pkgusr/p_gcc/VERSION \ + && echo "a package user's own file was excluded too" )" + if [ -z "$_o" ]; then + ok "verify does not report the build's own notes as unclaimed files" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- the two tools share one tree snapshot ---------------------------------- # +# `lfs` takes the last snapshot outside the chroot at the end of chapter 6; +# `lfs-helper` diffs against it for the first package inside. It is a handover, +# and it works only while both name the same file. 1.9.0 moved the reader to +# progress/tree-snapshot and left the writer on .snapshot at the top level, so +# the handover snapshot was silently lost and the first chroot package diffed +# against nothing. +python3 - "$LFS_TOOL" "$(dirname "$LFS_TOOL")/lfs-helper" <<'PYSNAP' +import re, sys +lfs = open(sys.argv[1]).read() +problems = [] +if '".snapshot"' in lfs or '".snapshot-dirs"' in lfs: + problems.append("lfs still writes the snapshot to .snapshot at the top level") +if 'pkgusr_state(lfs, "progress")' not in lfs: + problems.append("lfs does not write the snapshot into progress/") +for want in ('"tree-snapshot"', '"tree-snapshot-dirs"'): + if want not in lfs: + problems.append("lfs does not name %s" % want) +try: + helper = open(sys.argv[2]).read() +except OSError: + helper = "" +if helper: + m = re.search(r'^SNAPSHOT="([^"]+)"', helper, re.M) + if not m or "tree-snapshot" not in m.group(1): + problems.append("lfs-helper reads a different snapshot file") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS both tools read and write the same tree snapshot") +PYSNAP +_count_rc $? + +# ---- add-user names the account through the chokepoint ---------------------- # +# It used its argument raw for the ACCOUNT and pkg_owner_name only for the HOME. +# Fine while every caller passes an already-prefixed owner -- cmd_build does. +# last_build_step.sh does not: `lfs-helper add-user wget` produced an account +# called `wget` living in /usr/src/pkgusr/p_wget, and the sanity report found +# both halves without being able to connect them: +# !! accounts without a known prefix: wget urllib3 requests ... +# ? /usr/src/pkgusr/p_wget/ has no matching account (stray directory) +# !! group wget (gid 10082) is outside every convention +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _au="$(_slice_fn "$helper_src" cmd_add_user)" + _p="" + case "$(printf '%s\n' "$_au" | grep -vE '^[[:space:]]*#')" in + *'name="$(pkg_owner_name "$name")"'*) ;; + *) _p="$_p;add-user creates the account from the raw name" ;; + esac + # the normalisation must come BEFORE the account is created, or it renames + # nothing + _i1="$(printf '%s\n' "$_au" | grep -n 'name="$(pkg_owner_name' | head -1 | cut -d: -f1)" + _i2="$(printf '%s\n' "$_au" | grep -n 'useradd ' | head -1 | cut -d: -f1)" + if [ -n "$_i1" ] && [ -n "$_i2" ] && [ "$_i1" -gt "$_i2" ]; then + _p="$_p;add-user normalises the name after creating the account" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "add-user gives the account and its home the same name" + fi + + # last_build_step.sh is the caller that exposed it -- it passes bare names + _lbs="$(dirname "$LFS_TOOL")/last_build_step.sh" + if [ -f "$_lbs" ]; then + grep -q 'lfs-helper add-user' "$_lbs" \ + && ok "the final step still creates its users through add-user" \ + || bad "the final step no longer goes through add-user" + fi +fi + +# ---- the sanity report reads the sorted state, and knows when a build is done # +# Two findings on a perfect 105/105 tree, both from the report and neither from +# the build: +# +# adopted.list: MISSING (the adoption gate will never close) +# -- it read $STATE/adopted.list; the file is progress/adopted.list +# +# !! /usr/bin is already STICKY -- sealed before the build finished +# -- the build HAD finished, and seal-install-dirs is supposed to do that +# +# The sticky bit means opposite things on either side of that line: during the +# build it is a bug, afterwards its ABSENCE is. +san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$san" ]; then + _p="" + grep -q 'PROG="\$STATE/progress"' "$san" \ + || _p="$_p;the sanity report still reads the flat state layout" + grep -q 'BUILD_FINISHED' "$san" \ + || _p="$_p;the sanity report does not know whether the build finished" + grep -q 'the build finished but the seal never fired' "$san" \ + || _p="$_p;a finished build with no seal is not reported" + # an empty manifest is a config step that installed nothing, not an orphan + grep -q 'grep -q . "\$m" 2>/dev/null || continue' "$san" \ + || _p="$_p;empty manifests are still reported as missing an account" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the sanity report reads the sorted state and judges the seal in context" + fi +fi + +# ---- every tool can say which build it is ----------------------------------- # +# Section 0 of the sanity report exists because a stale copy in the chroot has +# cost more debugging time here than anything else. packagemanager_install had +# no --version, so that row read +# packagemanager_install Please provide an install script with $2 +pmi="$(dirname "$LFS_TOOL")/packagemanager_install" +if [ -f "$pmi" ]; then + _v="$(bash "$pmi" --version 2>&1)" + case "$_v" in + packagemanager_install*build*) ok "every tool reports its own build id" ;; + *) bad "packagemanager_install cannot report its build id: $_v" ;; + esac +fi + +# ---- no finding that repairing makes worse ---------------------------------- # +# Two passes disagreed about a mail spool, and each undid the other: +# +# lfs-helper verify --fix builduser /var/mail/lfs lfs -> root +# lfs-helper verify 1 root-owned file(s) that NO manifest claims: +# /var/mail/lfs +# +# The build-user pass gave it to root because the build user held it; the +# unclaimed pass then reported that root holds a file no manifest claims. +# Repairing it produced the next run's finding. +# +# Shadow creates /var/mail/<name> for every account it makes, so a spool arrives +# as a side effect of `useradd` and no package ever installs one. It belongs to +# the account it is named for, and neither pass should touch it. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr; WRAPPERS=/usr/lib/pkgusr + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + command -v is_never_claimed >/dev/null 2>&1 \ + || { echo "there is no is_never_claimed"; exit 0; } + for f in /var/mail/lfs /var/mail/p_gcc; do + is_never_claimed "$f" || echo "$f is still fought over by two passes" + done + # not the whole of /var, though -- a package's files there are its own + is_never_claimed /var/lib/something \ + && echo "the exclusion is too broad: it swallowed /var/lib" )" + if [ -z "$_o" ]; then + ok "a mail spool belongs to its account, and no pass claims it" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the general property: nothing the build-user pass hands to root may then be + # reported by the unclaimed pass, or --fix and verify never agree + case "$(_slice_fn "$helper_src" _vfy_build_user_leftovers)" in + *is_never_claimed*) ok "the build-user pass skips files no package may own" ;; + *) bad "the build-user pass can create a finding for the unclaimed pass" ;; + esac +fi + +# ---- every door to `chown -R lfs` has the same guard ------------------------ # +# There were FOUR places that hand the build tree to the `lfs` user, and only +# one of them had the check that refuses on a tree past chapter 7: +# +# _chown_tree_to_lfs book 4.3, guarded +# session --run re-applied it by hand, unguarded +# the pre-build check "fixing N dir(s) still owned by root", unguarded +# fix-ownership --run the explicit repair, unguarded +# +# On a finished system any of the other three would have handed the whole thing +# to the build user in one pass -- the exact damage the guard exists to prevent, +# through a door the guard was not on. It already happened once via a replayed +# step 4.2. +python3 - "$LFS_TOOL" <<'PYCHOWN' +import re, sys +raw = open(sys.argv[1]).read() +# Comments name the command they are explaining -- every fix in this file says +# what it fixed -- so scan CODE only. Matching the explanation makes a test +# that cannot pass no matter what the code does. +src = "\n".join("" if l.lstrip().startswith("#") else l + for l in raw.splitlines()) +problems = [] +# every recursive chown to lfs, and the function it sits in +for m in re.finditer(r"chown -R lfs", src): + start = src.rfind("\ndef ", 0, m.start()) + if start < 0: + problems.append("a chown -R lfs sits outside any function") + continue + end = src.find("\ndef ", m.end()) + body = src[start:end if end > 0 else len(src)] + name = re.match(r"\ndef ([a-zA-Z_0-9]+)", body).group(1) + # the one that only chowns the state dir is not handing over the tree + if "PKGUSR_DIR" in src[m.start()-120:m.end()+120]: + continue + if "_handover_is_safe" not in body: + problems.append("%s hands the tree to the build user unguarded" % name) +if problems: + for p in sorted(set(problems)): + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS every path that gives the tree to the build user is guarded") +PYCHOWN +_count_rc $? + +# ---- session says what to do next ------------------------------------------- # +# It ended on `export LFS=/mnt/lfs # <-- run this in your shell` and stopped. +# Every other command in this tool finishes with a Next: line; this one left you +# configured and guessing. +python3 - "$LFS_TOOL" <<'PYNEXT' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def cmd_bs_session\(.*?(?=\n\ndef )", src, re.S | re.M) +problems = [] +if not m: + problems.append("cmd_bs_session is gone") +else: + body = m.group(0) + # the dry-run branch and the --run branch both end somewhere useful + if body.count('"Next:"') < 2: + problems.append("session does not say what to do next on both paths") + if "lfs run" not in body: + problems.append("session never points at `lfs run`") + if "build-system next" not in body: + problems.append("session never offers `build-system next`") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS session ends by saying what to run next") +PYNEXT +_count_rc $? + +# ---- comm gets sorted input, in one collation ------------------------------- # +# The snapshot diff is `comm -13 <(before) <(after)`. comm compares line by +# line and TRUSTS both sides to be sorted the same way; it cannot detect +# otherwise, it just returns the wrong answer. Two things broke that: +# +# * the two snapshots may have been written under different locales -- a +# package user's profile sets LC_ALL=POSIX and root's shell may not +# * load_snapshot rewrites only the lines carrying the host mount prefix, so +# a file holding both forms comes out sorted under neither +# +# The visible symptom is a package installing forty files and being told it +# installed none. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in snapshot snapshot_dirs load_snapshot; do + _slice_fn "$helper_src" "$_fn" | grep -q 'LC_ALL=C sort' \ + || _p="$_p;$_fn does not pin the collation comm depends on" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both sides of the snapshot diff are sorted the same way" + fi + + # the mixed-prefix case, end to end + _sd="$T/snapsort"; mkdir -p "$_sd" + printf '%s\n' /mnt/lfs/usr/bin/z /usr/bin/a /mnt/lfs/usr/bin/b > "$_sd/snap" + _got="$( SNAP_ROOT=/; LFS_HOST_MOUNT=/mnt/lfs + eval "$(_slice_fn "$helper_src" load_snapshot)" + load_snapshot "$_sd/snap" | tr '\n' ' ' )" + case "$_got" in + "/usr/bin/a /usr/bin/b /usr/bin/z ") ok "a snapshot holding both path forms comes back sorted" ;; + *) bad "load_snapshot returns unsorted input to comm: [$_got]" ;; + esac +fi + +# ---- "installed NO files" has to say which of the three it is --------------- # +# There are exactly three ways to reach zero and they need different fixes: +# nothing was written, it was written somewhere excluded from tracking, or it +# was written over files another package owns and dropped by the filter. +# Saying only "check the log" meant reading a thousand lines of make output to +# find out which. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cb="$(_slice_fn "$helper_src" cmd_build)" + _p="" + case "$_cb" in + *'new files (snapshot diff)'*) ;; + *) _p="$_p;the zero-files failure does not report the snapshot diff count" ;; + esac + case "$_cb" in + *'files touched since stamp'*) ;; + *) _p="$_p;the zero-files failure does not report the touched count" ;; + esac + case "$_cb" in + *'already belongs to another package'*) ;; + *) _p="$_p;the zero-files failure does not distinguish a dropped install" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package that installed nothing says which kind of nothing" + fi +fi + +# ---- a re-run over its own work is not a silent failure --------------------- # +# The zero-files check counts what THIS RUN wrote, which is the wrong question +# when a package is built a second time. Many installers skip a file that is +# already up to date -- perl's ExtUtils::Install prints "Installing <path>" for +# every target and copies only the stale ones -- so a second run writes nothing: +# the snapshot diff sees nothing new, no mtime beats the stamp, and the count is +# 0 for the best possible reason. +# +# xml-parser hit exactly that: forty files on disk, every one owned by +# p_xml-parser, and the step failed as a no-op. Counting cannot tell that from +# a real failure. The tree can. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _package_already_owns_files | grep -q . \ + || _p="$_p;there is no _package_already_owns_files" + case "$(_slice_fn "$helper_src" cmd_build)" in + *_package_already_owns_files*) ;; + *) _p="$_p;a re-run over its own work still fails as a silent no-op" ;; + esac + # it must ask about the OWNER, and only inside install directories -- looking + # at the whole filesystem would be slow and would count build scratch + _pa="$(_slice_fn "$helper_src" _package_already_owns_files)" + case "$_pa" in + *'-user "$owner"'*) ;; + *) _p="$_p;the check does not look for files this package owns" ;; + esac + case "$_pa" in + *install_dirs_list*) ;; + *) _p="$_p;the check scans outside the install directories" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package that already owns its files is not failed as a no-op" + fi + + # both directions, against a real tree + if [ "$(id -u)" = 0 ]; then + _ow="$T/owns"; mkdir -p "$_ow/usr/lib" + echo x > "$_ow/usr/lib/thing" + _r="$( SNAP_ROOT="$_ow" + user_exists() { [ "$1" = p_test ]; } + install_dirs_list() { printf '%s\n' /usr/lib; } + eval "$(_slice_fn "$helper_src" _package_already_owns_files)" + # owned by root -> this package installed nothing, which IS a failure + _package_already_owns_files p_test && echo "false-pass" + # owned by the package -> it installed, whenever that was + chown 4242 "$_ow/usr/lib/thing" 2>/dev/null + user_exists() { [ "$1" = p_test ]; } + stat -c %U "$_ow/usr/lib/thing" >/dev/null 2>&1 || echo "fixture broken" )" + case "$_r" in + "") ok "a package owning nothing is still reported as a failure" ;; + *) bad "the check passes a package that installed nothing: $_r" ;; + esac + rm -rf "$_ow" + fi +fi + +# ---- a script outside the tool can ask for an account NAME ------------------ # +# `pkgusr-home` was exposed and the account name was not, so a shell script +# could ask where a package lives but not what its user is called. +# last_build_step.sh did the only thing left, which was assume: +# +# lfs-helper add-user wget +# chown -R "wget:wget" ... -> chown: invalid user: 'wget:wget' +# +# Right by accident while add-user used the raw name; broken the moment add-user +# started applying the prefix, as everything else already did. Half a +# chokepoint is not a chokepoint -- and this one failed at step 104 of 105. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" cmd_owner_name | grep -q 'pkg_owner_name' \ + || _p="$_p;there is no way to ask for a package's account name" + grep -q 'owner-name)' "$helper_src" \ + || _p="$_p;owner-name is not reachable as a command" + grep -q 'lfs-helper owner-name <package>' "$helper_src" \ + || _p="$_p;owner-name is not in the help" + # it must agree with pkgusr-home about the same package + _n="$(bash "$helper_src" owner-name wget 2>/dev/null)" + _h="$(bash "$helper_src" pkgusr-home wget 2>/dev/null)" + case "$_h" in + */"$_n") ;; + *) _p="$_p;owner-name and pkgusr-home disagree: '$_n' vs '$_h'" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package's account name and its home come from the same place" + fi +fi + +# and the final step must ASK rather than rebuild the name itself +_lbs="$(dirname "$LFS_TOOL")/last_build_step.sh" +if [ -f "$_lbs" ]; then + _p="" + _n="$(grep -c 'lfs-helper owner-name' "$_lbs")" + [ "${_n:-0}" -ge 2 ] \ + || _p="$_p;the final step still assumes the account name ($_n of 2 sites)" + # every chown/su in there must use the resolved name, never the bare argument + case "$(grep -c 'lfs-helper add-user' "$_lbs")" in + 0) _p="$_p;the final step no longer creates its users through add-user" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the final step asks for the account name it was given" + fi +fi + +# ---- the ESP hint prints a path you can actually type ----------------------- # +# Two sources feed it and they disagree about the prefix: `lsblk -rno NAME` +# gives a bare name, `blkid -o device` gives a full path. Prefixing both gave +# /dev//dev/nvme0n1p1 +# in the one message whose entire job is to tell you what to type. +python3 - "$LFS_TOOL" <<'PYESP' +import re, sys +src = open(sys.argv[1]).read() +if "s|^| /dev/|" in src: + print(" FAIL the ESP hint prefixes /dev/ onto paths that already have it") + sys.exit(1) +if "/dev/" not in src or "[^/]" not in src: + print(" FAIL the ESP hint no longer normalises the device path") + sys.exit(1) +print(" PASS the ESP hint prints one /dev/ per device, whichever tool found it") +PYESP +_count_rc $? + +# ---- a bootloader is opt-in ------------------------------------------------- # +# The default was `refind`, so every build generated a rEFInd step -- and most +# people running this already have a working ESP and a bootloader that finds it. +# The machine booted in order to build LFS at all. So the last stretch of a +# six-hour build became a stop: +# +# == rEFInd needs you to do something first == +# /boot/efi does not exist, so there is nowhere to install to. +# +# A question nobody asked to be asked. A bootloader is a decision about the +# whole machine, not a package. +python3 - "$LFS_TOOL" <<'PYBL' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def bootloader\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("bootloader() is gone") +else: + body = "\n".join("" if l.lstrip().startswith("#") else l + for l in m.group(0).splitlines()) + body = "".join(body.split('"""')[::2]) + if '"refind"' in body: + problems.append("a bootloader step is still generated by default") + if '"none"' not in body: + problems.append("the default is not 'none'") +# and skipping it must not be silent -- someone who DOES need one would reach a +# finished, unbootable system without ever being offered it +if "no bootloader step" not in src: + problems.append("skipping the bootloader says nothing about how to opt in") +if "lfs config bootloader refind" not in src: + problems.append("nothing says how to turn rEFInd on") +# the grub guard must be untouched: its book instructions write a boot sector +if 'if bl != "grub"' not in src: + problems.append("the grub script is no longer withheld unless chosen") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS no bootloader is set up unless you ask for one") +PYBL +_count_rc $? + + +# ---- the sort and the comparison must agree --------------------------------- # +# 1.9.6 pinned LC_ALL=C on the sorts feeding comm and stopped there. That is +# half a fix: comm validates ordering in ITS OWN locale, so C-sorted input +# compared under en_GB.UTF-8 is "not sorted" as far as comm is concerned -- +# +# comm: file 1 is not in sorted order +# comm: file 2 is not in sorted order +# comm: input is not in sorted order +# +# and the diff it returns is then simply wrong. Pinning either side alone +# guarantees they disagree. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _bare="$(grep -nE '(^|[^=[:alnum:]_])comm ' "$helper_src" \ + | grep -vE 'LC_ALL=C comm' | grep -vE '^[0-9]+:[[:space:]]*#')" + if [ -n "$_bare" ]; then + printf '%s\n' "$_bare" | while IFS= read -r m; do + bad "comm without LC_ALL=C: ${m%%:*}" + done + else + ok "every comm compares in the locale its input was sorted in" + fi + # and every sort whose output reaches a comm is pinned too + _p="" + for _fn in snapshot snapshot_dirs load_snapshot _vfy_unclaimed; do + _slice_fn "$helper_src" "$_fn" | grep -q 'LC_ALL=C sort' \ + || _p="$_p;$_fn sorts in the ambient locale" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every sort feeding a comm is pinned to the same locale" + fi +fi + + +# ---- the scripts in the tree must match the tools outside it ---------------- # +# The step order and the generated scripts live in the TREE; the tools live +# outside it. Nothing connected the two, so a tree generated by an older +# version went on running steps that version had: +# +# ===== [104/105] last-step ===== +# chown: invalid user: 'wget:wget' +# +# under an lfs-helper (1.10.0) that had removed `last-step` entirely and no +# longer knew what it was. The chroot TOOLS already carry a build id for +# exactly this reason -- a stale copy has cost more debugging time here than +# anything else -- and the generated scripts did not. +python3 - "$LFS_TOOL" <<'PYSTAMP' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +if "generated-by" not in src: + problems.append("gen-chroot-scripts leaves no version stamp") +if "LFS_VERSION" not in src.split("generated-by")[1][:400]: + problems.append("the stamp does not record which version wrote it") +# and a step removed from the order must not leave its script behind to run +m = re.search(r"^def cmd_bs_gen_chroot_scripts\(.*?(?=\n\ndef )", src, re.S | re.M) +body = m.group(0) if m else "" +if "stale" not in body or "os.remove" not in body: + problems.append("scripts for removed steps are left in the tree") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS generated scripts record the version that wrote them") +PYSTAMP +_count_rc $? + +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _warn_if_scripts_are_stale | grep -q 'LFS_HELPER_VERSION' \ + || _p="$_p;lfs-helper does not compare its version against the scripts'" + case "$(_slice_fn "$helper_src" cmd_build_all)" in + *_warn_if_scripts_are_stale*) ;; + *) _p="$_p;build-all runs an order it never checked" ;; + esac + # a missing script must blame the likely cause, not just offer to regenerate + case "$(_slice_fn "$helper_src" cmd_build)" in + *'no longer exists in this version'*) ;; + *) _p="$_p;a step removed from the tools reports only a missing file" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a step order older than the tools is reported, not obeyed" + fi + + # end to end: an unstamped tree, and a mismatched one + _st="$T/stamp"; mkdir -p "$_st/progress" + _r="$( STATE_PROGRESS="$_st/progress"; LFS_HELPER_VERSION="9.9.9" + eval "$(_slice_fn "$helper_src" _warn_if_scripts_are_stale)" + warn() { echo "WARNED: $*"; } + _warn_if_scripts_are_stale >/dev/null 2>&1 && echo "unstamped-passed" + printf 'version=9.9.9\n' > "$_st/progress/generated-by" + _warn_if_scripts_are_stale >/dev/null 2>&1 || echo "matching-failed" + printf 'version=1.0.0\n' > "$_st/progress/generated-by" + _warn_if_scripts_are_stale >/dev/null 2>&1 && echo "mismatch-passed" )" + case "$_r" in + "") ok "an unstamped or mismatched step order is caught, a matching one is not" ;; + *) printf '%s\n' "$_r" | while IFS= read -r m; do + [ -n "$m" ] && bad "stale-script check is wrong: $m" + done ;; + esac + rm -rf "$_st" +fi + echo -echo " PASS: $PASS FAIL: $FAIL (final)" -[ "$FAIL" -eq 0 ] +echo " PASS: $(_pass_total) FAIL: $(_fail_total) (final)" +[ "$(_fail_total)" -eq 0 ]