Fixed cfg steps And stuff
Christian Immanuel · 2026-08-28 17:12 · d612779c40732d9248da4cc32d39c3c2cc708fbc
HANDOFF.md | 377 +++++++++++++++++++++++-----
Makefile | 31 ++-
README.md | 29 ++-
blfs | 2 +-
lfs | 207 ++++++++++-----
lfs-helper | 413 +++++++++++++++++++++++++-----
packagemanager | 348 +++++++++++++++++++++++---
packagemanager_install | 57 ++++-
test_lfs_crosschain.sh | 664 +++++++++++++++++++++++++++++++++++++++++++++----
9 files changed, 1812 insertions(+), 316 deletions(-)
diff --git a/HANDOFF.md b/HANDOFF.md
index c7307b1..63269ee 100644
--- a/HANDOFF.md
+++ b/HANDOFF.md
@@ -11,15 +11,25 @@ Attach: the five scripts (`lfs`, `lfs-helper`, `packagemanager`,
## Current state
-All five tools at **1.10.1**. Build ids — check these match what is installed:
+All five tools at **1.11.6**. Build ids — check these match what is installed:
- lfs 8b2cb98
- lfs-helper c2f6fbc
- packagemanager d051474
- blfs 36a4717
- packagemanager_install 1c068ed
+ lfs f1f6164
+ lfs-helper 3e99aa1
+ packagemanager df7bb26
+ blfs c479327
+ packagemanager_install 1d6c685
-569 regression tests, 568 pass.
+589 regression tests, 584 pass in a bare checkout.
+
+**This is a build candidate.** Everything since 1.11.0 is desk work — no
+change has met a real build. The remaining open items need one to make
+progress, so the next move is a full run, not more code. The five failures are
+environment-only: four need the book HTML beside the tool, one needs
+`skel-u_xdg/.bash_profile`.
+
+**A full 104/104 build has completed on 1.10.1** and verified clean: 66265
+paths owned by the right package, every account prefixed, every home owned by
+its user, install directories sealed, `check-toolchain` usable.
**A full build on 1.9.3 verified clean: 66265 paths owned by the right
package, nothing else outstanding.** The one failure is `the XDG profile is not
@@ -37,7 +47,7 @@ directories, the build scratch and the state directory have all moved.
lfs build-system session --reconfigure # several new questions
lfs run
# inside the chroot:
- lfs-helper --version # MUST read c2f6fbc
+ lfs-helper --version # MUST read 7a7b240
lfs-helper build-all
Afterwards `lfs-helper verify` is the real check — it reconciles ownership
@@ -188,9 +198,9 @@ prefix and the root are both answered from it, so a name cannot be given one
kind's prefix and placed in another kind's root.
`lfs-helper pkgusr-home <name>` exposes the rule to shell scripts, so nothing
-else re-derives it. `packagemanager_install`, `lfs-completion.bash` and
-`last_build_step.sh` all go through a chokepoint now; a test fails if any of
-them builds `/usr/src/$something` by hand.
+else re-derives it. `packagemanager_install` and `lfs-completion.bash` go
+through a chokepoint now; a test fails if either builds `/usr/src/$something`
+by hand.
**The chokepoint must resolve to the ACCOUNT home, not just the root.**
Callers pass a mixture and always will: `build_pkg` has the step name
@@ -360,6 +370,7 @@ because the next one will look like them.
| Handover guarded by one inode | `_owner_is_root` checks `$LFS/usr`; the pass it guards covers eight trees |
| uid range used to identify the build user | host `lfs` was 10753, inside the package range |
| Adoption reached from two directions | an up-front pass and a mid-loop "late" pass with its own flag |
+| Ownership scans kept their own exclusion lists | build trees moved into the package homes in 1.7.6, the snapshot scans followed and the ownership scans did not — `verify` ran for minutes walking every unpacked source tree |
| Generated scripts had no version stamp | the step order lives in the tree, the tools outside it — a tree generated by an older version went on running `last-step` under a lfs-helper that had removed it |
| Collector export written and read in two places | 1.9.0 sorted the state dir; the writer stayed at the top level, the reader moved into `groups/` |
| `pkgusr-home` was exposed, `owner-name` was not | a script could ask where a package lives but not what its user is called, so `last_build_step.sh` assumed — `chown: invalid user: 'wget:wget'` at step 104 of 105 |
@@ -509,6 +520,64 @@ finished, unbootable system without ever having been offered one.
GRUB is unchanged and still withheld unless explicitly chosen: its book
instructions run `grub-install /dev/sda`, which writes a disk's boot sector.
+## A command that works in silence looks like one that hung (1.10.3)
+
+`lfs-helper verify` ran for five minutes printing nothing, on a tree it had
+already verified clean, and was reported as a hang. It was not. Two faults:
+
+**A fork per path, twice over.** `stat -c %U` for each path, and
+`is_never_claimed` running `< <(never_claim_list)` — a process substitution —
+for each path too. Across 66265 manifest paths that is ~130000 processes.
+
+- `_never_claim_load` reads the list once into `_NEVER_CLAIM`; matching is then
+ pure shell.
+- Owners come from **one batched `stat` per manifest** via `xargs -0`.
+ Measured on 8000 paths: **12.6s → 0.15s**.
+
+**No output until every pass had finished.** Each pass now announces itself
+before it runs, and the long one prints `[n/m] <package>` live.
+
+Two traps worth remembering, both of which produced code that was *correct and
+slow* — the hardest kind of wrong to notice:
+
+- **`stat` does not expand escapes.** `stat -c '%U\t%n'` prints a literal
+ backslash-t, so every line came back as one field, the map stayed empty, and
+ the loop fell through to the per-path `stat` it was written to replace. Use a
+ real tab (`$'\t'`).
+- **I first pasted the batching into `cmd_build`**, not the verify pass — both
+ contain `for man in ...` over manifests. The tests caught it.
+
+## Scratch, and the one list that names it (1.10.2)
+
+`scan_prune_paths` / `scan_prune_set` name everything no ownership scan should
+walk: `/dev /proc /sys /run /tmp`, `$LFS/sources`, `$BUILD_ROOT`, `$STATE`, and
+**every package's unpacked source tree** (`<home>/$PKGUSR_BUILD_SUBDIR`).
+
+There were five such lists and they had already diverged. Build trees moved from
+`/build` into each package user's home in 1.7.6; the snapshot scans followed and
+the ownership scans did not. So `lfs-helper verify` began walking every unpacked
+source tree in the system — minutes of silence — and a finished tree filled the
+sanity report with tcl's own documentation:
+
+ !! files with no owner (first 40):
+ /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm
+
+A tarball can carry any uid it likes. Unpacked sources are not installed,
+nothing owns them, and asking who does has no answer.
+
+**`SCAN_PRUNE` is an ARRAY, not a string spliced through `eval`.** The patterns
+contain `*`, and `eval` lets the shell expand them against the real filesystem
+before `find` ever sees them — `/usr/src/pkgusr/*/src/*` becomes a handful of
+literal directory names and the prune silently matches almost nothing. That is
+what the first version of this fix did, and it looked like it worked.
+
+`lfs-sanity.sh` prunes the same paths, or it reports what `verify` does not.
+
+Also: a config step that writes **root-owned** files (`cfg_clock` → `/etc/adjtime`)
+is not an orphaned manifest. It installs no software, so it has no account, and
+there is nothing adoption could ever want. Eleven such findings on a perfect
+tree.
+
## The tree remembers which version generated it (1.10.1)
`gen-chroot-scripts` writes `progress/generated-by` with the version, build id
@@ -556,6 +625,215 @@ it. Set it empty to turn it off.
back it should be a directory of drop-in scripts, not a template copied once
that then drifts from the tool that generated it.
+## The kernel and the bootloader are yours (1.11.6)
+
+Both are decisions about the whole machine rather than about a package, and
+getting either wrong costs the system doing the building. So the build does
+neither, and ends by **saying** so — `_say_what_is_yours_to_finish`, beside the
+login and strip warnings — rather than leaving it to be discovered after a
+reboot.
+
+`bootloader` was already `none` by default and stays that way. The prompt now
+says what that means: nothing is written to any ESP, boot sector or partition
+table, and the machine keeps booting exactly as it does today. rEFInd is opt-in
+by name, and the closing note prints how to ask for it.
+
+## $LFS may never be the running system (1.11.5)
+
+Every path this tool writes is `$LFS/something`. Nothing checked that the tree
+was not the host's own, so a config holding `/`, an `export LFS=/` in the wrong
+shell, or a typo in the interview was enough for
+`build-system restart --run` to delete the top-level directories of the machine
+doing the building, and for `_verify_lfs_ownership` to run `chown -R lfs /usr`
+on it.
+
+`_is_host_path` is the rule; `_refuse_host_tree` is the hard stop. It is called
+from `require_lfs_for_root`, the single door every command resolves `$LFS`
+through, and from `_verify_lfs_ownership`, which resolved `$LFS` itself and was
+therefore the one destructive path with no check at all. The interview refuses
+the same paths at the prompt, where you can just type another one.
+
+**Both spellings are checked.** On a merged-`/usr` system `/bin` is a symlink
+to `usr/bin`, so `realpath("/bin")` is `/usr/bin` and a check on the resolved
+path alone let `$LFS=/bin` through. The first version of the test caught this.
+
+**Inside a system tree counts.** `/usr/src/lfs` is not a build tree, it is a
+directory in the system's own `/usr`. `/home` and `/srv` are deliberately not
+treated that way: `/home/you/lfs` is an ordinary place to build, and a check
+that refuses `$LFS/usr` refuses every build there is.
+
+What was already safe, and stays so: `restart` refuses while the chroot's bind
+mounts are up, because deleting through `/dev` would reach the host. GRUB
+scripts are not generated at all unless GRUB is the chosen bootloader, so
+nothing that could run `grub-install /dev/sda` is left lying around. The rEFInd
+step never formats, partitions or writes a raw device; it adds files to an
+already-mounted ESP, backs the config up, adds a menu entry beside your
+existing bootloader, and refuses to overwrite an existing rEFInd without
+`REFIND_OVERWRITE=1`.
+
+## A suppressed failure is a bug waiting to be found (1.11.4)
+
+`soft <what> -- <cmd ...>` runs a command that may fail, and **says so** when
+it does, without stopping the build. Every ownership and permission call whose
+failure changes what the system looks like goes through it: the wrappers'
+own root:root, the install directories' group-writable bit, the staging tree's
+inherited owner and mode, `passwd`/`group` backups, the `lfs` build user, and
+the mode `verify --fix` counts as fixed.
+
+Fifteen call sites. `2>/dev/null || true` is not banned — `mkdir -p` on a
+directory that exists is fine — but an ownership call that keeps it must carry
+a comment on the line immediately above saying **SUPPRESSED DELIBERATELY** or
+**benign:**, and a test counts the ones that do not.
+
+Any comment was not enough as a rule. The surrounding prose explains what a
+call does, not why its failure may be thrown away, so a reintroduced
+suppression inherits whatever comment happened to be above it — the first
+version of the test passed a mutation that put one straight back.
+
+The two deliberate ones are `mkstate` and `ensure_pkgusr_roots`: both run
+before book 7.6, when the `install` group does not exist and the call cannot
+succeed. Reporting them would print a failure on every invocation for the
+first third of a build, and that is how a report becomes noise.
+
+## The build says what it did not do (1.11.4)
+
+`strip` is collected by the interview, carried into the chroot environment as
+`LFS_STRIP`, and acted on by nothing. `_warn_if_strip_was_asked_for` is now the
+build's second-to-last word, beside `_warn_if_no_login` — the place where the
+things you still have to do yourself are collected. It fires only if you asked.
+
+Still not implemented, and the reason stands: stripping rewrites every binary,
+and here a file's owner is the record of which package installed it, so it has
+to run as that owner, package by package.
+
+## One door per decision (1.11.3)
+
+Item 6 does not need a rewrite. `grant_dir_to_user` already showed the shape:
+**ask `lfs-helper` when it is on the system, keep the local copy as the
+fallback for when it is not.** Three more decisions now go through it.
+
+**The wrappers.** `packagemanager_install` shipped its own 480-line copy and
+wrote it to `/tmp/pkgusr-wrappers.$$` on every run. The wrappers *are* the rule
+about what a package user may do — chown skipped, chgrp skipped, `install -d`
+on an existing directory allowed to succeed — so that was a second answer to
+the question, and `/etc/pkgusr/bash_profile` pointed at neither copy: it names
+`/usr/lib/pkgusr`, which is where lfs-helper writes them. Now
+`_wrapper_dir_from_lfs_helper` asks, and writes them through
+`lfs-helper make-wrappers` if they are missing or incomplete — which also
+repairs a tree whose wrappers predate a fix. The local copy stays, for a system
+with no lfs-helper.
+
+**Only a temporary directory is deleted.** Both call sites ended with
+`rm -rf "$_wrapdir"`. Pointed at the shared directory that removes the wrappers
+from every package user on the system, so the cleanup is now conditional on
+having built a temporary one.
+
+**The account home.** `pkgusr_home_for` re-derived the layout — root per kind,
+prefix, `cfg_` special case. Correct today, wrong the moment the layout is
+configured differently, which it can be. It asks `lfs-helper pkgusr-home` now.
+
+**site-packages.** `pip_install_module` did its own `groupadd` + `chgrp -R` +
+`chmod -R g+w`: a third implementation, and the only one that never asked what
+group the directory already carried. It goes through `lfs-helper grant-dir`.
+That is the debt 1.11.2 left.
+
+Two new commands make the doors reachable: `lfs-helper wrapper-dir` and
+`lfs-helper make-wrappers --run`.
+
+**The dispatch repeated six entries** — `verify`, `install-as`, `pkgusr-info`,
+`pkgusr-home`, `owner-name`, `clean-state` — and `case` takes the first match,
+so the second set was unreachable code that looked like a second decision. A
+test now fails on any repeat.
+
+What is left of item 6 is the build loop itself: `cmd_build` and
+`packagemanager_install` still each drive unpack/build/install/configure. That
+is where staging (item 4) lives, and it is the part that needs a real build to
+prove.
+
+## packagemanager setup installs, not just declares (1.11.2)
+
+`--run` was a stub. It now does the bootstrap half of the old final step, and
+`--sources` is unchanged, so `get-sources` still asks it the same question.
+
+- **wget** comes from `packagemanager install wget --run`, invoked as a
+ command. A build plan, a script, a package user and a manifest — there is no
+ "just the wget part" of that which is not a second implementation of it.
+- **The wheels** go through `pip_install_module`, extracted from `cmd_pip` so
+ both commands run the same code. `--no-deps`, in the order `_SETUP_WHEELS`
+ lists them, `--no-index --find-links /sources`: a fresh system has no CA
+ certificates, so a pip index lookup is a hang rather than an error.
+- **Idempotent.** It reads `<dist>-<version>.dist-info` in site-packages —
+ what pip itself writes, so the check is an identity rather than a guess —
+ and skips what is there. `--force` redoes it.
+- **It refuses** when the wheels are not on disk, naming each one and saying
+ they came from `get-sources` before the reboot.
+
+**A bug fixed on the way in.** `cmd_pip` took `_sanitise_user_name(mod)` and
+used it everywhere. `create_package_user` normalises internally, so the account
+it created was `p_requests` while `usermod` and `su` were handed `requests`:
+
+ usermod: user 'requests' does not exist
+
+Nobody hit it because nothing called `packagemanager pip` on a real system.
+Same species as `chown: invalid user: 'wget:wget'` — half the code asking for
+the name, half assembling it.
+
+Still owed: site-packages access is granted by `chgrp -R`/`chmod -R g+w` inside
+`pip_install_module` rather than by `lfs-helper grant-dir`. Correct group,
+right result, wrong door — fold it in with item 6.
+
+## last-step is cleaned out of the tree (1.11.1)
+
+`last_build_step.sh` stopped existing in 1.10.0, but three things still spoke
+as if it did:
+
+- **`make install` copied it into `/etc/lfs`** and aborted, because the file is
+ not in the repo. Installing was broken and nothing said so.
+- **`build-all` sealed the install directories early** when the step name was
+ `last-step` — dead since the final step became `init-accounts`, which runs as
+ root and installs nothing. The seal after the loop was already doing the work.
+- **Three tests looked for the file beside `lfs`** and skipped silently when it
+ was absent, which is every run since 1.10.0.
+
+The name stays in `_is_not_a_package`'s match: a tree generated before 1.10.0
+still carries it in `steporder` and in its manifests, and the rule must still
+say no. It is marked as legacy there.
+
+`make clean` exists now, and `make install` no longer hides a failed test
+install behind `2>/dev/null || true` — the first of item 10.
+
+## The build's last word is whether you can log in (1.11.0)
+
+`init-accounts` asks for a root password, but it is skippable — a scripted run
+has no terminal, and an interactive one can be answered with a blank line. So
+`_warn_if_no_login` is the **last thing `build-all` does**, because it is the
+last moment it can still be fixed. After the reboot the chroot is gone and the
+way back is booting the host and mounting the tree by hand.
+
+It checks the **result**, not that the step ran: `/etc/shadow` for a real root
+password, or any account between uid 1000 and 9999 with one.
+
+ !! NOBODY CAN LOG INTO THIS SYSTEM.
+ Fix it NOW, while you are still in here:
+ lfs-helper init-accounts --force
+
+`lfs-helper check-login` runs it on its own.
+
+## wget belongs to packagemanager setup (1.11.0)
+
+`packagemanager setup --sources` prints every URL setup needs, and
+`lfs build-system get-sources` asks it. **One list, owned by the tool that
+installs from it.**
+
+The first version of this had `bootstrap_packages` in `lfs`'s own config,
+resolving wget from the BLFS book separately from the tool that installs it —
+a second thing to go stale, which is the mistake this project keeps making.
+That key is gone.
+
+`setup` itself is a stub: `--sources` works, `--run` says it is not implemented
+and exits non-zero. That is deliberate — `get-sources` needs the list *now*, so
+a fresh build still downloads what the finished system will need.
+
## packagemanager setup: what it owes
Three guarantees left the suite with `last_build_step.sh` and belong to `setup`.
@@ -575,21 +853,6 @@ Also: wget needs `check_certificate = off` in `/etc/wgetrc` with the
`lfs-temporary-no-verify` marker, because a fresh system has no CA certificates
until BLFS `make-ca` — and you need wget to fetch make-ca.
-## The final step is yours, so fixes do not reach it
-
-`/etc/lfs/last_build_step.sh` is created once from a template embedded in `lfs`
-and then **belongs to the person** — it is meant to be edited, so it is never
-overwritten. Which means a fix to the template reaches nobody who already has a
-copy, and the stale copy fails at step 104 of 105, after six hours, with an
-error that looks like it came from the tools:
-
- chown: invalid user: 'wget:wget'
-
-`_LAST_STEP_STALE` lists known-broken patterns and
-`_warn_if_last_step_is_stale` reports them at generation time with the exact
-repair. A fixed copy stays quiet, or the warning becomes noise. **Fixing the
-template alone is not fixing the bug** — there are two copies by design.
-
## Things that belong to nobody
`never_claim_list` — files no package may own, however many write to them:
@@ -721,29 +984,11 @@ Then `bash lfs-sanity.sh` and read the `!!` lines.
## Open items
-1. **NEXT JOB: `packagemanager setup`.** `last_build_step.sh` does two
- unrelated things: a *bootstrap* (builds `wget`, installs seven Python wheels
- so `lfs` and `blfs` can parse the books at all) and *configuration* (root
- password, login account). Only the second belongs in a file whose header
- says "EDIT THIS FILE. It is yours." The bootstrap should become
- `packagemanager setup`: idempotent, re-runnable, testable — today it runs
- exactly once, at the end of a six-hour build.
-
- Requirements to preserve: wheels not sdists (a modern sdist needs its build
- backend and no index is reachable — `BackendUnavailable: Cannot import
- 'hatchling.build'`); dependency order with `--no-deps`; `PIP_USER=0
- PYTHONNOUSERSITE=1`; each module as its own package user; site-packages
- reached via `lfs-helper grant-dir` joining the collector group the directory
- already carries, never the `install` group; and the `/etc/wgetrc`
- `lfs-temporary-no-verify` marker, which is how the tools later find and
- remove that weakening once `make-ca` exists.
-
- **Trap:** `lfs build-system get-sources` parses the `SOURCES=()` array
- *out of `last_build_step.sh`*. If the wheel list moves, `get-sources` must
- still find it. There is a test on that path: *the final step's SOURCES are
- parsed (never executed)*.
-
-2. **`strip` is asked for but does nothing.** The answer is collected and the
+1. ~~`packagemanager setup`~~ — **done in 1.11.2.** What is left of it is the
+ grant-dir door, folded into item 6.
+
+2. **`strip` is asked for but does nothing** — and since 1.11.4 the build says
+ so at the end, so it is no longer silent. Implementing it: The answer is collected and the
prompt says NOT YET IMPLEMENTED. Not a book copy-paste: stripping rewrites
each binary, and here a file's owner is the record of which package
installed it, so it must run as that owner or the record drifts. Book 8.85
@@ -763,11 +1008,18 @@ Then `bash lfs-sanity.sh` and read the `!!` lines.
the build loop twice.
5. **Root password** — the one decision the interview cannot hold; a plaintext
- config would be worse. `last_build_step.sh` still prompts, and prints
- `chroot /mnt/lfs /usr/bin/passwd root` when non-interactive.
-
-6. **`lfs-helper` and `packagemanager_install` are two implementations of one
- thing.** Nearly every bug this project has had was the same species: one
+ config would be worse. Handled as far as it can be: `init-accounts` prompts,
+ and `_warn_if_no_login` checks the *result* as build-all's last word. A
+ non-interactive run therefore finishes with a system nobody can log into,
+ loudly. Nothing further is planned.
+
+6. **The build loop is still two implementations.** Everything else went
+ through one door in 1.11.3 — wrappers, account home, collector grants — and
+ what remains is the loop itself: `cmd_build` and `packagemanager_install`
+ each drive unpack/build/install/configure. Item 4 (staging) lives inside it.
+ Needs a real build to prove, so it is not a desk change.
+
+ The original framing, still true of the loop: Nearly every bug this project has had was the same species: one
decision held in several places, free to disagree — `cfg_bootscripts`,
`adopted.list`, the package-user prefix, the `/sources` permissions, the
account's kind. `install-as` is the entry point that makes unifying them
@@ -790,12 +1042,10 @@ Then `bash lfs-sanity.sh` and read the `!!` lines.
has no fallback for a tree built before the move. Harmless given that no
tree is migrated; worth deleting the ambiguity if migration is ever written.
-10. **A sweep for `2>/dev/null || true` is still overdue.** Six suppressed
- errors were made visible in an earlier session and they were the cause of
- nearly every bug that session. `ensure_pkgusr_roots` adds two more
- suppressed `chown`/`chmod` calls — deliberately, because the roots being
- wrong is not fatal and `verify --fix` repairs it, but it is the same
- pattern and should be counted in the sweep.
+10. ~~The `2>/dev/null || true` sweep~~ — **done in 1.11.4** for ownership and
+ permission calls, with `soft` and a test on the marker. `packagemanager_install`
+ still has seven, all benign (chmod on a scratch directory, `disown` on a
+ backgrounded mandb); they have not been reviewed one by one.
## Useful commands
@@ -814,7 +1064,7 @@ lfs-helper pkgusr-info # where each package came from
## Tests
```sh
-bash test_lfs_crosschain.sh ./lfs # 569 tests, 568 pass
+bash test_lfs_crosschain.sh ./lfs # 577 tests, 576 pass
```
Counts are kept in files, not shell variables, so subshell increments survive.
@@ -840,8 +1090,9 @@ no longer calls something. The comment explaining *why* it no longer calls it
names the thing, so a naive grep matches the explanation and the test can never
pass. Strip comments first, or anchor on the call syntax.
-Run it from a directory holding ALL five scripts plus `last_build_step.sh` —
-about 100 tests silently skip if the others are not beside `lfs`.
+Run it from a directory holding ALL five scripts, plus `lfs-sanity.sh` and
+`skel-u_xdg/.bash_profile` — about 100 tests silently skip if the others are
+not beside `lfs`.
## Working style that worked
diff --git a/Makefile b/Makefile
index d37b440..b395a75 100644
--- a/Makefile
+++ b/Makefile
@@ -3,6 +3,7 @@
# make install -> /usr
# make install PREFIX=/usr/local -> /usr/local
# make uninstall
+# make clean -> scratch files, not installed ones
# make check -> run the regression tests
#
# DESTDIR is honoured, so this works for staged installs and for installing
@@ -22,12 +23,11 @@ SYSCONFDIR ?= $(DESTDIR)/etc
# The four tools, plus the bash engine packagemanager calls.
TOOLS := lfs lfs-helper packagemanager packagemanager_install blfs
COMPLETION := lfs-completion.bash
-LAST_STEP := last_build_step.sh
TESTS := test_lfs_crosschain.sh
INSTALL ?= install
-.PHONY: all install uninstall check help
+.PHONY: all install uninstall check clean help
all: help
@@ -35,6 +35,7 @@ help:
@echo "make install [PREFIX=...] [DESTDIR=...] install the tools"
@echo "make uninstall [PREFIX=...] remove them"
@echo "make check run the tests"
+ @echo "make clean remove scratch files"
@echo ""
@echo "installs into: $(PREFIX)/bin"
@@ -63,18 +64,15 @@ install:
"$(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile"; \
echo " installed (XDG_RUNTIME_DIR is filled in on first use)"; \
fi
- @echo "==> last build step -> $(SYSCONFDIR)/lfs"
- $(INSTALL) -d $(SYSCONFDIR)/lfs
- @if [ -f "$(SYSCONFDIR)/lfs/$(LAST_STEP)" ]; then \
- echo " kept your existing $(SYSCONFDIR)/lfs/$(LAST_STEP)"; \
- $(INSTALL) -m 644 $(LAST_STEP) "$(SYSCONFDIR)/lfs/$(LAST_STEP).new"; \
- echo " new version alongside it as $(LAST_STEP).new"; \
- else \
- $(INSTALL) -m 755 $(LAST_STEP) "$(SYSCONFDIR)/lfs/$(LAST_STEP)"; \
- fi
@echo "==> tests -> $(SHAREDIR)"
$(INSTALL) -d $(SHAREDIR)
- $(INSTALL) -m 644 $(TESTS) $(SHAREDIR)/ 2>/dev/null || true
+# The tests are optional -- say so when they are absent, rather than hiding a
+# real failure. `2>/dev/null || true` hid both cases alike.
+ @if [ -f "$(TESTS)" ]; then \
+ $(INSTALL) -m 644 $(TESTS) $(SHAREDIR)/ && echo " $(TESTS)"; \
+ else \
+ echo " (not shipped: $(TESTS))"; \
+ fi
@echo ""
@echo "Installed. Check with:"
@echo " $(PREFIX)/bin/lfs --version"
@@ -88,10 +86,17 @@ uninstall:
rm -rfv $(SHAREDIR)
@echo ""
@echo "Left alone (they are yours):"
- @echo " $(SYSCONFDIR)/lfs/$(LAST_STEP)"
@echo " $(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile"
@echo " /usr/share/lfs books, settings, snapshots"
@echo " /etc/pkgusr packagemanager settings"
check:
@bash $(TESTS) ./lfs
+
+# Nothing here is compiled, so clean removes only what running the tools here
+# leaves behind: Python bytecode and the test suite's scratch directories.
+# It never touches an installed tree -- that is what uninstall is for.
+clean:
+ rm -rf __pycache__ *.pyc
+ rm -rf /tmp/lfstest.* /tmp/lfstest-count.*
+ @echo "clean. (installed files are untouched -- use 'make uninstall')"
diff --git a/README.md b/README.md
index 3d76163..86b60d5 100644
--- a/README.md
+++ b/README.md
@@ -4,8 +4,8 @@ Build and maintain an LFS/BLFS system where **every package is owned by its own
user**.
> **100% vibecode, but tested.** Prompted into existence rather than hand
-> written. 568 regression tests. It builds my own system — read it before
-> you point it at yours.
+> written. Every fix carries a regression test. It builds my own system —
+> read it before you point it at yours.
---
@@ -48,7 +48,7 @@ make uninstall
`/usr` is the default: the chroot invokes these by name, and
`packagemanager` runs `packagemanager_install` off `$PATH`.
-Installs the four tools, the completion script, and `last_build_step.sh`.
+Installs the four tools, `packagemanager_install`, and the completion script.
**Requires:** Python 3.9+, bash, coreutils, tar. Book parsing needs
`beautifulsoup4` and `requests`.
@@ -78,6 +78,18 @@ failure, or a cancel.
## Use the built system
+A fresh system has no download tool, and `lfs` and `blfs` need `requests` and
+`beautifulsoup4` to read the books. Bootstrap both, offline, from what
+`get-sources` already downloaded:
+
+```sh
+packagemanager setup # what it would do
+packagemanager setup --run
+```
+
+Each module is installed as its own package user. Run it again after fixing a
+failure; it skips what is already there.
+
```sh
packagemanager install <pkg> --recursive --run
packagemanager update <pkg> --run
@@ -199,7 +211,7 @@ let any package user delete another package's home.
The tools print a fingerprint of their own contents:
```sh
-lfs --version # lfs 1.7.1 (build 4f5f344)
+lfs --version # name, version, build id
lfs-helper --version # inside the chroot
```
@@ -214,7 +226,7 @@ enter it.
bash test_lfs_crosschain.sh ./lfs
```
-568 tests. Each encodes a bug that actually happened, with a comment explaining
+Each test encodes a bug that actually happened, with a comment explaining
what broke.
Run it from a directory holding all five scripts — about 100 tests skip
@@ -224,7 +236,12 @@ without them.
## Caveats
-- The kernel and the bootloader's kernel line are yours to finish.
+- **The kernel and the bootloader are yours.** Nothing here configures, builds
+ or installs a kernel, and nothing is written to any ESP, boot sector or
+ partition table. The machine keeps booting exactly as it does now. The build
+ ends by saying so and printing what to run.
+- A rEFInd entry can be added beside your existing bootloader if you ask for it
+ (`lfs config bootloader refind`). It only ever adds to a mounted ESP.
- `lfs-helper` and `packagemanager_install` duplicate some logic. Unifying them
is worthwhile but not done.
- Tested on x86_64 only.
diff --git a/blfs b/blfs
index f851ef0..943b4ec 100644
--- a/blfs
+++ b/blfs
@@ -118,7 +118,7 @@ CACHE_VERSION = 4
# tool + generated-script format versions. Bump SCRIPT_VERSION when the shape
# of generated install scripts changes (phases, dispatcher, ...), and
# CACHE_VERSION when the parsed/cached data structure changes.
-BLFS_VERSION = "1.10.1"
+BLFS_VERSION = "1.11.6"
def _build_id():
diff --git a/lfs b/lfs
index 721b1a1..ac5c67b 100644
--- a/lfs
+++ b/lfs
@@ -60,7 +60,7 @@ except ImportError:
" packagemanager pip install beautifulsoup4\n")
sys.exit(2)
-LFS_VERSION = "1.10.1"
+LFS_VERSION = "1.11.6"
def _build_id():
@@ -1222,6 +1222,84 @@ def _lfs_dir():
return os.environ.get("LFS") or load_config().get("lfs_mount")
+# THE ONE THING THAT COULD DESTROY THE MACHINE YOU ARE BUILDING ON.
+#
+# Every path in this tool is `$LFS/something`. If $LFS is the running system,
+# `$LFS/usr` is the HOST's /usr -- and `build-system restart --run` deletes the
+# top-level directories of the tree it is given. Nothing checked that the tree
+# was not the host's own: a config holding `/` as the mount point, an
+# `export LFS=/` in the wrong shell, or a typo in the interview was enough.
+#
+# The chroot's bind mounts are already guarded (restart refuses while /dev is
+# mounted, because deleting through it would reach the host's /dev). This is
+# the other half: refusing when the tree IS the host.
+#
+# It lives in require_lfs_for_root because that is the single door every
+# command resolves $LFS through -- put it anywhere else and the next command
+# added is the one that misses it.
+_HOST_DIRS = ("/", "/usr", "/etc", "/boot", "/bin", "/sbin", "/lib", "/lib64",
+ "/var", "/home", "/root", "/opt", "/srv", "/run", "/dev",
+ "/proc", "/sys",
+ # merged-/usr: /bin is a symlink to usr/bin, so realpath("/bin")
+ # is "/usr/bin" and the names above never match it. A check that
+ # only looked at the resolved path let $LFS=/bin through.
+ "/usr/bin", "/usr/sbin", "/usr/lib", "/usr/lib64", "/usr/share",
+ "/usr/include", "/usr/local", "/usr/src")
+
+# Trees that belong to the running system whole. Anything inside one of these
+# is the host, however deep -- $LFS=/usr/src/lfs is not a build tree, it is a
+# directory in the system's own /usr. /home and /srv are deliberately NOT
+# here: /home/you/lfs is a perfectly ordinary place to build.
+_HOST_TREES = ("/usr/", "/etc/", "/boot/", "/bin/", "/sbin/", "/lib/",
+ "/lib64/", "/proc/", "/sys/", "/dev/", "/run/", "/var/")
+
+
+def _is_host_path(p):
+ """True if this path is part of the running system. The test behind both
+ the interview's refusal and the hard stop below, so they cannot disagree."""
+ try:
+ real = os.path.realpath(p)
+ except OSError:
+ return False
+ # BOTH spellings. On a merged-/usr system /bin resolves to /usr/bin, so
+ # checking only the resolved path misses the name the person typed, and
+ # checking only the name misses where it actually points.
+ given = os.path.abspath(os.path.expanduser(str(p))).rstrip("/") or "/"
+ if real in _HOST_DIRS or given in _HOST_DIRS:
+ return True
+ for t in _HOST_TREES:
+ if real.startswith(t) or given.startswith(t):
+ return True
+ try:
+ if os.path.samefile(real, "/"):
+ return True
+ except OSError:
+ pass
+ for d in ("/usr", "/etc", "/boot"):
+ cand = os.path.join(real, d.lstrip("/"))
+ try:
+ if os.path.exists(cand) and os.path.samefile(cand, d):
+ return True
+ except OSError:
+ continue
+ return False
+
+
+def _refuse_host_tree(lfs):
+ """Hard stop if $LFS is, or contains, the running system."""
+ if not _is_host_path(lfs):
+ return lfs
+ real = os.path.realpath(lfs)
+ sys.stderr.write(
+ "\n!! REFUSING: %s is part of the running system.\n"
+ " Every path this tool writes is $LFS/something, so this would\n"
+ " build into, and `restart` would delete, the machine you are\n"
+ " sitting on.\n"
+ "\n Set it to the mount point of the LFS partition:\n"
+ " lfs build-system session\n" % real)
+ sys.exit(2)
+
+
def require_lfs_for_root():
"""Ensure an LFS mount is known (book rule: root needs $LFS after 2.4). Falls
back to the saved session config and exports it so child shells inherit it."""
@@ -1232,6 +1310,7 @@ def require_lfs_for_root():
" Configure it once: lfs build-system session\n"
" or for this shell: export LFS=/mnt/lfs\n")
sys.exit(2)
+ _refuse_host_tree(d)
os.environ["LFS"] = d
return d
@@ -1301,10 +1380,10 @@ _SESSION_KEYS = [
("locale", "system locale (see 'locale -a')", "en_US.UTF-8"),
("paper_size", "paper size for groff: A4 or letter", "A4"),
# Everything below is a decision the build used to discover halfway
- # through, or at the very end -- `last_build_step.sh` blocked on `passwd
- # root` after a six-hour build, and its own comment admits the
- # non-interactive path leaves the system "unbootable-but-known". Asking
- # here is what lets `lfs run` and `build-all` finish without a person.
+ # through, or at the very end -- the old final step blocked on `passwd
+ # root` after a six-hour build, and its non-interactive path left the
+ # system "unbootable-but-known". Asking here is what lets `lfs run` and
+ # `build-all` finish without a person.
("timezone", "time zone for the built system (e.g. Europe/Berlin, "
"or UTC)", "UTC"),
# Deleting a working cross-toolchain should never happen by default, so
@@ -1320,10 +1399,11 @@ _SESSION_KEYS = [
("strip", "strip debug symbols at the end? Saves several GB, "
"but rewrites every binary -- slower, and harder to "
"debug a crash afterwards. NOT YET IMPLEMENTED: the "
- "answer is stored and nothing acts on it (yes/no)",
+ "answer is stored and the build says so at the end "
+ "rather than acting on it (yes/no)",
"no"),
- # The account you will actually log in as. last_build_step.sh reads this;
- # without it, a freshly booted system has only root, with no password.
+ # The account you will actually log in as. The `init-accounts` step reads
+ # this; without it, a freshly booted system has only root, with no password.
("main_user", "login account to create on the built system "
"(blank for none)", ""),
]
@@ -1353,6 +1433,17 @@ def _prompt_session_config(cfg, force=False):
continue
val = input(f" {desc} [{cur}]: ").strip()
val = val if val else cur
+ # Refuse a mount point that is the running system, HERE, where you can
+ # simply type another one. The guard in require_lfs_for_root catches
+ # it later too, but by then it is stored, every command refuses, and
+ # the fix is to work out which config key to edit.
+ if k == "lfs_mount" and val:
+ while _is_host_path(val):
+ sys.stderr.write(
+ " !! %s is part of the running system -- refusing.\n"
+ " The LFS mount point is an empty directory the LFS\n"
+ " partition mounts on, e.g. /mnt/lfs.\n" % val)
+ val = input(f" {desc} [{cur}]: ").strip() or cur
if k == "collector_import_file" and val:
# Accepting a path that is not there means the groups silently
# never get imported and every sharing question comes back.
@@ -1598,6 +1689,12 @@ def _verify_lfs_ownership():
lfs = _lfs_dir()
if not lfs:
return
+ # This function runs `chown -R lfs` on $LFS/usr, $LFS/lib64, $LFS/var and
+ # $LFS/etc. It resolves $LFS itself rather than going through
+ # require_lfs_for_root, so it was the one destructive path with no check
+ # that the tree is not the running system -- and on a host, /usr has no
+ # package users, so _handover_is_safe would have said yes.
+ _refuse_host_tree(lfs)
import pwd as _pwd
try:
want = _pwd.getpwnam("lfs").pw_uid
@@ -2838,17 +2935,15 @@ CONFIG_KEYS = [
("editor", "editor for --edit; default: $EDITOR or vim"),
("sudo", "prefix suggested root commands with 'sudo' "
"(on/off; default off)"),
- ("bootstrap_packages",
- "BLFS packages the built system needs before it can "
- "fetch anything at all (default: wget). Their tarballs "
- "are downloaded by get-sources, while there is still a "
- "network; `packagemanager setup` installs them on the "
- "booted system. Blank for none"),
- ("bootloader", "which bootloader to set up: none (default), refind, "
- "or grub. GRUB's book instructions run "
- "'grub-install /dev/sda', which writes to a disk's boot "
- "sector -- with 'refind' that section is skipped "
- "entirely and a rEFInd script is generated instead"),
+ ("bootloader", "which bootloader to set up: none (DEFAULT -- nothing "
+ "is written to any ESP, boot sector or partition "
+ "table, and the machine keeps booting exactly as it "
+ "does now), refind, or grub. GRUB's book instructions "
+ "run 'grub-install /dev/sda', which writes to a disk's "
+ "boot sector -- with 'refind' that section is skipped "
+ "entirely and a rEFInd script is generated instead, "
+ "which only ADDS an entry beside your existing "
+ "bootloader"),
("esp", "path of the EFI System Partition inside the built "
"system (e.g. /boot/efi). rEFInd installs there; it is "
"never formatted, and an existing installation is left "
@@ -3476,7 +3571,7 @@ def write_chroot_env(lfs, quiet=False):
f'# languages to keep -- see `lfs-helper prune-locales`\n'
f'export LFS_LOCALES="{cfg.get("locales") or "en de"}"\n'
f'# answers collected by `lfs build-system session`, so the\n'
- f'# chroot side never has to stop and ask. last_build_step.sh\n'
+ f'# chroot side never has to stop and ask. `init-accounts`\n'
f'# reads LFS_MAIN_USER instead of blocking on a prompt after\n'
f'# a six-hour build.\n'
f'export LFS_TIMEZONE="{cfg.get("timezone") or "UTC"}"\n'
@@ -4081,60 +4176,36 @@ def _gen_blfs_extra(outdir, order, unresolved):
-def bootstrap_packages():
- """BLFS packages the built system needs before it can fetch anything.
-
- Just wget by default: it is the one thing LFS leaves out that makes a fresh
- system unable to download its own next package. `packagemanager setup`
- installs it; this is only about having the tarball on disk in time.
- """
- v = load_config().get("bootstrap_packages")
- if v is None:
- return ["wget"]
- return [p for p in str(v).replace(",", " ").split() if p]
-
-
def bootstrap_sources():
- """Their download URLs, asked of the BLFS book.
+ """What the built system needs downloaded before it can fetch anything.
- `blfs` runs on the host and keeps its own book store, so this works before
- the chroot exists. Asking the book rather than hardcoding URLs means there
- is one place that knows where wget comes from, and it is the same place
- `packagemanager` will ask when it installs it.
+ ASKED OF `packagemanager setup`, which owns that decision -- wget and the
+ Python modules the tools need are its job to install, so the list of what
+ to download is its to declare. This command only has to make sure they are
+ on disk before the chroot is sealed, because get-sources is the last moment
+ there is still a network.
+
+ Two tools, one list. A copy here would be a second thing to go stale, and
+ the first version of this did exactly that: `bootstrap_packages` in lfs's
+ own config, resolving wget separately from the tool that installs it.
"""
import subprocess
- urls = []
- for pkg in bootstrap_packages():
- try:
- out = subprocess.run(["blfs", "sources", pkg],
- capture_output=True, text=True, timeout=120)
- except (OSError, subprocess.SubprocessError):
- warn("! could not run `blfs sources %s` -- is blfs installed?" % pkg)
- continue
- got = [l.strip() for l in (out.stdout or "").splitlines()
- if l.strip().startswith(("http://", "https://", "ftp://"))]
- if not got:
- warn("! the BLFS book gave no download for '%s'." % pkg)
- warn(" A fresh system will have no way to fetch anything. Either")
- warn(" fetch a book (blfs fetch) or drop it:")
- warn(" lfs config bootstrap_packages ''")
- continue
- urls += got
+ try:
+ out = subprocess.run(["packagemanager", "setup", "--sources"],
+ capture_output=True, text=True, timeout=180)
+ except (OSError, subprocess.SubprocessError):
+ warn("! could not run `packagemanager setup --sources`.")
+ warn(" The built system will have no way to fetch anything until you")
+ warn(" put a download tool on it by hand.")
+ return []
+ urls = [l.strip() for l in (out.stdout or "").splitlines()
+ if l.strip().startswith(("http://", "https://", "ftp://"))]
+ if not urls:
+ warn("! `packagemanager setup --sources` listed nothing.")
+ for line in (out.stderr or "").splitlines():
+ warn(" %s" % line)
return urls
-
-
-
-
-
-
-
-
-
-
-
-
-
def _write_tool_stamps(lfs, tools):
"""Record the md5 of every tool we copied in.
diff --git a/lfs-helper b/lfs-helper
index c8cf943..6503e33 100644
--- a/lfs-helper
+++ b/lfs-helper
@@ -22,7 +22,7 @@ set -u
# Bump on every change that alters behaviour. `lfs-helper --version` makes it
# possible to tell at a glance whether the copy inside the chroot is the one
# that was just fixed -- guessing at that has wasted a lot of time.
-LFS_HELPER_VERSION="1.10.1"
+LFS_HELPER_VERSION="1.11.6"
# A fingerprint of this file's own contents. A hand-maintained version number
# goes stale the moment someone forgets to bump it -- and the reason for
@@ -185,6 +185,35 @@ say() { echo "$*"; }
ok() { echo "${C_OK}$*${C_OFF}"; }
# yellow on stderr: something needs attention, and is not part of normal flow
warn() { echo "${C_WARN}$*${C_OFF}" >&2; }
+
+# `cmd ... 2>/dev/null || true` -- the pattern that has cost this project more
+# debugging than any other. A whole tree came out with every package home
+# root:root 755, and not one line of output said why; the reason it took a
+# filesystem check, an su test and three rounds of guessing was that the
+# failure had been suppressed at the point it happened.
+#
+# There are two honest reasons to write it, and they are different:
+#
+# soft <what> -- <cmd ...> the command MAY fail and the build carries on,
+# but the failure is SAID. Use this wherever a
+# failure changes what the system looks like:
+# ownership, modes, group membership, backups.
+#
+# <cmd> 2>/dev/null || true only where failing is NORMAL and means nothing:
+# mkdir -p on a directory that exists, appending
+# to a log that may not be writable, sorting a
+# file in place. Every remaining one carries a
+# comment saying which.
+soft() {
+ local what="$1"; shift
+ [ "${1:-}" = "--" ] && shift
+ local err rc
+ err="$("$@" 2>&1 >/dev/null)"; rc=$?
+ [ "$rc" = 0 ] && return 0
+ warn "!! $what"
+ [ -n "$err" ] && warn " $err"
+ return 0
+}
# yellow on stdout: needs attention, but IS part of the flow -- a retry, a
# question, a directory being granted. Same colour because it means the same
# thing; stdout because it belongs in sequence with the build output.
@@ -214,7 +243,13 @@ mkstate() {
mkdir -p "$STATE" "$SCRIPTS" "$MANIFESTS" "$LOGS" \
"$STATE_PROGRESS" "$STATE_GROUPS" "$STATE_CONF" || return 0
if [ "$(id -u)" = 0 ]; then
+ # SUPPRESSED DELIBERATELY. mkstate runs at the top of every command,
+ # including ones from before book 7.6, when the `install` group does
+ # not exist yet and this cannot succeed. Reporting it would print a
+ # failure on every invocation for the first third of a build, which is
+ # how a report becomes noise. seal-install-dirs sets it for real.
set_install_dir_owner "$STATE" 2>/dev/null || true
+ # SUPPRESSED DELIBERATELY: same moment, same reason as the line above.
real_chmod 1775 "$STATE" 2>/dev/null || true
fi
}
@@ -655,7 +690,8 @@ cmd_init_pkgusr() {
# o+t (sticky) is deliberately deferred to seal-install-dirs, because
# it would stop package users replacing files the temporary system
# installed as root.
- real_chmod g+w "$d" 2>/dev/null || true
+ soft "$d is not group-writable -- package users cannot install into it" \
+ -- real_chmod g+w "$d"
n=$((n+1))
done < <(install_dirs_list)
ok "# $n install directories are now root:install (group-writable)"
@@ -798,11 +834,14 @@ EOF
# Steps that are not packages, and must never get a package user.
#
-# last-step, init-* and refind are actions, not software: they own no files of
-# their own and installing "as" them makes no sense. A `last-step` user
-# appeared because a manifest existed under that name, and everything the step
-# touched -- including wget's binary, installed as the wget user -- was then
-# handed to it.
+# init-* and refind are actions, not software: they own no files of their own
+# and installing "as" them makes no sense. A `last-step` user appeared because
+# a manifest existed under that name, and everything the step touched --
+# including wget's binary, installed as the wget user -- was then handed to it.
+#
+# `last-step` itself was removed in 1.10.0 and no tool generates it any more.
+# It stays in the match below for trees generated before that: their steporder
+# and manifests still carry the name, and the rule must still say no.
#
# cfg_* is NOT decidable from the name. Most chapter-9 "configuration"
# sections write a file (/etc/hostname, /etc/hosts) and are root's work, but
@@ -1493,7 +1532,11 @@ ensure_pkgusr_roots() {
# another package's home. With it, only the owner can. Group write then
# grants only the ability to CREATE, which nothing unprivileged does --
# `add-user` is root-only -- so this is consistency, not capability.
+ # SUPPRESSED DELIBERATELY, same reason as mkstate: the roots are
+ # created before the `install` group exists. `lfs-helper verify`
+ # checks the result afterwards, which is the honest place to find out.
set_install_dir_owner "$r" 2>/dev/null || true
+ # SUPPRESSED DELIBERATELY: same moment, same reason as the line above.
real_chmod 1775 "$r" 2>/dev/null || true
done < <(pkgusr_roots)
}
@@ -1741,7 +1784,8 @@ grant_dir_access() {
if [ -n "$curgrp" ] && [ "$curgrp" != root ] \
&& case "$curgrp" in "$COLLECTOR_PREFIX"*) true ;; *) false ;; esac \
&& id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx "$curgrp"; then
- real_chmod g+rwx "$dir" 2>/dev/null || true
+ soft "$dir is group $curgrp but not group-writable -- the grant will not take" \
+ -- real_chmod g+rwx "$dir"
say " $dir -> already group $curgrp, and '$user' is a member"
return 0
fi
@@ -1836,7 +1880,8 @@ grant_dir_access() {
# is already a member of the group by then, that grant is automatic (see
# the fast path at the top of this function).
real_chgrp "$grp" "$dir" 2>/dev/null || return 1
- real_chmod g+rwx "$dir" 2>/dev/null || true
+ soft "$dir is group $grp but not group-writable -- the grant will not take" \
+ -- real_chmod g+rwx "$dir"
say " $dir -> group $grp (owner $owner, + $user)"
mkdir -p "$(dirname "$GRANTED")"
grep -qxF "$grp|$user|$owner" "$GRANTED" 2>/dev/null \
@@ -2403,8 +2448,10 @@ EOF
# an install directory: root:install and group-writable during the build.
# Inheriting that would let any package user rewrite the wrappers -- that
# is, rewrite the rule that stops it changing ownership.
- chmod 0755 "$WRAPPERS" 2>/dev/null || true
- chown root:root "$WRAPPERS" "$WRAPPERS"/* 2>/dev/null || true
+ soft "could not set $WRAPPERS to 0755 -- a package user may be able to rewrite the wrappers" \
+ -- chmod 0755 "$WRAPPERS"
+ soft "could not give $WRAPPERS to root -- the wrappers are writable by whoever owns them now" \
+ -- chown root:root "$WRAPPERS" "$WRAPPERS"/*
}
# Re-apply the install-directory permissions.
@@ -2682,15 +2729,32 @@ NCLIST
printf '%s\n' "$STATE"
}
-is_never_claimed() {
- local f="${1%/}" p
- [ -n "$f" ] || return 1
+# The list, read ONCE into an array.
+#
+# is_never_claimed used to run `< <(never_claim_list)` on every call, which is a
+# process substitution -- a fork -- per path. `verify` calls it for every path
+# in every manifest: 66265 forks on a finished system, and that was only half of
+# them. The command ran for minutes in silence.
+_TAB=$'\t'
+_NEVER_CLAIM=()
+_never_claim_load() {
+ [ "${#_NEVER_CLAIM[@]}" -gt 0 ] && return 0
+ local p
while IFS= read -r p; do
[ -n "$p" ] || continue
p="${SNAP_ROOT%/}${p}"
- [ "$f" = "${p%/}" ] && return 0
- case "$f" in "${p%/}"/*) return 0 ;; esac
+ _NEVER_CLAIM+=( "${p%/}" )
done < <(never_claim_list)
+}
+
+is_never_claimed() {
+ local f="${1%/}" p
+ [ -n "$f" ] || return 1
+ _never_claim_load
+ for p in "${_NEVER_CLAIM[@]}"; do
+ [ "$f" = "$p" ] && return 0
+ case "$f" in "$p"/*) return 0 ;; esac
+ done
return 1
}
@@ -2845,8 +2909,12 @@ _install_staged_tree() {
continue
fi
mkdir -p "$dest" || { rc=1; break; }
- real_chown --reference="$d" "$dest" 2>/dev/null || true
- real_chmod --reference="$d" "$dest" 2>/dev/null || true
+ # A staged directory that does not inherit the live one's owner and
+ # mode is how a package quietly takes /usr/bin.
+ soft "$dest did not inherit the owner of $d" \
+ -- real_chown --reference="$d" "$dest"
+ soft "$dest did not inherit the mode of $d" \
+ -- real_chmod --reference="$d" "$dest"
done < <(find "$stage" -mindepth 1 -type d 2>/dev/null | sort)
[ "$rc" = 0 ] || return 1
@@ -3348,9 +3416,11 @@ cmd_build() {
real_chown "$owner:$owner" "$_stage" 2>/dev/null \
|| warn "# could not give $_stage to '$owner'"
fi
- real_chmod 755 "$_stage" 2>/dev/null || true
+ soft "$_stage is not 755 -- the package user may not be able to stage into it" \
+ -- real_chmod 755 "$_stage"
# the parent too, or the package user cannot traverse into it
- real_chmod 755 "$(dirname "$_stage")" 2>/dev/null || true
+ soft "$(dirname "$_stage") is not 755 -- the package user cannot traverse into the staging tree" \
+ -- real_chmod 755 "$(dirname "$_stage")"
else
_stage=""
fi
@@ -3383,7 +3453,8 @@ cmd_build() {
# already owned correctly and nothing has to be chowned afterwards
user_exists "$owner" && real_chown "$owner:$owner" "$_bldroot" 2>/dev/null
fi
- real_chmod 755 "$_bldroot" 2>/dev/null || true
+ soft "$_bldroot is not 755 -- the build may not be able to enter its own tree" \
+ -- real_chmod 755 "$_bldroot"
local envpass="LFS_CC_PHASE='$phase' BUILD_ROOT='$_bldroot'"
[ -n "$_stage" ] && envpass="$envpass DESTDIR='$_stage'"
@@ -3851,6 +3922,7 @@ cmd_build() {
echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper"
[ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script"
} > "$_home/VERSION"
+ # benign: a note in the package's own home, rewritten every install
real_chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true
fi
@@ -4028,6 +4100,33 @@ _needs_adoption() {
# Give one user access to one directory, using the normal rules.
+# The wrappers, on demand.
+#
+# `packagemanager_install` shipped its own 480-line copy and wrote it to a
+# fresh /tmp directory on every run -- a second set of wrappers, with its own
+# idea of what `chown` should do, for the same job. This is the door it asks
+# through instead. Idempotent: writing them again is how a fixed wrapper
+# reaches a system that already has the old one.
+cmd_make_wrappers() {
+ need_root
+ local run=0
+ case "${1:-}" in --run) run=1 ;; esac
+ if [ "$run" != 1 ]; then
+ say "# would write the package-user wrappers to $WRAPPERS"
+ say "# $(cd "$WRAPPERS" 2>/dev/null && echo "currently: $(ls | tr '\n' ' ')")"
+ say "# apply with: lfs-helper make-wrappers --run"
+ return 0
+ fi
+ make_wrappers
+ ok "wrappers written to $WRAPPERS"
+ printf '%s\n' "$WRAPPERS"
+}
+
+# Where they are, for a caller that only needs the path.
+cmd_wrapper_dir() {
+ printf '%s\n' "$WRAPPERS"
+}
+
cmd_grant_dir() {
need_root
local dir="${1:?usage: lfs-helper grant-dir <dir> <user> [--run]}"
@@ -4109,10 +4208,8 @@ _has_orphaned_files() {
local r="${SNAP_ROOT%/}" hit
# see _vfy_orphans: before 7.6 nothing resolves, so everything looks orphaned
_user_db_ready || return 1
- hit="$(find "${r:-/}" -xdev -nouser \
- -not -path "$r/dev/*" -not -path "$r/proc/*" \
- -not -path "$r/sys/*" -not -path "$r/run/*" \
- -not -path "$r/sources/*" \
+ scan_prune_set
+ hit="$(find "${r:-/}" -xdev -nouser "${SCAN_PRUNE[@]}" \
-print -quit 2>/dev/null)"
[ -n "$hit" ]
}
@@ -4472,7 +4569,7 @@ cmd_sort_users() {
continue
fi
if [ "$run" = 1 ]; then
- cp -p "$f" "$f.bak" 2>/dev/null || true
+ soft "no backup of $f -- editing it without one" -- cp -p "$f" "$f.bak"
cat "$tmp" > "$f"
rm -f "$tmp"
ok " $f sorted by id (previous kept as $f.bak)"
@@ -4607,7 +4704,10 @@ _vfy_manifest_ownership() {
# gave man-pages (the first chapter-8 package) uid 10000 and binutils --
# built first, in chapter 5 -- a much higher one, which makes the uids
# useless for reading the build order off the passwd file.
- for man in $(_manifests_in_build_order) $(_manifests_in_build_order dirs); do
+ local _mans _nman=0 _ntot
+ _mans="$(_manifests_in_build_order) $(_manifests_in_build_order dirs)"
+ _ntot="$(printf '%s\n' $_mans | grep -c .)"
+ for man in $_mans; do
[ -e "$man" ] || continue
[ -s "$man" ] || continue
case "$man" in *.dirs) kind=dir ;; *) kind=file ;; esac
@@ -4642,6 +4742,30 @@ _vfy_manifest_ownership() {
_vfy_n_fixed=$((_vfy_n_fixed + 1))
fi
+ # Which package, live. This pass walks every path in every manifest,
+ # and it used to print nothing until it had finished. A command that
+ # works for minutes in silence is indistinguishable from one that has
+ # hung -- and was reported as one.
+ _nman=$((_nman + 1))
+ [ -t 1 ] && printf '\r [%d/%d] %-24s\033[K' "$_nman" "$_ntot" "$name" >&2
+
+ # Every owner in ONE stat call, not one call per path.
+ #
+ # `stat -c %U` per path is a fork per path, and `verify` walks every
+ # path in every manifest -- 66265 of them on a finished system. With
+ # is_never_claimed forking too that was ~130000 processes, which is
+ # where the minutes went. xargs batches them into a handful.
+ # A REAL tab in the format string. `stat -c '%U\t%n'` prints a
+ # literal backslash-t -- stat does not expand escapes -- so every line
+ # came back as one field, the map stayed empty, and the loop below fell
+ # through to the per-path stat it was written to avoid. It would have
+ # been correct and slow, which is the hardest kind of wrong to notice.
+ declare -A _own_of=()
+ while IFS="$_TAB" read -r _o _f; do
+ [ -n "$_f" ] && _own_of["$_f"]="$_o"
+ done < <(grep -v '^[[:space:]]*$' "$man" \
+ | tr '\n' '\0' | xargs -0 -r stat -c "%U${_TAB}%n" 2>/dev/null)
+
while IFS= read -r p; do
[ -n "$p" ] || continue
p="$(strip_host_prefix "$p")"
@@ -4653,7 +4777,8 @@ _vfy_manifest_ownership() {
is_never_claimed "$p" && continue
case "$p" in "${BUILD_ROOT:-${SNAP_ROOT%/}/build}"/*) continue ;; esac
_vfy_n_checked=$((_vfy_n_checked + 1))
- cur="$(stat -c %U "$p" 2>/dev/null)" || continue
+ cur="${_own_of[$p]:-}"
+ [ -n "$cur" ] || cur="$(stat -c %U "$p" 2>/dev/null)" || continue
[ "$cur" = "$owner" ] && continue
# A shared install directory belongs to root:install, never to a
# package -- even when the package's manifest records creating it.
@@ -4676,6 +4801,8 @@ _vfy_manifest_ownership() {
fi
done < "$man"
done
+ [ -t 1 ] && printf '\r\033[K' >&2
+ return 0
}
# --- rule 2: shared install directories belong to root:install ------------- #
@@ -4715,8 +4842,12 @@ _vfy_install_dirs() {
"$(stat -c '%U:%G' "$live" 2>/dev/null) $mode" \
"root:install $want_mode"
if [ "$fix" = 1 ]; then
+ # COUNTED AS FIXED, so it must not fail in silence: the next
+ # run would find the same directory and report it again with
+ # no explanation of why the repair never took.
set_install_dir_owner "$live" || true
- real_chmod "$want_mode" "$live" 2>/dev/null || true
+ soft "could not set $live to $want_mode -- counted as fixed but it did not take" \
+ -- real_chmod "$want_mode" "$live"
_vfy_n_fixed=$((_vfy_n_fixed + 1))
fi
fi
@@ -4724,8 +4855,53 @@ _vfy_install_dirs() {
}
# --- rule 3: every file has an owner that exists here ---------------------- #
+# The paths no ownership scan should walk, as find arguments.
+#
+# ONE list, because there were five and they had already diverged: the snapshot
+# scans excluded the build scratch and the orphan scan did not, so when build
+# trees moved from /build into each package user's home (1.7.6) the orphan scan
+# started walking every unpacked source tree in the system. `lfs-helper verify`
+# went from seconds to minutes with no output, and the sanity report filled up
+# with tcl's own documentation:
+#
+# !! files with no owner (first 40):
+# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm
+#
+# A tarball can contain any uid it likes. Unpacked sources are scratch -- they
+# are not installed, nothing owns them, and asking who does has no answer.
+scan_prune_paths() {
+ local r="${SNAP_ROOT%/}"
+ printf '%s\n' \
+ "$r/dev" "$r/proc" "$r/sys" "$r/run" "$r/tmp" \
+ "$r/sources" "${BUILD_ROOT:-$r/build}" "$STATE"
+ # every package's unpacked source tree, wherever the homes are
+ local d
+ while IFS= read -r d; do
+ [ -n "$d" ] || continue
+ printf '%s\n' "$d/*/$PKGUSR_BUILD_SUBDIR"
+ done < <(pkgusr_roots)
+}
+
+# The same list as a find argument ARRAY, in SCAN_PRUNE.
+#
+# An array, not a string spliced through `eval`: the patterns contain `*`, and
+# eval lets the shell expand them against the real filesystem before find ever
+# sees them. `/usr/src/pkgusr/*/src/*` then becomes a handful of literal
+# directory names instead of a wildcard, and the prune silently matches almost
+# nothing -- which is exactly how it looked like it was working.
+SCAN_PRUNE=()
+scan_prune_set() {
+ local p
+ SCAN_PRUNE=()
+ while IFS= read -r p; do
+ [ -n "$p" ] || continue
+ SCAN_PRUNE+=( -not -path "$p" -not -path "$p/*" )
+ done < <(scan_prune_paths)
+}
+
_vfy_orphans() {
local fix="$1" r="${SNAP_ROOT%/}" p n=0
+ scan_prune_set
# NOT before book 7.6 has written /etc/passwd.
#
# `find -nouser` asks whether a uid resolves to a name. With no passwd file
@@ -4751,10 +4927,7 @@ _vfy_orphans() {
real_chown -h lfs:lfs "$p" 2>/dev/null \
&& _vfy_n_fixed=$((_vfy_n_fixed + 1))
fi
- done < <(find "${r:-/}" -xdev -nouser \
- -not -path "$r/dev/*" -not -path "$r/proc/*" \
- -not -path "$r/sys/*" -not -path "$r/run/*" \
- -not -path "$r/sources/*" 2>/dev/null)
+ done < <(find "${r:-/}" -xdev -nouser "${SCAN_PRUNE[@]}" 2>/dev/null)
[ "$n" -gt 5 ] && say " ... and $((n - 5)) more with no owner"
[ -n "$VERIFY_LOG" ] && say " all $n: grep ^no-owner $VERIFY_LOG"
}
@@ -4774,6 +4947,7 @@ _vfy_orphans() {
# which package they belong to.
_vfy_build_user_leftovers() {
local fix="$1" r="${SNAP_ROOT%/}" p n=0
+ scan_prune_set
_user_db_ready || return 0
user_exists lfs || return 0
while IFS= read -r p; do
@@ -4797,13 +4971,7 @@ _vfy_build_user_leftovers() {
# the chroot and should be root's. Its CONTENTS are not -- scratch has no
# owner worth arguing about. The state directory is under $SRCROOT, which
# is excluded wholesale.
- done < <(find "${r:-/}" -xdev -user lfs \
- -not -path "$r/dev/*" -not -path "$r/proc/*" \
- -not -path "$r/sys/*" -not -path "$r/run/*" \
- -not -path "$r/tmp/*" \
- -not -path "$r/sources" -not -path "$r/sources/*" \
- -not -path "${BUILD_ROOT:-$r/build}/*" \
- -not -path "$STATE/*" \
+ done < <(find "${r:-/}" -xdev -user lfs "${SCAN_PRUNE[@]}" \
-not -path "$SRCROOT/*" 2>/dev/null)
[ "$n" -gt 5 ] && say " ... and $((n - 5)) more still owned by the build user"
[ "$n" -gt 0 ] && [ -n "$VERIFY_LOG" ] \
@@ -4813,9 +4981,17 @@ _vfy_build_user_leftovers() {
_vfy_make_lfs_user() {
if have_shadow_tools; then
- groupadd -g "$LFS_BUILD_UID" lfs 2>/dev/null || true
- useradd -M -u "$LFS_BUILD_UID" -g lfs -s /bin/bash \
- -c "build user from outside the chroot" lfs 2>/dev/null || true
+ # Failing here is NORMAL on a second run -- the account exists. It
+ # is not normal on the first, and that case used to look identical.
+ if ! getent group lfs >/dev/null 2>&1; then
+ soft "could not create the 'lfs' group" \
+ -- groupadd -g "$LFS_BUILD_UID" lfs
+ fi
+ if ! id lfs >/dev/null 2>&1; then
+ soft "could not create the 'lfs' build user" \
+ -- useradd -M -u "$LFS_BUILD_UID" -g lfs -s /bin/bash \
+ -c "build user from outside the chroot" lfs
+ fi
else
printf 'lfs:x:%s:\n' "$LFS_BUILD_UID" >> "$ETC/group"
printf 'lfs:x:%s:%s:build user from outside the chroot:/:/bin/bash\n' \
@@ -4855,7 +5031,7 @@ _vfy_user_order() {
if ! cmp -s "$f" "$tmp"; then
_vfy_report order "$f" "build order" "id order"
if [ "$fix" = 1 ]; then
- cp -p "$f" "$f.bak" 2>/dev/null || true
+ soft "no backup of $f -- editing it without one" -- cp -p "$f" "$f.bak"
cat "$tmp" > "$f"
_vfy_n_fixed=$((_vfy_n_fixed + 1))
fi
@@ -5156,6 +5332,8 @@ _vfy_unclaimed() {
[ -n "$p" ] || continue
is_never_claimed "$p" || printf '%s\n' "$p" >> "$_keep"
done < "$found"
+ # benign: $_keep is a filtered copy of $found -- if the move fails the
+ # unfiltered list is still correct, just longer
mv -f "$_keep" "$found" 2>/dev/null || true
n="$(LC_ALL=C comm -23 "$found" "$list" 2>/dev/null | wc -l | tr -d ' ')"
@@ -5199,6 +5377,7 @@ cmd_verify() {
local problems=0 fixed=0
+ detail "# checking for accounts that share an id ..."
# ---- 1. two accounts sharing one id ----------------------------------- #
# Ownership is a number: a collision makes files report as the wrong name
# and a package unable to write its own files.
@@ -5207,6 +5386,7 @@ cmd_verify() {
warn " (verify cannot repair this: which name owns what is yours to decide)"
fi
+ detail "# checking the shared install directories ..."
# ---- 2. the shared install directories -------------------------------- #
# root:install, group-writable. A staged install used to stamp the package
# user's ownership onto these, handing /usr and /usr/bin to one package and
@@ -5217,6 +5397,7 @@ cmd_verify() {
# to be a second copy that did neither.
_vfy_install_dirs "$fix" || problems=$((problems+1))
+ detail "# checking every package owns the files it installed ..."
# ---- 3. every manifest's files belong to its package ------------------- #
#
# A SECOND copy of _vfy_manifest_ownership used to live here, inline, and
@@ -5246,9 +5427,11 @@ cmd_verify() {
ok "package files: $checked path(s) owned by the right package"
fi
+ detail "# looking for files no manifest claims ..."
# ---- 3b. files no manifest claims ------------------------------------- #
_vfy_unclaimed || problems=$((problems+1))
+ detail "# looking for files with no owner ..."
# ---- 4. files with no owner at all ------------------------------------ #
if _has_orphaned_files; then
warn "some files are owned by a uid with no passwd entry"
@@ -5272,6 +5455,7 @@ cmd_verify() {
ok "every file has a real owner"
fi
+ detail "# checking homes, the build user and the uid order ..."
# ---- 4b. the passes the dead copy had and this one did not ------------- #
# homes owned by their own user, the build user's leftovers, and the uid
# order. They were written, and reachable only from the definition bash
@@ -5281,6 +5465,7 @@ cmd_verify() {
_vfy_build_user_leftovers "$fix" || true
_vfy_user_order "$fix" || true
+ detail "# looking for left-over staging trees ..."
# ---- 5. staging trees left behind ------------------------------------- #
local st n_stage=0
for st in "$STATE"/stage/*; do
@@ -5760,19 +5945,12 @@ cmd_build_all() {
warn " lfs-helper build-all"
return 0
fi
- # Seal immediately BEFORE the final step. By now every package is
- # built, so the install directories no longer need to be writable by
- # everyone -- and last-step runs after the seal, which is the correct
- # order: it installs new files (allowed under the sticky bit) rather
- # than replacing other packages' (which is what the seal prevents).
- if [ "$s" = "last-step" ] && ! _install_dirs_are_sealed; then
- say ""
- say "Everything before the final step is built -- sealing the"
- say "install directories now, so packages can no longer overwrite"
- say "each other's files:"
- say ""
- _vfy_seal_install_dirs
- fi
+ # The seal happens once, after the loop -- see the end of this
+ # function. There used to be a second one here, firing just before
+ # `last-step` because that step still INSTALLED software (wget) and had
+ # to do it under the sticky bit. `last-step` is gone; the final step
+ # is now `init-accounts`, which runs as root and installs nothing, so
+ # there is nothing left to seal early for.
say ""
say "===== [$i/$total] $s ====="
@@ -5833,6 +6011,116 @@ cmd_build_all() {
# cosmetic, so it is safe to do without asking.
say ""
_vfy_user_order 1 || true
+
+ _warn_if_strip_was_asked_for
+ _warn_if_no_login
+ _say_what_is_yours_to_finish
+}
+
+# The two things this tool deliberately does not do.
+#
+# The kernel and the bootloader are decisions about the whole MACHINE, not
+# about a package, and getting either wrong costs you the system you are
+# building on -- so they are yours, on purpose, and the build ends by saying
+# so rather than leaving you to discover it after the reboot.
+#
+# rEFInd is opt-in for the same reason and stays that way: the machine booted
+# in order to build LFS at all, so it already has a working bootloader, and
+# adding a menu entry to it is not something a build should do because nobody
+# said no.
+_say_what_is_yours_to_finish() {
+ say ""
+ say "# Two things are yours to finish, on purpose:"
+ say "#"
+ say "# 1. THE KERNEL. Nothing here configures, builds or installs one."
+ say "# Book 10.3, from inside the chroot:"
+ say "# cd /sources && tar -xf linux-*.tar.xz && cd linux-*"
+ say "# make menuconfig && make && make modules_install"
+ say "# cp -iv arch/x86/boot/bzImage /boot/vmlinuz-lfs"
+ say "#"
+ say "# 2. THE BOOTLOADER. Nothing was written to any ESP, boot sector"
+ say "# or partition table, and your current bootloader is untouched."
+ say "# Add an entry for this system by hand, or let the tools add a"
+ say "# rEFInd entry beside your existing one:"
+ say "# lfs config bootloader refind"
+ say "# lfs config esp /boot/efi"
+ say "# lfs-helper build refind --force"
+ say "# That only ever ADDS to a mounted ESP: it never formats, never"
+ say "# writes a raw device, and never replaces what boots you today."
+}
+
+# You asked for stripping. Nothing did it, and nothing will say so.
+#
+# `lfs build-system session` stores the answer and the prompt says NOT YET
+# IMPLEMENTED -- but that was six hours ago, at the top of an interview, and
+# the build otherwise finishes as though every answer had been acted on. A
+# setting that is collected, carried into the chroot environment, and then
+# silently ignored is worse than one that was never offered.
+#
+# Said at the END, next to the login warning, because that is where the things
+# you still have to do yourself are collected.
+_warn_if_strip_was_asked_for() {
+ [ "${LFS_STRIP:-0}" = 1 ] || return 0
+ say ""
+ warn "# you asked for debug symbols to be stripped, and NOTHING DID IT."
+ warn "#"
+ warn "# Stripping rewrites every binary, and in this system a file's"
+ warn "# owner is the record of which package installed it -- so it has"
+ warn "# to run as that owner, package by package. That is not written"
+ warn "# yet, and doing it as root would hand the whole tree to root."
+ warn "#"
+ warn "# The system is complete and correct; it is only larger than you"
+ warn "# asked for. Book 8.85 also removes libtool .la files."
+ return 1
+}
+
+# Can anyone actually log into this system?
+#
+# THE LAST THING SAID, because it is the last thing that can still be fixed.
+# A fresh LFS has root with no password. Depending on the login manager that is
+# either "anyone can log in as root" or "nobody can log in at all", and you find
+# out after the reboot -- when the chroot is gone and there is no way back in
+# short of booting the host again and mounting the tree by hand.
+#
+# init-accounts asks for it, but it is skippable: a scripted run has no
+# terminal, and an interactive one can be answered with a blank line. So this
+# checks the RESULT rather than trusting that the step ran.
+_warn_if_no_login() {
+ local root_ok=0 user_ok=0 u
+ if grep -qE '^root:[^:*!]' "$ETC/shadow" 2>/dev/null; then
+ root_ok=1
+ fi
+ # any non-system account with a real password
+ while IFS=: read -r u _ uid _; do
+ case "$uid" in ''|*[!0-9]*) continue ;; esac
+ [ "$uid" -ge 1000 ] && [ "$uid" -lt 10000 ] || continue
+ grep -qE "^$u:[^:*!]" "$ETC/shadow" 2>/dev/null && { user_ok=1; break; }
+ done < "$ETC/passwd"
+
+ if [ "$root_ok" = 1 ] || [ "$user_ok" = 1 ]; then
+ say ""
+ ok "# you can log in after rebooting:"
+ [ "$root_ok" = 1 ] && ok "# root has a password"
+ [ "$user_ok" = 1 ] && ok "# '$u' has a password"
+ return 0
+ fi
+
+ say ""
+ warn "=============================================================="
+ warn "!! NOBODY CAN LOG INTO THIS SYSTEM."
+ warn ""
+ warn " root has no password, and no other account has one either."
+ warn " Depending on the login manager that means either anyone can"
+ warn " log in as root, or nobody can log in at all -- and you would"
+ warn " find out after the reboot, with the chroot gone."
+ warn ""
+ warn " Fix it NOW, while you are still in here:"
+ warn " lfs-helper init-accounts --force"
+ warn ""
+ warn " or, from outside the chroot:"
+ warn " chroot /mnt/lfs /usr/bin/passwd root"
+ warn "=============================================================="
+ return 1
}
# Are the install directories already sticky? Checking a handful of the ones
@@ -6333,6 +6621,8 @@ Look around / tidy up:
lfs-helper verify [--fix] does the tree match the manifests?
lfs-helper pkgusr-info [<name>] where each package came from
lfs-helper pkgusr-home <name> print an account's home directory
+ lfs-helper wrapper-dir print where the build wrappers live
+ lfs-helper make-wrappers --run (re)write the build wrappers
lfs-helper owner-name <package> print the account name for a package
lfs-helper manifests [<name>] what each package installed
lfs-helper which-package <path> which package installed this file
@@ -6380,12 +6670,8 @@ case "${1:-}" in
clean-state) shift; cmd_clean_state "$@" ;;
which-package) shift; cmd_which_package "$@" ;;
grant-dir) shift; cmd_grant_dir "$@" ;;
- verify) shift; cmd_verify "$@" ;;
- install-as) shift; cmd_install_as "$@" ;;
- pkgusr-info) shift; cmd_pkgusr_info "$@" ;;
- pkgusr-home) shift; cmd_pkgusr_home "$@" ;;
- owner-name) shift; cmd_owner_name "$@" ;;
- clean-state) shift; cmd_clean_state "$@" ;;
+ make-wrappers) shift; cmd_make_wrappers "$@" ;;
+ wrapper-dir) shift; cmd_wrapper_dir "$@" ;;
export-groups) shift; cmd_export_groups "$@" ;;
import-groups) shift; cmd_import_groups "$@" ;;
prune-locales) shift; cmd_prune_locales "$@" ;;
@@ -6397,6 +6683,7 @@ case "${1:-}" in
# itself the moment book 7.6 has created the user database; this is for
# doing it by hand, or checking whether it has happened.
establish-ownership) shift; cmd_establish_ownership "$@" ;;
+ check-login) shift; _warn_if_no_login ;;
init-accounts) shift; cmd_init_accounts "$@" ;;
# The repair commands people were told to run for years. verify is the
# ONE implementation; these are aliases, not second copies -- a command
diff --git a/packagemanager b/packagemanager
index e7f2844..1a0e769 100644
--- a/packagemanager
+++ b/packagemanager
@@ -109,7 +109,7 @@ TMP_DIR = "/tmp/packagemanager"
CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf")
# Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure
# changes (so stale plans are ignored); TOOL_VERSION is the program version.
-TOOL_VERSION = "1.10.1"
+TOOL_VERSION = "1.11.6"
def _build_id():
@@ -3391,6 +3391,77 @@ def _pip_owner_of_python():
return "python"
+# One module, installed the package-user way. Shared by `packagemanager pip`
+# and `packagemanager setup`, because they want exactly the same thing and a
+# second copy of it is how the two would drift apart.
+def pip_install_module(mod, site, grp, find_links=None, upgrade=False,
+ no_deps=False, target=None):
+ """Install ONE Python module as its own package user. True on success.
+
+ `target` is an explicit file to install (a wheel in the sources directory);
+ without it pip is given the module NAME and finds it however it can.
+ """
+ # THE ACCOUNT NAME COMES FROM THE CHOKEPOINT.
+ #
+ # This read `_sanitise_user_name(mod)` and used the result for every call
+ # below. create_package_user normalises internally, so the account it made
+ # was `p_requests` -- and then usermod and su were handed the bare
+ # `requests`, which does not exist:
+ # usermod: user 'requests' does not exist
+ # The same species as `chown: invalid user: 'wget:wget'`: half the code
+ # asking for the name, half assuming it.
+ user = pkgusr_name(_sanitise_user_name(mod))
+ printHeader(f"{mod} -> package user '{user}'")
+
+ if not _user_exists(user):
+ if not create_package_user(user):
+ printError(f"could not create {user}")
+ return False
+ report_created_user(user)
+
+ # THE COLLECTOR GROUP IS GRANTED THROUGH ONE DOOR.
+ #
+ # lfs-helper owns the rules: join the group the directory already carries,
+ # ask which group should own it when there is none, never create one for
+ # root, never hand a package's tree to the install group. This did its own
+ # groupadd/chgrp -R/chmod -R instead -- a third implementation, after
+ # lfs-helper's and packagemanager_install's, and the only one that never
+ # asked what the directory already carried.
+ if shutil.which("lfs-helper"):
+ _run(["lfs-helper", "grant-dir", site, user, "--run"])
+ else:
+ if not _group_exists(grp):
+ printInfo(f"creating collector group {grp}")
+ _run([_cmd_path("groupadd"), grp])
+ _run([_cmd_path("usermod"), "-a", "-G", grp, user])
+ if _group_exists(grp):
+ _run(["chgrp", "-R", grp, site])
+ _run(["chmod", "-R", "g+w", site])
+
+ # PIP_USER=0 / PYTHONNOUSERSITE=1: without these pip quietly falls back
+ # to ~/.local when site-packages is not writable, and then reports
+ # "Requirement already satisfied" from there ever after -- so the module
+ # is installed for exactly one user and importable by nobody else.
+ env = ["env", "PIP_USER=0", "PYTHONNOUSERSITE=1"]
+ pip_args = ["pip3", "install"]
+ if upgrade:
+ pip_args.append("--upgrade")
+ if no_deps:
+ pip_args.append("--no-deps")
+ if find_links:
+ pip_args += ["--no-index", "--find-links", find_links]
+ pip_args.append(target or mod)
+
+ cmd = ["su", "-s", "/bin/bash", user, "-c",
+ " ".join(shlex.quote(x) for x in env + pip_args)]
+ printInfo(" ".join(pip_args) + f" (as {user})")
+ if _run(cmd, show_output=True):
+ printSuccess(f"{mod} installed as package-user {user}")
+ return True
+ printError(f"{mod} failed -- see the output above")
+ return False
+
+
def cmd_pip(args):
"""Install or update a Python module AS A PACKAGE USER.
@@ -3404,8 +3475,7 @@ def cmd_pip(args):
if not site:
printError("no system site-packages directory found -- is Python installed?")
return 1
- owner = _pip_owner_of_python()
- grp = collector_prefix() + owner
+ grp = collector_prefix() + _pip_owner_of_python()
modules = args.modules
if not modules:
@@ -3413,43 +3483,10 @@ def cmd_pip(args):
return 1
for mod in modules:
- user = _sanitise_user_name(mod)
- printHeader(f"{mod} -> package user '{user}'")
-
- if not _user_exists(user):
- if not create_package_user(user):
- printError(f"could not create {user}")
- continue
- report_created_user(user)
-
- # the collector group is what lets this user write into Python's tree
- if not _group_exists(grp):
- printInfo(f"creating collector group {grp}")
- _run([_cmd_path("groupadd"), grp])
- _run(["usermod", "-a", "-G", grp, user])
- if _group_exists(grp):
- _run(["chgrp", "-R", grp, site])
- _run(["chmod", "-R", "g+w", site])
-
- # PIP_USER=0 / PYTHONNOUSERSITE=1: without these pip quietly falls back
- # to ~/.local when site-packages is not writable, and then reports
- # "Requirement already satisfied" from there ever after -- so the module
- # is installed for exactly one user and importable by nobody else.
- env = ["env", "PIP_USER=0", "PYTHONNOUSERSITE=1"]
- pip_args = ["pip3", "install"]
- if args.upgrade:
- pip_args.append("--upgrade")
- if args.find_links:
- pip_args += ["--no-index", "--find-links", args.find_links]
- pip_args.append(mod)
-
- cmd = ["su", "-s", "/bin/bash", user, "-c",
- " ".join(shlex.quote(x) for x in env + pip_args)]
- printInfo(" ".join(pip_args) + f" (as {user})")
- if _run(cmd, show_output=True):
- printSuccess(f"{mod} installed as package-user {user}")
- else:
- printError(f"{mod} failed -- see the output above")
+ pip_install_module(mod, site, grp,
+ find_links=args.find_links,
+ upgrade=args.upgrade,
+ no_deps=getattr(args, "no_deps", False))
return 0
def cmd_add_user(args):
@@ -3806,6 +3843,215 @@ def cmd_user_list(args):
print(f"\n{len(names)} user(s).")
+_SETUP_HELP = """\
+Bootstrap the tooling on a freshly booted system.
+
+LFS ships no download tool, so a new system cannot fetch even the sources for
+its own next package. And `lfs` and `blfs` need `requests` and
+`beautifulsoup4` to read the books, so without them the tools cannot run on the
+system they just built.
+
+ packagemanager setup --sources what it needs downloaded
+ packagemanager setup --run install it
+
+The tarballs must be on disk BEFORE the reboot, while the host still has a
+network -- `lfs build-system get-sources` asks this command for the list.
+"""
+
+# What a freshly booted system needs before it can fetch anything itself.
+#
+# wget comes from the BLFS BOOK: the book already knows its URL and keeps it
+# current, and a second list of download links is a second thing to go stale.
+_SETUP_BLFS_PACKAGES = ["wget"]
+
+# The Python modules `lfs` and `blfs` need to parse the books.
+#
+# WHEELS, not source tarballs, and not from the book. A modern sdist needs its
+# build backend -- beautifulsoup4 wants hatchling, requests wants setuptools --
+# and with no package index reachable pip cannot fetch one:
+# BackendUnavailable: Cannot import 'hatchling.build'
+# A wheel is already built, so pip only unpacks it. All of these are pure
+# Python, so the "any" wheel works on every architecture.
+#
+# Dependencies first: setup installs with --no-deps, so pip will not pull them.
+_SETUP_WHEELS = [
+ "https://files.pythonhosted.org/packages/py3/u/urllib3/urllib3-2.2.3-py3-none-any.whl",
+ "https://files.pythonhosted.org/packages/py3/c/charset_normalizer/charset_normalizer-3.4.0-py3-none-any.whl",
+ "https://files.pythonhosted.org/packages/py3/i/idna/idna-3.10-py3-none-any.whl",
+ "https://files.pythonhosted.org/packages/py3/c/certifi/certifi-2024.8.30-py3-none-any.whl",
+ "https://files.pythonhosted.org/packages/py3/s/soupsieve/soupsieve-2.6-py3-none-any.whl",
+ "https://files.pythonhosted.org/packages/py3/r/requests/requests-2.32.3-py3-none-any.whl",
+ "https://files.pythonhosted.org/packages/py3/b/beautifulsoup4/beautifulsoup4-4.12.3-py3-none-any.whl",
+]
+
+
+def setup_sources():
+ """Every URL this command needs, resolved once."""
+ import subprocess
+ urls = []
+ for pkg in _SETUP_BLFS_PACKAGES:
+ try:
+ out = subprocess.run(["blfs", "sources", pkg],
+ capture_output=True, text=True, timeout=120)
+ except (OSError, subprocess.SubprocessError):
+ printWarning("could not run `blfs sources %s` -- is blfs installed?"
+ % pkg)
+ continue
+ got = [l.strip() for l in (out.stdout or "").splitlines()
+ if l.strip().startswith(("http://", "https://", "ftp://"))]
+ if not got:
+ printWarning("the BLFS book gave no download for '%s' "
+ "(try: blfs fetch)" % pkg)
+ continue
+ urls += got
+ return urls + _SETUP_WHEELS
+
+
+# Where the tarballs and wheels are, on the booted system.
+#
+# Book 3.1's /sources, kept: `lfs build-system get-sources` put them there
+# while the host still had a network, and that is the last moment there was
+# one. Overridable, because a tree can be mounted anywhere.
+def setup_sources_dir():
+ return os.environ.get("LFS_SOURCES_DIR") or "/sources"
+
+
+def _wheel_dist(url):
+ """(distribution, version, filename) from a wheel URL.
+
+ PEP 427 fixes the filename: {dist}-{version}-{python}-{abi}-{platform}.whl,
+ so the first two fields are exactly what pip records in site-packages as
+ <dist>-<version>.dist-info. That is what makes the check below an
+ identity, not a guess.
+ """
+ fn = url.rsplit("/", 1)[-1]
+ parts = fn[:-4].split("-") if fn.endswith(".whl") else fn.split("-")
+ if len(parts) < 2:
+ return fn, "", fn
+ return parts[0], parts[1], fn
+
+
+def _wheel_installed(site, dist, version):
+ return bool(site) and os.path.isdir(
+ os.path.join(site, "%s-%s.dist-info" % (dist, version)))
+
+
+def _setup_state():
+ """What is already true, so setup can be run twice and do nothing twice."""
+ site = _pip_site_dir()
+ wheels = []
+ for url in _SETUP_WHEELS:
+ dist, ver, fn = _wheel_dist(url)
+ wheels.append((dist, ver, fn, url, _wheel_installed(site, dist, ver)))
+ return {
+ "site": site,
+ "wget": shutil.which("wget"),
+ "wheels": wheels,
+ }
+
+
+def _setup_install_wget(yes):
+ """Install wget through the ordinary BLFS path.
+
+ Invoked as a command rather than called as a function: `packagemanager
+ install wget --run` is a build plan, a script, a package user and a
+ manifest, and there is no version of "just the wget part" of that which is
+ not a second implementation of it.
+ """
+ cmd = [sys.executable, os.path.abspath(__file__), "install", "wget", "--run"]
+ if yes:
+ cmd.append("--yes")
+ printInfo(" $ " + " ".join(cmd[2:]))
+ try:
+ return subprocess.run(cmd).returncode == 0
+ except (OSError, subprocess.SubprocessError) as e:
+ printError("could not install wget: %s" % e)
+ return False
+
+
+def cmd_setup(args):
+ if args.sources:
+ for u in setup_sources():
+ print(u)
+ return
+
+ st = _setup_state()
+ srcdir = getattr(args, "sources_dir", None) or setup_sources_dir()
+ force = getattr(args, "force", False)
+ todo_wget = not st["wget"] or force
+ todo_wheels = [w for w in st["wheels"] if force or not w[4]]
+
+ printHeader("packagemanager setup")
+ print()
+ print(" site-packages : %s" % (st["site"] or "NOT FOUND -- is Python installed?"))
+ print(" sources : %s" % srcdir)
+ print()
+ print(" wget : %s" % ("installed at %s" % st["wget"] if st["wget"]
+ else "MISSING -- will build from the BLFS book"))
+ for dist, ver, fn, url, have in st["wheels"]:
+ print(" %-13s %s" % (dist, "%s installed" % ver if have
+ else "%s to install" % ver))
+
+ if not st["site"]:
+ printError("\nNo system site-packages directory: nothing can be installed.")
+ sys.exit(1)
+
+ if not todo_wget and not todo_wheels:
+ print()
+ printSuccess("Nothing to do -- wget and every module are already installed.")
+ printInfo("Re-do it anyway with: packagemanager setup --run --force")
+ return
+
+ if not args.run:
+ print()
+ printInfo("Dry run -- to apply, re-run with --run.")
+ return
+
+ # The wheels are already on disk and pip must not reach for an index: this
+ # system has no CA certificates yet, so anything that tries is a hang, not
+ # an error.
+ missing = [fn for _d, _v, fn, _u, _h in todo_wheels
+ if not os.path.isfile(os.path.join(srcdir, fn))]
+ if missing:
+ printError("\nThese wheels are not in %s:" % srcdir)
+ for fn in missing:
+ print(" %s" % fn)
+ printWarning("They had to be downloaded before the reboot:")
+ printWarning(" lfs build-system get-sources --run (on the host)")
+ sys.exit(1)
+
+ failed = []
+ if todo_wget:
+ printHeader("wget")
+ if not _setup_install_wget(getattr(args, "yes", False)):
+ failed.append("wget")
+ else:
+ # A system with no certificates cannot verify anything it fetches.
+ # This is the same call the tool makes on every invocation; doing it
+ # here means the very next download works.
+ ensure_wget_workaround()
+
+ if todo_wheels:
+ grp = collector_prefix() + _pip_owner_of_python()
+ # IN ORDER, with --no-deps: pip cannot reach an index to resolve
+ # anything, so the order in _SETUP_WHEELS is the dependency order and
+ # each module must already have what it imports.
+ for dist, ver, fn, url, _have in todo_wheels:
+ if not pip_install_module(dist, st["site"], grp,
+ find_links=srcdir, no_deps=True,
+ target=os.path.join(srcdir, fn)):
+ failed.append(dist)
+
+ print()
+ if failed:
+ printError("setup finished with %d failure(s): %s"
+ % (len(failed), ", ".join(failed)))
+ printInfo("Fix them and run it again -- it skips what is already done.")
+ sys.exit(1)
+ printSuccess("setup complete.")
+ printInfo("Check it: python3 -c 'import requests, bs4' && wget --version | head -1")
+
+
def cmd_user_create(args):
u = _user_prefixed(args.name)
if _user_exists(u):
@@ -5502,6 +5748,24 @@ def main():
help="wait for it instead of running as a disowned task")
p.set_defaults(func=cmd_reload_pkg_list)
+ p = sub.add_parser(
+ "setup",
+ help="bootstrap the tooling on a freshly booted system",
+ description=_SETUP_HELP,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--sources", action="store_true",
+ help="print the download URLs this needs, one per line, "
+ "and exit (used by `lfs build-system get-sources`)")
+ p.add_argument("--run", action="store_true", help="apply (default: dry run)")
+ p.add_argument("--force", action="store_true",
+ help="re-install everything, including what is already there")
+ p.add_argument("--sources-dir", dest="sources_dir",
+ help="where the downloaded tarballs and wheels are "
+ "(default: /sources)")
+ p.add_argument("--yes", action="store_true",
+ help="answer yes to the questions the wget build asks")
+ p.set_defaults(func=cmd_setup)
+
p = sub.add_parser("pip", help="install or update a Python module as a "
"package user (not as root)")
p.add_argument("action", choices=["install", "update", "upgrade"],
@@ -5510,9 +5774,13 @@ def main():
p.add_argument("--find-links", metavar="DIR",
help="install from local wheels in DIR instead of the network "
"(there is no network inside the chroot)")
+ p.add_argument("--no-deps", action="store_true",
+ help="do not let pip pull dependencies -- name them yourself, "
+ "in order (what `setup` does, offline)")
p.set_defaults(func=lambda a: cmd_pip(
argparse.Namespace(modules=a.modules,
upgrade=(a.action in ("update", "upgrade")),
+ no_deps=a.no_deps,
find_links=a.find_links)))
p = sub.add_parser("add-user",
diff --git a/packagemanager_install b/packagemanager_install
index fd513ad..87513e4 100644
--- a/packagemanager_install
+++ b/packagemanager_install
@@ -14,7 +14,7 @@
# packagemanager_install package_short_name path_to_install_file
#
-PACKAGEMANAGER_INSTALL_VERSION="1.10.1"
+PACKAGEMANAGER_INSTALL_VERSION="1.11.6"
# A fingerprint of this file's own contents, the same as the other four tools.
#
@@ -64,6 +64,15 @@ pkgusr_home_for() {
case "$n" in
"") printf '%s' "$PKGUSR_ROOT"; return ;;
esac
+ # ASK, if there is anything to ask. lfs-helper owns this rule and the
+ # layout it depends on is configurable; a second copy of it here is right
+ # only until someone changes the layout. The rest of this function is the
+ # fallback for a system with no lfs-helper.
+ if command -v lfs-helper >/dev/null 2>&1; then
+ local _h
+ _h="$(lfs-helper pkgusr-home "$n" 2>/dev/null)"
+ case "$_h" in /*) printf '%s' "$_h"; return ;; esac
+ fi
if [ -n "$PKGUSR_PREFIX" ]; then
case "$n" in "$PKGUSR_PREFIX"*) ;; *) n="${PKGUSR_PREFIX}${n}" ;; esac
fi
@@ -99,8 +108,30 @@ home_user_dir="/home/${home_user}"
# install -d on an existing directory -> succeed, change nothing
# chmod on a directory we do not own -> skip, keep going
# chown/chgrp on anything -> skip (a package user may not)
-# Anything else is passed straight through. This is the same approach
-# lfs-helper uses during the chroot build.
+# Anything else is passed straight through.
+#
+# ONE SET OF WRAPPERS, NOT TWO. lfs-helper writes these to /usr/lib/pkgusr,
+# where they outlive the build and where the package-user profile already
+# points; this script wrote its OWN copy to a fresh /tmp directory on every
+# run. Two implementations of "what may a package user do", free to disagree
+# about it -- the species of bug that costs this project the most. Ask for
+# them when lfs-helper is on the system, which it is on anything these tools
+# built; the copy below is the fallback for a system without it.
+_wrapper_dir_from_lfs_helper() {
+ command -v lfs-helper >/dev/null 2>&1 || return 1
+ local d
+ d="$(lfs-helper wrapper-dir 2>/dev/null)" || return 1
+ [ -n "$d" ] || return 1
+ # Missing or incomplete -- write them. Idempotent, so this also repairs a
+ # tree whose wrappers predate a fix to them.
+ if [ ! -x "$d/install" ] || [ ! -x "$d/chown" ]; then
+ lfs-helper make-wrappers --run >/dev/null 2>&1 || return 1
+ fi
+ [ -x "$d/install" ] || return 1
+ printf '%s' "$d"
+}
+
+# The temporary copy: only for a system with no lfs-helper.
_make_build_wrappers() {
local d="$1"
mkdir -p "$d" || return 1
@@ -571,14 +602,16 @@ installPkg() {
chown $user_name: $install_user_dir/install_*
chown -R $user_name: $install_user_dir/log/
# pipefail so tee doesn't hide a failing build/install; capture the real code
- _wrapdir="/tmp/pkgusr-wrappers.$$"
- _make_build_wrappers "$_wrapdir" || _wrapdir=""
- chmod 755 "$_wrapdir" 2>/dev/null || true
+ _wrapdir="$(_wrapper_dir_from_lfs_helper)" && _wraptmp=0 || {
+ _wrapdir="/tmp/pkgusr-wrappers.$$"; _wraptmp=1
+ _make_build_wrappers "$_wrapdir" || _wrapdir=""
+ chmod 755 "$_wrapdir" 2>/dev/null || true
+ }
su - -c "set -o pipefail; \
${_wrapdir:+PATH='$_wrapdir':\$PATH; export PATH; } \
bash ~/install_$user_name $parameter 2>&1 | tee ~/log/packagemanager_install-$date_log.log" $user_name
install_rc=${PIPESTATUS[0]}
- [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir"
+ [ "${_wraptmp:-0}" = 1 ] && [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir"
if echo "$user_name" | grep "^u_"; then
gpasswd -d $user_name install # remove install group if nimgnu_group
@@ -787,14 +820,16 @@ diagnose_install_failure() {
auto_grant_from_log "$_log" "$user_name" || break
_round=$((_round + 1))
echo -e "\033[0;33m# retrying $user_name ($parameter), round $_round ...\033[0m"
- _wrapdir="/tmp/pkgusr-wrappers.$$"
- _make_build_wrappers "$_wrapdir" || _wrapdir=""
- chmod 755 "$_wrapdir" 2>/dev/null || true
+ _wrapdir="$(_wrapper_dir_from_lfs_helper)" && _wraptmp=0 || {
+ _wrapdir="/tmp/pkgusr-wrappers.$$"; _wraptmp=1
+ _make_build_wrappers "$_wrapdir" || _wrapdir=""
+ chmod 755 "$_wrapdir" 2>/dev/null || true
+ }
su - -c "set -o pipefail; \
${_wrapdir:+PATH='$_wrapdir':\$PATH; export PATH; } \
bash ~/install_$user_name $parameter 2>&1 | tee ~/log/packagemanager_install-$date_log.log" $user_name
install_rc=${PIPESTATUS[0]}
- [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir"
+ [ "${_wraptmp:-0}" = 1 ] && [ -n "${_wrapdir:-}" ] && rm -rf "$_wrapdir"
done
if [ "$install_rc" != "0" ]; then
diff --git a/test_lfs_crosschain.sh b/test_lfs_crosschain.sh
index baf9205..d13d604 100644
--- a/test_lfs_crosschain.sh
+++ b/test_lfs_crosschain.sh
@@ -1526,40 +1526,162 @@ fi
# ---- what the built system needs before it can fetch anything --------------- #
# LFS ships no download tool, so a freshly booted system cannot get the sources
-# for its own next package. The tarball has to be on disk before the chroot is
-# sealed, and get-sources is the last moment there is still a network.
+# for its own next package. The tarballs have to be on disk before the chroot
+# is sealed, and get-sources is the last moment there is still a network.
#
-# Resolved from the BLFS BOOK, not a hardcoded URL: the book already knows where
-# wget comes from and keeps it current, and a second list of links is a second
-# thing to go stale.
+# The LIST belongs to `packagemanager setup`, which is what installs them. A
+# copy in lfs would be a second thing to go stale -- the first version of this
+# did exactly that, with a `bootstrap_packages` config key in lfs resolving wget
+# separately from the tool that installs it.
python3 - "$LFS_TOOL" <<'PYBOOT'
-import sys, importlib.machinery as m
-lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module()
+import re, sys
src = open(sys.argv[1]).read()
problems = []
-for fn in ("bootstrap_packages", "bootstrap_sources"):
- if not hasattr(lfs, fn):
- problems.append("there is no %s" % fn)
-if hasattr(lfs, "bootstrap_packages"):
- lfs.load_config = lambda: {}
- if lfs.bootstrap_packages() != ["wget"]:
- problems.append("wget is not the default bootstrap package")
- lfs.load_config = lambda: {"bootstrap_packages": ""}
- if lfs.bootstrap_packages():
- problems.append("bootstrap packages cannot be turned off")
- lfs.load_config = lambda: {"bootstrap_packages": "wget curl"}
- if lfs.bootstrap_packages() != ["wget", "curl"]:
- problems.append("more than one bootstrap package is not accepted")
-if "blfs" not in src or '"sources"' not in src:
- problems.append("the URLs are not resolved from the BLFS book")
+if not re.search(r"^def bootstrap_sources\(", src, re.M):
+ problems.append("there is no bootstrap_sources")
+m = re.search(r"^def bootstrap_sources\(.*?(?=\n\ndef )", src, re.S | re.M)
+if m:
+ body = m.group(0)
+ if "packagemanager" not in body or "--sources" not in body:
+ problems.append("lfs does not ask packagemanager what it needs")
+ # lfs must not keep its own copy of the list. CODE only -- the docstring
+ # explains why the list lives elsewhere, and naming it there is the point.
+ code = "\n".join("" if l.lstrip().startswith("#") else l
+ for l in body.splitlines())
+ code = "".join(code.split('"""')[::2])
+ if "wget" in code:
+ problems.append("lfs still names the packages itself")
+if "bootstrap_packages" in src.replace("`bootstrap_packages`", ""):
+ problems.append("the old bootstrap_packages config key is still here")
if problems:
for p in problems:
print(" FAIL %s" % p)
sys.exit(1)
-print(" PASS the bootstrap tarball is resolved from the book, not hardcoded")
+print(" PASS lfs asks packagemanager what the new system needs downloaded")
PYBOOT
_count_rc $?
+# and packagemanager must answer, without needing a booted system
+pm_src="$(dirname "$LFS_TOOL")/packagemanager"
+if [ -f "$pm_src" ]; then
+ _p=""
+ _out="$(python3 "$pm_src" setup --sources 2>/dev/null)"
+ _n="$(printf '%s\n' "$_out" | grep -c '^https\?://')"
+ [ "${_n:-0}" -ge 7 ] \
+ || _p="$_p;setup --sources listed only $_n urls (expected the wheels at least)"
+ # WHEELS, not sdists: a modern sdist needs its build backend and pip cannot
+ # fetch one with no index -- BackendUnavailable: Cannot import hatchling.build
+ printf '%s\n' "$_out" | grep -q 'requests-.*\.whl' \
+ || _p="$_p;requests is not taken as a wheel"
+ printf '%s\n' "$_out" | grep -q 'beautifulsoup4-.*\.whl' \
+ || _p="$_p;beautifulsoup4 is not taken as a wheel"
+ # and its dependencies, because it installs with --no-deps
+ for _d in urllib3 charset_normalizer idna certifi soupsieve; do
+ printf '%s\n' "$_out" | grep -q "$_d" \
+ || _p="$_p;$_d is missing, and --no-deps will not pull it in"
+ done
+ # wget comes from the BOOK, not a hardcoded URL
+ grep -q '_SETUP_BLFS_PACKAGES' "$pm_src" \
+ || _p="$_p;wget is not resolved from the BLFS book"
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "packagemanager setup declares what it needs, wheels and all"
+ fi
+fi
+
+# ---- setup INSTALLS what it declares (1.11.2) ------------------------------ #
+# `--sources` worked and `--run` said "not implemented". Everything below is a
+# guarantee the old last_build_step.sh held and dropped when it was removed.
+if [ -f "$pm_src" ]; then
+python3 - "$pm_src" <<'PYSETUP'
+import sys, re, importlib.machinery as m
+src = open(sys.argv[1]).read()
+pm = m.SourceFileLoader('pm', sys.argv[1]).load_module()
+problems = []
+
+def body(fn):
+ mm = re.search(r"^def %s\(.*?(?=\n\ndef |\n\n# )" % fn, src, re.S | re.M)
+ return mm.group(0) if mm else ""
+
+# ONE implementation. `pip` and `setup` want the same thing -- a module
+# installed as its own package user -- and a second copy of it is how the two
+# would drift apart, which is the species of bug this project keeps having.
+if not re.search(r"^def pip_install_module\(", src, re.M):
+ problems.append("there is no shared pip_install_module")
+if "pip_install_module" not in body("cmd_pip"):
+ problems.append("cmd_pip does not use the shared implementation")
+if "pip_install_module" not in body("cmd_setup"):
+ problems.append("setup does not use the shared implementation")
+
+# THE ACCOUNT NAME IS ASKED FOR, NEVER ASSEMBLED.
+# create_package_user normalises internally, so an unprefixed name creates
+# `p_requests` and then hands `requests` to usermod and su:
+# usermod: user 'requests' does not exist
+# Same species as `chown: invalid user: 'wget:wget'` at step 104 of 105.
+b = body("pip_install_module")
+if not re.search(r"pkgusr_name\(\s*_sanitise_user_name", b):
+ problems.append("the pip account name skips the naming chokepoint")
+
+# offline, and in order. pip cannot reach an index on a fresh system: no CA
+# certificates, so a lookup is a hang rather than an error.
+sb = body("cmd_setup")
+if "no_deps=True" not in sb:
+ problems.append("setup lets pip resolve dependencies it cannot fetch")
+if "find_links" not in sb:
+ problems.append("setup does not install from the local wheels")
+if "--no-index" not in b:
+ problems.append("pip is still allowed to reach for an index")
+
+# wget comes from the ordinary install path, not a private copy of it
+if "install" not in sb or "wget" not in sb:
+ problems.append("setup no longer installs wget")
+if re.search(r"^def _setup_install_wget\(", src, re.M):
+ wb = body("_setup_install_wget")
+ if '"install"' not in wb:
+ problems.append("wget is installed by some path other than `install`")
+
+# IDEMPOTENT. The old final step ran exactly once, at the end of a six-hour
+# build; this one has to be safe to run again after fixing whatever failed.
+if "dist-info" not in src:
+ problems.append("setup cannot tell what is already installed")
+d, v, fn = pm._wheel_dist(
+ "https://x/packages/py3/c/charset_normalizer/charset_normalizer-3.4.0-py3-none-any.whl")
+if (d, v) != ("charset_normalizer", "3.4.0"):
+ problems.append("the wheel filename is parsed as %r %r" % (d, v))
+if fn != "charset_normalizer-3.4.0-py3-none-any.whl":
+ problems.append("the wheel filename is not recovered from the url")
+
+if problems:
+ for x in problems:
+ print(" FAIL %s" % x)
+ sys.exit(1)
+print(" PASS setup installs wget and the wheels, offline, as package users")
+PYSETUP
+_count_rc $?
+
+# It must REFUSE rather than half-run when the wheels were never downloaded.
+# There is no network on a fresh system, so "try it and see" is a hang.
+_sbx="$T/setup-empty"; mkdir -p "$_sbx"
+_o="$(LFS_SOURCES_DIR="$_sbx" python3 "$pm_src" setup --run --yes 2>&1)"; _rc=$?
+_p=""
+[ "$_rc" -ne 0 ] || _p="$_p;setup --run succeeded with no wheels on disk"
+printf '%s\n' "$_o" | grep -q 'get-sources' \
+ || _p="$_p;it does not say where the wheels were supposed to come from"
+printf '%s\n' "$_o" | grep -q 'requests-.*\.whl' \
+ || _p="$_p;it does not name the files it could not find"
+if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+else
+ ok "setup stops when the wheels are missing, and says which"
+fi
+fi
+
+
# the dry run must print EVERY url, not a sample
grep -q "print the WHOLE list" "$LFS_TOOL" \
@@ -3553,6 +3675,255 @@ PY60
else
bad "the local copy runs even when lfs-helper is available"
fi
+
+ # ---- one set of wrappers, not two (1.11.3) ------------------------------ #
+ # The wrappers ARE the rule about what a package user may do: chown skipped,
+ # chgrp skipped, `install -d` on an existing directory allowed to succeed.
+ # lfs-helper writes them to /usr/lib/pkgusr, where they outlive the build and
+ # where the package-user profile already points. This script wrote its own
+ # 480-line copy into a fresh /tmp directory on EVERY run: a second answer to
+ # the same question, and the profile pointed at neither of them.
+ _p=""
+ grep -q '^_wrapper_dir_from_lfs_helper()' "$pmi" \
+ || _p="$_p;there is no way to ask lfs-helper for the wrappers"
+ grep -q 'lfs-helper wrapper-dir' "$pmi" \
+ || _p="$_p;the wrapper directory is assumed rather than asked for"
+ # every call site must go through the resolver first
+ _n_ask="$(grep -c '_wrapper_dir_from_lfs_helper' "$pmi")"
+ _n_own="$(grep -c '_make_build_wrappers "\$_wrapdir"' "$pmi")"
+ [ "${_n_ask:-0}" -gt "${_n_own:-0}" ] \
+ || _p="$_p;a call site still builds its own wrappers without asking"
+ # and the SHARED directory must never be deleted afterwards -- that would
+ # take the wrappers away from every package user on the system
+ grep -q '_wraptmp' "$pmi" \
+ || _p="$_p;the cleanup does not distinguish the shared dir from a temporary one"
+ if grep -q 'rm -rf "$_wrapdir"' "$pmi" && ! grep -q '_wraptmp.*=.*1.*rm -rf "$_wrapdir"' "$pmi"; then
+ _p="$_p;the wrapper directory is removed unconditionally"
+ fi
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "both tools use one set of wrappers, and only a temporary one is deleted"
+ fi
+
+ # the account home is asked for, not re-derived from the layout
+ _hf="$(sed -n '/^pkgusr_home_for() {/,/^}/p' "$pmi")"
+ case "$_hf" in
+ *"lfs-helper pkgusr-home"*) ok "the account home comes from the chokepoint" ;;
+ *) bad "packagemanager_install re-derives where an account lives" ;;
+ esac
+fi
+
+# ---- the kernel and the bootloader are yours (1.11.6) --------------------- #
+# Both are decisions about the whole MACHINE, and getting either wrong costs
+# the system you are building on. The build must not do them by default, and
+# must not let you find that out after the reboot.
+_h="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$_h" ]; then
+ _p=""
+ grep -q '_say_what_is_yours_to_finish' "$_h" \
+ || _p="$_p;the build never says the kernel and bootloader are yours"
+ _sf="$(sed -n '/^_say_what_is_yours_to_finish() {/,/^}/p' "$_h")"
+ printf '%s' "$_sf" | grep -qi 'kernel' \
+ || _p="$_p;the closing note does not mention the kernel"
+ printf '%s' "$_sf" | grep -qi 'refind' \
+ || _p="$_p;the closing note does not say how to add a bootloader entry"
+ # it is a note, not an action: nothing in it may write anything
+ printf '%s' "$_sf" | grep -qE '^\s*(cp|mv|rm|dd|mkfs|chmod|chown|install) ' \
+ && _p="$_p;the closing note runs commands instead of printing them"
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "the build ends by saying the kernel and bootloader are yours"
+ fi
+fi
+
+# and no bootloader is set up unless it was asked for, by name
+python3 - "$LFS_TOOL" <<'PYBL'
+import sys, re, importlib.machinery as m
+lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module()
+problems = []
+# the default must be "none" with NO config at all -- not "none" because some
+# other key happened to be unset
+lfs.load_config = lambda: {}
+if lfs.bootloader() != "none":
+ problems.append("the default bootloader is %r, not 'none'" % lfs.bootloader())
+lfs.load_config = lambda: {"bootloader": ""}
+if lfs.bootloader() != "none":
+ problems.append("an empty bootloader setting does not mean 'none'")
+# and the prompt must say the machine keeps booting as it does now
+src = open(sys.argv[1]).read()
+m2 = re.search(r'\("bootloader",.*?\),\n\s*\("esp"', src, re.S)
+if m2 and "DEFAULT" not in m2.group(0):
+ problems.append("the prompt does not say that doing nothing is the default")
+if problems:
+ for p in problems: print(" FAIL %s" % p)
+ sys.exit(1)
+print(" PASS no bootloader is touched unless it was asked for by name")
+PYBL
+_count_rc $?
+
+# ---- $LFS may never be the running system (1.11.5) ------------------------ #
+# Every path this tool writes is $LFS/something. With $LFS set to the host --
+# a config holding "/", an `export LFS=/` in the wrong shell, a typo in the
+# interview -- `restart --run` deletes the top-level directories of the machine
+# you are sitting on, and _verify_lfs_ownership runs `chown -R lfs /usr`.
+# The chroot bind mounts were guarded; the tree itself was not.
+python3 - "$LFS_TOOL" <<'PYHOST'
+import sys, os, tempfile, importlib.machinery as m
+lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module()
+problems = []
+
+for fn in ("_is_host_path", "_refuse_host_tree"):
+ if not hasattr(lfs, fn):
+ problems.append("there is no %s" % fn)
+if problems:
+ for p in problems: print(" FAIL %s" % p)
+ sys.exit(1)
+
+# the host, by every spelling
+for p in ("/", "/usr", "/etc", "/boot", "/var", "/home", "/bin", "/lib"):
+ if not lfs._is_host_path(p):
+ problems.append("%s is not recognised as the running system" % p)
+
+# by another name: a symlink to /
+d = tempfile.mkdtemp()
+link = os.path.join(d, "root-link")
+try:
+ os.symlink("/", link)
+ if not lfs._is_host_path(link):
+ problems.append("a symlink to / is not recognised")
+except OSError:
+ pass
+
+# a path INSIDE a system tree is the system too, however deep
+for p in ("/usr/src/lfs", "/etc/lfs", "/var/tmp/lfs"):
+ if not lfs._is_host_path(p):
+ problems.append("%s is not recognised as part of the system" % p)
+
+# ...and a real build tree must still be allowed, even though it has the same
+# layout. A check that refuses /mnt/lfs/usr refuses every build there is.
+tree = os.path.join(d, "lfs")
+os.makedirs(os.path.join(tree, "usr"), exist_ok=True)
+os.makedirs(os.path.join(tree, "etc"), exist_ok=True)
+for p in (tree, "/mnt/lfs", "/home/someone/lfs", "/media/disk/lfs"):
+ if lfs._is_host_path(p):
+ problems.append("a legitimate build tree is refused: %s" % p)
+
+# ONE rule, not two: the interview and the hard stop must agree, or a path
+# refused by one is accepted by the other.
+src = open(sys.argv[1]).read()
+import re
+body = re.search(r"def _refuse_host_tree\(lfs\):.*?\n\n\ndef ", src, re.S)
+if body and "_is_host_path" not in body.group(0):
+ problems.append("the hard stop keeps its own copy of the rule")
+if "_is_host_path" not in re.search(r"for k, desc, default in _SESSION_KEYS:.*?\n\n", src, re.S).group(0):
+ problems.append("the interview accepts a mount point the tools will refuse")
+
+# the destructive paths must go through it
+for fn in ("require_lfs_for_root", "_verify_lfs_ownership"):
+ b = re.search(r"def %s\(.*?(?=\n\ndef )" % fn, src, re.S)
+ if b and "_refuse_host_tree" not in b.group(0):
+ problems.append("%s does not check the tree is not the host" % fn)
+
+if problems:
+ for p in problems: print(" FAIL %s" % p)
+ sys.exit(1)
+print(" PASS $LFS is refused when it is the running system, by any spelling")
+PYHOST
+_count_rc $?
+
+# ---- an answer that is collected must be acted on, or said (1.11.4) ------- #
+# `strip` is asked for during the interview and nothing does it. The prompt
+# says NOT YET IMPLEMENTED, but that was six hours and a whole build ago, and
+# the build otherwise ends as though every answer had been acted on.
+_h="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$_h" ]; then
+ _p=""
+ grep -q '_warn_if_strip_was_asked_for' "$_h" \
+ || _p="$_p;nothing tells you the strip answer was ignored"
+ # it must READ the answer, not assume it
+ _sf="$(sed -n '/^_warn_if_strip_was_asked_for() {/,/^}/p' "$_h")"
+ printf '%s' "$_sf" | grep -q 'LFS_STRIP' \
+ || _p="$_p;the strip notice does not read LFS_STRIP"
+ # and only when it was actually asked for
+ printf '%s' "$_sf" | grep -q 'return 0' \
+ || _p="$_p;the strip notice fires even when nobody asked for it"
+ # said at the END, with the other things you still have to do
+ _i1="$(grep -n '_warn_if_strip_was_asked_for$' "$_h" | tail -1 | cut -d: -f1)"
+ _i2="$(grep -n '^ _warn_if_no_login$' "$_h" | tail -1 | cut -d: -f1)"
+ [ -n "$_i1" ] && [ -n "$_i2" ] \
+ || _p="$_p;the notice is not part of the build's last word"
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "a build that ignored your strip answer says so at the end"
+ fi
+fi
+
+# ---- a suppressed failure is a bug waiting to be found (1.11.4) ----------- #
+# `cmd 2>/dev/null || true` has cost this project more debugging than anything
+# else: a tree came out with every package home root:root 755 and not one line
+# said why. The pattern is not banned -- mkdir -p on a directory that exists is
+# fine -- but a failure that CHANGES WHAT THE SYSTEM LOOKS LIKE must be said.
+_h="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$_h" ]; then
+ _p=""
+ grep -q '^soft() {' "$_h" || _p="$_p;there is no way to report a non-fatal failure"
+ # the ownership and permission calls are the ones that matter
+ _leak="$(grep -nE '(real_chmod|real_chown|real_chgrp|chown|chgrp) .*2>/dev/null \|\| true' "$_h" \
+ | grep -v 'SUPPRESSED DELIBERATELY' | wc -l)"
+ # ANY comment above is not enough -- the surrounding prose explains what the
+ # call does, not why its failure may be thrown away, and a reintroduced
+ # suppression inherits whatever comment happened to be there. The line
+ # immediately above must say which of the two cases this is, in those words.
+ _undoc=0
+ while IFS= read -r _ln; do
+ [ -n "$_ln" ] || continue
+ _n="${_ln%%:*}"
+ sed -n "$((_n-1))p" "$_h" \
+ | grep -qE 'SUPPRESSED DELIBERATELY|benign:' || _undoc=$((_undoc+1))
+ done <<< "$(grep -nE '(real_chmod|real_chown|real_chgrp) .*2>/dev/null \|\| true' "$_h")"
+ [ "${_undoc:-0}" = 0 ] \
+ || _p="$_p;$_undoc suppressed ownership call(s) are not marked deliberate or benign"
+ # and the two that are deliberate must say so in those words
+ grep -q 'SUPPRESSED DELIBERATELY' "$_h" \
+ || _p="$_p;the deliberate suppressions are not marked as such"
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "every suppressed ownership failure says why it is suppressed"
+ fi
+fi
+
+# lfs-helper must actually offer the doors the other tool now knocks on.
+_h="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$_h" ]; then
+ _p=""
+ for _c in wrapper-dir make-wrappers grant-dir pkgusr-home owner-name; do
+ grep -qE "^ $_c\)" "$_h" \
+ || _p="$_p;lfs-helper has no '$_c' command"
+ done
+ # `case` takes the first match, so a repeated entry is unreachable code
+ # pretending to be a second decision. Six of them were.
+ _dups="$(sed -n '/^case "${1:-}" in/,/^esac/p' "$_h" \
+ | grep -oE '^ [a-z][a-z0-9|-]*\)' | sort | uniq -d)"
+ [ -z "$_dups" ] || _p="$_p;the dispatch repeats: $(echo $_dups | tr -d ')')"
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "every shared decision is reachable as one lfs-helper command"
+ fi
fi
# ---- editing the wrong install script fails silently ------------------------ #
@@ -6073,7 +6444,7 @@ if [ -f "$helper_src" ]; then
fi
# nothing outside the chokepoints may build an account path by hand
-for _f in packagemanager_install lfs-completion.bash last_build_step.sh; do
+for _f in packagemanager_install lfs-completion.bash; do
_p="$(dirname "$LFS_TOOL")/$_f"
[ -f "$_p" ] || continue
if grep -qE '"?/usr/src/\$[a-z_]+' "$_p"; then
@@ -6872,6 +7243,7 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper"
if [ -f "$helper_src" ]; then
_o="$( SNAP_ROOT=/
eval "$(_slice_fn "$helper_src" never_claim_list)"
+ eval "$(_slice_fn "$helper_src" _never_claim_load)"
eval "$(_slice_fn "$helper_src" is_never_claimed)"
for f in /usr/share/info/dir /etc/ld.so.cache /var/cache/ldconfig/aux-cache; do
is_never_claimed "$f" || echo "$f can still be claimed by a package"
@@ -7457,6 +7829,7 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper"
if [ -f "$helper_src" ]; then
_o="$( ETC=/etc; SNAP_ROOT=/
eval "$(_slice_fn "$helper_src" never_claim_list)"
+ eval "$(_slice_fn "$helper_src" _never_claim_load)"
eval "$(_slice_fn "$helper_src" is_never_claimed)"
for f in /etc/passwd /etc/passwd- /etc/group /etc/group- \
/etc/shadow /etc/gshadow /etc/.pwd.lock; do
@@ -7844,6 +8217,7 @@ if [ -f "$helper_src" ]; then
_o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr
WRAPPERS=/usr/lib/pkgusr
eval "$(_slice_fn "$helper_src" never_claim_list)"
+ eval "$(_slice_fn "$helper_src" _never_claim_load)"
eval "$(_slice_fn "$helper_src" is_never_claimed)"
command -v is_never_claimed >/dev/null 2>&1 \
|| { echo "there is no is_never_claimed"; exit 0; }
@@ -7900,7 +8274,7 @@ _count_rc $?
# ---- add-user names the account through the chokepoint ---------------------- #
# It used its argument raw for the ACCOUNT and pkg_owner_name only for the HOME.
# Fine while every caller passes an already-prefixed owner -- cmd_build does.
-# last_build_step.sh does not: `lfs-helper add-user wget` produced an account
+# last_build_step.sh did not: `lfs-helper add-user wget` produced an account
# called `wget` living in /usr/src/pkgusr/p_wget, and the sanity report found
# both halves without being able to connect them:
# !! accounts without a known prefix: wget urllib3 requests ...
@@ -7929,13 +8303,11 @@ if [ -f "$helper_src" ]; then
ok "add-user gives the account and its home the same name"
fi
- # last_build_step.sh is the caller that exposed it -- it passes bare names
- _lbs="$(dirname "$LFS_TOOL")/last_build_step.sh"
- if [ -f "$_lbs" ]; then
- grep -q 'lfs-helper add-user' "$_lbs" \
- && ok "the final step still creates its users through add-user" \
- || bad "the final step no longer goes through add-user"
- fi
+ # last_build_step.sh was the caller that exposed it -- it passed bare names.
+ # It is gone (1.10.0) and this checked a file that no longer exists, so it
+ # never ran again. The requirement outlives it: `packagemanager setup` is
+ # the next caller to create accounts from outside the tool, and it must go
+ # through add-user rather than useradd. Assert that when setup lands.
fi
# ---- the sanity report reads the sorted state, and knows when a build is done #
@@ -8003,6 +8375,7 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper"
if [ -f "$helper_src" ]; then
_o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr; WRAPPERS=/usr/lib/pkgusr
eval "$(_slice_fn "$helper_src" never_claim_list)"
+ eval "$(_slice_fn "$helper_src" _never_claim_load)"
eval "$(_slice_fn "$helper_src" is_never_claimed)"
command -v is_never_claimed >/dev/null 2>&1 \
|| { echo "there is no is_never_claimed"; exit 0; }
@@ -8265,25 +8638,10 @@ if [ -f "$helper_src" ]; then
fi
fi
-# and the final step must ASK rather than rebuild the name itself
-_lbs="$(dirname "$LFS_TOOL")/last_build_step.sh"
-if [ -f "$_lbs" ]; then
- _p=""
- _n="$(grep -c 'lfs-helper owner-name' "$_lbs")"
- [ "${_n:-0}" -ge 2 ] \
- || _p="$_p;the final step still assumes the account name ($_n of 2 sites)"
- # every chown/su in there must use the resolved name, never the bare argument
- case "$(grep -c 'lfs-helper add-user' "$_lbs")" in
- 0) _p="$_p;the final step no longer creates its users through add-user" ;;
- esac
- if [ -n "$_p" ]; then
- printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
- [ -n "$m" ] && bad "$m"
- done
- else
- ok "the final step asks for the account name it was given"
- fi
-fi
+# The caller that exposed this was last_build_step.sh, which is gone (1.10.0).
+# The rule is not: any script outside the tool that needs an account name must
+# ASK -- `lfs-helper owner-name` -- instead of gluing a prefix on itself.
+# `packagemanager setup` inherits it; assert it there when setup lands.
# ---- the ESP hint prints a path you can actually type ----------------------- #
# Two sources feed it and they disagree about the prefix: `lsblk -rno NAME`
@@ -8459,6 +8817,210 @@ if [ -f "$helper_src" ]; then
rm -rf "$_st"
fi
+# ---- unpacked sources are scratch, and no scan should walk them ------------- #
+# Build trees moved from /build into each package user's home in 1.7.6. The
+# snapshot scans followed; the ownership scans did not. So `lfs-helper verify`
+# began walking every unpacked source tree in the system -- it ran for minutes
+# with no output -- and a finished tree filled the sanity report with tcl's own
+# documentation:
+#
+# !! files with no owner (first 40):
+# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm
+#
+# A tarball can carry any uid it likes. Unpacked sources are not installed,
+# nothing owns them, and asking who does has no answer.
+helper_src="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$helper_src" ]; then
+ _p=""
+ _slice_fn "$helper_src" scan_prune_paths | grep -q 'PKGUSR_BUILD_SUBDIR' \
+ || _p="$_p;the prune list does not cover the per-package build trees"
+ # ONE list -- there were five and they had already diverged
+ for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do
+ _slice_fn "$helper_src" "$_fn" | grep -q 'SCAN_PRUNE' \
+ || _p="$_p;$_fn does not use the shared prune list"
+ done
+ # an ARRAY, not a string through eval: the patterns contain *, and eval lets
+ # the shell expand them before find ever sees them
+ case "$(_slice_fn "$helper_src" scan_prune_set)" in
+ *'SCAN_PRUNE+=('*) ;;
+ *) _p="$_p;the prune list is not built as an array" ;;
+ esac
+ if _slice_fn "$helper_src" _vfy_orphans | grep -q 'eval find'; then
+ _p="$_p;the orphan scan splices its prune list through eval"
+ fi
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "every ownership scan skips the same scratch, from one list"
+ fi
+
+ # end to end: a build tree is pruned, a real file is not
+ _pt="$T/prune"
+ mkdir -p "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html" "$_pt/usr/bin"
+ touch "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real"
+ if [ "$(id -u)" = 0 ]; then
+ chown 4321 "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real"
+ _r="$( SNAP_ROOT="$_pt"; STATE="$_pt/usr/src/lfs-pkgusr"
+ PKGUSR_BUILD_SUBDIR=src
+ PKGUSR_ROOT="$_pt/usr/src/pkgusr"; CFGUSR_ROOT="$_pt/usr/src/cfg"
+ pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; }
+ eval "$(_slice_fn "$helper_src" scan_prune_paths)"
+ eval "$(_slice_fn "$helper_src" scan_prune_set)"
+ scan_prune_set
+ find "$_pt" -xdev -nouser "${SCAN_PRUNE[@]}" 2>/dev/null \
+ | sed "s|$_pt||" | sort | tr '\n' ' ' )"
+ case "$_r" in
+ "/usr/bin/real ") ok "an unpacked source tree is skipped, an installed file is not" ;;
+ *) bad "the prune list is wrong: got [$_r]" ;;
+ esac
+ fi
+ rm -rf "$_pt"
+fi
+
+# the sanity report has to skip the same thing, or it reports what verify does not
+san="$(dirname "$LFS_TOOL")/lfs-sanity.sh"
+if [ -f "$san" ]; then
+ _p=""
+ grep -q 'PKGROOT/\*/src' "$san" \
+ || _p="$_p;the sanity report still walks the package build trees"
+ # a config step writing root-owned files is not an orphaned manifest
+ grep -q 'writes ROOT-owned files' "$san" \
+ || _p="$_p;root-owned config files are still reported as missing an account"
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "the sanity report skips scratch and does not invent findings"
+ fi
+fi
+
+# ---- verify must not work in silence, or one fork at a time ----------------- #
+# `lfs-helper verify` ran for over five minutes printing nothing, on a tree it
+# had already verified clean. Two separate faults:
+#
+# * a fork per path, twice over -- `stat -c %U` and a process substitution
+# inside is_never_claimed -- across 66265 manifest paths
+# * no output at all until every pass had finished
+#
+# A command that works for minutes in silence is indistinguishable from one that
+# has hung, and was reported as one.
+helper_src="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$helper_src" ]; then
+ _p=""
+ # the never-claim list is read once, not per call
+ _slice_fn "$helper_src" is_never_claimed | grep -q '< <(never_claim_list)' \
+ && _p="$_p;is_never_claimed forks for every path it is asked about"
+ _slice_fn "$helper_src" _never_claim_load | grep -q . \
+ || _p="$_p;the never-claim list is not cached"
+ # owners come from one batched stat, not one per path
+ case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in
+ *'xargs -0 -r stat'*) ;;
+ *) _p="$_p;the ownership pass still runs one stat per path" ;;
+ esac
+ # a REAL tab: stat does not expand \t, so '%U\tname' comes back as one field
+ # and the map silently stays empty -- correct but slow, the hardest wrong to see
+ # comments stripped: the fix NAMES the broken format while explaining it
+ case "$(_slice_fn "$helper_src" _vfy_manifest_ownership \
+ | grep -vE '^[[:space:]]*#')" in
+ *'%U\t%n'*) _p="$_p;the stat format uses a literal backslash-t" ;;
+ esac
+ # and it must say what it is doing, before it does it
+ _nd="$(_slice_fn "$helper_src" cmd_verify | grep -c 'detail "# ')"
+ [ "${_nd:-0}" -ge 5 ] \
+ || _p="$_p;verify announces only $_nd of its passes"
+ case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in
+ *'[%d/%d]'*) ;;
+ *) _p="$_p;the long pass shows no progress" ;;
+ esac
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "verify says what it is doing, and does it in one pass not 66000"
+ fi
+
+ # the batched lookup must actually map something -- an empty map is the bug
+ _pf="$T/statmap"; mkdir -p "$_pf"
+ touch "$_pf/a" "$_pf/b" "$_pf/c"
+ printf '%s\n' "$_pf/a" "$_pf/b" "$_pf/c" > "$_pf/man"
+ _n="$( _TAB=$'\t'
+ declare -A own=()
+ while IFS="$_TAB" read -r o nm; do
+ [ -n "$nm" ] && own["$nm"]="$o"
+ done < <(grep -v '^[[:space:]]*$' "$_pf/man" \
+ | tr '\n' '\0' | xargs -0 -r stat -c "%U${_TAB}%n" 2>/dev/null)
+ echo "${#own[@]}" )"
+ [ "$_n" = 3 ] \
+ && ok "the batched owner lookup maps every path it is given" \
+ || bad "the batched owner lookup mapped $_n of 3 paths"
+ rm -rf "$_pf"
+fi
+
+# ---- the build must not end quietly with no way to log in ------------------- #
+# init-accounts asks for a root password, but it is skippable: a scripted run
+# has no terminal, and an interactive one can be answered with a blank line.
+# So the LAST thing build-all says is whether anyone can actually log in --
+# because it is the last moment it can still be fixed. After the reboot the
+# chroot is gone, and the way back is booting the host and mounting the tree.
+helper_src="$(dirname "$LFS_TOOL")/lfs-helper"
+if [ -f "$helper_src" ]; then
+ _p=""
+ _slice_fn "$helper_src" _warn_if_no_login | grep -q . \
+ || _p="$_p;nothing checks whether the system can be logged into"
+ # it must be the LAST thing build-all does
+ _ba="$(_slice_fn "$helper_src" cmd_build_all)"
+ case "$_ba" in
+ *_warn_if_no_login*) ;;
+ *) _p="$_p;build-all can finish without saying whether you can log in" ;;
+ esac
+ _last="$(printf '%s\n' "$_ba" | grep -vE '^[[:space:]]*(#|$)' | tail -3)"
+ case "$_last" in
+ *_warn_if_no_login*) ;;
+ *) _p="$_p;the login check is not the last thing build-all does" ;;
+ esac
+ # it must check the RESULT, not that the step ran
+ case "$(_slice_fn "$helper_src" _warn_if_no_login)" in
+ *shadow*) ;;
+ *) _p="$_p;it trusts init-accounts instead of checking for a password" ;;
+ esac
+ if [ -n "$_p" ]; then
+ printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "the build's last word is whether you can log in afterwards"
+ fi
+
+ # all three outcomes, against a real shadow file
+ _lg="$T/login"; mkdir -p "$_lg"
+ _run_login() { # <shadow contents> -> "ok" or "warn"
+ ( ETC="$_lg"; say(){ :; }; ok(){ :; }; warn(){ echo warn; }
+ printf 'root:x:0:0::/root:/bin/bash\nbob:x:1000:1000::/home/bob:/bin/bash\n' > "$_lg/passwd"
+ printf '%s' "$1" > "$_lg/shadow"
+ eval "$(_slice_fn "$helper_src" _warn_if_no_login)"
+ _warn_if_no_login | head -1 ) 2>/dev/null
+ }
+ _bad=""
+ [ -z "$(_run_login 'root:$6$a:1::::::
+bob:!:1::::::')" ] || _bad="$_bad;a root password is not accepted"
+ [ -z "$(_run_login 'root:*:1::::::
+bob:$6$b:1::::::')" ] || _bad="$_bad;a user password is not accepted"
+ [ -n "$(_run_login 'root:*:1::::::
+bob:!:1::::::')" ] || _bad="$_bad;a system nobody can log into is not reported"
+ if [ -n "$_bad" ]; then
+ printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do
+ [ -n "$m" ] && bad "$m"
+ done
+ else
+ ok "a locked-out system is reported; a usable one is not"
+ fi
+ rm -rf "$_lg"
+fi
+
echo
echo " PASS: $(_pass_total) FAIL: $(_fail_total) (final)"
[ "$(_fail_total)" -eq 0 ]