Packagemanager-LFS-PackageUser-System git · main
Package-user based package manager for LFS/BLFS
Shell 63.7% Markdown 35.8%git clone https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System.gitwget https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System/archive/Packagemanager-LFS-PackageUser-System.tar.gzStack sorting And bug fixes
BUILD.md | 5 +- HANDOFF.md | 446 ++++++++++++++++++++++- README.md | 41 ++- blfs | 120 ++++++- kernel.conf | 32 +- lfs | 2 +- lfs-helper | 246 ++++++++++++- lfs-kernel | 957 +++++++++++++++++++++++++++++++++++++++++++++++++- lfs-phases | 59 +++- packagemanager | 181 +++++++++- stacks/apps.stack | 30 +- stacks/diag.stack | 14 +- stacks/machine.conf | 17 + stacks/net-wifi.conf | 60 ++++ stacks/net-wifi.sh | 223 +++++++++++- stacks/services.stack | 35 +- 16 files changed, 2393 insertions(+), 75 deletions(-) diff --git a/BUILD.md b/BUILD.md index 5234eb8..c33bde9 100644 --- a/BUILD.md +++ b/BUILD.md @@ -300,7 +300,8 @@ generated, so it can be edited before the first build. | `HEADERS`, `V4L2LOOPBACK`, `BOOTLOGO` | as you like | ```sh -lfs-kernel # build, install, record as p_linux +lfs-kernel # the plan; nothing is built +lfs-kernel --run # build, install, record as p_linux lfs-kernel verify # image, modules, depmod, headers, firmware ``` @@ -402,7 +403,7 @@ packagemanager update <pkg> --run # what the book has newer packagemanager which-package /usr/bin/foo packagemanager reclaim <pkg> --run # give files back another package took packagemanager user create firefox --shared --share-dir --launcher # an app account -lfs-kernel latest # try a newer kernel beside this one +lfs-kernel latest --run # try a newer kernel beside this one ``` A new BLFS release: `blfs fetch`, `packagemanager blfs set-default`, diff --git a/HANDOFF.md b/HANDOFF.md index 360f94c..2754d00 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -7,13 +7,13 @@ account per package, so every file on the system has an owner, and Written for a **SysV** system (no systemd) built from the **systemd** BLFS book, because BLFS stopped publishing a SysV edition after 12.4. - all six tools at 1.14.321 (lfs-backup has its own: 1.3.0) - lfs cb920be - lfs-helper 3422951 - packagemanager 3517a41 - blfs f8ce360 - lfs-phases 2c5f6a3 - lfs-kernel 219215c (no version line of its own) + all six tools at 1.14.340 (lfs-backup has its own: 1.3.0) + lfs 2f1eb17 + lfs-helper db72098 + packagemanager 8265dde + blfs 80426a3 + lfs-phases 026ca06 + lfs-kernel fce4ce0 (no version line of its own) lfs-backup c73acaa (1.3.0; replaces lfs-config) lfs-sysvbook 62ebe64 (1.0.0; untouched since it was written) @@ -3777,6 +3777,438 @@ nested-repo-path to flat-request, absent), the ownership failure, and blocks - Not run: the regression suite (sandbox slot). `bash -n` / `py_compile` and `--version` only. +## 1.14.340 -- a man page with colons in its name + +XML::Parser: six repair rounds, all of them granting directories, and the +install still failed. The report said + + /usr/share/man/man3/XML missing; parent /usr/share/man/man3 is drwxrwxr-t + +-- a file that has never existed. The real name is +`/usr/share/man/man3/XML::Parser.3`, and both path extractors cut it at the +first colon: their character class was `[A-Za-z0-9._/+-]`. So every Perl +man page has been reported under a truncated name, and the file that +actually blocked the install was never named. A trailing colon (from the +message's own "path: Permission denied") is stripped now. + +- THE STICKY BIT IS THE POINT. /usr/share/man/man3 is drwxrwxr-t, so only + a file's OWNER may replace it -- and ExtUtils::Install replaces by + unlinking first. The file belongs to p_xml-parser, an older account for + the same module (this one is p_perl-xml-parser, a different book anchor). + No directory grant can ever fix that, which is why six rounds did not. +- takeover_files_from_log now reads perl's wording -- `Cannot forceunlink + <path>`, `Can't unlink/remove/rename <path>` -- as a whole token, colons + and all. The file is handed to the package that is installing it. +- When two or more files move from an account whose name is this one's + minus a language prefix (xml-parser <- perl-xml-parser), the box says + they are one upstream module under two anchors and offers + `packagemanager remove xml-parser --purge --run` for afterwards. +- Tested: both wordings extracted with their colons intact. + +## 1.14.339 -- install_last is written BEFORE the build + +The answer to 1.14.338's open question, and the reason the stack needed a +package named by hand. + +- lfs-helper copies the script to `install_last` BEFORE running it ("RECORD + WHAT IS INSTALLED BEFORE OVERWRITING THE RECORD"). So a build that dies + at configure leaves one behind. `User.state`, with no manifest and a page + that declares no programs or libraries to check -- cairomm is exactly that + -- then falls through to `if self.meta.get("name_version"): installed`. +- cairomm-1.0 failed at configure with 0 files tracked, was recorded + installed by that rule, and every later plan skipped it as a dependency + that was already there. pangomm then failed on + Run-time dependency cairomm-1.0 found: NO + and only naming cairomm by hand got the chain moving. +- `_install_finished(u)`: a manifest with real paths, or `.pkgusr-installed` + (the marker the runner writes AFTER a successful install), is evidence. + A script copied in before the build is not. The planner skips a + dependency only with that evidence; without it the package goes into the + plan, and the plan says so: + N package(s) are recorded as installed but have no finished-install + record: cairomm-1.0 +- Tested end to end with three shapes: install_last only (planned), + .pkgusr-installed (skipped), a manifest (skipped). +- NOT changed: `state` itself, which many commands read. The rule here is + the PLANNER's, where a wrong "installed" costs a broken chain and a + needless rebuild costs minutes. + +## 1.14.338 -- "recorded as installed" is not "on disk" + +pangomm could not find cairomm-1.0, and the box said cairomm "IS installed +here ... if the .pc is absent, it installed without it". Two very +different things hide behind that: a package that installed and later lost +a file, and one that is RECORDED as installed and put nothing on disk (a +build that failed after its account and home were made). The manifest +answers it in one read, so the box now reads it: + + * manifest empty -> "nothing of it is on disk. Its last build did + not finish, and whatever marked it installed + was wrong." + * manifest, but no .pc -> "records N file(s), and NONE of them is a .pc" + * manifest with .pc -> each one printed, "(on disk)" or "MISSING from + disk", and both the rebuild and the + pkg-config-path checks + +Tested against all three shapes. + +STILL OPEN on that machine: why cairomm-1.0 was skipped as installed at +all. Its previous build failed in the BUILD phase with 0 files tracked, +which `state` should read as "not installed" (no manifest, no +install_last). Either something wrote install_last for a build that never +installed, or a DIFFERENT cairomm (the 1.16 API is its own package) is +being mistaken for it. The next run's box will say which. + +## 1.14.337 -- the build named the option; now the tool reads it + + Run-time dependency Boost found: NO (tried system) + ERROR: Building the test programs requires Boost Test ... and Fontconfig. + Install them or use -Dbuild-tests=false or -Dbuild-tests=if-dependencies-found + +- cairomm stops at configure over TEST programs, which want Boost -- and + Boost is in avoid.stack on purpose. The dependency is never coming; the + option meson prints is the whole answer, and it was on the screen while + the chain sat stopped. +- `_meson_opt_from_log` takes the FIRST `use -D<opt>=<value>` the build + suggested, adds it to the script's `meson setup` line (old script kept as + .bak-<date>) and the retry loop runs the phase again. Only an option the + build itself named, only one that turns something OFF (=false, =disabled, + =no, =if-dependencies-found), only once. The gate lets such a failure + reach the loop, and the "no repair attempted" line no longer claims the + only repairs are permissions. +- A regenerated script loses the edit, so the note prints the machine.conf + section to make it permanent -- and machine.conf now ships + build-tests = false for cairomm-1.0, pangomm-2.48 and glibmm-2.68, which + are the three in this chain with the same Boost test dependency. +- Tested: the option is added once, the second call is a no-op, and a + suggestion that turns something ON is refused. + +## 1.14.336 -- the test suite was in the BUILD block all along + +XML::Parser died in the BUILD phase: + + make: *** [Makefile:1005: test_dynamic] Error 2 + Failed 60/62 test programs. + +- Every Perl module in BLFS uses "the standard build and installation + instructions", which are ONE block: + perl Makefile.PL && + make && + make test + The test suite is not a block to skip -- it is the last LINE of the block + that builds the package, joined with &&. So the block-level test filter + could never see it, and every perl module on this machine has been + running its test suite inside the build. Class::Inspector's passed, + which is why nobody noticed; XML::Parser's fail (60 of 62 files cannot + load the module) and took the gparted chain with them. +- _strip_test_lines drops such a line, leaves the book's own text in a + comment above it, and removes the now-dangling `&&` from the line before. + A block that is ONLY tests still goes through the block-level guard with + its reason. `--test` (PKGUSR_TESTS=1) still runs the suite -- in the + TEST phase, where a failure is reported and not fatal. +- Checked against seven blocks: the perl one, a plain configure+make, a + bare `make check`, `make && LC_ALL=... make check_local`, `make install`, + LVM2's two-target install, and meson+ninja+`ninja test`. +- Script format v19, so every generated script is rebuilt from the book. + +Worth knowing: XML::Parser's tests failing to LOAD the module may mean its +Expat binding is not linking to the installed expat. It is not in the way +of the build any more; after the install, check it: + perl -MXML::Parser -e 'print "ok\n"' + +## 1.14.335 -- a PDF that was never built is not a failed install + +- parted installed 97 files -- every binary, every library -- and then + failed: + install: cannot stat 'doc/parted.pdf': No such file or directory + The book's last line is + install -v -m644 doc/{FAT,API,parted.{html,txt,pdf,ps,dvi}} \ + /usr/share/doc/parted-3.7 + and parted.pdf/ps/dvi and API exist only when texlive built them. + texlive is in avoid.stack, so they never exist here -- for parted, and + for every other page with a doc block like it. +- The install PHASE now defines an `install` function (and only that phase, + which runs in a child process of its own -- nothing outside ever sees + it). For a destination under share/doc it runs the book's command + unchanged first; only if that fails does it retry without the sources + that are not there, and say which they were. Everything else goes + straight to /usr/bin/install. A doc command that fails for any OTHER + reason is re-run and its real error reported. +- Tested: the book's block with two files missing (installs the two that + exist, rc 0, names the others), all sources missing, `-m 644` as two + words, `install -dm755` on a doc directory, and a non-doc destination. +- kernel.conf ships CONFIG_SET="LEGACY_TIOCSTI=n EXFAT_FS=m" -- the kernel + half of exfatprogs, and what this machine already has. (It was the only + difference left in kernel.conf.new.) + +## 1.14.334 -- `mount -o remount,dev /tmp` in an install phase + +LVM2 stopped the gparted chain with + + mount: /tmp: must be superuser to use mount. + !! phase 'install' FAILED (exit 32) + +and the report said "no line matched the usual error patterns". + +- WHERE IT CAME FROM. BLFS 13.0's LVM2 page has three blocks between the + build and the install: a partial install for the test suite, then + `mount -o remount,dev /tmp`, then `LC_ALL=... make check_local`. The + middle one prepares /tmp so the TESTS may create device nodes. All three + went into the install phase. +- `make check_local` was missed by the test filter for two reasons, both + now fixed: the pattern wanted a word boundary after "check" (and + "check_local" has none), and it did not allow the `LC_ALL=...` in front. +- A new rule, judged on the commands and not on the prose: a block + containing mount, umount, modprobe, insmod, mknod, swapon or swapoff is + never part of installing a package. A package user cannot run those and + no collector group can grant them; they are the system's business. The + script says so where the block would have been. +- lfs-helper recognises "must be superuser" and the like: it names the line + and says the step wanted something root-only, instead of falling through + to "no pattern matched". +- apps.stack: `book gparted ignore=lvm2`. LVM2 is only RECOMMENDED by + parted, and this laptop has no logical volumes. +- Tested: the block classifier against all seven blocks of the LVM2 page + (mount -> root-only, check_local -> test, install/configure -> kept). + +## 1.14.333 -- the wpa service reads one directory; we wrote the other + +The boot log, not dhcpcd: + + Starting dhcpcd on the wlp3s0 interface... FAIL + wpa_supplicant configuration file + /etc/sysconfig/wpa_supplicant-wlan0.conf not present FAIL + Bringing up the eth0 interface... Interface eth0 doesn't exist. FAIL + +- THE PATH. This machine's wpa service reads + /etc/sysconfig/wpa_supplicant-<iface>.conf; net-wifi wrote and looked in + /etc/sysconfig/wpa_supplicant/ -- the subdirectory other bootscript + releases use. So wpa_supplicant never started, nothing associated, and + dhcpcd had no link to ask on. The path is now READ out of $svcdir/wpa + ("the wpa service reads: ...") and used for the check and for COPY_WPA; + a config in either layout is found and copied to the one that is read. +- Per interface, the step now says whether the config the service will look + for exists, and prints the one-line copy when it does not. +- DHCP_START without -b is reported: dhcpcd in the foreground waits for a + lease on a wireless link that is not associated yet, which is the + "Starting dhcpcd ... FAIL" line. Our writer has always used -b. +- A config for an interface the kernel does not have is a FAIL line at + every boot. It is reported with the sed that turns ONBOOT off, and + DISABLE_MISSING=yes in net-wifi.conf does it for all of them (backed up; + nothing is deleted -- the card may be back tomorrow). +- Tested in a sandbox with a fake service script, a fake /sys/class/net and + a temporary /etc/sysconfig: the path is detected, the copy lands where + the service reads, and a stale eth0 file is reported. + +## 1.14.332 -- ExtUtils::Install says it in its own words + +- File::ShareDir stopped the plan with + ERROR: Can't create '/usr/lib/perl5/5.42/site_perl/File' + Do not have write permissions on '/usr/lib/perl5/5.42/site_perl/File' + and the box said "no automatic repair attempted: the failure does not + look like a permission or missing-directory problem". It is exactly a + permission problem; ExtUtils::Install simply never writes the words + "Permission denied" for it. Two greps were blind to it: + * the _perm_ish gate (so zero rounds ran), and + * unwritable_dirs_from_log's tail filter (so even a round would have + had nothing to grant). + Both now know `Can't create '<path>'`, `Do not have write permissions` + and `Cannot mkdir`. Checked against the real log text: the gate fires + and the extractor yields /usr/lib/perl5/5.42/site_perl, the directory + the grant needs. +- Round two of the same lesson as 1.14.331: a Perl module's install writes + into three trees owned by three different packages, and every one of them + reports the refusal differently. + +## 1.14.331 -- perllocal.pod is the info/dir of the perl tree + +- `perl-class-inspector` installed both its modules and both its man pages + and was then reported FAILED, for a line make itself had marked + "(ignored)": + /usr/lib/perl5/5.42/core_perl/perllocal.pod: Permission denied + Every Perl module APPENDS to that file in its doc_install step, so it + belongs to whichever module was installed first (p_xml-parser, 664, group + p_xml-parser) and every module after that cannot write it. Granting the + DIRECTORY to the perl collector group -- which the auto-repair did -- + cannot help: the file's own group is one package's. +- It is in shared_files_list now, with the info/dir family: written by many + packages, installed by none. share_shared_files makes it root:install + 664, which every package user can append to. The version is globbed + (/usr/lib/perl5/*/*/perllocal.pod). +- shared_file_dirs SKIPS the perl directories: the file is shared, the + directory is not -- /usr/lib/perl5/<ver>/core_perl is p_perl's install + directory and is in its manifest. Handing it to root:install would make + `verify` right to complain. +- The "exited 0 but the log says otherwise" gate no longer fails a phase + for a shared registry file it could not update. It says so, runs + share_shared_files, and lets the package's own files stand. A real + `install: cannot create ...: Permission denied` still fails, tested + against both logs. + +## 1.14.330 -- an installed dependency is not a thing to upgrade + +- `install gparted --run --recursive` planned FIFTY-THREE packages: gparted, + its missing dependencies -- and updates of bubblewrap, alsa-lib, gtk3, + gtk4, librsvg, glycin, GnuTLS and thirty more that were installed and + working, purely because the book had moved on. Asking for one program is + not asking to rebuild the desktop, and each rebuild is a chance to break + a machine that was fine. A DEPENDENCY that is installed is now left + alone whatever its version; a package NAMED on the command line is + unchanged; `--update-deps` restores the old behaviour. The plan says how + many it left and how to move them on purpose. + (Unit-tested: gparted with gtk3 installed-but-behind plans 2 packages, + and 3 with --update-deps.) +- exfatprogs is a `tar` entry now, not `book`: the BLFS index carries no + exFAT package at all (checked). 1.4.2 from the project's release + tarball, ./configure with an autogen.sh fallback. The relan/exfat the + book mentions is the FUSE implementation -- a different thing, and only + for a kernel without EXFAT_FS. + +## 1.14.329 -- gparted + +- apps.stack: `book gparted`, last in the file (it is the heaviest entry -- + the planner brings gtkmm3 and parted). Confirmed a BLFS page against the + stable index: Gparted-1.7.0. +- It has no graphical privilege step here: the book's page uses polkit and + polkit is avoided on this machine (it arrives only through colord and + systemd). So: run `gparted` from a ROOT terminal inside the session. + The comment in the stack says so. +- The filesystem tools it drives are their own packages; exfatprogs and + btrfs-progs (services.stack) are what stop entries being greyed out. + +## 1.14.328 -- exfatprogs + +- services.stack: `book exfatprogs` (mkfs.exfat, fsck.exfat, exfatlabel), + beside btrfs-progs. No `when=`: the card that needs it is not in the + machine when the stack runs. The kernel side stays separate -- + `lfs-kernel enable exfat` sets EXFAT_FS=m. +- Not tested against the book here: if the page's anchor is not + `exfatprogs`, the stack says NOT IN BOOK and `blfs search exfat` gives + the real one. + +## 1.14.327 -- net-wifi writes what holds no secret, and prefers an antenna + +- `cfg net-wifi` reported and wrote nothing, so a deleted + /etc/sysconfig/ifconfig.<iface> meant pasting a file out of its output. + It now reads stacks/net-wifi.conf and, with WRITE=yes, writes that file + for every wireless interface that has none -- marked as ours, never over + one of yours. The wpa config is still never written: it holds the key. +- PREFER (patterns, best first) decides the dhcpcd route metric: the first + interface PRESENT gets METRIC_PREFERRED, the rest METRIC_OTHER, so a USB + antenna takes over while it is plugged in and the built-in card carries + on when it is gone. A file the step wrote is re-metered when PREFER + changes (backed up); one of yours is reported, not edited. +- USB_RENAME writes a udev rule naming an adapter by its USB ids + (0cf3:9271 -> wl-ext), because its bus name changes with the socket and + ifconfig.<name> cannot follow. It applies when the device next appears. +- COPY_WPA copies an existing wpa_supplicant-<iface>.conf to an interface + that has none (chmod 600), and says so every time -- a key is duplicated. +- A config naming an interface the kernel does not have is now reported. +- diag.stack: usbutils and pciutils (lsusb, lspci) -- the first commands + for a device that did not appear. +- Tested with a fake /sys/class/net (NET_WIFI_SYSFS) and a temporary + /etc/sysconfig: first write, the metric flip when the preferred interface + appears, the wpa copy, the orphan-config note, the udev rule text. + +## 1.14.326 -- a build needs --run, a module can be loaded, a scan is pruned + +- `lfs-kernel` alone no longer builds. It prints what a build WOULD do -- + version, tree state (fetch / continue / already built), config source and + CONFIG_SET, where the image and modules go, firmware, headers, + v4l2loopback -- and says `--run`. `--run` is taken from anywhere on the + line. An hour of compiling that replaces the booting kernel should not + start from a bare word; every other tool here already waited. +- `lfs-kernel load <module> [--boot]` / `unload`: modprobe, confirm it in + /proc/modules (and say when a symbol is built IN rather than missing), + show the dmesg lines that mention it or firmware, and with --boot add it + to /etc/sysconfig/modules (backed up, no duplicates). `modules` points at + it instead of telling people to type modprobe themselves. +- `reload-pkg-list` scans with its own pruned find instead of the hint's + `list_package`: -xdev, and never /proc, /sys, /dev, /run, /tmp, /var/tmp, + /var/cache, /mnt, /media, the snapshots, /sources, /build or the PACKAGE + SOURCE ROOTS. That last one is where the time went -- ninety unpacked + source trees, the kernel's own among them. Still run as the package user + (so unreadable directories are skipped), with a line saying so and + `--root` for the thorough scan. The seconds are now printed. +- Tested: the plan for an already-built tree, load/unload with a stub + modprobe and the boot file, the pruned find, py_compile / bash -n. + +## 1.14.325 -- where are we, and a firmware report that needs no network + +- `lfs-kernel status`: what is running and what is known to boot, whether + the tree's .config carries every CONFIG_SET entry, whether the build is + FINISHED / UNFINISHED (object files, no current image) / not started, how + many modules are in the tree and in /lib/modules, what the ESP holds, and + whether the installed config differs from the tree's. Reads only; no + network; works with the ESP unmounted. Written because an interrupted + build leaves nothing on the machine that says how far it got. +- `firmware --missing` asked kernel.org which kernel is newest, for a + question /lib/modules and the boot log answer -- and EXITED on a machine + with no network. It uses the running kernel's version now. +- Tested: status against a tree with objects only and with a bzImage, + CONFIG_SET read-back, the unmounted-ESP path. + +## 1.14.324 -- a typo built a kernel, and the command line lost to the file + +Two bugs in lfs-kernel's argument handling, both old, both found by one +paste: `lfs-kernel search AR9271` rebuilt and reinstalled 7.2.6. + +- ANY unrecognised word became VERSION. It is now refused, with the + command list and a did-you-mean; only `latest` or something shaped like + 7.2.6 is a version. `search` and `set` became real top-level commands + (they are what a person types) and go to `config search` / `config set`. +- WORSE: `VERSION="$1"` was set BEFORE `. "$CONF"`, and kernel.conf's own + VERSION= then overwrote it. So `lfs-kernel 6.12.40` built whatever the + file said (latest, here) and `lfs-kernel verify 7.2.4` reported on another + kernel -- both documented, neither working, since the config file grew a + VERSION line. The command line is re-applied after the source now. +- `lfs-kernel prune` is named in two comments and has never existed; the + comments are corrected rather than the command invented. +- Tested: unknown word refused, search alias, an explicit version surviving + the config file. + +## 1.14.323 -- hardware by name, and the modules a kernel really has + +- `lfs-kernel enable <name>`: a table of hardware (ath9k-htc, ath9k, ath10k, + ath11k, alx, atl1c, iwlwifi, rtl8xxxu, bluetooth-usb, btrfs, exfat, ntfs3, + nvme, usb-serial, v4l2-usb) -> the symbols AND the firmware it needs, + written into CONFIG_SET and FIRMWARE_ONLY. A bare symbol works too: the + tree says whether it is tristate (=m) or bool (=y). `--list` shows them. +- `lfs-kernel modules [text]` -- what /lib/modules holds for this kernel, + what is built in, what is loaded, with each module's description. +- `config search` now matches a symbol's NAME or its PROMPT and prints the + prompt. It used to list every symbol of every Kconfig file that merely + mentioned the word: 40 lines of MIPS console options for "atheros". +- `config save [--good]` seeds the archive from the running kernel without a + build -- what CONFIG_FROM=last-good needs before the first build ever runs. + `config from <source>` sets CONFIG_FROM and refuses last-good when there + is none. One writer (`_conf_set_var`) for every kernel.conf edit. +- Tested: enable (name, bare symbol, unknown), search prompts, save/from, + modules against a fake /lib/modules. No kernel built here. + +## 1.14.322 -- btrfs-progs when it applies, and the kernel configs are kept + +- Stack entries take `when="<shell>"`: judged and run only where that + command succeeds. `book btrfs-progs when=...` is in services.stack -- + mounted btrfs, or a btrfs partition on any disk (so it is right in a + chroot too). +- lfs-kernel keeps every config it builds with in /etc/pkgusr/kernel-configs + (root's, outside the tree, so `prune` and `packagemanager remove linux` + cannot take it): `config-<ver>-<date>` (newest CONFIG_KEEP=5), + `last-good.config` and `prev-good.config` -- the last two that BOOTED, + rotated where last-good is written, i.e. only from a kernel that ran. + The archive is written BEFORE the compile, so a build that dies keeps its + answers. +- CONFIG_FROM takes last-good | prev-good | saved, and `running` falls back + to last-good instead of exiting when /proc/config.gz is unreadable. +- `lfs-kernel config` -- show | list | diff | search <text> | set SYM=VAL | + unset SYM. set/unset edit CONFIG_SET in kernel.conf with a backup; + search reads the current config and the tree's Kconfig files and says + that a NET_VENDOR_*/WLAN_VENDOR_* symbol alone builds no driver. +- Tested: the config subcommands against a fake tree and config directory + (show/list/diff/search/set/unset), stack parse of the when= entry, + py_compile / bash -n. No kernel was built here. + ## 1.14.321 -- git reinstall, a stack on the svn book, @setup - `install <gitpkg> --reinstall [--clean]` builds from the package's own diff --git a/README.md b/README.md index a81bf07..0cf7cef 100644 --- a/README.md +++ b/README.md @@ -72,12 +72,15 @@ lfs-helper build-all # inside: chapters 7-9 packagemanager bootstrap --run # wget, Python modules, BLFS book, certificates packagemanager stack # list the stacks packagemanager stack sway --run # then network, services; apps and tools when wanted -lfs-kernel # the kernel, from /etc/pkgusr/kernel.conf +lfs-kernel --run # the kernel, from /etc/pkgusr/kernel.conf ``` `lfs run` and `build-all` continue from wherever they stopped. Run them again after a reboot, a failure, or a cancel. +An entry can carry `when="<shell>"`: it is built only where that command +succeeds. `btrfs-progs` in services.stack is built on btrfs machines only. + A stack can read its packages from another book. `from-book svn` in a stack file puts every book entry below it on the development book. Dependencies outside the stack stay on the default book. `packagemanager stack <name> @@ -175,7 +178,7 @@ dependency failed is skipped, not attempted. Run it again after fixing anything; it does only what is left. ```sh -packagemanager install <pkg> --recursive --run +packagemanager install <pkg> --recursive --run # missing deps only; installed ones are left alone packagemanager update <pkg> --run packagemanager install <gitpkg> --reinstall --clean --run # git packages: no book lookup packagemanager which-package /usr/bin/foo @@ -224,6 +227,18 @@ login. The step also sources the fragment from `/etc/bashrc`; any shell can refresh itself with `. /etc/profile.d/shared-dirs.sh`, which also drops the variables whose directory is gone. +### Wireless + +`packagemanager stack network --only net-wifi --run` reports what a wireless +link still needs and, with `WRITE=yes` in +`/etc/pkgusr/stacks/net-wifi.conf`, writes the parts that hold no secret: an +`/etc/sysconfig/ifconfig.<iface>` per wireless interface, and a udev rule +that gives a USB adapter the same name in every port. `PREFER` names the +interfaces best first — the first one present gets the low route metric, so +an external antenna is used while it is plugged in. The wpa_supplicant +config is never written (it holds your passphrase), but an existing one is +copied to a new interface when `COPY_WPA=yes`. + --- ## Services @@ -357,12 +372,30 @@ own package (`p_linux-firmware`) and the headers update the package that owns ```sh lfs-kernel list # what kernel.org offers -lfs-kernel # build the version in /etc/pkgusr/kernel.conf -lfs-kernel 6.12.40 # that version, once +lfs-kernel # what a build would do (nothing is built) +lfs-kernel --run # build the version in /etc/pkgusr/kernel.conf +lfs-kernel 6.12.40 --run # that version, once +lfs-kernel load ath9k_htc --boot # load a module now and at every boot lfs-kernel verify # is everything installed? lfs-kernel firmware --missing [--fix] # firmware this machine wants and lacks +lfs-kernel status # how far the last build got, what is installed +lfs-kernel enable --list # hardware this tool knows by name +lfs-kernel enable ath9k-htc # its symbols into CONFIG_SET, its firmware + # into FIRMWARE_ONLY -- kernel.conf, backed up +lfs-kernel modules [text] # what this kernel has, and what is loaded +lfs-kernel config # where the next .config comes from +lfs-kernel config search atheros # symbols by name or prompt +lfs-kernel config save --good # keep the running config as a known-good one +lfs-kernel config from last-good # build from it, not from /proc +lfs-kernel config diff # prev-good -> last-good ``` +Every config the tool builds with is copied to `/etc/pkgusr/kernel-configs`, +outside the source tree, so removing the package cannot take it: the last +`CONFIG_KEEP` builds, plus +`last-good.config` and `prev-good.config` — the two that booted. +`CONFIG_FROM=last-good` starts from one of them. + Everything it does is configured in `/etc/pkgusr/kernel.conf`, which is installed rather than generated so it can be edited first. `ESP` and `KERNEL_NAME` together decide whether a build replaces the working kernel or diff --git a/blfs b/blfs index 88f08d7..592baac 100755 --- a/blfs +++ b/blfs @@ -129,7 +129,7 @@ CACHE_VERSION = 56 # v56: a skipped build variant does not keep its install st # tool + generated-script format versions. Bump SCRIPT_VERSION when the shape # of generated install scripts changes (phases, dispatcher, ...), and # CACHE_VERSION when the parsed/cached data structure changes. -BLFS_VERSION = "1.14.321" +BLFS_VERSION = "1.14.340" def _build_id(): @@ -147,7 +147,7 @@ def _build_id(): except OSError: return "unknown" -SCRIPT_VERSION = 17 # v17: the runner moved out into lfs-phases +SCRIPT_VERSION = 19 # v19: a test line chained into the build block is dropped def _tty_out(): @@ -1490,7 +1490,13 @@ def _emit_command_elements(elements, out, nested): out.append(SKIP_MARK + _g) out.append(_commented(text)) else: - out.append(text) + text, _dropped = _strip_test_lines(text) + if _dropped: + out.append(SKIP_MARK + "the test suite, chained into " + "this block -- `--test` runs it") + out.append(_commented("\n".join(_dropped))) + if text.strip(): + out.append(text) elif name == "div": classes = list(el.get("class", [])) if "admon" in classes: @@ -1556,7 +1562,13 @@ _COND_GUARD_RE = re.compile(r"^(if\b|optionally\b|should you\b)", re.I) _TEST_BLOCK_RE = re.compile( r"^(to test\b|to check\b|the test suite\b|tests? (can|may) be run\b)" r"|\bcan be tested\b|\bto test the (results|installation|build)\b" - r"|\bto run the test", re.I) + r"|\bto run the test" + # LVM2 13.0: "If you wish to run the tests, install a minimal set of + # ... first" -- an INSTALL command that is part of the test suite, and + # the only thing marking it as such is the sentence above it. + r"|\bif you (wish|want) to run the test" + r"|\bbefore running the test" + r"|\bfor the test suite\b", re.I) # THE COMMANDS SAY WHAT THEY ARE, AND PROSE IS A SECOND-HAND ACCOUNT. # @@ -1571,7 +1583,17 @@ _TEST_BLOCK_RE = re.compile( # commands, so a rewording upstream cannot bring the bug back. _TEST_CONTENT_RE = re.compile( r"""(?mx) - ^\s*(make|ninja)\s+(-\w+\s+)*(check|test)\b # make check / ninja test + ^\s*(\w+=\S*\s+)*(make|ninja)\s+(-\w+\s+)*(check|test)\b + # make check / ninja test + | ^\s*(\w+=\S*\s+)*(make|ninja)\s+(-\w+\s+)*(check|test)[_-]\w+ + # LC_ALL=... make check_local + # (the old pattern needed a + # word boundary after + # "check", which + # "check_local" has not -- + # and no env assignment in + # front, which LVM2's page + # has) | ^\s*meson\s+test\b | ^\s*ctest\b | ^\s*py\.test\b | ^\s*pytest\b @@ -1582,6 +1604,84 @@ _TEST_CONTENT_RE = re.compile( """) +# A COMMAND THAT ONLY ROOT CAN RUN IS NOT AN INSTALL STEP. +# +# LVM2's page (13.0) puts three blocks between the build and the install: +# a partial install for the test suite, then +# mount -o remount,dev /tmp +# and then `make check_local`. The middle one is not about installing +# anything -- it prepares /tmp so the tests may create device nodes -- and +# it ran in the install phase, as the package user: +# mount: /tmp: must be superuser to use mount. +# !! phase 'install' FAILED (exit 32) +# A package user cannot mount, load a module or make a device node, and no +# BLFS page needs it to: those commands belong to the system, not to the +# package. (`make check_local` was missed too -- the test pattern required +# a word boundary after "check", and "check_local" has none.) +_ROOT_ONLY_RE = re.compile( + r"""(?mx) + ^\s*mount\b | ^\s*umount\b + | ^\s*modprobe\b | ^\s*insmod\b + | ^\s*mknod\b + | ^\s*swapon\b | ^\s*swapoff\b + """) + + +def _root_only_reason(pre): + """Why this block cannot run as the package user, or None.""" + m = _ROOT_ONLY_RE.search(pre.get_text()) + if not m: + return None + hit = " ".join(m.group(0).split()) + return ("it needs root and the system, not this package (`%s`) -- run it " + "yourself if you want it" % hit) + + +# A TEST COMMAND CHAINED INTO THE BUILD BLOCK. +# +# Every Perl module in BLFS uses "the standard build and installation +# instructions", which are ONE block: +# perl Makefile.PL && +# make && +# make test +# So the test suite is not a block to skip -- it is the last line of the +# block that builds the package, joined with &&. XML::Parser's tests fail +# on this machine (62 files, 60 of them cannot even load the module) and +# took the whole gparted chain down in the BUILD phase; Class::Inspector's +# passed, which is why nobody noticed the tests were running at all. +# +# Tests run here only when they are asked for (PKGUSR_TESTS=1, which is +# `--test`), and then the runner runs them ITSELF, in the test phase. So +# the line comes out of the build, with the book's own text left in a +# comment, and the trailing && of the line before it goes too. +_TEST_LINE_RE = re.compile( + r"""^\s*(\w+=\S*\s+)*(make|ninja|meson|ctest|py\.test|pytest)\b[^&|;]*? + \b(check|test)([_-]\w+)?\b[^&|;]*$""", re.X) + + +def _strip_test_lines(text): + """Drop `make test` / `make check` lines from a block that also builds. + + Returns (text, dropped-lines). A block that is ONLY tests comes back + empty and its caller keeps treating it as a test block.""" + lines = text.splitlines() + keep, dropped = [], [] + for ln in lines: + if _TEST_LINE_RE.match(ln.strip()) and not re.search(r"\binstall\b", ln): + dropped.append(ln.strip()) + continue + keep.append(ln) + if not dropped: + return text, [] + # the line before a dropped last line ends in `&&` (or `;`) that now + # chains to nothing -- and `make &&` on its own is a syntax error + while keep and not keep[-1].strip(): + keep.pop() + if keep: + keep[-1] = re.sub(r"\s*(&&|\|\||;)\s*$", "", keep[-1]) + return "\n".join(keep), dropped + + def _test_content_reason(pre): """Why this block is a test suite, judged by its commands, or None.""" txt = pre.get_text() @@ -1609,7 +1709,7 @@ def _conditional_guard(pre): "optional" wins over both: an optional tool is exactly what we do not install.""" # content first: it does not depend on the book's wording or nesting - _by_content = _test_content_reason(pre) + _by_content = _root_only_reason(pre) or _test_content_reason(pre) if _by_content: return _by_content prev = pre.find_previous(["p", "pre", "h3", "h4"]) @@ -1711,6 +1811,14 @@ def extract_commands_phased(sec): stream.append(SKIP_MARK + guard) stream.append(_commented(text)) continue + text, _dropped = _strip_test_lines(text) + if _dropped: + stream.append(SKIP_MARK + "the test suite, chained into this " + "block with && -- run it with `--test`, which " + "runs it in the test phase") + stream.append(_commented("\n".join(_dropped))) + if not text.strip(): + continue stream.append(text) else: # h3 / h4 title = " ".join(el.get_text().split()) diff --git a/kernel.conf b/kernel.conf index e779b54..053db7a 100644 --- a/kernel.conf +++ b/kernel.conf @@ -39,9 +39,25 @@ PKG=linux # kernel link is memory-hungry. JOBS=-j14 -# CONFIG_FROM: running (/proc/config.gz) | file:<path> | keep (the tree's own) +# CONFIG_FROM: where the .config starts from. +# running /proc/config.gz -- the kernel carrying the machine now. +# If it cannot be read (no CONFIG_IKCONFIG_PROC), the saved +# last-good config is used instead. +# last-good the config of the last kernel that BOOTED +# prev-good the one before it -- the way back from a bad answer +# saved the newest config this tool built with, booted or not +# file:<path> any file +# keep the tree's own .config, untouched +# Every build's config is copied to /etc/pkgusr/kernel-configs, which is +# outside the source tree and survives `packagemanager remove linux`: +# lfs-kernel config list what is saved +# lfs-kernel config diff prev-good -> last-good CONFIG_FROM=running +# CONFIG_KEEP: how many per-build configs to keep in that directory. +# last-good.config and prev-good.config are NOT counted and never pruned. +CONFIG_KEEP=5 + # CONFIG_REFRESH: a config from one kernel is never complete for another -- # every version adds symbols the old file has no answer for, and something # has to decide them: @@ -63,13 +79,25 @@ CONFIG_REFRESH=olddefconfig # Each is read back from the final .config; one the kernel refuses (a missing # dependency, a symbol this version does not have) is named and stops the build. # +# Edit it here, or let the tool do it -- same file, with a backup: +# lfs-kernel enable --list hardware by name +# lfs-kernel enable ath9k-htc its symbols AND its firmware +# lfs-kernel config search atheros which symbol that is +# lfs-kernel config set NET_VENDOR_ATHEROS=y ALX=m +# lfs-kernel config unset ALX +# A NET_VENDOR_* / WLAN_VENDOR_* symbol only opens a submenu: name the +# driver under it too, or nothing is built. +# # LEGACY_TIOCSTI=n no program may type into a terminal it does not own. # With it allowed, shadow's su (4.20+) detaches the shell # of every `su - <user>` run by root -- "cannot set # terminal process group ... no job control" -- because # that user could otherwise inject commands into root's # terminal. Off, su keeps the terminal and job control. -CONFIG_SET="LEGACY_TIOCSTI=n" +# EXFAT_FS=m the kernel side of exFAT: mkfs.exfat and fsck.exfat +# (exfatprogs, services.stack) work on a filesystem +# the kernel cannot mount without this. +CONFIG_SET="LEGACY_TIOCSTI=n EXFAT_FS=m" # ---- firmware ------------------------------------------------------------ # diff --git a/lfs b/lfs index 00d6793..9a3a0c9 100755 --- a/lfs +++ b/lfs @@ -60,7 +60,7 @@ except ImportError: " packagemanager pip install beautifulsoup4\n") sys.exit(2) -LFS_VERSION = "1.14.321" +LFS_VERSION = "1.14.340" def _build_id(): diff --git a/lfs-helper b/lfs-helper index 20582ad..929b8db 100755 --- a/lfs-helper +++ b/lfs-helper @@ -49,7 +49,7 @@ set -u # Bump on every change that alters behaviour. `lfs-helper --version` makes it # possible to tell at a glance whether the copy inside the chroot is the one # that was just fixed -- guessing at that has wasted a lot of time. -LFS_HELPER_VERSION="1.14.321" +LFS_HELPER_VERSION="1.14.340" # A fingerprint of this file's own contents. A hand-maintained version number # goes stale the moment someone forgets to bump it -- and the reason for @@ -1157,9 +1157,33 @@ PHASES="$STATE_PROGRESS/phases" # job's message can be thousands of lines above the end, so showing the tail # (what you see on screen) is usually useless. show_build_errors() { - local log="$1" hits="" + local log="$1" hits="" _rootline="" [ -f "$log" ] || return 0 say "" + # A COMMAND ONLY ROOT CAN RUN, IN A BUILD THAT IS NOT ROOT. + # + # mount: /tmp: must be superuser to use mount. + # !! phase 'install' FAILED (exit 32) + # ...and the report said "no line matched the usual error patterns", + # because nothing here looks like a compiler error. It is not a + # permission to GRANT either: no collector group makes mount work. The + # command does not belong in a package's install at all (it was LVM2's + # "remount /tmp so the TESTS can make device nodes"), so say exactly + # that and name the line. + _rootline="$(grep -m1 -E "must be superuser|only root can|Operation not permitted \(are you root|must be run as root|requires root privileges" \ + "$log" 2>/dev/null)" + if [ -n "$_rootline" ]; then + fail "--- this step tried to do something only root may do ---" + say " $_rootline" + say " A package builds and installs as its own user. Mounting a" + say " filesystem, loading a module or making a device node is the" + say " system's business, not a package's -- and no collector group" + say " can grant it. Such a command usually comes from a book block" + say " that belongs to the TEST suite (blfs 1.14.334 drops those)." + say " Regenerate the script, and run the command yourself if you" + say " actually want what it does:" + say " packagemanager update <pkg> --reinstall --regenerate --run" + fi hits="$(grep -nE "(^|[^-])\\berror\\b:|\\*\\*\\* \\[|Error [0-9]+|No such file or directory|command not found|undefined reference" \ "$log" 2>/dev/null | grep -vE "Error 1$|Error 2$" | head -12)" if [ -n "$hits" ]; then @@ -2455,8 +2479,12 @@ unwritable_dirs_from_log() { # of it and then running `su ... test -w` on each is minutes of CPU for an # answer that is always in the last few lines. local _tail + # The same three wordings as the gate above: ExtUtils::Install's + # "Can't create '<dir>'" / "Do not have write permissions on '<dir>'" + # and tar's "Cannot mkdir". Without them this list came back EMPTY for + # every Perl module, and the repair had nothing to grant. _tail="$(tail -c 524288 "$log" 2>/dev/null \ - | grep -aE "Permission denied|Operation not permitted|cannot (create|remove|touch|open)|couldn't copy|No such file or directory" \ + | grep -aE "Permission denied|Operation not permitted|cannot (create|remove|touch|open)|[Cc]an'?t create|[Dd]o not have write permissions|[Cc]annot mkdir|couldn't copy|No such file or directory" \ | grep -avE "^[[:space:]]*(File \"|Traceback|at .* line )" \ | head -n 100)" # A RELATIVE PATH NEEDS THE `cd` THAT PRECEDED IT. @@ -2575,7 +2603,7 @@ unwritable_dirs_from_log() { printf '%s\n' "$_cmakedirs" printf '%s\n' "$_cdpaths" printf '%s\n' "$_tail" | grep -aoE "'[^']+'|\"[^\"]+\"" | tr -d "'\"" - printf '%s\n' "$_tail" | grep -aoE "/[A-Za-z0-9._/+-]{6,}"; } \ + printf '%s\n' "$_tail" | grep -aoE "/[A-Za-z0-9._/+:-]{6,}" | sed 's/:*$//'; } \ | while IFS= read -r p; do case "$p" in /*) ;; *) continue ;; esac if ! under_dir "$p" "$SNAP_ROOT"; then @@ -2594,6 +2622,44 @@ unwritable_dirs_from_log() { done } +# A BUILD OPTION THE BUILD ITSELF NAMES. +# +# cairomm 1.14.6 stops at configure: +# Run-time dependency Boost found: NO (tried system) +# ERROR: Building the test programs requires Boost Test ... and Fontconfig. +# Install them or use -Dbuild-tests=false or -Dbuild-tests=if-dependencies-found +# Boost is in avoid.stack -- deliberately, it is an hour of build and this +# desktop wants none of it -- so the dependency is never coming, and the +# option meson prints is the whole answer. A gparted chain that stops for +# an hour on a TEST program nobody asked for is the failure; the fix was on +# the screen the whole time. +# +# Only an option the tool suggested, only one that turns something OFF, only +# if it is not already there, and the old script is kept beside it. +_meson_opt_from_log() { # <log> <script> <name> + local log="$1" script="$2" pkg="${3:-}" opt + [ -f "$log" ] && [ -f "$script" ] || return 1 + opt="$(grep -ohE "use -D[A-Za-z0-9_.-]+=[A-Za-z0-9_.-]+" "$log" 2>/dev/null \ + | head -n1 | sed 's/^use //')" + [ -n "$opt" ] || return 1 + case "$opt" in + *=false|*=disabled|*=no|*=if-dependencies-found) ;; + *) return 1 ;; # never turn something ON + esac + grep -qF -- "$opt" "$script" && return 1 + grep -q "meson setup" "$script" || return 1 + cp -p "$script" "$script.bak-$(date +%Y%m%d-%H%M%S)" || return 1 + sed -i "s|meson setup|meson setup $opt|" "$script" || return 1 + note "# the build named the option itself -- adding it and trying again:" + note "# $opt" + note "# in $script (the old one is beside it as .bak-<date>)" + note "# A REGENERATED SCRIPT LOSES IT. To keep it, put it in" + note "# /etc/pkgusr/stacks/machine.conf:" + note "# [${pkg:-<package>}]" + note "# ${opt#-D}" | sed 's/=/ = /' + return 0 +} + # Grant <user> access to everything in the log it could not write, and say # which groups that took. Returns 0 if anything was granted. @@ -2716,6 +2782,7 @@ takeover_files_from_log() { user_exists "$owner" || continue if real_chown -h "$user:$user" "$f" 2>/dev/null; then warn "!! $f belonged to $owner -- given to $user, which is installing it" + LAST_TAKEOVER_OWNER="$owner" n=$((n + 1)) fi done <<< "$( @@ -2723,7 +2790,39 @@ takeover_files_from_log() { grep -oE "cannot create regular file '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 grep -oE "cannot unlink '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 grep -oE "cannot overwrite '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 + # PERL SAYS IT WITHOUT QUOTES, AND THE PATH CONTAINS COLONS. + # + # Cannot forceunlink /usr/share/man/man3/XML::Parser.3: Permission denied + # + # ExtUtils::Install replaces a file by unlinking it first, and in a + # STICKY directory (/usr/share/man/man3 is drwxrwxr-t) only the + # file's owner may do that -- so granting the directory, which the + # repair did six times, can never help. The file belongs to an + # older account for the same module (p_xml-parser; this one is + # p_perl-xml-parser), and handing it over is the fix. + # The path is taken as a whole token: XML::Parser.3 has colons in + # it, so splitting on ":" loses it. + grep -oE "[Cc]annot forceunlink [^ ]+" "$log" 2>/dev/null \ + | sed 's/^[Cc]annot forceunlink //; s/:$//' + grep -oE "[Cc]an'?t (unlink|remove|rename) [^ ]+" "$log" 2>/dev/null \ + | sed -E "s/^[Cc]an'?t (unlink|remove|rename) //; s/:\$//" } | sort -u)" + # AN OLDER ACCOUNT FOR THE SAME PACKAGE. When several files move from + # one package user whose name is this one's with a language prefix + # dropped (xml-parser <- perl-xml-parser), the two are the same upstream + # module installed twice under two book anchors. Taking the files is + # right; leaving the old account to claim them in `verify` is not. + if [ "$n" -ge 2 ] && [ -n "${LAST_TAKEOVER_OWNER:-}" ]; then + local _a="${user#p_}" _b="${LAST_TAKEOVER_OWNER#p_}" + case "$_a" in + perl-"$_b"|python-"$_b"|"$_b"-perl|"$_b"|py-"$_b") + warn "!! $LAST_TAKEOVER_OWNER looks like an older account for the same" + warn " module as $user -- two book anchors, one upstream package." + warn " The files are $user's now. The old account still claims" + warn " them; when this build is done, take it out of the way:" + warn " packagemanager remove ${_b} --purge --run" ;; + esac + fi [ "$n" -gt 0 ] } @@ -4493,7 +4592,16 @@ diagnose_paths_from_log() { [ -f "$log" ] || return 0 local paths # tail only, and cap the results: see unwritable_dirs_from_log for why - paths="$(tail -c 524288 "$log" 2>/dev/null | grep -aoE "/[A-Za-z0-9._/+-]{6,}" \ + # A PERL MAN PAGE HAS COLONS IN ITS NAME. + # + # /usr/share/man/man3/XML::Parser.3 was reported as + # /usr/share/man/man3/XML missing; parent ... is drwxrwxr-t + # -- the class below had no ":" in it, so the name was cut at the first + # one, and the report named a file that has never existed while the one + # that could not be replaced went unmentioned. A trailing colon (the + # message's own "path: Permission denied") is stripped afterwards. + paths="$(tail -c 524288 "$log" 2>/dev/null | grep -aoE "/[A-Za-z0-9._/+:-]{6,}" \ + | sed 's/:*$//' \ | grep -vE "^/(proc|sys|dev|tmp)/" | sort -u \ | while IFS= read -r p; do case "$p" in @@ -5907,8 +6015,34 @@ run_phase_as() { # ninja prints `ninja: build stopped: subcommand failed.` with no # `***`, so it needs its own branch -- the one place this check was # supposed to work, and did not. - _swallowed="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^ninja: build stopped|^install: cannot create|: Permission denied$|^CMake Error|^-- Configuring incomplete|^configure: error:|^meson\.build:[0-9]+:[0-9]+: ERROR:|^ERROR: Unhandled python|^error: could not compile|^fatal: could not read Username|^fatal: repository .* not found|^fatal: Authentication failed|BackendUnavailable: Cannot import|^ModuleNotFoundError: No module named" \ - "$log" 2>/dev/null)" + # A SHARED REGISTRY IS NOT THE PACKAGE. + # + # perl-class-inspector installed both its modules and both its man + # pages, and was then failed for this line: + # .../core_perl/perllocal.pod: Permission denied + # which make itself had already marked "(ignored)". perllocal.pod + # is a list of what has been installed, written by every module and + # owned by the first one; not updating it loses a paragraph of + # bookkeeping, not a file of this package's. The same is true of + # every entry in shared_files_list. They are excluded from this + # test -- and reported, because the reason they cannot be written + # is worth fixing (share_shared_files does it on the next repair). + local _ign _shared_hit="" + _ign="$(mktemp)" + { shared_files_list; printf '%s\n' perllocal.pod; } > "$_ign" 2>/dev/null + _shared_hit="$(grep -Ff "$_ign" "$log" 2>/dev/null \ + | grep -m1 -E ": Permission denied$|Operation not permitted")" + _swallowed="$(grep -vFf "$_ign" "$log" 2>/dev/null \ + | grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^ninja: build stopped|^install: cannot create|: Permission denied$|^CMake Error|^-- Configuring incomplete|^configure: error:|^meson\.build:[0-9]+:[0-9]+: ERROR:|^ERROR: Unhandled python|^error: could not compile|^fatal: could not read Username|^fatal: repository .* not found|^fatal: Authentication failed|BackendUnavailable: Cannot import|^ModuleNotFoundError: No module named")" + rm -f "$_ign" + if [ -z "$_swallowed" ] && [ -n "$_shared_hit" ]; then + warn "a shared registry file could not be updated:" + warn " $_shared_hit" + note " that file is written by every package of its family and" + note " owned by the first one; this package's OWN files installed." + note " Fixed for the next one: lfs-helper fix-perms $name --run" + share_shared_files + fi if [ -n "$_swallowed" ]; then warn "!! this phase exited 0, but its log says otherwise:" warn " $_swallowed" @@ -6987,7 +7121,16 @@ cmd_build() { # Partition first" -- must never be answered by handing directories away. local _perm_ish=0 # "file INSTALL cannot": CMake wraps "Permission denied" over two lines. - grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|couldn't copy|read-only file system|file INSTALL cannot" \ + # PERL SAYS IT IN ITS OWN WORDS, AND THEY MATCHED NOTHING. + # + # ERROR: Can't create '/usr/lib/perl5/5.42/site_perl/File' + # Do not have write permissions on '/usr/lib/perl5/5.42/site_perl/File' + # make: *** [Makefile:777: pure_site_install] Error 13 + # + # ExtUtils::Install never writes "Permission denied" for this case, so + # the gate said "the failure does not look like a permission problem", + # ran zero rounds, and stopped a plan whose fix is one group. + grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|can(no|')?t (create|make|mkdir)|do not have write permissions|couldn't copy|read-only file system|file INSTALL cannot" \ "$log" 2>/dev/null && _perm_ish=1 # A BOOK COMMAND THAT WORKS ONCE. # @@ -7015,13 +7158,17 @@ cmd_build() { # permission problem and granting will never fix it -- but it IS # automatically fixable, so it must reach the retry loop. [ -n "$(missing_dirs_from_log "$log")" ] && _perm_ish=1 + # ...and neither is "use -Dbuild-tests=false", which is the build telling + # us how to finish without a dependency this machine does not have. + grep -qE "use -D[A-Za-z0-9_.-]+=" "$log" 2>/dev/null && _perm_ish=1 # SAY WHY THERE WAS NO RETRY. cmake failed twice on the same missing # directory with no attempt and no explanation, which left the only # evidence -- "did the loop run?" -- invisible. A decision not to act is # a decision, and it has to be reviewable. if [ "$rc" != 0 ] && [ "$auto_fix" = 1 ] && [ "$_perm_ish" != 1 ]; then - detail "# no automatic repair attempted: the failure does not look like" - detail "# a permission or missing-directory problem" + detail "# no automatic repair attempted: the failure is not one this" + detail "# tool knows how to fix (a permission, a missing directory," + detail "# or a build option the build itself names)" fi # Grants are per-directory now (a collector group is permission to install @@ -7053,6 +7200,7 @@ cmd_build() { && _fixed_something=1 fi create_missing_dirs_from_log "$log" "$owner" && _fixed_something=1 + _meson_opt_from_log "$log" "$s" "$name" && _fixed_something=1 # the package's own leftovers from a previous install of the same # version: removing them IS the fix, and it counts as one, so the # phase is retried @@ -7700,13 +7848,47 @@ cmd_build() { fail "!" if [ -n "$_dep_provider" ]; then fail "! '$_dep' is the pkg-config name of '$_dep_provider'," - fail "! which IS installed here. So this is a pkg-config" - fail "! problem, not a missing package:" - fail "! pkg-config --exists --print-errors '$_dep'" - fail "! ls /usr/lib/pkgconfig/$_dep.pc" - fail "! If the .pc is absent, $_dep_provider installed without it" - fail "! -- rebuild it:" - fail "! packagemanager install $_dep_provider --run --reinstall" + fail "! which is RECORDED as installed here." + # ASK THE MANIFEST, DO NOT GUESS. + # + # "If the .pc is absent, it installed without it" left the + # reader to find out which of two very different things had + # happened: a package that installed and then lost a file, + # or one that is recorded as installed and put nothing on + # disk at all (a build that failed after its account and + # home were made). The manifest says which, in one read. + local _php _plst _pn=0 _ppc="" + _php="$(pkgusr_home_for "$_dep_provider" 2>/dev/null)" + _plst="$_php/pkg.lst" + if [ -s "$_plst" ]; then + _pn="$(grep -c . "$_plst" 2>/dev/null || echo 0)" + _ppc="$(grep -E '\.pc$' "$_plst" 2>/dev/null | head -n3)" + fi + if [ "${_pn:-0}" -eq 0 ]; then + fail "! ...but its manifest is EMPTY: nothing of it is on" + fail "! disk. Its last build did not finish, and whatever" + fail "! marked it installed was wrong. Build it again:" + fail "! packagemanager install $_dep_provider --run --reinstall" + elif [ -z "$_ppc" ]; then + fail "! It records $_pn file(s), and NONE of them is a .pc" + fail "! -- so it installed without the pkg-config file this" + fail "! build needs. Rebuild it:" + fail "! packagemanager install $_dep_provider --run --reinstall" + else + fail "! It records $_pn file(s), including:" + printf '%s\n' "$_ppc" | while IFS= read -r _l; do + if [ -e "$_l" ]; then + fail "! $_l (on disk)" + else + fail "! $_l -- MISSING from disk" + fi + done + fail "! If the one this build wants is missing, rebuild it;" + fail "! if it is there, the build did not look where it is:" + fail "! pkg-config --exists --print-errors '$_dep'" + fail "! pkg-config --variable pc_path pkg-config" + fail "! packagemanager install $_dep_provider --run --reinstall" + fi fi _pcdir="$(timeout 10 find "${SNAP_ROOT%/}/usr/lib" "${SNAP_ROOT%/}/usr/share" \ -maxdepth 3 -name "$_dep.pc" 2>/dev/null | head -1)" @@ -11440,6 +11622,22 @@ EOF # the people who must update it. A shared directory is root:install and # group-writable; a shared file is the same rule, one level down. shared_files_list() { + # perllocal.pod IS THE info/dir OF THE PERL TREE. + # + # Every Perl module's `make install` runs doc_install, which APPENDS a + # paragraph to /usr/lib/perl5/<ver>/core_perl/perllocal.pod. So the + # file belongs to whichever module was installed first -- here + # p_xml-parser, mode 664, group p_xml-parser -- and every module after + # that fails on it: + # /usr/lib/perl5/5.42/core_perl/perllocal.pod: Permission denied + # Granting the DIRECTORY to the perl collector group does not help: the + # file's own group is one package's. It is rewritten by many packages + # and installed by none, which is exactly this list's rule. + # The version is in the path, so it is globbed rather than spelled. + local _p + for _p in "${SNAP_ROOT%/}"/usr/lib/perl5/*/*/perllocal.pod; do + [ -e "$_p" ] && printf '%s\n' "$_p" + done printf '%s\n' \ "${SNAP_ROOT%/}/usr/share/info/dir" \ "${SNAP_ROOT%/}/etc/xml/catalog" \ @@ -11474,9 +11672,19 @@ shared_files_list() { # The directories those files live in. They may NOT be sticky -- see below. shared_file_dirs() { - local f + local f d while IFS= read -r f; do - [ -n "$f" ] && dirname "$f" + [ -n "$f" ] || continue + d="$(dirname "$f")" + # THE FILE IS SHARED; THE DIRECTORY IS NOT. + # + # /usr/lib/perl5/<ver>/core_perl holds perl's OWN modules and is + # p_perl's install directory, recorded in its manifest. Only + # perllocal.pod inside it is written by everyone. Handing the whole + # directory to root:install would take a directory away from the + # package that installed it and make `verify` right to complain. + case "$d" in */usr/lib/perl5/*) continue ;; esac + printf '%s\n' "$d" done < <(shared_files_list) | sort -u } diff --git a/lfs-kernel b/lfs-kernel index bcd52d9..92d97bb 100755 --- a/lfs-kernel +++ b/lfs-kernel @@ -1,13 +1,36 @@ #!/bin/bash # lfs-kernel -- build and install a kernel. # -# lfs-kernel build and install the configured version +# lfs-kernel what a build would do (nothing is built) +# lfs-kernel --run build and install the configured version +# lfs-kernel load <module> [--boot] +# load it now, and at every boot with --boot # lfs-kernel list what kernel.org offers right now -# lfs-kernel 6.12.40 that version, once (config untouched) -# lfs-kernel latest the newest stable, once +# lfs-kernel 6.12.40 --run that version, once (config untouched) +# lfs-kernel latest --run the newest stable, once # lfs-kernel detect-esp print the EFI directory this would use # lfs-kernel verify [ver] check that everything is installed # lfs-kernel firmware install what FIRMWARE_ONLY names, no kernel build +# lfs-kernel status how far the last build got, what is installed +# lfs-kernel enable --list hardware this tool knows by name +# lfs-kernel enable ath9k-htc +# put what that hardware needs into kernel.conf +# lfs-kernel modules [text] what this kernel has, and what is loaded +# lfs-kernel config what the next build's .config comes from +# lfs-kernel config list the saved configs (the last two that BOOTED) +# lfs-kernel config diff [a] [b] +# what changed between two saved configs +# lfs-kernel config search <text> +# which symbols that text means, and their state +# lfs-kernel config set SYM=y|m|n [...] +# add them to CONFIG_SET in kernel.conf +# lfs-kernel config save [--good] +# keep the running config; --good: it BOOTS +# lfs-kernel config from last-good +# build from that config, not from the running +# kernel +# lfs-kernel config unset SYM [...] +# take them out again # lfs-kernel firmware --missing [--fix] [--all] # firmware this machine asks for and has not got, # from the boot log, this tree's modules and the @@ -46,6 +69,24 @@ _detect_esp() { findmnt -rn -t vfat,msdos -o TARGET 2>/dev/null | head -n1 } +# A BUILD IS NOT SOMETHING YOU START BY TYPING FOUR WORDS. +# +# `lfs-kernel` alone compiled a kernel, installed it over the one the +# machine boots, installed the modules, fetched firmware and copied the +# Documentation -- an hour, from a bare command. Every other tool here +# prints the plan and waits for --run; this one now does too. --run is +# taken from anywhere on the line, so `lfs-kernel 7.2.6 --run` and +# `lfs-kernel --run` both read naturally. +_RUN=0 +_rest=() +for _a in "$@"; do + case "$_a" in + --run) _RUN=1 ;; + *) _rest+=("$_a") ;; + esac +done +set -- ${_rest[@]+"${_rest[@]}"} + case "${1:-}" in list) echo "# kernel.org offers:" @@ -55,7 +96,24 @@ case "${1:-}" in detect-esp) printf '%s\n' "$(_detect_esp)"; exit 0 ;; verify) - shift; _VERIFY=1; VERSION="${1:-}" ;; + shift; _VERIFY=1; _CLI_VERSION="${1:-}" ;; + enable) + # hardware by name: what symbols it needs, written into kernel.conf + shift; _ENABLE=1; _ENARGS=("$@") ;; + modules|mods) + # what this kernel actually has, and what is loaded + shift; _MODCMD=1; _MODARGS=("$@") ;; + load|unload) + # load a module now, and (--boot) at every boot + _LOADCMD="$1"; shift; _LOADARGS=("$@") ;; + status|state) + # how far the last build got, and what is installed -- reads only + shift; _STATUS=1 ;; + config) + # the config, without building anything: what it will be made of, + # what the last two booted kernels used, and the one place this + # machine's own answers are written down (CONFIG_SET) + shift; _CFGCMD=1; _CFGARGS=("$@") ;; firmware) # firmware alone, with no kernel build: the blobs are a separate # package and a missing one is a run-time problem, so fixing it must @@ -76,8 +134,37 @@ case "${1:-}" in # new command silently pushes out of range. sed -n '2,/^# Settings:/p' "$0" | sed '$d' | sed 's/^# \?//'; exit 0 ;; "") ;; + search|find) + # `lfs-kernel search AR9271` is what a person types, and it used to + # mean "build the kernel called search" -- see the version check + # below for what that really did. It is the config search. + shift; _CFGCMD=1; _CFGARGS=("search" "$@") ;; + set) + shift; _CFGCMD=1; _CFGARGS=("set" "$@") ;; -*) echo "!! unknown option: $1 (try: lfs-kernel --help)"; exit 1 ;; - *) VERSION="$1" ;; + *) + # A WORD THAT IS NOT A VERSION IS NOT A VERSION. + # + # Anything unrecognised became the version to build, and the version + # from kernel.conf then overwrote it (below) -- so a typo, or a + # subcommand this tool does not have, silently started a FULL BUILD + # of the configured kernel. `lfs-kernel search AR9271` rebuilt and + # reinstalled 7.2.6, fetched the firmware and said nothing about the + # word `search` at all. + case "$1" in + latest|[0-9]*.[0-9]*) _CLI_VERSION="$1" ;; + *) echo "!! lfs-kernel: '$1' is neither a command nor a version." + echo " commands: status, list, enable, modules, load," + echo " config, search, firmware, verify, detect-esp" + echo " a version looks like 7.2.6, or the word latest." + case "$1" in + conf*|cfg) echo " did you mean: lfs-kernel config" ;; + mod*|lsmod) echo " did you mean: lfs-kernel modules" ;; + en*|add) echo " did you mean: lfs-kernel enable --list" ;; + fw|firm*) echo " did you mean: lfs-kernel firmware --missing" ;; + esac + exit 1 ;; + esac ;; esac # The config is INSTALLED, not written on first run: it is a file to edit @@ -91,6 +178,13 @@ if [ ! -f "$CONF" ]; then fi . "$CONF" +# THE COMMAND LINE WINS. `lfs-kernel 6.12.40` sets VERSION, and then this +# file -- sourced after it -- set VERSION=latest straight over the top: the +# documented "that version, once" built whatever kernel.conf said instead, +# and `verify 7.2.4` reported on another kernel entirely. Re-apply it here, +# where nothing can overwrite it again. +[ -n "${_CLI_VERSION:-}" ] && VERSION="$_CLI_VERSION" + # A CONFIG FROM AN OLDER RELEASE. # # kernel.conf is never overwritten once edited, so an installed copy can @@ -136,8 +230,22 @@ _latest_stable() { _releases | awk '$1=="stable"{print $2; exit}'; } # "latest" would ask kernel.org for a version it is never going to look at, # and fail on a machine with no network while reporting on one that is # perfectly installed. -if [ "${_VERIFY:-0}" = 1 ]; then +if [ "${_VERIFY:-0}" = 1 ] || [ "${_CFGCMD:-0}" = 1 ] \ + || [ "${_ENABLE:-0}" = 1 ] || [ "${_MODCMD:-0}" = 1 ] \ + || [ "${_STATUS:-0}" = 1 ] || [ -n "${_LOADCMD:-}" ]; then : +elif [ "${_FWONLY:-0}" = 1 ] && [ "$VERSION" = latest ]; then + # A FIRMWARE REPORT IS ABOUT THIS MACHINE, NOT ABOUT KERNEL.ORG. + # + # `lfs-kernel firmware --missing` asked the website which kernel is + # newest -- for a question answered entirely by /lib/modules and the + # boot log -- printed "newest stable on kernel.org", and on a machine + # with no network it EXITED instead of reporting. + VERSION="$(uname -r | sed "s/-.*//")" + if [ ! -d "/lib/modules/$VERSION" ] && [ ! -d "/lib/modules/$(uname -r)" ]; then + VERSION="$(_latest_stable)" + [ -n "$VERSION" ] || { echo "!! no /lib/modules for the running kernel, and kernel.org is unreachable"; exit 1; } + fi elif [ "$VERSION" = latest ]; then VERSION="$(_latest_stable)" [ -n "$VERSION" ] || { echo "!! could not reach $MIRROR_JSON -- set VERSION in $CONF"; exit 1; } @@ -152,12 +260,21 @@ fi acct="p_$PKG" # `verify` must not CREATE anything -- a report that makes an account is not # a report. It only reads $acct's home below. -if [ "${_VERIFY:-0}" != 1 ] && ! id "$acct" >/dev/null 2>&1; then +if [ "${_VERIFY:-0}" != 1 ] && [ "${_CFGCMD:-0}" != 1 ] \ + && [ "${_ENABLE:-0}" != 1 ] && [ "${_MODCMD:-0}" != 1 ] \ + && [ "${_STATUS:-0}" != 1 ] && [ -z "${_LOADCMD:-}" ] \ + && ! id "$acct" >/dev/null 2>&1; then packagemanager add-user "$PKG" || { echo "!! could not create $acct"; exit 1; } fi -[ "${_VERIFY:-0}" = 1 ] || lfs-helper fix-home "$acct" --run >/dev/null 2>&1 || true +[ "${_VERIFY:-0}" = 1 ] || [ "${_CFGCMD:-0}" = 1 ] \ + || [ "${_ENABLE:-0}" = 1 ] || [ "${_MODCMD:-0}" = 1 ] \ + || [ "${_STATUS:-0}" = 1 ] || [ -n "${_LOADCMD:-}" ] \ + || lfs-helper fix-home "$acct" --run >/dev/null 2>&1 || true SRC="$(getent passwd "$acct" 2>/dev/null | cut -d: -f6)/src" -[ "${_VERIFY:-0}" = 1 ] || install -d -o "$acct" -g "$acct" -m 755 "$SRC" +[ "${_VERIFY:-0}" = 1 ] || [ "${_CFGCMD:-0}" = 1 ] \ + || [ "${_ENABLE:-0}" = 1 ] || [ "${_MODCMD:-0}" = 1 ] \ + || [ "${_STATUS:-0}" = 1 ] || [ -n "${_LOADCMD:-}" ] \ + || install -d -o "$acct" -g "$acct" -m 755 "$SRC" # Everything that touches the tree runs AS THE PACKAGE USER: a root-owned # file left in its home would break the next build and every rule about who @@ -168,6 +285,739 @@ tree="$SRC/linux-$VERSION" tarball="$SRC/linux-$VERSION.tar.xz" major="${VERSION%%.*}" +# -------------------------------------------------------------------------- +# THE CONFIGS ARE KEPT, OUTSIDE THE TREE. +# +# A .config lives in a source tree, and a source tree is deleted by hand or +# by `packagemanager remove linux`, which takes the account's home, +# and a failed build can leave one unusable. What is lost with it is the +# only record of every answer this machine has ever given -- hours of +# menuconfig, and the difference between a kernel that boots and one that +# does not. +# +# So every config this tool builds with is copied to $KCFG_DIR, which is +# root's, small (a config is ~200 kB) and has nothing to do with the tree: +# +# config-<version>-<date> every build, newest CONFIG_KEEP kept +# last-good.config the config of the kernel that BOOTED +# prev-good.config the one before that +# +# The last two are the ones worth having: they are known to work on this +# hardware. CONFIG_FROM=last-good starts from one, `config diff` says what +# changed between them, and `config set` is how an answer is added without +# hand-editing a file that the next build overwrites. +# -------------------------------------------------------------------------- +KCFG_DIR="${KCFG_DIR:-/etc/pkgusr/kernel-configs}" +: "${CONFIG_KEEP:=5}" + +_kcfg_readable() { # a config file with something in it + [ -s "$1" ] && grep -q '^CONFIG_' "$1" 2>/dev/null +} + +_kcfg_newest() { # the newest archived build config, or empty + ls -1t "$KCFG_DIR"/config-* 2>/dev/null | head -n1 +} + +_kcfg_ver() { # the kernel version a config file was built for + sed -n 's/^# Linux\/[^ ]* \([0-9][^ ]*\) Kernel Configuration.*/\1/p' "$1" \ + 2>/dev/null | head -n1 +} + +_kcfg_archive() { # <config> <version> -- keep a copy of what we built with + local src="$1" ver="$2" dst + _kcfg_readable "$src" || return 0 + install -d -m 755 "$KCFG_DIR" || return 0 + dst="$KCFG_DIR/config-$ver-$(date +%Y%m%d-%H%M%S)" + cp "$src" "$dst" 2>/dev/null || return 0 + chmod 644 "$dst" + echo "# config kept as $dst" + # PRUNE THE BUILD SNAPSHOTS ONLY. last-good/prev-good are not in this + # pattern and are never deleted here. + ls -1t "$KCFG_DIR"/config-* 2>/dev/null | tail -n +"$((CONFIG_KEEP + 1))" \ + | while read -r _old; do rm -f "$_old"; done +} + +_kcfg_mark_good() { # <config> <version> -- this kernel has booted + local src="$1" ver="$2" + _kcfg_readable "$src" || return 0 + install -d -m 755 "$KCFG_DIR" || return 0 + # the same version again is the same kernel: refresh it, do not push the + # OTHER working config out with a copy of this one + if [ -f "$KCFG_DIR/last-good.config" ] \ + && [ "$(_kcfg_ver "$KCFG_DIR/last-good.config")" != "$ver" ]; then + cp "$KCFG_DIR/last-good.config" "$KCFG_DIR/prev-good.config" 2>/dev/null + chmod 644 "$KCFG_DIR/prev-good.config" 2>/dev/null + fi + cp "$src" "$KCFG_DIR/last-good.config" 2>/dev/null || return 0 + chmod 644 "$KCFG_DIR/last-good.config" + echo "# the config of $ver is now $KCFG_DIR/last-good.config" +} + +# THE CONFIG THIS MACHINE IS RUNNING ON, wherever it can be read from. +_kcfg_current() { + local c + for c in "$tree/.config" "$KCFG_DIR/last-good.config" "$(_kcfg_newest)"; do + [ -n "$c" ] && _kcfg_readable "$c" && { printf '%s' "$c"; return 0; } + done + if [ -r /proc/config.gz ]; then + c="$(mktemp)"; gunzip < /proc/config.gz > "$c" 2>/dev/null \ + && { printf '%s' "$c"; return 0; } + rm -f "$c" + fi + return 1 +} + +_kcfg_state() { # <config> <SYMBOL> -- what that symbol is, in words + local f="$1" sym="CONFIG_${2#CONFIG_}" l + l="$(grep -E "^($sym=|# $sym is not set)" "$f" 2>/dev/null | head -n1)" + [ -n "$l" ] || { printf 'absent'; return; } + case "$l" in "# $sym is not set") printf 'not set' ;; *) printf '%s' "${l#*=}" ;; esac +} + +# ONE WRITER FOR kernel.conf. `config set`, `config from` and `enable` all +# change one KEY="value" line in a file the user edits by hand, so they all +# back it up the same way and none of them rewrites anything else. +_conf_set_var() { # <KEY> <value> + local key="$1" val="$2" bak tmp + bak="$CONF.bak-$(date +%Y%m%d-%H%M%S)" + cp -p "$CONF" "$bak" || { echo "!! could not back up $CONF"; return 1; } + if grep -qE "^[[:space:]]*$key=" "$CONF"; then + tmp="$(mktemp)" + awk -v v="$key=\"$val\"" -v k="^[[:space:]]*$key=" ' + $0 ~ k && !done { print v; done = 1; next } + { print }' "$CONF" > "$tmp" && cat "$tmp" > "$CONF" + rm -f "$tmp" + else + printf '\n%s="%s"\n' "$key" "$val" >> "$CONF" + fi + echo "# $CONF (backup: $(basename "$bak"))" + echo "# $key=\"$val\"" +} + +# WHAT A SYMBOL IS, ACCORDING TO THE TREE: bool, tristate, or unknown -- +# which decides whether `enable` may say =m. +_ksym_type() { # <SYMBOL> + local sym="${1#CONFIG_}" + [ -d "$tree" ] || return 0 + grep -rhA3 --include='Kconfig*' -E "^[[:space:]]*(menu)?config[[:space:]]+$sym\$" \ + "$tree" 2>/dev/null \ + | grep -m1 -oE '^[[:space:]]*(bool|tristate|string|int|hex)' \ + | tr -d '[:space:]' +} + +# THE SYMBOLS A KCONFIG FILE DEFINES, with their prompt -- the only way a +# search can say "ATH9K_HTC: Atheros HTC based wireless cards" instead of +# printing every symbol in a file that happens to mention the word. +_ksym_scan() { # <pattern> -- SYMBOL<TAB>type<TAB>prompt, for matches only + local pat="$1" + [ -d "$tree" ] || return 0 + grep -rlis --include='Kconfig*' -- "$pat" "$tree" 2>/dev/null | head -n 40 \ + | xargs -r awk -v pat="$pat" ' + function low(x) { return tolower(x) } + /^[[:space:]]*(menu)?config[[:space:]]+[A-Za-z0-9_]+/ { + sym = $2; next } + sym != "" && /^[[:space:]]*(bool|tristate|string|int|hex)([[:space:]]|$)/ { + type = $1 + p = $0 + # the prompt is the FIRST quoted string on the line, if any: + # `tristate "Atheros HTC based wireless cards support"` + if (match(p, /"[^"]*"/)) + p = substr(p, RSTART + 1, RLENGTH - 2) + else + p = "" + if (index(low(sym), low(pat)) || index(low(p), low(pat))) + print sym "\t" type "\t" p + sym = ""; next } + { if ($0 ~ /^[[:space:]]*$/) sym = "" }' 2>/dev/null \ + | sort -u +} + +# THE MODULES THIS KERNEL HAS: which version's /lib/modules to read. +_kmod_ver() { + local r; r="$(uname -r)" + [ -d "/lib/modules/$r" ] && { printf '%s' "$r"; return 0; } + ls -1 /lib/modules 2>/dev/null | sort -V | tail -n1 +} + +# THE NEWEST SOURCE TREE, for a `config` command that names no version. +_kcfg_tree() { + [ -d "${SRC:-}" ] || return 0 + ls -1d "$SRC"/linux-*/ 2>/dev/null | grep -v '\-headers' | sort -V | tail -n1 \ + | sed 's:/$::' +} + +if [ "${_CFGCMD:-0}" = 1 ]; then + [ -d "$tree" ] || tree="$(_kcfg_tree)" + _sub="${_CFGARGS[0]-show}" + case "$_sub" in + show|"") + echo "# the next build's .config" + echo "# CONFIG_FROM = $CONFIG_FROM" + echo "# CONFIG_REFRESH = ${CONFIG_REFRESH:-olddefconfig}" + echo "# CONFIG_SET = ${CONFIG_SET:-(nothing)}" + echo "# kept in $KCFG_DIR (CONFIG_KEEP=$CONFIG_KEEP builds)" + echo + for _f in last-good prev-good; do + if [ -f "$KCFG_DIR/$_f.config" ]; then + printf '# %-16s %s (%s)\n' "$_f.config" \ + "$(_kcfg_ver "$KCFG_DIR/$_f.config")" \ + "$(date -r "$KCFG_DIR/$_f.config" '+%Y-%m-%d %H:%M')" + else + printf '# %-16s -- none yet\n' "$_f.config" + fi + done + echo + echo "# add an option: lfs-kernel config set NET_VENDOR_ATHEROS=y" + echo "# find its name: lfs-kernel config search atheros" + echo "# then build: lfs-kernel" + exit 0 ;; + list|ls) + if [ ! -d "$KCFG_DIR" ]; then + echo "# no saved configs yet ($KCFG_DIR) -- the next build writes one" + exit 0 + fi + echo "# $KCFG_DIR" + for _f in "$KCFG_DIR"/last-good.config "$KCFG_DIR"/prev-good.config; do + [ -f "$_f" ] && printf ' %-34s %-10s %s\n' "$(basename "$_f")" \ + "$(_kcfg_ver "$_f")" "$(date -r "$_f" '+%Y-%m-%d %H:%M')" + done + ls -1t "$KCFG_DIR"/config-* 2>/dev/null | while read -r _f; do + printf ' %-34s %-10s %s\n' "$(basename "$_f")" \ + "$(_kcfg_ver "$_f")" "$(date -r "$_f" '+%Y-%m-%d %H:%M')" + done + echo "# start from one: CONFIG_FROM=file:<path> in $CONF" + echo "# or CONFIG_FROM=last-good / prev-good / saved" + exit 0 ;; + diff) + _a="${_CFGARGS[1]:-$KCFG_DIR/prev-good.config}" + _b="${_CFGARGS[2]:-$KCFG_DIR/last-good.config}" + [ -f "$_a" ] || _a="$KCFG_DIR/$_a" + [ -f "$_b" ] || _b="$KCFG_DIR/$_b" + for _f in "$_a" "$_b"; do + [ -f "$_f" ] || { echo "!! no such config: $_f"; echo " lfs-kernel config list"; exit 1; } + done + echo "# $_a ($(_kcfg_ver "$_a")) -> $_b ($(_kcfg_ver "$_b"))" + echo "# < only in the first, > only in the second" + diff <(grep -E '^(CONFIG_|# CONFIG_)' "$_a" | sort) \ + <(grep -E '^(CONFIG_|# CONFIG_)' "$_b" | sort) \ + | grep -E '^[<>]' | sort -k2 || echo "# identical" + exit 0 ;; + search|find) + _pat="${_CFGARGS[1]:-}" + [ -n "$_pat" ] || { echo "!! lfs-kernel config search <text>"; exit 1; } + _cfg="$(_kcfg_current || true)" + if [ -n "$_cfg" ]; then + echo "# '$_pat' in $_cfg" + grep -iE "^(CONFIG_[A-Z0-9_]*${_pat}[A-Z0-9_]*=|# CONFIG_[A-Z0-9_]*${_pat}[A-Z0-9_]* is not set)" \ + "$_cfg" | head -n 25 | sed 's/^/ /' + fi + echo + if [ -d "$tree" ]; then + # BY NAME OR BY PROMPT, NOT BY FILE. Listing every symbol of + # every Kconfig that mentions the word gave 40 lines of MIPS + # console options for "atheros". A symbol is a hit when its own + # name or its prompt says so, and the prompt is what tells you + # which one you want. + echo "# symbols in $tree (state, type, what it is)" + _hits="$(_ksym_scan "$_pat")" + if [ -n "$_hits" ]; then + printf '%s\n' "$_hits" | head -n 30 | while IFS="$(printf '\t')" read -r _sym _ty _pr; do + printf ' %-26s %-9s %-9s %s\n' "$_sym" \ + "$( [ -n "$_cfg" ] && _kcfg_state "$_cfg" "$_sym" )" \ + "$_ty" "$_pr" + done + _n="$(printf '%s\n' "$_hits" | wc -l)" + [ "$_n" -gt 30 ] && echo " ... $((_n - 30)) more" + else + echo " (nothing whose name or prompt says '$_pat')" + fi + else + echo "# no kernel tree here to search; unpack one with: lfs-kernel <version>" + fi + echo + echo "# a NET_VENDOR_* / WLAN_VENDOR_* symbol only opens the submenu:" + echo "# the DRIVER under it has to be set too." + echo "# known hardware, by name: lfs-kernel enable --list" + echo "# set a symbol: lfs-kernel config set ATH9K_HTC=m" + [ -d "$tree" ] \ + && echo "# read the help text: cd $tree && make menuconfig (/ searches)" + exit 0 ;; + save) + # SEED THE ARCHIVE WITHOUT A BUILD. Before the first build there is + # nothing saved, so CONFIG_FROM=last-good has nothing to read -- and + # the config worth keeping is already on this machine. + _src="${_CFGARGS[1]:-}" + _good=0 + for _a in ${_CFGARGS[@]+"${_CFGARGS[@]}"}; do + [ "$_a" = "--good" ] && _good=1 + done + [ "$_src" = "--good" ] && _src="" + if [ -z "$_src" ]; then + if [ -r /proc/config.gz ]; then + _src="$(mktemp)"; gunzip < /proc/config.gz > "$_src" + _what="the running kernel" + elif _kcfg_readable "$tree/.config"; then + _src="$tree/.config"; _what="$tree" + else + echo "!! nothing to save: no /proc/config.gz and no .config in a tree" + exit 1 + fi + else + _what="$_src" + fi + _kcfg_readable "$_src" || { echo "!! $_src is not a kernel config"; exit 1; } + _ver="$(_kcfg_ver "$_src")" + [ -n "$_ver" ] || _ver="$(uname -r | sed 's/-.*//')" + echo "# saving $_what (kernel $_ver)" + _kcfg_archive "$_src" "$_ver" + if [ "$_good" = 1 ]; then + _kcfg_mark_good "$_src" "$_ver" + else + echo "# it is a build snapshot, not a known-good config." + echo "# --good marks it as one (use it for a kernel that BOOTS)." + fi + echo "# start from it next time: lfs-kernel config from last-good" + exit 0 ;; + from) + _val="${_CFGARGS[1]:-}" + case "$_val" in + running|last-good|good|prev-good|prev|saved|last|keep|file:*) ;; + "") echo "# CONFIG_FROM = $CONFIG_FROM" + echo "# running | last-good | prev-good | saved | keep | file:<path>" + exit 0 ;; + *) echo "!! CONFIG_FROM=$_val is not one of:" + echo " running last-good prev-good saved keep file:<path>"; exit 1 ;; + esac + case "$_val" in + last-good|good) _kcfg_readable "$KCFG_DIR/last-good.config" \ + || { echo "!! there is no $KCFG_DIR/last-good.config yet." + echo " Make one from the kernel running now:" + echo " lfs-kernel config save --good"; exit 1; } ;; + esac + _conf_set_var CONFIG_FROM "$_val" || exit 1 + exit 0 ;; + set|unset) + _items=("${_CFGARGS[@]:1}") + [ "${#_items[@]}" -gt 0 ] || { + echo "!! lfs-kernel config $_sub SYMBOL[=VALUE] ..."; exit 1; } + # the current list, as words + _new="" + for _w in ${CONFIG_SET:-}; do + _keep=1 + for _i in "${_items[@]}"; do + _isym="CONFIG_${_i%%=*}"; _isym="CONFIG_${_isym#CONFIG_}" + _wsym="CONFIG_${_w%%=*}"; _wsym="CONFIG_${_wsym#CONFIG_}" + [ "$_isym" = "$_wsym" ] && _keep=0 + done + [ "$_keep" = 1 ] && _new="$_new $_w" + done + if [ "$_sub" = set ]; then + for _i in "${_items[@]}"; do + case "$_i" in + *=*) ;; + *) echo "!! '$_i' has no value -- SYMBOL=y|m|n|<number>|\"text\""; exit 1 ;; + esac + _val="${_i#*=}" + case "$_val" in + y|m|n|[0-9]*|\"*) ;; + *) echo "# note: '$_val' is taken as a string value" ;; + esac + _new="$_new ${_i#CONFIG_}" + done + fi + _new="$(printf '%s' "$_new" | sed 's/^ *//')" + _conf_set_var CONFIG_SET "$_new" || exit 1 + echo + echo "# nothing is built yet. Apply it:" + echo "# lfs-kernel # rebuild and install" + echo "# Each entry is read back from the final .config: one the" + echo "# kernel refuses (missing dependency, no such symbol) is" + echo "# named and stops the build." + exit 0 ;; + *) echo "!! lfs-kernel config: show | list | diff | search | set | unset" + echo " | save [--good] | from <source>" + exit 1 ;; + esac +fi + +# -------------------------------------------------------------------------- +# HARDWARE BY NAME. +# +# "I have an AR9271 USB adapter" is the question; CONFIG_ATH9K_HTC plus +# WLAN_VENDOR_ATH plus the htc_9271 firmware is the answer, and nothing on +# the machine connects the two. menuconfig will not either: it wants the +# symbol you do not know yet. +# +# Each line is: <name>|<symbols>|<firmware>|<what it is> +# `enable <name>` puts the symbols into CONFIG_SET and the firmware into +# FIRMWARE_ONLY, both in kernel.conf, both backed up. A symbol this kernel +# refuses is reported by the build, which reads every CONFIG_SET entry back. +_KDEV_TABLE=' +ath9k-htc|WLAN_VENDOR_ATH=y ATH9K_HTC=m|ath9k_htc htc_9271*|Atheros AR9271/AR7010 USB wifi (TP-Link TL-WN722N v1, Alfa AWUS036NHA) +ath9k|WLAN_VENDOR_ATH=y ATH9K=m ATH9K_PCI=y|ath9k_htc|Atheros AR9xxx PCI/PCIe wifi +ath10k|WLAN_VENDOR_ATH=y ATH10K=m ATH10K_PCI=m|ath10k|Qualcomm Atheros QCA988x PCI wifi +ath11k|WLAN_VENDOR_ATH=y ATH11K=m ATH11K_PCI=m|ath11k|Qualcomm QCA6390/WCN685x wifi +alx|NET_VENDOR_ATHEROS=y ALX=m|-|Atheros AR816x/AR817x, Killer E2200 ethernet +atl1c|NET_VENDOR_ATHEROS=y ATL1C=m|-|Atheros AR813x/AR815x ethernet +iwlwifi|WLAN_VENDOR_INTEL=y IWLWIFI=m IWLMVM=m|intel iwlwifi-*.ucode|Intel wifi (AX200 and friends) +rtl8xxxu|WLAN_VENDOR_REALTEK=y RTL8XXXU=m|rtlwifi rtw88|Realtek RTL8188/8192 USB wifi +bluetooth-usb|BT=m BT_HCIBTUSB=m BT_RFCOMM=m BT_BNEP=m|rtl_bt intel|USB Bluetooth (most laptop and dongle radios) +btrfs|BTRFS_FS=m BTRFS_FS_POSIX_ACL=y|-|the btrfs filesystem (stack: btrfs-progs) +exfat|EXFAT_FS=m|-|exFAT, the usual format of big SD cards +ntfs3|NTFS3_FS=m NTFS3_LZX_XPRESS=y|-|NTFS read/write (the in-kernel driver) +nvme|BLK_DEV_NVME=y|-|NVMe SSDs +usb-serial|USB_SERIAL=m USB_SERIAL_CP210X=m USB_SERIAL_FTDI_SIO=m USB_SERIAL_CH341=m|-|USB serial adapters (CP210x, FTDI, CH340) +v4l2-usb|MEDIA_SUPPORT=m MEDIA_USB_SUPPORT=y USB_VIDEO_CLASS=m|-|USB webcams (UVC) +' + +_kdev_line() { # <name> -- the table line, or empty + printf '%s\n' "$_KDEV_TABLE" | awk -F'|' -v n="$1" 'NF && $1 == n {print; exit}' +} + +_kdev_list() { + echo "# hardware this tool knows by name:" + printf '%s\n' "$_KDEV_TABLE" | awk -F'|' 'NF { printf " %-15s %s\n", $1, $4 }' + echo + echo "# enable one: lfs-kernel enable ath9k-htc" + echo "# or a symbol: lfs-kernel enable ATH9K_HTC" + echo "# what is set: lfs-kernel config show modules: lfs-kernel modules" +} + +if [ "${_ENABLE:-0}" = 1 ]; then + [ -d "$tree" ] || tree="$(_kcfg_tree)" + _args=("${_ENARGS[@]}") + if [ "${#_args[@]}" = 0 ] || [ "${_args[0]}" = "--list" ] || [ "${_args[0]}" = "list" ]; then + _kdev_list; exit 0 + fi + _syms=""; _fw=""; _unknown="" + for _a in "${_args[@]}"; do + _row="$(_kdev_line "$_a")" + if [ -n "$_row" ]; then + _syms="$_syms $(printf '%s' "$_row" | cut -d'|' -f2)" + _f="$(printf '%s' "$_row" | cut -d'|' -f3)" + [ "$_f" = "-" ] || _fw="$_fw $_f" + echo "# $_a: $(printf '%s' "$_row" | cut -d'|' -f4)" + continue + fi + # not a name in the table: take it as a symbol, and ask the tree + # whether it can be a module before saying =m + case "$_a" in + *=*) _syms="$_syms ${_a#CONFIG_}"; continue ;; + esac + _sym="$(printf '%s' "${_a#CONFIG_}" | tr '[:lower:]' '[:upper:]')" + case "$(_ksym_type "$_sym")" in + tristate) _syms="$_syms $_sym=m" + echo "# $_sym: a tristate symbol -- set to m (a module)" ;; + bool) _syms="$_syms $_sym=y" + echo "# $_sym: a bool symbol -- set to y (built in)" ;; + "") _unknown="$_unknown $_a" ;; + *) _syms="$_syms $_sym=y" ;; + esac + done + if [ -n "$_unknown" ]; then + echo "!! not a name I know, and not a symbol in this tree:$_unknown" + echo " lfs-kernel enable --list # the names" + echo " lfs-kernel config search <text> # the symbols" + echo " lfs-kernel enable SYM=m # if you are sure" + exit 1 + fi + # merge into CONFIG_SET, last value wins, order kept + _new="" + for _w in ${CONFIG_SET:-}; do + _keep=1 + for _i in $_syms; do + [ "CONFIG_${_i%%=*}" = "CONFIG_${_w%%=*}" ] && _keep=0 + done + [ "$_keep" = 1 ] && _new="$_new $_w" + done + for _i in $_syms; do _new="$_new $_i"; done + _new="$(printf '%s' "$_new" | sed 's/^ *//')" + _conf_set_var CONFIG_SET "$_new" || exit 1 + # the firmware, if this hardware needs some and it is not already named + if [ -n "$_fw" ] && [ "${FIRMWARE_FROM:-none}" != none ]; then + _fwnew="${FIRMWARE_ONLY:-}" + if [ -n "$_fwnew" ]; then # empty means "everything": leave it + for _f in $_fw; do + case " $_fwnew " in *" $_f "*) ;; *) _fwnew="$_fwnew $_f" ;; esac + done + [ "$_fwnew" = "${FIRMWARE_ONLY:-}" ] \ + || _conf_set_var FIRMWARE_ONLY "$_fwnew" + fi + fi + echo + echo "# nothing is built yet. Apply it:" + echo "# lfs-kernel # rebuild, install, and the firmware" + echo "# then, after the reboot:" + echo "# lfs-kernel modules ${_args[0]}" + echo "# lfs-kernel firmware --missing" + exit 0 +fi + +if [ "${_MODCMD:-0}" = 1 ]; then + _mver="$(_kmod_ver)" + [ -n "$_mver" ] || { echo "!! no /lib/modules -- no kernel is installed here"; exit 1; } + _md="/lib/modules/$_mver" + _pat="${_MODARGS[0]-}" + _loaded="$(cut -d' ' -f1 /proc/modules 2>/dev/null | tr '_' '-' | sort -u)" + _all="$(find "$_md" -name '*.ko' -o -name '*.ko.*' 2>/dev/null \ + | sed 's:.*/::; s:\.ko.*::' | tr '_' '-' | sort -u)" + _builtin="$(sed 's:.*/::; s:\.ko.*::' "$_md/modules.builtin" 2>/dev/null \ + | tr '_' '-' | sort -u)" + _nall="$(printf '%s\n' "$_all" | grep -c . || true)" + _nld="$(printf '%s\n' "$_loaded" | grep -c . || true)" + _nbi="$(printf '%s\n' "$_builtin" | grep -c . || true)" + echo "# kernel $_mver: $_nall module(s) built, $_nld loaded, $_nbi built in" + [ "$_mver" = "$(uname -r)" ] \ + || echo "# (the running kernel is $(uname -r) -- this is another one)" + if [ -z "$_pat" ]; then + echo "# loaded now:" + printf '%s\n' "$_loaded" | grep . | tr '\n' ' ' | fold -s -w 72 | sed 's/^/ /' + echo + echo "# one subject: lfs-kernel modules ath" + echo "# not built yet: lfs-kernel config search ath9k" + exit 0 + fi + _hit="$(printf '%s\n' "$_all" "$_builtin" | sort -u \ + | grep -i -- "$(printf '%s' "$_pat" | tr '_' '-')" || true)" + if [ -z "$_hit" ]; then + echo "# no module of this kernel is called anything like '$_pat'." + echo "# is it even configured? lfs-kernel config search $_pat" + echo "# what it would be called: lfs-kernel enable --list" + exit 1 + fi + echo "# modules matching '$_pat'" + printf '%s\n' "$_hit" | while read -r _m; do + _st="built" + printf '%s\n' "$_builtin" | grep -qx -- "$_m" && _st="built-in" + printf '%s\n' "$_loaded" | grep -qx -- "$_m" && _st="LOADED" + _desc="$(modinfo -F description "$(printf '%s' "$_m" | tr '-' '_')" \ + 2>/dev/null | head -n1)" + printf ' %-24s %-9s %s\n' "$_m" "$_st" "${_desc:-}" + done + echo + echo "# load one now: lfs-kernel load <name>" + echo "# and at boot: lfs-kernel load <name> --boot" + echo "# (a USB device loads its driver when you plug it in, if udev runs)" + exit 0 +fi + +# -------------------------------------------------------------------------- +# WHERE ARE WE? +# +# A kernel build is long enough to be interrupted -- Ctrl-C, a reboot, a +# session that timed out -- and afterwards nothing on the machine says how +# far it got. The tree holds a .config that may or may not carry your +# answers, object files that may or may not be a finished image, and +# /lib/modules and the ESP each hold a kernel from some earlier day. +# +# `lfs-kernel status` reads all four and says, in order: what is running, +# what the tree has, what is installed, and what to do next. It builds +# nothing, needs no network, and does not care whether the ESP is mounted. +# -------------------------------------------------------------------------- +if [ "${_STATUS:-0}" = 1 ]; then + [ -d "$tree" ] || tree="$(_kcfg_tree)" + _ver="$(basename "${tree:-}" 2>/dev/null | sed 's/^linux-//')" + _run="$(uname -r)"; _runv="$(echo "$_run" | sed 's/-.*//')" + echo "### lfs-kernel status" + echo + echo "# running now" + echo " kernel $_run" + if [ -s /etc/pkgusr/kernel.last-good ]; then + echo " known to boot $(cat /etc/pkgusr/kernel.last-good)" + else + echo " known to boot -- nothing recorded yet" + fi + if [ -d "/lib/modules/$_run" ]; then + echo " its modules /lib/modules/$_run" + else + echo " its modules MISSING (/lib/modules/$_run is not there)" + fi + echo + echo "# the config" + echo " CONFIG_FROM $CONFIG_FROM" + for _f in last-good prev-good; do + if [ -f "$KCFG_DIR/$_f.config" ]; then + echo " $_f $(_kcfg_ver "$KCFG_DIR/$_f.config") ($(date -r "$KCFG_DIR/$_f.config" '+%Y-%m-%d %H:%M'))" + else + echo " $_f -- none" + fi + done + # DOES THE TREE'S CONFIG CARRY WHAT YOU ASKED FOR? This is the question + # after an interrupted run: the answers are applied BEFORE the compile, + # so they are in the file even when no kernel came out of it. + if [ -n "${CONFIG_SET:-}" ] && [ -f "${tree:-}/.config" ]; then + _miss="" + for _cs in $CONFIG_SET; do + _sym="CONFIG_${_cs%%=*}"; _sym="CONFIG_${_sym#CONFIG_}"; _val="${_cs#*=}" + case "$_val" in + n) grep -qE "^$_sym=(y|m)" "$tree/.config" && _miss="$_miss $_cs" ;; + *) grep -qx "$_sym=$_val" "$tree/.config" || _miss="$_miss $_cs" ;; + esac + done + if [ -z "$_miss" ]; then + echo " CONFIG_SET all $(echo $CONFIG_SET | wc -w) in $tree/.config" + else + echo " CONFIG_SET NOT in the tree's .config:$_miss" + echo " (the next build applies them)" + fi + fi + echo + echo "# the tree" + if [ -z "${tree:-}" ] || [ ! -d "$tree" ]; then + echo " none in $SRC -- nothing has been unpacked here" + else + echo " $tree" + _bz="$tree/arch/x86/boot/bzImage" + if [ ! -f "$tree/.config" ]; then + echo " config none yet" + elif [ -f "$_bz" ] && [ "$_bz" -nt "$tree/.config" ]; then + echo " build FINISHED -- bzImage is newer than .config" + echo " $(date -r "$_bz" '+%Y-%m-%d %H:%M')" + elif [ -f "$tree/vmlinux" ] || find "$tree" -maxdepth 2 -name '*.o' -print -quit 2>/dev/null | grep -q .; then + echo " build UNFINISHED -- object files, no current image" + echo " (an interrupted make; lfs-kernel continues" + echo " from here, it does not start over)" + else + echo " build not started" + fi + _nko="$(find "$tree" -name '*.ko' 2>/dev/null | wc -l)" + echo " modules built $_nko in the tree" + fi + echo + echo "# installed" + if [ -n "$_ver" ] && [ -d "/lib/modules/$_ver" ]; then + echo " /lib/modules/$_ver $(test -s "/lib/modules/$_ver/modules.dep" && echo "(depmod has run)" || echo "-- NO modules.dep")" + echo " $(find "/lib/modules/$_ver" -name '*.ko*' 2>/dev/null | wc -l) module(s)" + elif [ -n "$_ver" ]; then + echo " /lib/modules/$_ver -- not there: this kernel was never installed" + fi + if [ -d "${ESP:-}" ]; then + if [ -f "$ESP/$KERNEL_NAME" ]; then + echo " $ESP/$KERNEL_NAME $(date -r "$ESP/$KERNEL_NAME" '+%Y-%m-%d %H:%M')" + fi + for _k in "$ESP/$KERNEL_NAME"-*; do + [ -f "$_k" ] && echo " $(basename "$_k")" + done + if [ -n "$_ver" ] && [ -f "$ESP/config-$_ver" ] && [ -f "$tree/.config" ]; then + if cmp -s "$ESP/config-$_ver" "$tree/.config"; then + echo " the installed kernel was built from the tree's CURRENT config" + else + echo " the installed config DIFFERS from the tree's .config --" + echo " what is in the tree has not been built and installed yet" + fi + fi + else + echo " $ESP is not mounted -- cannot say what is on the EFI partition" + fi + echo + echo "# next" + if [ -n "$_ver" ] && [ -f "$tree/.config" ] \ + && { [ ! -f "$ESP/config-$_ver" ] || ! cmp -s "$ESP/config-$_ver" "$tree/.config"; }; then + echo " lfs-kernel # build and install what the tree now has" + echo " an interrupted build continues; nothing is thrown away." + else + echo " nothing owed: the tree, /lib/modules and the ESP agree." + fi + echo " lfs-kernel modules ath # is a driver there, is it loaded" + echo " lfs-kernel config show # what the next build would use" + exit 0 +fi + +# -------------------------------------------------------------------------- +# LOADING A MODULE, NOW AND AT EVERY BOOT. +# +# A driver built as a module does nothing until something loads it. For a +# USB device udev does that when you plug it in; for everything else, and +# on a machine where udev is not running, it is modprobe -- and then the +# same name in /etc/sysconfig/modules, which is the file LFS's `modules` +# bootscript reads. Two places, one command. +# -------------------------------------------------------------------------- +if [ -n "${_LOADCMD:-}" ]; then + _MODFILE="${MODULES_FILE:-/etc/sysconfig/modules}" + _boot=0; _names=() + for _a in ${_LOADARGS[@]+"${_LOADARGS[@]}"}; do + case "$_a" in + --boot) _boot=1 ;; + -*) echo "!! lfs-kernel $_LOADCMD: unknown option '$_a'"; exit 1 ;; + *) _names+=("$_a") ;; + esac + done + if [ "${#_names[@]}" = 0 ]; then + echo "!! lfs-kernel $_LOADCMD <module> [...] [--boot]" + echo " which modules there are: lfs-kernel modules <text>" + exit 1 + fi + _rc=0 + for _m in "${_names[@]}"; do + _m="$(printf '%s' "$_m" | tr '-' '_')" + if [ "$_LOADCMD" = unload ]; then + if modprobe -r "$_m" 2>&1; then + echo "# $_m: unloaded" + else + echo "!! $_m: could not be unloaded (in use? lsmod | grep $_m)" + _rc=1 + fi + if [ -f "$_MODFILE" ] && grep -qE "^[[:space:]]*$_m([[:space:]]|$)" "$_MODFILE"; then + cp -p "$_MODFILE" "$_MODFILE.bak-$(date +%Y%m%d-%H%M%S)" + _t="$(mktemp)" + grep -vE "^[[:space:]]*$_m([[:space:]]|$)" "$_MODFILE" > "$_t" \ + && cat "$_t" > "$_MODFILE" + rm -f "$_t" + echo "# and taken out of $_MODFILE" + fi + continue + fi + if grep -qE "^${_m}[[:space:]]" /proc/modules 2>/dev/null; then + echo "# $_m: already loaded" + elif modprobe "$_m" 2>&1; then + if grep -qE "^${_m}[[:space:]]" /proc/modules 2>/dev/null; then + echo "# $_m: loaded" + else + # BUILT IN IS NOT A FAILURE. modprobe succeeds silently for + # a symbol that is =y; the module simply never appears in + # /proc/modules because it is part of the image. + echo "# $_m: modprobe returned success but it is not in" + echo "# /proc/modules -- it is probably built INTO the kernel" + echo "# (lfs-kernel modules $_m says which)" + fi + else + echo "!! $_m: modprobe refused it." + echo " is it built? lfs-kernel modules $_m" + echo " is it set? lfs-kernel config search $_m" + _rc=1 + continue + fi + # WHAT THE KERNEL SAID ABOUT IT, which is where a missing firmware + # blob or an unsupported device id turns up. + if command -v dmesg >/dev/null 2>&1; then + _say="$(dmesg 2>/dev/null | tail -n 40 \ + | grep -iE "${_m}|${_m//_/-}|firmware" | tail -n 5)" + [ -n "$_say" ] && printf '%s\n' "$_say" | sed 's/^/ /' + fi + if [ "$_boot" = 1 ]; then + if [ -f "$_MODFILE" ] && grep -qE "^[[:space:]]*$_m([[:space:]]|$)" "$_MODFILE"; then + echo "# $_MODFILE already lists it" + else + [ -f "$_MODFILE" ] \ + && cp -p "$_MODFILE" "$_MODFILE.bak-$(date +%Y%m%d-%H%M%S)" + if [ ! -f "$_MODFILE" ]; then + install -d "$(dirname "$_MODFILE")" + { + echo "# /etc/sysconfig/modules -- one module per line," + echo "# loaded at boot by the 'modules' bootscript." + echo "# Arguments may follow the name." + } > "$_MODFILE" + fi + printf '%s\n' "$_m" >> "$_MODFILE" + echo "# added to $_MODFILE (loaded at every boot)" + fi + else + echo "# at every boot as well: lfs-kernel load $_m --boot" + fi + done + exit $_rc +fi + # -------------------------------------------------------------------------- # the ESP has to be the EFI partition, not a directory of the same name. # Every copy below would SUCCEED on the root filesystem and the firmware @@ -1095,6 +1945,44 @@ fi echo "# kernel $VERSION -> $ESP (tree $tree, $JOBS)" +# ---- the plan, when --run was not given ---------------------------------- +if [ "${_RUN:-0}" != 1 ]; then + echo + echo "### what a build would do (nothing has been built)" + echo " version $VERSION" + if [ -d "$tree" ]; then + _bz="$tree/arch/x86/boot/bzImage" + if [ -f "$tree/.config" ] && [ -f "$_bz" ] && [ "$_bz" -nt "$tree/.config" ]; then + echo " tree $tree (already built -- make would do little)" + elif [ -f "$tree/.config" ]; then + echo " tree $tree (a build was started; make CONTINUES)" + else + echo " tree $tree (unpacked, not configured)" + fi + else + echo " tree $tree -- would be fetched and unpacked" + fi + echo " config from $CONFIG_FROM, refreshed with ${CONFIG_REFRESH:-olddefconfig}" + [ -n "${CONFIG_SET:-}" ] && echo " then set $CONFIG_SET" + echo " installs $ESP/$KERNEL_NAME (previous kept as ${KERNEL_NAME}_old)" + echo " /lib/modules/$VERSION, and the config beside the image" + case "${FIRMWARE_FROM:-none}" in + none) echo " firmware not touched (FIRMWARE_FROM=none)" ;; + *) echo " firmware ${FIRMWARE_FROM}: ${FIRMWARE_ONLY:-everything}" ;; + esac + [ "${HEADERS:-no}" = yes ] \ + && echo " headers /usr/include, as ${HEADERS_PKG:-linux-headers}" + case "${V4L2LOOPBACK:-no}" in yes|1|true) + echo " also v4l2loopback, built against this tree" ;; + esac + echo + echo " This is an hour of compiling on most machines, and it REPLACES" + echo " what $ESP/$KERNEL_NAME boots." + echo " Go ahead: lfs-kernel${_CLI_VERSION:+ $_CLI_VERSION} --run" + echo " Where we are: lfs-kernel status" + exit 0 +fi + # -------------------------------------------------------------------------- # THE COMMANDS. Everything above chooses the version and checks the paths; # everything below is the build, and is meant to be read. @@ -1122,13 +2010,43 @@ if [ -n "${BOOTLOGO:-}" ]; then fi # .config +# running /proc/config.gz -- the kernel carrying the machine +# last-good the config of the last kernel that BOOTED ($KCFG_DIR) +# prev-good the one before it -- the way back from a bad answer +# saved the newest config this tool built with, booted or not +# file:<path> any file; keep the tree's own .config +_cfg_src="" case "$CONFIG_FROM" in - running) _as_pkg "gunzip < /proc/config.gz > .config" || { - echo "!! /proc/config.gz unreadable -- set CONFIG_FROM=file:<path>"; exit 1; } ;; + running) + if [ -r /proc/config.gz ]; then + _as_pkg "gunzip < /proc/config.gz > .config" || exit 1 + elif _kcfg_readable "$KCFG_DIR/last-good.config"; then + # NOT A FAILURE WORTH STOPPING FOR. A kernel built without + # CONFIG_IKCONFIG_PROC has no /proc/config.gz, and the config of + # the last kernel that booted is the same answer from a file. + echo "# /proc/config.gz is not readable (no CONFIG_IKCONFIG_PROC?)" + _cfg_src="$KCFG_DIR/last-good.config" + else + echo "!! /proc/config.gz unreadable, and no saved config either." + echo " Set CONFIG_FROM=file:<path> in $CONF, or see:" + echo " lfs-kernel config list" + exit 1 + fi ;; + last-good|good) _cfg_src="$KCFG_DIR/last-good.config" ;; + prev-good|prev) _cfg_src="$KCFG_DIR/prev-good.config" ;; + saved|last) _cfg_src="$(_kcfg_newest)" ;; file:*) _as_pkg "cp -v '${CONFIG_FROM#file:}' .config" || exit 1 ;; keep) [ -f "$tree/.config" ] || { echo "!! no .config in $tree"; exit 1; } ;; *) echo "!! CONFIG_FROM=$CONFIG_FROM not understood"; exit 1 ;; esac +if [ -n "$_cfg_src" ]; then + _kcfg_readable "$_cfg_src" || { + echo "!! CONFIG_FROM=$CONFIG_FROM, but $_cfg_src is not a config." + echo " lfs-kernel config list # what there is" + exit 1; } + echo "# starting from $_cfg_src (kernel $(_kcfg_ver "$_cfg_src"))" + _as_pkg "cp '$_cfg_src' .config" || exit 1 +fi # ---- refreshing the config for a NEW kernel ------------------------------ # # A config from one kernel is never complete for another: every version adds @@ -1209,6 +2127,11 @@ if [ -n "${CONFIG_SET:-}" ]; then echo "# all $(echo $CONFIG_SET | wc -w) set, read back from .config" fi +# THE CONFIG IS KEPT BEFORE ANYTHING IS COMPILED. It is finished here -- +# source, refresh and CONFIG_SET all applied -- and a build that dies at +# hour two must not take the answers with it. +_kcfg_archive "$tree/.config" "$VERSION" + # what a Wayland compositor needs; a warning, not a refusal for opt in CONFIG_DRM CONFIG_DRM_FBDEV_EMULATION CONFIG_INPUT_EVDEV CONFIG_SYSVIPC; do grep -qE "^$opt=(y|m)" "$tree/.config" || echo "!! $opt is not set (sway needs it)" @@ -1324,7 +2247,8 @@ depmod -a "$VERSION" || exit 1 # perfectly installed. The kernel is on disk at this point; record it. # (Run again after the headers, which add /usr/include.) _record() { - echo "# recording what $acct owns (scans the filesystem; this is slow) ..." + echo "# recording what $acct owns (a scan AS $acct, skipping /proc, the" + echo "# package source trees and the caches; --root if something is missed)" packagemanager reload-pkg-list "$PKG" >/dev/null 2>&1 || return 0 local _h; _h="$(getent passwd "$acct" | cut -d: -f6)" echo "# recorded $(grep -c . "$_h/pkg.lst" 2>/dev/null) file(s) for $acct" @@ -1434,8 +2358,9 @@ esac # A kernel that boots is the only way back from one that does not, and its # TREE is what you need to rebuild or bisect -- not just the bzImage. The # version that is running right now is recorded as last-good before this one -# is; nothing here ever removes that tree, and `lfs-kernel prune` (below) -# refuses to. +# is; nothing here ever removes that tree. (An older comment here promised +# a `lfs-kernel prune`; there is none, and before 1.14.324 typing it started +# a full build instead of saying so.) # ONLY A KERNEL THAT HAS BOOTED IS KNOWN TO BOOT. # # The old `elif` wrote the version just built into last-good whenever the @@ -1450,9 +2375,12 @@ if [ "$_running" = "$VERSION" ]; then # the kernel just built is the one running: it demonstrably boots printf '%s\n' "$VERSION" > "$_lastgood" echo "# $VERSION is running, so it is now the kernel known to boot" + _kcfg_mark_good "$tree/.config" "$VERSION" elif [ -d "$SRC/linux-$_running" ]; then printf '%s\n' "$_running" > "$_lastgood" echo "# keeping $SRC/linux-$_running as the last kernel known to boot" + # ...and ITS config is the one known to work, not the one just built + _kcfg_mark_good "$SRC/linux-$_running/.config" "$_running" else echo "# the running kernel is $_running, and there is no tree for it here," echo "# so nothing is recorded as known-good. $VERSION will be recorded" @@ -1468,5 +2396,6 @@ echo "# $VERSION installed as $ESP/$KERNEL_NAME" echo "# previous kernel kept as $ESP/${KERNEL_NAME}_old" echo "# its source kept at $SRC/linux-$(cat "$_lastgood" 2>/dev/null)" echo "# owned by $acct (packagemanager verify | grep $PKG)" +echo "# its config kept in $KCFG_DIR (lfs-kernel config list)" echo "# running kernel is $(uname -r) -- reboot to use $VERSION" [ "$_fw_ok" = 1 ] || echo "# NOTE: firmware was NOT installed (see the message above)" diff --git a/lfs-phases b/lfs-phases index 2e3b0eb..f97d577 100755 --- a/lfs-phases +++ b/lfs-phases @@ -34,7 +34,7 @@ # scripts are also SOURCED by lfs-helper to read their metadata, and this file # comes along -- it must not shadow anything in the sourcing shell. -PKGUSR_LIB_VERSION="1.14.321" +PKGUSR_LIB_VERSION="1.14.340" _pp_build_id() { local f="${BASH_SOURCE[0]}" @@ -570,9 +570,66 @@ _pp_warn_unbuilt() { return 0 } +# DOCUMENTATION THAT WAS NEVER BUILT IS NOT A FAILED INSTALL. +# +# parted's page ends with the book's doc block: +# install -v -m644 doc/{FAT,API,parted.{html,txt,pdf,ps,dvi}} \ +# /usr/share/doc/parted-3.7 +# parted.pdf, parted.ps, parted.dvi and API are built only when texlive is +# installed, and texlive is in avoid.stack. So `install` stopped on +# install: cannot stat 'doc/parted.pdf': No such file or directory +# after the package itself -- 97 files, every binary and library -- was +# already on disk. A missing PDF is not a failed install. +# +# Only for a destination under share/doc, only after the command has been +# tried EXACTLY as the book wrote it, and only for sources that are not +# there: everything else is passed through to /usr/bin/install untouched. +# The function is defined inside this phase, which runs in a child process +# of its own, so nothing outside it ever sees an `install` that is not the +# real one. +_pp_install_docs() { + local last="" a; for a in "$@"; do last="$a"; done + case "$last" in + */share/doc/*|*/share/doc|*/share/gtk-doc/*) ;; + *) command install "$@"; return $? ;; + esac + command install "$@" 2>/dev/null && return 0 + local rc=$? + local keep=() prev="" i=0 n=$# missing="" have=0 + for a in "$@"; do + i=$((i + 1)) + if [ "$i" -eq "$n" ]; then keep+=("$a"); continue; fi + case "$prev" in + -m|-o|-g|-t|--mode|--owner|--group|--target-directory) + keep+=("$a"); prev=""; continue ;; + esac + case "$a" in -*) keep+=("$a"); prev="$a"; continue ;; esac + prev="" + if [ -e "$a" ] || [ -L "$a" ]; then + keep+=("$a"); have=$((have + 1)) + else + missing="$missing $a" + fi + done + # nothing missing -> the failure is a real one, and it is the book's + # command that must be reported, not ours + if [ -z "$missing" ]; then + command install "$@" + return $? + fi + if [ "$have" -gt 0 ]; then + command install "${keep[@]}" || { command install "$@"; return $?; } + fi + _pp_note "not installed, because this build did not produce them:$missing" + _pp_note " they are documentation (the book's doc block builds them" + _pp_note " only with texlive and friends); the package itself is in." + return 0 +} + _pp_phase_install() { _pp_enter_build || return 1 _pp_warn_unbuilt + install() { _pp_install_docs "$@"; } install_pkg } diff --git a/packagemanager b/packagemanager index 92749ce..8a75981 100755 --- a/packagemanager +++ b/packagemanager @@ -188,7 +188,7 @@ TMP_DIR = "/tmp/packagemanager" CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf") # Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure # changes (so stale plans are ignored); TOOL_VERSION is the program version. -TOOL_VERSION = "1.14.321" +TOOL_VERSION = "1.14.340" def _build_id(): @@ -4619,6 +4619,32 @@ def _explain_not_in_book(target): printInfo(f" Nothing similar found either. Try: blfs search {target}") +def _install_finished(u): + """Positive evidence that an install FINISHED -- not that one started. + + THIS IS WHY THE STACK NEEDED HAND-HOLDING. `install_last` is copied + into the package home BEFORE the build runs (lfs-helper: "RECORD WHAT + IS INSTALLED BEFORE OVERWRITING THE RECORD"), so a build that dies at + configure leaves one behind. For a package whose page declares no + programs or libraries to check -- cairomm is one -- `state` then has + nothing else to go on and answers "installed". + + cairomm-1.0 failed at configure with 0 files tracked, was recorded + installed by that rule, and every later plan SKIPPED it as a dependency + that was already there. pangomm found out the hard way: + Run-time dependency cairomm-1.0 found: NO + and the person had to name it by hand to get the chain moving. + + A manifest with real paths, or the marker the runner writes after a + SUCCESSFUL install, is evidence. A script that was copied in before + the build is not. + """ + if getattr(u, "pkg_list", None): + return True + home = getattr(u, "home", "") or "" + return bool(home) and os.path.isfile(os.path.join(home, ".pkgusr-installed")) + + def build_install_plan(args): """Return (plan, missing, why). plan = [(anchor, name_version, reason)] in install order (deps first); reason in {new, update, reinstall}. why maps @@ -4657,6 +4683,8 @@ def build_install_plan(args): # plan that has shrunk 181 -> 151 -> 135 across three runs reads as three # identical fresh starts. The progress was real and invisible. build_install_plan.already = set() + build_install_plan.behind = [] # installed deps the book has moved past + build_install_plan.unproven = [] # "installed", with nothing to show for it _not_found = [] # Reading the book can take many seconds the first time (it is indexed # once per version). Without a word on screen that looks like a hang, so @@ -4676,10 +4704,33 @@ def build_install_plan(args): if _PLAN_BOOK and _book_override(anchor) != _PLAN_BOOK: nv = default_book_version(anchor) or nv u = gather_user(anchor) - up_to_date = (u.state == "installed" - and (not u.version or u.version == nv)) + _done = u.state == "installed" and _install_finished(u) + if u.state == "installed" and not _done: + # recorded, but nothing says the install ever finished + build_install_plan.unproven.append(anchor) + up_to_date = (_done and (not u.version or u.version == nv)) if up_to_date: build_install_plan.already.add(anchor) + # AN INSTALLED DEPENDENCY IS NOT A THING TO UPGRADE. + # + # `install gparted --recursive` planned 53 packages: gparted, its + # real missing dependencies -- and UPDATES of bubblewrap, alsa-lib, + # gtk3, gtk4, librsvg, glycin and thirty others that are installed + # and working, purely because the book has moved on since they were + # built. Asking for one program is not asking to rebuild the + # desktop, and each of those rebuilds is a chance to break a + # machine that was fine. + # + # So: a DEPENDENCY that is installed is left alone, whatever version + # it is. A package NAMED on the command line still goes through + # (that is an instruction), --update-deps restores the old + # behaviour, and `update --stack <name>` is the deliberate way to + # move a whole chain forward. + elif (_done and anchor not in _named + and not getattr(args, "update_deps", False)): + build_install_plan.already.add(anchor) + build_install_plan.behind.append((anchor, u.version or "?", nv)) + return if args.reinstall: plan.append((anchor, nv, "reinstall" if up_to_date else ("update" if u.state == "installed" else "new"))) @@ -4951,6 +5002,29 @@ def cmd_install(args, _replanned=False): return f"depends on {parent}" return f"{kind} by {parent}" + _unproven = getattr(build_install_plan, "unproven", []) or [] + if _unproven: + printWarning(" %d package(s) are recorded as installed but have no" + " finished-install record:" % len(_unproven)) + printWarning(" " + ", ".join(sorted(set(_unproven))[:8]) + + (" ..." if len(set(_unproven)) > 8 else "")) + printInfo(" Their install_last was written before a build that did" + " not finish, so they") + printInfo(" are in this plan rather than skipped -- which is what" + " lets the stack run") + printInfo(" through without naming them by hand.") + _behind = getattr(build_install_plan, "behind", []) or [] + if _behind: + printInfo(" %d installed dependency(ies) are older than the book and" + " are NOT touched:" % len(_behind)) + printInfo(" " + ", ".join("%s (%s -> %s)" % (a, iv, bv) + for a, iv, bv in _behind[:6]) + + (" ..." if len(_behind) > 6 else "")) + printInfo(" that is deliberate: one program is not a reason to" + " rebuild what works.") + printInfo(" move them forward on purpose: packagemanager update" + " <name> --run") + printInfo(" or build this plan against them: add --update-deps") if getattr(args, "select", False): plan = _select_plan_rows( plan, lambda r: f"{r[1]:<30} {r[2]:<10} {why_str(r[0])}") @@ -7916,7 +7990,32 @@ def _python_module_here(name): return False -def write_pkg_list(user, background=True): +# WHAT A MANIFEST SCAN MUST NOT WALK. +# +# `reload-pkg-list p_linux` took minutes, and the time went nowhere near the +# installed files: the scan walked /usr/src/pkgusr -- every package user's +# unpacked source tree, the kernel's own quarter-million files among them -- +# plus /proc, /sys, the snapshots and the caches. None of it can contain an +# INSTALLED file by definition: a package's home is excluded from its own +# manifest, and no package installs into another's source tree. +_SCAN_SKIP = ["/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", + "/var/cache", "/mnt", "/media", "/lost+found", + "/.snapshots", "/snapshots", "/sources", "/build"] + + +def _scan_prune_args(): + """find(1) arguments that skip those directories (and stay on one fs).""" + skips = list(_SCAN_SKIP) + [d.rstrip("/") for d in pkgusr_roots()] + out = ["-xdev", "("] + for i, d in enumerate(skips): + if i: + out.append("-o") + out += ["-path", d] + out += [")", "-prune", "-o"] + return out + + +def write_pkg_list(user, background=True, as_root=False): """Regenerate /usr/src/<user>/pkg.lst. Runs list_package AS the package user (su -) when possible, so it only scans directories that user owns -- much faster than scanning the whole filesystem as root.""" @@ -7970,16 +8069,29 @@ def write_pkg_list(user, background=True): " /\\/lib\\/modules\\/[^/]+\\/source\\// { next } " " { print }'" % shlex.quote(home.rstrip("/") + "/")) - if _can_su(user): + # THE SCAN ITSELF, PRUNED. `list_package` is the hint's script and walks + # everything; this is the same question asked of the same filesystem + # without the directories that cannot hold an installed file. It is run + # AS the package user, which also skips every directory that account + # cannot read -- and that is the difference between minutes and seconds + # on a machine with 90 source trees. --root scans as root instead: it + # reaches directories the account cannot enter (0700 dirs owned by + # somebody else), and takes the long way round. + _find = " ".join(shlex.quote(x) for x in + ["find", "/"] + _scan_prune_args() + + ["-user", user, "-print"]) + _scan = f"{_find} 2>/dev/null | LC_ALL=C sort -u" + if _can_su(user) and not as_root: # su - starts in the user's home, so pkg.lst.new/pkg.lst are ~/... - inner = (f"{lp} {quser} 2>/dev/null | {_filter} > pkg.lst.new" + inner = (f"{_scan} | {_filter} > pkg.lst.new" f" && mv pkg.lst.new pkg.lst") cmd = ["su", "-", user, "-c", inner] else: tmp = shlex.quote(os.path.join(home, "pkg.lst.new")) dst = shlex.quote(os.path.join(home, "pkg.lst")) - cmd = ["sh", "-c", f"{lp} {quser} 2>/dev/null | {_filter} > {tmp}" - f" && mv {tmp} {dst}"] + cmd = ["sh", "-c", f"{_scan} | {_filter} > {tmp}" + f" && mv {tmp} {dst}" + f" && chown {quser} {dst} 2>/dev/null; true"] if background: # detached / disowned subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, @@ -8004,8 +8116,12 @@ def cmd_reload_pkg_list(args): printInfo(f"{acct}: pkg.lst regeneration started" f" (disowned background task).") continue - printInfo(f"{acct}: scanning for files it owns ...") - if not write_pkg_list(user, background=False): + _as_root = getattr(args, "root", False) + printInfo(f"{acct}: scanning for files it owns " + + ("as root (every directory) ..." if _as_root + else f"as {acct} ...")) + _t0 = time.time() + if not write_pkg_list(user, background=False, as_root=_as_root): printError(f"{acct}: could not regenerate pkg.lst") rc = 1 continue @@ -8016,7 +8132,18 @@ def cmd_reload_pkg_list(args): except OSError: n = 0 if n: - printSuccess(f"{acct}: {n} file(s) recorded in {manifest}") + printSuccess(f"{acct}: {n} file(s) recorded in {manifest}" + f" ({time.time() - _t0:.0f}s)") + if not _as_root: + # SAY WHAT THE FAST SCAN CANNOT SEE. A file this package + # owns inside a directory it may not read is not in this + # list, and nothing else would ever say so. + printInfo(f" scanned as {acct}: directories it cannot enter" + f" were skipped, and so were") + printInfo(f" {', '.join(_SCAN_SKIP[:6])} ... and the package" + f" source roots.") + printInfo(f" If something is missing: packagemanager" + f" reload-pkg-list {user} --root") else: printWarning(f"{acct}: no files found -- it owns nothing on disk.") printInfo(" Either it was never installed, or its files ended up" @@ -12437,6 +12564,10 @@ def cmd_env(args): # # File format, one entry per line: # book <target> [key=value ...] a BLFS book package (deps resolved) +# <any> ... when="<shell>" only when that command succeeds -- +# a package this machine needs only +# sometimes (btrfs-progs on a btrfs +# system). It is asked every run. # git <name> url=<repo> [ref=<ref>] [prog=<installed-program>] # tar <name> url=<tarball-url> pkg=<filename> [prog=|have=] [build=|install=] # check <name> root script <stackdir>/<name>.sh; @@ -13173,6 +13304,28 @@ def cmd_stack(args): entries = [e for e in entries if e["target"] in _only] printInfo("running only: %s" % ", ".join(e["target"] for e in entries)) print() + # WHEN=: AN ENTRY THIS MACHINE DOES NOT NEED. + # + # btrfs-progs belongs in the stack of a btrfs machine and nowhere else, + # and a second stack file per filesystem is not an answer. The test is + # a shell command, run here, every time -- so a disk added later is + # picked up without editing anything. + _when_out = [] + _kept = [] + for e in entries: + _w = e["opts"].get("when") + if _w and subprocess.run(["/bin/bash", "-c", _w], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL).returncode != 0: + _when_out.append((e["target"], _w)) + continue + _kept.append(e) + entries = _kept + if _when_out and not getattr(args, "status", False): + for _t, _w in _when_out: + printInfo(" %s: not for this machine -- when=%s" % (_t, _w)) + print() + rows = [] for e in entries: e["_key"] = "%s %s" % (e["kind"], e["target"]) @@ -14682,6 +14835,9 @@ def main(): p.add_argument("--recursive", action="store_true", help="also resolve and build the package's dependencies " "(default: just the named package(s))") + p.add_argument("--update-deps", action="store_true", + help="also rebuild dependencies that are installed but " + "older than the book (default: leave them alone)") p.add_argument("-f", "--force", action="store_true", help="(deprecated: non-recursive is now the default; kept for " "compatibility)") @@ -14800,6 +14956,9 @@ def main(): p.set_defaults(func=cmd_remove) p = sub.add_parser("reload-pkg-list", help="regenerate /usr/src/<name>/pkg.lst") + p.add_argument("--root", action="store_true", + help="scan as root: slower, but reaches directories the " + "package user cannot enter") p.add_argument("packages", nargs="+") p.add_argument("--background", action="store_true", help="run detached and return at once (the old default)") diff --git a/stacks/apps.stack b/stacks/apps.stack index 3c74a12..b274c71 100644 --- a/stacks/apps.stack +++ b/stacks/apps.stack @@ -1,10 +1,12 @@ -# apps.stack -- desktop applications: a document viewer and an image viewer. -# stack-version: 1.14.320 +# apps.stack -- desktop applications: a document viewer, an image viewer and +# a partition editor. +# stack-version: 1.14.334 # # WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack # here: the exclusions are a property of the machine, not of one stack. include avoid + # packagemanager stack apps --run # # Programs you open, not the desktop they open on. Run it AFTER sway.stack: @@ -92,3 +94,27 @@ git imv url=https://github.com/exec64/imv.git ref=v4.3.0 \ install="ninja -C build install" \ expect="command -v imv-wayland >/dev/null 2>&1 || ldd /usr/bin/imv 2>/dev/null | grep -q wayland" +# GPARTED -- partitions, with a window. It is a BLFS page (Gparted-1.7.0), +# so the planner brings gtkmm3 and parted with it; both are C++/GTK builds +# and this is not a five-minute entry. +# +# IT RUNS AS ROOT, FROM A TERMINAL. The book's page uses polkit for the +# graphical "type your password" step, and polkit is in avoid.stack (it +# arrives only through colord and systemd, which this machine does not +# have). Without it there is no menu entry that can ask: start it from a +# root shell. `gparted` alone then works; it needs $DISPLAY/$WAYLAND_DISPLAY, +# so run it from a terminal inside the session. +# +# The partition TOOLS it drives are separate packages -- exfatprogs and +# btrfs-progs are in services.stack; what is missing shows up greyed out in +# gparted's filesystem menu, not as an error. +# LVM2 IS NOT PART OF A PARTITION EDITOR. The book RECOMMENDS it for +# parted, so the planner pulls it in -- and on this machine, which has no +# logical volumes, it is an hour of build and a package whose install wants +# root (13.0's page puts `mount -o remount,dev /tmp` in the middle of it). +# ignore= keeps it out of this entry's plan only; install it deliberately +# the day this laptop grows an LVM volume: +# packagemanager install lvm2 --run --recursive +book gparted \ + ignore=lvm2 \ + expect="command -v gparted >/dev/null" diff --git a/stacks/diag.stack b/stacks/diag.stack index 11dbeb7..1dbae23 100644 --- a/stacks/diag.stack +++ b/stacks/diag.stack @@ -1,9 +1,19 @@ -# diag.stack -- GPU diagnostic tools: glxinfo, eglinfo, vulkaninfo, vainfo, nvtop. -# stack-version: 1.14.320 +# diag.stack -- what the hardware is (lsusb, lspci) and what the GPU does +# (glxinfo, eglinfo, vulkaninfo, vainfo, nvtop). +# stack-version: 1.14.327 # # WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack. include avoid +# WHAT HARDWARE IS THERE AT ALL. `lsusb` is the first command for a USB +# device that did not appear -- it says whether the machine sees the device +# before any question about drivers or firmware makes sense. lspci is its +# counterpart for the cards inside, and both are one small BLFS page each. +book usbutils \ + expect="command -v lsusb >/dev/null" +book pciutils \ + expect="command -v lspci >/dev/null" + # # Replay with: packagemanager stack diag --run # diff --git a/stacks/machine.conf b/stacks/machine.conf index 7a506c5..2317c02 100644 --- a/stacks/machine.conf +++ b/stacks/machine.conf @@ -109,6 +109,23 @@ gbm = enabled [iptables] configure_args = --enable-nftables +# cairomm/pangomm/glibmm build TEST PROGRAMS by default, and those need +# Boost Test. Boost is in avoid.stack -- an hour of build for a desktop +# that wants none of it -- so meson stops at configure: +# ERROR: Building the test programs requires Boost Test ... and Fontconfig +# Install them or use -Dbuild-tests=false +# The library itself needs neither. (lfs-helper takes this option from the +# error by itself since 1.14.337; this is where it STAYS, because a +# regenerated script would lose it.) +[cairomm-1.0] +build-tests = false + +[pangomm-2.48] +build-tests = false + +[glibmm-2.68] +build-tests = false + # GST_DEBUG=<level> ONLY WORKS WITH THIS. The book builds the core with # -D gst_debug=false, which compiles the whole debug-log system out; this # replaces that switch (the script's -D gst_debug=false becomes =true). diff --git a/stacks/net-wifi.conf b/stacks/net-wifi.conf new file mode 100644 index 0000000..40fa6d1 --- /dev/null +++ b/stacks/net-wifi.conf @@ -0,0 +1,60 @@ +# net-wifi.conf -- read by `cfg net-wifi`. +# +# packagemanager stack network --only net-wifi --run +# +# The step REPORTS what a wireless link still needs, and -- with WRITE=yes -- +# writes the parts that hold no secret. It never touches a file it did not +# write: a config of yours is reported, never edited. +# +# WHAT IT WILL NOT WRITE: the wpa_supplicant configuration, because that one +# holds your passphrase. It prints the two commands that make it. + +# WRITE: write /etc/sysconfig/ifconfig.<iface> for every wireless interface +# that has none. That file names the interface, the service and the DHCP +# client -- no SSID, no key. +WRITE=yes + +# PREFER: interface names, best first; shell patterns are allowed. The first +# one that EXISTS gets the low route metric, so while it is plugged in the +# machine uses it and falls back to the next when it is gone. Leave empty to +# let dhcpcd decide. +# +# An external antenna usually beats the card inside the lid, which is why +# the USB name comes first here. +PREFER="wl-ext wlan*" +METRIC_PREFERRED=100 +METRIC_OTHER=300 + +# A USB ADAPTER'S NAME DEPENDS ON THE PORT IT IS IN. +# +# udev names it from where it sits on the bus -- wlp7s0f4u2 in one socket, +# something else in the next -- and an ifconfig.<name> file cannot follow +# that. USB_RENAME writes a udev rule that names this adapter by its USB +# ids instead, so the name is the same in every port. +# +# The ids are the ones dmesg prints when you plug it in: +# usb 6-2: New USB device found, idVendor=0cf3, idProduct=9271 +# (0cf3:9271 is the Atheros AR9271 -- TL-WN722N v1, Alfa AWUS036NHA, UB91C.) +# USB_NAME must NOT be wlanN: that is the kernel's own namespace and udev +# refuses to rename into it. +USB_RENAME=yes +USB_VENDOR=0cf3 +USB_PRODUCT=9271 +USB_NAME=wl-ext + +# DISABLE_MISSING: a config file for an interface this kernel does not have +# makes the boot print +# Bringing up the eth0 interface... Interface eth0 doesn't exist. FAIL +# yes sets ONBOOT="no" in those files (backed up first) so the boot stops +# trying. Nothing is ever deleted: the card may be back tomorrow, and +# `ifup <iface>` still brings it up by hand. +# +# Leave it at no if you unplug a USB adapter often and would rather have it +# come up automatically the moment it is there at boot. +DISABLE_MISSING=no + +# COPY_WPA: when a new interface has no wpa_supplicant config and another +# interface on this machine has one, copy it (chmod 600). Same machine, same +# networks, your own key -- it is a copy, and the step says so every time. +# An existing file is never overwritten. +COPY_WPA=yes diff --git a/stacks/net-wifi.sh b/stacks/net-wifi.sh index 157f59d..0ee69c6 100644 --- a/stacks/net-wifi.sh +++ b/stacks/net-wifi.sh @@ -22,10 +22,23 @@ # todo something only you can write -> printed, exit 0 fail=0 warn() { echo "!! $*"; fail=1; } +did() { echo "++ $*"; } todo() { echo "-> $*"; todos=$((todos + 1)); } note() { echo "# $*"; } todos=0 +# ---- 0. the settings ------------------------------------------------------ +CONF="${NET_WIFI_CONF:-/etc/pkgusr/stacks/net-wifi.conf}" +[ -r "$CONF" ] && . "$CONF" +: "${WRITE:=no}"; : "${PREFER:=}"; : "${COPY_WPA:=no}" +: "${METRIC_PREFERRED:=100}"; : "${METRIC_OTHER:=300}" +: "${USB_RENAME:=no}"; : "${USB_NAME:=wl-ext}" +SYSCONF="${NET_WIFI_SYSCONF:-/etc/sysconfig}" +NETSYS="${NET_WIFI_SYSFS:-/sys/class/net}" +UDEVDIR="${NET_WIFI_UDEVDIR:-/etc/udev/rules.d}" +MARKER="# written by stacks/net-wifi.sh -- safe to delete, regenerated" +[ -r "$CONF" ] && note "settings: $CONF" || note "no $CONF -- reporting only" + # ---- 1. the binaries ------------------------------------------------------ for b in /sbin/wpa_supplicant /usr/sbin/wpa_supplicant; do [ -x "$b" ] && wpa="$b" && break @@ -36,11 +49,43 @@ command -v dhcpcd >/dev/null 2>&1 \ command -v ip >/dev/null 2>&1 || command -v ifconfig >/dev/null 2>&1 \ || warn "neither ip nor ifconfig is installed" +# ---- 1b. a stable name for the USB adapter -------------------------------- +# Written before the interfaces are listed, because the name it gives is the +# one the rest of this script should be configuring. +_usb_rule="$UDEVDIR/80-wifi-$USB_NAME.rules" +case "$USB_RENAME" in yes|1|true) + if [ -z "${USB_VENDOR:-}" ] || [ -z "${USB_PRODUCT:-}" ]; then + warn "USB_RENAME=yes but USB_VENDOR/USB_PRODUCT are not set in $CONF" + elif [ -e "$_usb_rule" ] && ! grep -qF "$MARKER" "$_usb_rule"; then + note "$_usb_rule exists and is not ours -- left alone" + else + install -d "$UDEVDIR" + cat > "$_usb_rule" <<RULE +$MARKER +# Name the $USB_VENDOR:$USB_PRODUCT wireless adapter "$USB_NAME" whatever port it +# is in. Without this its name comes from its place on the bus (wlp7s0f4u2), +# which changes with the socket -- and /etc/sysconfig/ifconfig.<name> cannot +# follow that. +SUBSYSTEM=="net", ACTION=="add", ATTRS{idVendor}=="$USB_VENDOR", ATTRS{idProduct}=="$USB_PRODUCT", NAME="$USB_NAME" +RULE + chmod 644 "$_usb_rule" + did "wrote $_usb_rule ($USB_VENDOR:$USB_PRODUCT -> $USB_NAME)" + # IT DOES NOT APPLY TO AN INTERFACE THAT IS ALREADY THERE. Renaming + # happens when the device appears, so a plugged-in adapter keeps the + # name it was given until it is unplugged or the machine reboots. + if [ ! -e "$NETSYS/$USB_NAME" ] \ + && ls -1 "$NETSYS" 2>/dev/null | grep -q '^wl.*u[0-9]'; then + note " unplug the adapter and plug it back in to take the name" + note " (or reboot; udev renames a device when it APPEARS)" + fi + fi ;; +esac + # ---- 2. the interfaces the kernel actually has ---------------------------- # The name matters: ifconfig.<name> must match the interface, and predictable # names (wlp3s0) are not the wlan0/wifi0 that examples use. ifaces="" -for d in /sys/class/net/*; do +for d in "$NETSYS"/*; do [ -e "$d/wireless" ] || [ -e "$d/phy80211" ] || continue ifaces="$ifaces $(basename "$d")" done @@ -70,6 +115,31 @@ if [ -d "$svcdir" ]; then note " from blfs-bootscripts' install-service-dhcpcd target:" note " packagemanager stack services --only blfs-bootscripts --run" ;; esac + # WHERE THE SERVICE LOOKS FOR THE CONFIG -- ASK THE SCRIPT. + # + # This machine's boot said: + # wpa_supplicant configuration file + # /etc/sysconfig/wpa_supplicant-wlan0.conf not present FAIL + # ...while the file sat in /etc/sysconfig/wpa_supplicant/ -- the + # SUBDIRECTORY, which is where other bootscript releases keep it. So + # wpa_supplicant never started, the card never associated, and dhcpcd + # had nothing to ask for an address on. Two directories, one word + # apart, and no output anywhere said which one this machine reads. + # The service script names it; read it rather than remember it. + if [ -r "$svcdir/wpa" ]; then + _wpa_pat="$(grep -ohE '/[A-Za-z0-9_./${}-]*wpa_supplicant[A-Za-z0-9_.${}-]*\.conf' \ + "$svcdir/wpa" 2>/dev/null | head -n1)" + if [ -n "$_wpa_pat" ]; then + WPA_DIR="$(dirname "$_wpa_pat")" + # NET_WIFI_SYSCONF overrides /etc/sysconfig (the tests use it); + # a path read out of the service script has to follow it too. + case "$WPA_DIR" in + /etc/sysconfig*) WPA_DIR="$SYSCONF${WPA_DIR#/etc/sysconfig}" ;; + esac + note "the wpa service reads: $_wpa_pat" + note " -> $WPA_DIR/wpa_supplicant-<iface>.conf" + fi + fi # which variable the wpa service reads to chain onward -- from the script if [ -r "$svcdir/wpa" ]; then chain="$(grep -oE '\$\{?[A-Z_]*SERVICE[A-Z_]*' "$svcdir/wpa" \ @@ -83,11 +153,79 @@ else warn "no $svcdir: blfs-bootscripts is not installed" fi +# A SECOND ADAPTER JOINS THE SAME NETWORKS. +# +# The wpa service reads wpa_supplicant-<iface>.conf, so a new interface has +# nothing to join with even though this machine has known the SSID and the +# key for months. Copying your own file is the whole job; it is said out +# loud every time, because a key is being duplicated. +# The directory the SERVICE reads, when it told us; otherwise the one this +# script has always assumed. +_wpadir="${WPA_DIR:-$SYSCONF/wpa_supplicant}" +case "$COPY_WPA" in yes|1|true) + for i in $ifaces; do + _dst="$_wpadir/wpa_supplicant-$i.conf" + [ -e "$_dst" ] && continue + _srcconf="" + # A CONFIG IN THE OTHER DIRECTORY IS STILL YOUR CONFIG. The two + # layouts -- /etc/sysconfig/wpa_supplicant-<iface>.conf and + # /etc/sysconfig/wpa_supplicant/wpa_supplicant-<iface>.conf -- have + # both been right at some point, and a machine can easily have the + # file in the one its service does not read. + for _c in "$_wpadir"/wpa_supplicant-*.conf \ + "$SYSCONF"/wpa_supplicant-*.conf \ + "$SYSCONF"/wpa_supplicant/wpa_supplicant-*.conf; do + [ -r "$_c" ] || continue + case "$(basename "$_c")" in *__*__*|*example*|*template*|*sample*) continue ;; esac + grep -q 'psk\|key_mgmt=NONE' "$_c" 2>/dev/null || continue + _srcconf="$_c"; break + done + [ -n "$_srcconf" ] || continue + install -d -m 755 "$_wpadir" + cp "$_srcconf" "$_dst" && chmod 600 "$_dst" \ + && did "copied $(basename "$_srcconf") -> $(basename "$_dst") (chmod 600; it holds your key)" + done ;; +esac + # ---- 4. the per-interface configuration ----------------------------------- for i in $ifaces; do - f="/etc/sysconfig/ifconfig.$i" + f="$SYSCONF/ifconfig.$i" + # WHICH INTERFACE WINS. The first PREFER pattern that matches something + # this machine has gets the low metric; everything else gets the high + # one, so an external antenna takes over while it is plugged in and the + # card inside carries on when it is not. + _metric="$METRIC_OTHER" + for _p in $PREFER; do + _hit="" + for _c in $ifaces; do + case "$_c" in $_p) _hit="$_c"; break ;; esac + done + [ -n "$_hit" ] || continue + [ "$_hit" = "$i" ] && _metric="$METRIC_PREFERRED" + break + done + if [ ! -r "$f" ] && [ "$WRITE" = yes ]; then + _chain_var="$(printf '%s' "${chain:-WPA_SERVICE}" | awk '{print $1}')" + install -d "$SYSCONF" + cat > "$f" <<IFCFG +$MARKER +# $i -- what ifup reads. No SSID and no key are in here; those live in +# $SYSCONF/wpa_supplicant/wpa_supplicant-$i.conf, which nothing writes for +# you. -m <metric> decides which link the machine prefers when two are up: +# the lower number wins. +ONBOOT="yes" +IFACE="$i" +SERVICE="wpa" +${_chain_var:-WPA_SERVICE}="dhcpcd" +DHCP_START="-b -m $_metric" +DHCP_STOP="-k" +IFCFG + chmod 644 "$f" + did "wrote $f (route metric $_metric$([ "$_metric" = "$METRIC_PREFERRED" ] && echo " -- preferred"))" + fi if [ ! -r "$f" ]; then todo "write $f -- ifup $i has nothing to read" + note " or let this step do it: WRITE=yes in $CONF" # THIS FILE HOLDS NO SECRET. Only the wpa_supplicant conf does, so # there is no reason to make the reader work out the syntax: print # the file, with the interface that exists and the hand-off variable @@ -122,6 +260,73 @@ for i in $ifaces; do done grep -qE '^[[:space:]]*ONBOOT=.?yes' "$f" \ || note "$f: ONBOOT is not yes, so boot will not bring it up" + # THE CONFIG THE SERVICE WILL LOOK FOR, FOR THIS INTERFACE. + if grep -qE '^[[:space:]]*SERVICE=.?wpa' "$f"; then + if [ -r "$_wpadir/wpa_supplicant-$i.conf" ]; then + note "$f: wpa config $_wpadir/wpa_supplicant-$i.conf" + else + todo "no $_wpadir/wpa_supplicant-$i.conf -- at boot this reads:" + note " wpa_supplicant configuration file ... not present FAIL" + note " and nothing associates, so dhcpcd has no link to ask on." + _other="$(ls -1 "$SYSCONF"/wpa_supplicant-*.conf \ + "$SYSCONF"/wpa_supplicant/wpa_supplicant-*.conf \ + 2>/dev/null | head -n1)" + [ -n "$_other" ] \ + && note " you have one for another interface: $_other" \ + && note " cp '$_other' '$_wpadir/wpa_supplicant-$i.conf' && chmod 600 \$_" \ + || note " wpa_passphrase '<SSID>' '<key>' > $_wpadir/wpa_supplicant-$i.conf" + fi + fi + # dhcpcd IN THE FOREGROUND MISSES ITS OWN LINK. A wireless interface + # is up before it is associated; a dhcpcd without -b waits for a lease + # that cannot arrive yet and the boot line says FAIL. -b puts it in + # the background, where it keeps trying and succeeds a second later. + if grep -qE '^[[:space:]]*DHCP_START=' "$f" \ + && ! grep -qE '^[[:space:]]*DHCP_START=.*-b' "$f"; then + todo "$f: DHCP_START has no -b, so dhcpcd waits for a lease it cannot" + note " have yet -- that is the 'Starting dhcpcd ... FAIL' at boot." + note " sed -i 's/^DHCP_START=.*/DHCP_START=\"-b\"/' '$f'" + fi + # OURS, AND NOW WRONG. A file this step wrote may carry yesterday's + # metric after PREFER changed; one of yours is never touched. + if grep -qF "$MARKER" "$f" 2>/dev/null \ + && ! grep -qE "^DHCP_START=.*-m $_metric([^0-9]|$)" "$f"; then + if [ "$WRITE" = yes ]; then + cp -p "$f" "$f.bak-$(date +%Y%m%d-%H%M%S)" + sed -i "s/^DHCP_START=.*/DHCP_START=\"-b -m $_metric\"/" "$f" + did "$f: route metric is now $_metric" + else + todo "$f says another metric than PREFER does ($_metric)" + fi + fi +done + +# A CONFIG FOR AN INTERFACE THAT IS NOT HERE. After a rename (or a card +# taken out) its ifconfig file stays, and ifup at boot fails on a name the +# kernel does not have. Named, never deleted: it may be the adapter you +# unplugged this morning. +for f in "$SYSCONF"/ifconfig.*; do + [ -f "$f" ] || continue + case "$f" in *.bak-*) continue ;; esac + _n="$(basename "$f" | sed 's/^ifconfig\.//')" + [ -e "$NETSYS/$_n" ] && continue + grep -qE '^[[:space:]]*ONBOOT=.?yes' "$f" 2>/dev/null || continue + # AT BOOT THIS IS A RED LINE ON THE SCREEN. + # Bringing up the eth0 interface... Interface eth0 doesn't exist. FAIL + # The file is for a card that was taken out, an interface that was + # renamed, or an adapter that is simply not plugged in right now -- so + # it is never deleted here. ONBOOT=no stops the boot from trying. + if [ "${DISABLE_MISSING:-no}" = yes ]; then + cp -p "$f" "$f.bak-$(date +%Y%m%d-%H%M%S)" + sed -i 's/^[[:space:]]*ONBOOT=.*/ONBOOT="no"/' "$f" + did "$f: $_n is not here -- ONBOOT=no (it was failing at every boot)" + note " bring it up by hand when it is: ifup $_n" + else + todo "$f names $_n, which this kernel does not have -- it FAILS at boot" + note " removable, renamed, or gone? Either delete the file, or:" + note " sed -i 's/^ONBOOT=.*/ONBOOT=\"no\"/' '$f'" + note " (DISABLE_MISSING=yes in $CONF does that for every one of them)" + fi done # ---- 4b. the firewall: userspace and kernel must agree -------------------- @@ -163,7 +368,9 @@ echo # ---- 5. the wpa_supplicant configuration ---------------------------------- found_conf="" -for c in /etc/sysconfig/wpa_supplicant/wpa_supplicant-*.conf \ +for c in "$_wpadir"/wpa_supplicant-*.conf \ + "$SYSCONF"/wpa_supplicant-*.conf \ + "$SYSCONF"/wpa_supplicant/wpa_supplicant-*.conf \ /etc/wpa_supplicant/wpa_supplicant*.conf /etc/wpa_supplicant.conf; do [ -r "$c" ] && found_conf="$found_conf $c" done @@ -208,6 +415,16 @@ elif [ "$todos" -gt 0 ]; then else echo "# wireless: the pieces are all in place; bring it up with" echo "# ifup $(set -- $ifaces; echo "$1")" + if [ -n "$PREFER" ]; then + for _p in $PREFER; do + for _c in $ifaces; do + case "$_c" in $_p) + echo "# preferred while it is there: $_c (metric $METRIC_PREFERRED)" + break 2 ;; + esac + done + done + fi fi # -> lines do not fail the step; !! lines do exit $fail diff --git a/stacks/services.stack b/stacks/services.stack index a16e234..1d28750 100644 --- a/stacks/services.stack +++ b/stacks/services.stack @@ -1,5 +1,5 @@ # services.stack -- the boot-time services a SysV desktop needs. -# stack-version: 1.14.320 +# stack-version: 1.14.330 # # WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack # here: the exclusions are a property of the machine, not of one stack. @@ -52,6 +52,39 @@ git bash-completion url=https://github.com/scop/bash-completion ref=2.18.0 \ cfg shell-env \ expect='test -f /etc/bashrc && test -f /etc/locale.conf && { test ! -d /etc/profile.d || test -f /etc/profile.d/i18n.sh; }' +# BTRFS, ON A BTRFS MACHINE ONLY. `btrfs check`, `scrub`, `balance`, +# subvolumes and snapshots all come from btrfs-progs, and without it a btrfs +# root can be mounted and nothing else -- no repair, no snapshot, and +# mkinitramfs/fsck have no helper. when= asks the machine: a filesystem +# mounted now, or a partition on any disk (so it is also right in a chroot +# whose target is btrfs and in the installer, before the mount). +book btrfs-progs \ + when="grep -qw btrfs /proc/mounts || lsblk -no FSTYPE 2>/dev/null | grep -qx btrfs || blkid -t TYPE=btrfs >/dev/null 2>&1" \ + expect="command -v btrfs >/dev/null && btrfs --version >/dev/null" + +# EXFAT, WHICH ARRIVES IN A POCKET. mkfs.exfat / fsck.exfat / exfatlabel +# for the cards and sticks that come formatted that way -- no when= test, +# because the medium is not in the machine when the stack runs. +# +# NOT A BLFS PAGE. The book carries no exFAT package at all (checked +# against the stable index, 1.14.330); what it links to in passing is +# relan/exfat, the FUSE implementation -- a different thing. This is the +# userspace for the KERNEL driver, which is the modern path and the one +# this machine uses: +# lfs-kernel enable exfat # EXFAT_FS=m, then lfs-kernel --run +# relan/exfat (mount.exfat-fuse) is only worth having on a kernel without +# that driver. +# +# The release tarball is an autotools dist, so ./configure is in it; the +# build line falls back to autogen.sh if a future tarball is a bare export. +tar exfatprogs \ + url=https://github.com/exfatprogs/exfatprogs/releases/download/1.4.2/exfatprogs-1.4.2.tar.xz \ + pkg=exfatprogs-1.4.2.tar.xz \ + prog=mkfs.exfat \ + build="test -x ./configure || ./autogen.sh; ./configure --prefix=/usr --sbindir=/usr/sbin; make" \ + install="make install" \ + expect="command -v mkfs.exfat >/dev/null && command -v fsck.exfat >/dev/null" + book acpid # power/lid/button events -- what a laptop needs # ONE GROUP CARRIES THE PRIVILEGE. halt/poweroff/reboot must become root, so