Significance of failed tests in Emacs 28.1 ("make check" et al.)

For the past few months, I've used Emacs 28.1 (compiled from the official release tarball) with no *apparent* bugs. I've primarily used Debian Unstable, but I've also tried this on Fedora 36 and Arch. Changing from gcc11 to gcc12 doesn't change the situation. It seems that no matter what dependencies I install or which configuration options I provide, my compiled Emacs will never pass a handful of the tests provided by `make check-all`. Why is this the case? Has anyone managed to run 'make check-all' without any resulting failures, and if so, how? Are the failed tests in levels above 'make check' telling me anything significant? --Editing to fix formatting and (hopefully) simplify my question-- Here's the result of running make check-all after compiling with the `--with-xwidgets` option: SUMMARY OF TEST RESULTS ----------------------- Files examined: 375 Ran 5568 tests, 1 failed to run, 5456 results as expected, 13 unexpected, 99 skipped 1 files did not finish: misc/test-custom-noloads.log 7 files contained unexpected results: src/thread-tests.log lisp/progmodes/python-tests.log lisp/mail/rmailmm-tests.log lisp/emacs-lisp/cconv-tests.log lisp/cedet/srecode/fields-tests.log lisp/cedet/srecode/document-tests.log lisp/autorevert-tests.log make[2]: *** [Makefile:336: check-doit] Error 2 make[2]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make[1]: *** [Makefile:315: check-all] Error 2 make[1]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make: *** [Makefile:994: check-all] Error 2 All the tests that fail are performed only when I run `make check-all`. They aren't run for `make check` or `make check-expensive`. I'll post more information in the comments shortly.

For the life of me I couldn't get the code to display properly on [old.reddit.com](https://old.reddit.com), but it should work correctly on the 'new' reddit.

It is because you're using backticks instead of indenting the text you want to be in a code block. Both methods work on new reddit, but only indenting works on old reddit.

First, at least one of the results above says "0 unexpected", which means success. And you don't say which tests failed, so it's hard to say anything intelligent about the possible reasons.

As /u/TheKrister2 says, the *only* code formatting method which works for all of reddit is to switch the reddit editor to markdown mode and indent the code by 4 spaces (you can use `M-4 C-x C-i` on a region followed by `M-x untabify` to achieve this in Emacs). Remove backticks and/or any other code formatting syntax.

I think my initial post was overly complicated. My intent was to show that the failed tests are only performed when I run `make check-all`, and not when I run `make check-expensive` or `make check`. That's why I included multiple test summaries, of which some showed '0 unexpected'. It should be easier to follow now. Here are two questions that I hope will help me understand the situation: * When building Emacs from source, should a user expect to only get 'expected' results for each test run by `make check-all`? * Does the fact that these tests are included only in `check-all` say anything about their importance? Or does it say only that they take a long time to run? Below is information about which tests failed. Happy to provide any other info that might prove useful. From src/thread-tests.log: 1 unexpected results: FAILED threads-join-error From lisp/progmodes/python-tests.log: 1 unexpected results: FAILED python-tests--python-nav-end-of-statement--infloop From lisp/mail/rmailmm-tests.log: 2 unexpected results: FAILED rmailmm-test-bulk-handler FAILED rmailmm-test-multipart-handler From lisp/emacs-lisp/cconv-tests.log: 2 unexpected results: PASSED cconv-tests-cl-iter-defun-:documentation PASSED cconv-tests-iter-defun-:documentation From lisp/cedet/srecode/fields-tests.log: 1 unexpected results: FAILED srecode-field-utest-impl From lisp/cedet/srecode/document-tests.log: 1 unexpected results: FAILED srecode-document-function-comment-extract-test From lisp/autorevert-tests.log: 5 unexpected results: FAILED auto-revert-test01-auto-revert-several-files-remote FAILED auto-revert-test02-auto-revert-deleted-file-remote FAILED auto-revert-test04-auto-revert-mode-dired-remote FAILED auto-revert-test05-global-notify-remote FAILED auto-revert-test06-write-file-remote

Thanks. I think it was the need for an empty line before the indented lines that was throwing me off. Should now be correctly formatted.

Thanks. I think it was the need for an empty line before the indented lines that was throwing me off. Should now be correctly formatted.

> the need for an empty line before the indented lines Thanks, I'll add that to my standard reply for this issue :)

`make check-all` runs *all* the tests, including ones that might randomly fail. The normal thing to try is `make check`, which runs 5444 tests for me here.

This is really helpful, thanks. That resolves my concerns about the unexpected results from `make check-all`. Does your advice also apply to tests run by `make check-expensive`, but not `make check`? I'd like to understand an unexpected test result that appears when I add `--with-native-compilation` to my configuration flags and then run `make check-expensive`. Again, this unexpected test result appears consistently on all distributions I've tried. SUMMARY OF TEST RESULTS ----------------------- Files examined: 375 Ran 5699 tests, 5602 results as expected, 1 unexpected, 96 skipped 1 files contained unexpected results: lisp/net/tramp-archive-tests.log make[2]: *** [Makefile:336: check-doit] Error 1 make[2]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make[1]: *** [Makefile:310: check-expensive] Error 2 make[1]: Leaving directory '/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test' make: *** [Makefile:994: check-expensive] Error 2 Unfortunately, I'm not yet able to understand the backtrace in `tramp-archive-tests.log`. Here are the relevant lines from that file: Test tramp-archive-test45-auto-load backtrace: signal(ert-test-failed (((should (string-match (format "tramp-archiv ert-fail(((should (string-match (format "tramp-archive loaded: %s[[: (if (unwind-protect (setq value-910 (apply fn-908 args-909)) (setq f (let (form-description-912) (if (unwind-protect (setq value-910 (app (let ((value-910 'ert-form-evaluation-aborted-911)) (let (form-descr (let* ((fn-908 #'string-match) (args-909 (condition-case err (let (( (let ((file (car --dolist-tail--))) (let* ((fn-908 #'string-match) ( (while --dolist-tail-- (let ((file (car --dolist-tail--))) (let* ((f (let ((--dolist-tail-- (list "/mock::foo" (concat tramp-archive-test (let ((default-directory (car --dolist-tail--))) (let ((--dolist-tai (while --dolist-tail-- (let ((default-directory (car --dolist-tail-- (let ((--dolist-tail-- (list temporary-file-directory (file-name-as- (let ((code "(progn \11 (message \"tramp-archive loaded: %%s\" . (let ((lexical-binding t)) (let ((value-901 (gensym "ert-form-evalua (closure (tramp-copy-size-limit t) nil (let ((lexical-binding t)) (l ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name tramp-archive-test45-auto-load :docum ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m ert-run-tests((not (tag :unstable)) #f(compiled-function (event-type ert-run-tests-batch((not (tag :unstable))) ert-run-tests-batch-and-exit((not (tag :unstable))) command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/tramp-archive-te command-line() normal-top-level() Test tramp-archive-test45-auto-load condition: (ert-test-failed ((should (string-match (format "tramp-archive loaded: %s[[:ascii:]]+tramp-archive loaded: %s" ... ...) (shell-command-to-string ...))) :form (string-match "tramp-archive loaded: t[[:ascii:]]+tramp-archive loaded: t" "Cannot open load file: Too many open files, debug\nError in kill-emacs-hook (#[0 \"\\301\\300\\302\\\"\\207\" [\"/home/src/emacs/emacs-28.1-tarball/emacs-28.1/test/lisp/net/tramp-archive-resources/foo.iso/emacs-testsuite-gH3jR3\" delete-directory t] 3]): (file-error \"Cannot open load file\" \"Too many open files\" \"tramp-archive\")\n") :value nil)) FAILED 21/22 tramp-archive-test45-auto-load (11.039797 sec) ... Ran 22 tests, 21 results as expected, 1 unexpected (2022-07-12 11:27:11-0400, 172.927142 sec) ... 1 unexpected results: FAILED tramp-archive-test45-auto-load Do you have any insight into why this test fails for builds compiled using `--with-native-compilation`?

You should only ever run `make check`. All the rest is for those pointy-hairy developers.

That pretty much settles it, then. Much appreciated.