diff --git a/.github/workflows/test-sanitizers.yml b/.github/workflows/test-sanitizers.yml index 11a339cd3..c6b3d8db0 100644 --- a/.github/workflows/test-sanitizers.yml +++ b/.github/workflows/test-sanitizers.yml @@ -65,7 +65,7 @@ jobs: - name: Run tests shell: bash run: | - make -j$procs test TARGETS= EXTRA_TARGETS= + make -j$procs vanilla-test TARGETS= EXTRA_TARGETS= - name: Report errors if: ${{ failure() }} @@ -76,4 +76,4 @@ jobs: - name: Run unit tests shell: bash run: | - make -j$procs unit-test ENABLE_LIBYOSYS=1 + make -j$procs unit-test diff --git a/.github/workflows/test-verific.yml b/.github/workflows/test-verific.yml index adc6f59d8..feba3c0f9 100644 --- a/.github/workflows/test-verific.yml +++ b/.github/workflows/test-verific.yml @@ -68,7 +68,7 @@ jobs: - name: Run Yosys tests run: | - make -j$procs test + make -j$procs vanilla-test - name: Run Verific specific Yosys tests run: | @@ -83,7 +83,7 @@ jobs: - name: Run unit tests shell: bash run: | - make -j$procs unit-test ENABLE_LTO=1 ENABLE_LIBYOSYS=1 + make -j$procs unit-test ENABLE_LTO=1 test-pyosys: needs: pre-job diff --git a/Makefile b/Makefile index 364e1ce8d..7ce3153df 100644 --- a/Makefile +++ b/Makefile @@ -977,9 +977,11 @@ makefile-tests/%: %/run-test.mk $(TARGETS) $(EXTRA_TARGETS) $(MAKE) -C $* -f run-test.mk +@echo "...passed tests in $*" -test: makefile-tests abcopt-tests seed-tests +test: vanilla-test unit-test + +vanilla-test: makefile-tests abcopt-tests seed-tests @echo "" - @echo " Passed \"make test\"." + @echo " Passed \"make vanilla-test\"." ifeq ($(ENABLE_VERIFIC),1) ifeq ($(YOSYS_NOVERIFIC),1) @echo " Ran tests without verific support due to YOSYS_NOVERIFIC=1." diff --git a/README.md b/README.md index 3b2f41768..df65a6a10 100644 --- a/README.md +++ b/README.md @@ -114,8 +114,8 @@ To build Yosys simply type 'make' in this directory. $ sudo make install Tests are located in the tests subdirectory and can be executed using the test -target. Note that you need gawk as well as a recent version of iverilog (i.e. -build from git). Then, execute tests via: +target. Note that you need gawk, a recent version of iverilog, and gtest. +Execute tests via: $ make test diff --git a/docs/source/yosys_internals/extending_yosys/test_suites.rst b/docs/source/yosys_internals/extending_yosys/test_suites.rst index 81a79e77f..c43dc3a84 100644 --- a/docs/source/yosys_internals/extending_yosys/test_suites.rst +++ b/docs/source/yosys_internals/extending_yosys/test_suites.rst @@ -8,7 +8,43 @@ Running the included test suite The Yosys source comes with a test suite to avoid regressions and keep everything working as expected. Tests can be run by calling ``make test`` from -the root Yosys directory. +the root Yosys directory. By default, this runs vanilla and unit tests. + +Vanilla tests +~~~~~~~~~~~~~ + +These make up the majority of our testing coverage. +They can be run with ``make vanilla-test`` and are based on calls to +make subcommands (``make makefile-tests``) and shell scripts +(``make seed-tests`` and ``make abcopt-tests``). Both use ``run-test.sh`` +files, but make-based tests only call ``tests/gen-tests-makefile.sh`` +to generate a makefile appropriate for the given directory, so only +afterwards when make is invoked do the tests actually run. + +Usually their structure looks something like this: +you write a .ys file that gets automatically run, +which runs a frontend like ``read_verilog`` or ``read_rtlil`` with +a relative path or a heredoc, then runs some commands including the command +under test, and then uses :doc:`/using_yosys/more_scripting/selections` +with ``-assert-count``. Usually it's unnecessary to "register" the test anywhere +as if it's similar to other tests it will be run together with the rest. + +Unit tests +~~~~~~~~~~ + +Running the unit tests requires the following additional packages: + +.. tab:: Ubuntu + + .. code:: console + + sudo apt-get install libgtest-dev + +.. tab:: macOS + + No additional requirements. + +Unit tests can be run with ``make unit-test``. Functional tests ~~~~~~~~~~~~~~~~ @@ -41,23 +77,6 @@ instructions `_. Then, set the :makevar:`ENABLE_FUNCTIONAL_TESTS` make variable when calling ``make test`` and the functional tests will be run as well. -Unit tests -~~~~~~~~~~ - -Running the unit tests requires the following additional packages: - -.. tab:: Ubuntu - - .. code:: console - - sudo apt-get install libgtest-dev - -.. tab:: macOS - - No additional requirements. - -Unit tests can be run with ``make unit-test``. - Docs tests ~~~~~~~~~~