Rewrite testing adapter

[kimadeline]

Meta-issue: #16936

Unittest

• Test discovery with unittest - Unittest discovery using the new test adapter #18253
• Test execution with unittest - Refactor unittest test execution #19098
• Test debugging with unittest

Pytest

• Prototyping pytest plugin Prototype pytest plugin for test discovery/execution. #19790
• Add pytest plugin Review and Commit Pytest Plugin (without enabling) #20077
• Test discovery with pytest
  • Write discovery logic for pytest plugin Finalize Pytest Discovery Logic  #20078
  • Display discovered test on UI Connect Pytest Discovery Plugin to Testing UI #20079
• Test execution with pytest
  • Create pytest run tests call Create Pytest Plugin Call for Run Test #20080
  • Write run tests plugin logic Pytest Run Tests Logic #20081
  • Display run test info on UI Surface Test Results to UI #20082
• Pytest error handling Error Handling Pytest Discovery #20083
• Complete pytest testing suite Write testing for pytest discovery #20085
• Enable pytest rewrite to users Enable Pytest Rewrite to Users #20086
• Test debugging with pytest

Cleanup

• Clean up unittest-related issues
• Clean up pytest-related issues
• Make sure there are tests for the test controller after the rewrite

Notes

Public issues

1. python.testing.cwd ignored when discovering or running tests #8678
2. Tests not discovered, but running tests anyway works fine. #16565
3. Handle exclude lists in a test run request #16929
4. Go to Test doesn't work for test class items in Test Explorer #17107
5. Tests are discovered correctly but then an incorrect argument is supplied to PyTest, leading test-result discovery to fail (Git Submodules?) #17903
6. Unittest discovery not working when sys.stdout is set in global scope #17957

Issues we expect this rewrite will fix

• Problem with test IDs
  • Folders named 'py' result in "Test result not found" #17414
  • VSCode not running discovered PyTest tests when path contains a symlink #17573
  • "Test result not found" if cwd is a subdirectory in the workspace #18030
  • Escape test ID`s in Testing API #17810
• Problems with other plugins or python or libs printing to output
  • Pytest discovery stuck in Output panel #18656
• Problems with test discovery
  • pytest discovery error #19809
  • Improve python test discovery error reporting #19978
  • Group test explorer test by workspace in multi-root workspace #20345
  • unexpected situation while processing a pytest node during test discovery #19804
  • Test discovery should be fault tolerant and display all successfully imported files #13301
• Problems with testing output
  • Debug output for test discovery has misleading argument #19820
  • Support an output stream from the testing adapter script. #6594
• Problems with remote setups
  • After switching to devcontainer (based on 3.10-bullseye python) can't go to test or run test #19857
  • Pytest test discovery over SSH does not work #19864
• Problems with testing args
  • PyTest optional args don't support wildcards #19918
  • Having PYTHONWARNINGS=default crashes pytest discovery #20259
  • Python unittest runner doesn't respect arguments to pass to unittest runner #20425
• Performance Concerns
  • Testing: pytest should be running multiple nodes in one single process #19796
• Problems with Running Tests
  • Use pytest hooks for test execution instead of "scraping" the JUnit XML results. #12973

pytest feedback on how to things the "right way"

• What is the preferred way for editors/tools to interact with pytest? pytest-dev/pytest#8458
• What is the minimum amount of pytest's CLI that should be supported by integrating editors? pytest-dev/pytest#8506
• visualstudio_py_testlauncher.py - format_exception() missing 3 required positional arguments: 'etype', 'value', and 'tb' #3305
• Red test failure output in editor gets wrapped over lines #18047

unittest missing features

• support for subtests status unittest runner thinks failing test is not run #17561
• Unittest tests will not run with unittest.main() in the test file #17342
• should_never_reach_here for particular test case when parametrized test has unmatched brackets or parenthesis #17676

[Bilalh]

It would be nice if other python testing frameworks such as behave (a BDD framework) https://behave.readthedocs.io/en/stable/index.html were supported by the testing adapter.

[kimadeline]

Hi @Bilalh, support for behave is not within scope of this issue. Feel free to open a feature request on the repo and we'll take a look at it. Thanks!