* Added the documentation regarding the new libraries test modes.
* Enable the fundamentals behind libraries tests compiled via crossgen2.
* Fixed a typo with SINGLE_FILE_TEST_RUNNER.
* Addressed review comments.
* Restored an accidentally deleted comment.
* Fixed wrong Crossgen2 path and added comment with link to an important tracking SDK bug.
* Draft.
* DevTools as IDE instructions.
* VS Code as IDE instructions.
* Correction: `WebRoot` is not required.
* Added: expect long wait & new pic without webRoot
* Renamed as suggested by @pavelsavara.
* The first bp is not automatic, user has to set it on their own.
* Set target SDK version
* Update Android docs
* Disable tests that do not pass with target API 31
* Check if the installed Android SDK is up-to-date
* Update skip explanation
* Use latest SDK
* Disable failing System.Net.Security test
Follow-up on #32451
Follow-up on #80074
In #43651, the "Windows" OS name was lowercased to achieve target frameworks with RIDs in them. At that time, the original plan was to lowercase all TargetOS values but that had to be cut because of resources. This PR finishes that and updates build systems, YML, markdowns and managed components.
To not break existing developer workflows, the passed-in -os value is automatically lower-cased in the script entry points and validated in the msbuild entry point.
These fixes were built for PR #74886; however, as that PR is so utterly large and unreviewable, I've pulled out the test infra changes for separate review
Changes
- Increase the number of trampolines in the llvm aot compilation process to 20,000 from 10,000 (This avoids running out of them in some of the hardware intrinsics tests
- Add a concept of striping tests when running under GC stress
- To use this new feature, specify <NumberOfStripesToUseInStress>N</NumberOfStripesToUseInStress> within the merged test assembly. If this value is set, then the tests within that merged test assembly will be run across N different work items instead of 1 when running under any form of GC stress based scenario. At this moment the largest supported value of N is 99
- Emit the testresults.xml file as a file which is exported from the tests. This is useful for debugging testresult.xml parsing failures
- Fix the testresults summary generator to never emit an empty CDATA string. If one is present the parser may fail the parse.
- In the XUnitWrapperGenerator fix the implementation of the Outerloop and ActiveIssue when used with a conditional member.
- Add PlatformDetection.IsMonoLLVMAOT, PlatformDetection.IsMonoLLVMFULLAOT, and PlatformDetection.IsMonoInterpreter boolean properties to the PlatformDetection type for use with the ActiveIssue attribute
- Add some documentation about project files for coreclr tests, as well as some documentation on the command line parameters for merged test runner assemblies
* Quality Week 2022: The First Step to a Better Documentation.
* Removed the Markdown Extension's additional development tags.
* Removed redundant stuff.
* Removed redundant stuff.
* Removed guide testing versions to avoid confusion that they would be the only supported versions.
* Addressed some Powershell comments, added a pointer to a yaml with links to other .NET sibling repos, and did a couple mini-corrections. Still got a few comments to address...
* Fixed a ';:' with Powershell's syntax.
* Addressed review comments: Redaction improvements, localization url's, improved code snippets, added info on individual subsets building configurations, changed COMPlus for DOTNET, and other misc improvements.
* Addressed remaining comments.
* Addressed some last comments regarding corerun, and the subset flags in the build scripts.
Co-authored-by: Ivan Diaz <bluehorizon186@gmail.com>
* Add test
* Add CHECKs
* Build scripts and test settings
* COMPlus -> DOTNET
* Fix filenames
* Missing >
* Missing quotes
* use %scriptPath%
* Rework properties, start two examples
* Arch demo
* Cleanup
* Remove tab
* Easier environment variables. Undo precommand changes.
* undo blank line
* clean CHECKs
* Draft of bash
* Bash, conditionals
* More variables, start on run.cmd/sh/py
* another output
* Support in cmd/bash for RunningDisasmChecks
* copy, factor, formatting
* Initial work to include FileCheck. Added SuperFileCheck.
* Able to build SuperFileCheck
* Do not DisasmCheck TypeEquality_r for now. Update some FileChecks to follow SuperFileCheck rules.
* Partially wiring up SuperFileCheck to tests
* Piping list of method names from SuperFileCheck to JitDisasm
* Handling bash a little bit
* Moving SuperFileCheck to tests/Common
* Few tweaks
* Building SuperFileCheck as part of the test build
* Tweaking a few things
* Fixed a bug
* Moving SuperFileCheck back to src\coreclr\tools. Removed checks from TypeEquality_r.
* Restore original logic in Runtime_73681
* Trying to add CI leg for disasmchecks
* Use x64 package if x86 platform detected for JIT tools package
* Remove innerloop for disasmchecks
* Trying to fix build. Only run in Windows for now.
* Update Runtime_73681.cs
Trying to fail test
* Trying to fix build
* Update Runtime_73681.cs
* Update Runtime_73681.cs
* Fixing a few issues
* Trying to run disasmchecks as part of CI
* Trying to run disasmchecks
* Trying to run disasmchecks
* Trying to run disasmchecks
* Revert a change
* Trying to run disasmchecks
* Trying to run disasmchecks
* build SuperFileCheck on non-windows
* few tweaks
* Trying to fix CI
* Including SuperFileCheck for tests
* Cleanup
* More cleanup
* Cleanup
* Changed SuperFileCheck to not publish everything. Changed SuperFileCheck's lookup for FileCheck.
* Invoking SuperFileCheck using dotnet
* Making the test pass
* Only run disasm checks for coreclr and not mono
* Using HasBatchDisasmCheck and HasBashDisasmCheck to determine to run the check
* Enabling filecheck on linux and osx
* Added more comments
* Added ARM64 specific test. Do not run SuperFileCheck if no methods were found.
* Added documentation. Changed disasm-output.
* Minor doc tweak
* Minor doc tweak
* Minor doc tweak
* Minor doc tweak
* Minor doc tweak
* Cleanup. Trying to fix linux
* Fixing test
* Add information on additional functionality
* cleanup
* Add FileCheck snippet
* Undo environment variable changes
* Feedback from Mark
* Cleanup
* Trying to fix linux test run
* Trying to fix linux test run
* A few missing changes from the original branch
* Enable OSX for disasm checks
* cleanup / comment
* Force test failure
* Update Runtime_73681.cs
* Set env vars after errorlevel check
* Reverting back on setting environment variables in test. Added new FileCheck test for mod optimization
* Force a failure by changing the register
* Ignore native binaries for superpmi collect
* Update Runtime_34937.cs
* Force the correct failure
* Update Runtime_34937.cs
* Update Runtime_34937.cs
* Adding specific OS check prefixes. Changed dump-input context amount
* Added getting fully qualified method names with wildcards for SuperFileCheck
* More tests. Fixed a few issues with generics.
* Disabling generic support
* Error if it cannot find enclosing type declaration
* Fixing build
* Remove namespac
* Bring generics back, but in a limited form
Co-authored-by: Mark Plesko <markples@microsoft.com>
- This will help simplifying the path subsets in
`eng/pipelines/common/evaluate-default-paths.yml`, which would then help
to avoid a lot of unrelated non-wasm builds on PRs
* [wasm] Modify workload to pick threading runtime packs
This change adds the 2 wasm threading runtime packs to the wasm workload. In order for a threading runtime pack to be chosen, WorkloadManifest.targets is also modified to override the runtime pack name when the following props are set:
WasmEnableThreads - full threading support and will load Microsoft.NETCore.App.Runtime.multithread.Mono.browser-wasm
WasmEnablePerfTrace - runtime only threading support and will load Microsoft.NETCore.App.Runtime.perftrace.Mono.browser-wasm
* Add error condition when both WasmEnableThreading and WasmEnablePerfTrace are true
* [wasm] Set default value for MonoWasmBuildVariant, for runtime pack nuget
* [wasm] Wasm.Build.Tests: build runtime pack nugets with different names
- The earlier approach of simply making copies of the existing runtime
pack nuget with different names doesn't work, and `dotnet workload
install` rejects it.
```
Installing pack Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm version 7.0.0-ci...
Workload installation failed. Rolling back installed packs...
```
Instead, now we build the missing nugets from the project with different
values for `$(MonoWasmBuildVariant)`.
- this handles local builds, and incremental builds also
- To skip building the missing nugets, for example, when you have all of
them available, then set `WasmSkipMissingRuntimePackBuild=true`.
* fix comment
Co-authored-by: Steve Pfister <steve.pfister@microsoft.com>
Co-authored-by: Ankit Jain <radical@gmail.com>
* Add document describing how to disable tests
Update libraries "filtering tests" doc with information and examples for
using `SkipOnCoreClrAttribute`
* Update for feedback
Added SkipOnMonoAttribute, CollectionAttribute, an example
of using multiple SkipOnCoreClrAttribute, and how the xunit
attributes for test disabling apply to the "converted"
src/tests tree.
* Use CMakeProjectReference instead of ProjectReference
Changing this permits usage of "\runtime\dotnet.cmd build" as opposed
to forcing users to use "\runtime\dotnet.cmd msbuild".
- Move all SetScriptCommands to a target, so they are placed after scenario in the generated script.
- Add WasmXHarnessArgsCli as an alternative variable that can be used from cli.
- Fix NodeJS symbolic links on Helix.
- Copy TestEchoMiddleware and RemoteLoopMiddleware even for Scenario=NodeJS.
- Enable System.Net.WebSockets.Client.Tests on NodeJS.
* Modify existing tests
* Enable process start and kill on MacCatalyst
* Typo
* Temporarily enable ProcessTests for MacCatalyst
* Temporarily enable ProcessTests for MacCatalyst - attempt 2
* Allow running Mac Catalyst builds in App Sandbox
* Allow enabling app sandbox for the other Mac Catalyst sample
* Add missing parameter to XCode project generator
* Remove unnecessary MacCatalyst detection
* Enable App Sandbox for Mac Catalyst tests
* Create a separate test branch for App Sandbox
* Remove the restriction to enable app sandbox just for Mac Catalyst apps
* Do not throw PNSE for Mac Catalyst
* Clean-up platform-specific conditions
* Build correct Process implementation for MacCatalyst
* Try to get more information from CI for further investigation of the failing build
* Revert "Try to get more information from CI for further investigation of the failing build"
This reverts commit fc63a370d688aa0d1dc890e6928a8d031d148e56.
* Add MacCatalyst target framework for System.Diagnostics.Process
* Add supported platform annotations for Mac Catalyst
* Fix annotations
* Remove incorrectly placed attributes
* Update attributes including the reference file
* Change platform attributes
* Generate app.entitlements in the BuildAppleAppBundles test build target
* Revert "Generate app.entitlements in the BuildAppleAppBundles test build target"
This reverts commit abbe2241586e3469c3d3e9d4ecd2cea2e09d3641.
* Enable AppSandbox when generating CMakeLists.txt for libraries tests
* Try implementing a workaround for app sandbox mode
* Fix app sandbox detection bug
* Add explanation comment
* Enable more tests for MacCatalyst
* Add apple app builder input validation
* Add Mac Catalyst w/ App Sandbox enabled to runtime-manual test pipeline
* Enable networking in App Sandbox mode
* Skip test which won't work on Mac Catalyst
* Skip some tests in app sandbox mode
* Update src/tasks/AppleAppBuilder/AppleAppBuilder.cs
Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
* Add a clarification comment for a networking entitlement
* Simplify supported platform condition
* Remove temporarily enabled test
* Remove unnecessary attributes
* Replace checking env variable with checking errno in libproc
* Update docs
* Remove unnecessary changes
* Temporarily enable running System.Diagnostics.Process.Tests for this PR
This reverts commit 02d370c2cf730bf67a3cf6fca8fcb8c5e4ff1ccd.
* Revert "Temporarily enable running System.Diagnostics.Process.Tests for this PR"
This reverts commit dc72f0f1d068314317d088d33ec87f9aa6447854.
* Fix job suffix in runtime-manual
* Remove attributes
* Revert "Remove attributes"
This reverts commit 704e9fa73cd40854d1becc2889569bab3c47d62d.
* Try changing the order of attributes to please CI
* Update ApiCompat baseline
* ApiCompat step 1: remove all attributes for MaxWorkingSet setter
* Revert "Update ApiCompat baseline"
This reverts commit a2ad03207c5f9f40b9dd2c335a1b0f31c37a0ac5.
* Revert "ApiCompat step 1: remove all attributes for MaxWorkingSet setter"
This reverts commit be72a3d2564afb89a83198ef010b3d4622799601.
* Update ApiCompat baseline
* Update ApiCompat baseline after pulling upstream main
* Remove trailing whitespace
Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
* [tests] If -debug is LLDB, add separator before corerun and args
For runtime test shell scripts, if a debugger is specified with `-debug=/usr/bin/lddb` or the like, add a separator so that `lldb` doesn't try to interpret `-p` as a PID argument.
Also update mono workflow doc
* Also support gdb as the debugger
* Remove EOL tfms and upgrade to net7.0
* Make linker tests net7 aware
Make installer tests run on net7.0
Don't attempt to use live apphost
Fix payloads
Regenerate test files
Fix package testing
* Implement support for passing a .env file to corerun to easily set environment variables for stress testing.
* Load the parsed dotenv into the process.
* Implement support for # comments
* Clean up code
* Avoid copy
* PR feedback
* Update docs and bash/batch scripts to support the dotenv files.
* Use what clang recommends to not copy
* Apply suggestions from code review
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
New tests that extend the System.Private.CoreLib API should add to the ref assemblies in src/libraries and the tests themselves should be library tests.
Updated the workflow doc: the Clr.Native subset is all that is need if we're interested in testing Mono.
Contributes to https://github.com/dotnet/runtime/issues/58266
* [tests] Use RuntimeFlavor to build ReferenceSystemPrivateCoreLib tests
When building tests that reference System.Private.CoreLib, use the one from the specified RuntimeFlavor (or fall back to coreclr if it's unset - although src/tests/build.sh sets it always)
* Update docs: We only need to build the Clr.Native subset if we're interested in testing Mono.
* flavor the pipelines to build the runtime-specific System.Private.CoreLib for the runtime tests.
* Don't build ReferenceSystemPrivateCoreLib tests with Mono
We should avoid adding new ReferenceSystemPrivateCoreLib tests
This change adds four new options to the runtime test build command:
1) test <test project path relative to src\tests> - build just that
one test.
2) dir <directory relative to src\tests> - build all tests in the
directory.
3) tree <directory relative to src\tests> - build all tests in the
given subtree.
4) all - use clean rebuild (i.e. don't apply incrementalism)
when building the tests.
Thanks
Tomas
* Update instructions for running runtime tests on Android
* Update docs/workflow/testing/mono/testing.md
Co-authored-by: Juan Hoyos <juan.hoyos@microsoft.com>
Co-authored-by: Juan Hoyos <juan.hoyos@microsoft.com>
* Build IJW assemblies against LKG SDK ref pack.
* Allow building against the live ref assemblies by passing `-cmakeargs -DCPP_CLI_LIVE_REF_ASSEMBLIES=1` to the test build script.
* Add test instructions
* Error handling
* Add support for the stdcall-mangled _CorDllMain entrypoint on x86 since MSVC uses it if it exists.
* Write out ref-pack path.
