* Add warning on unmaintained testing doc page
* Update testing.md
Some example text that seems more clear to me, but only offered as a suggestion.
Feel free to adjust it or if you want to use it as-is please double check what I wrote is accurate : )
I think the useful elements are:
1. Being explicit about what workflow steps need to happen in total
2. Being explicit about which commands are covering the entire workflow and which ones are only covering a part of it
3. Show the simple "do-it-all" options first before showing more complex partial options. Glancing at the first example and blindly copying it should land in the pit of success.
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
* Update testing docs with x86 instructions
* Apply suggestions from code review
Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com>
* add more examples
* Apply suggestions from code review
Co-authored-by: Dan Moseley <danmose@microsoft.com>
* Update docs/workflow/testing/libraries/testing.md
Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com>
Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
* corerun supports user defined properties from command line.
* Update workflow help doc for corerun.
Co-authored-by: Elinor Fung <elfung@microsoft.com>
Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.
## This is done by:
1. building the test assemblies on the build machine
- the wasm part of the build is not executed on the build machine,
because it has the AOT build part
2. Zip up the test assembly+friends, *and* any bits required to run the wasm
app build for that on helix (eg. emsdk, wasm app targets, cross compiler etc)
3. Send all this to helix, and use a custom `aot-build.proj`
- which recreates all the build inputs for the `WasmBuildApp` target
using the paths for the assets on helix
- then we can run `WasmBuildApp` for the build, resulting in a wasm app
bundle.
4. Run the tests!
- We already have the bits required for building wasm apps on helix, supported
for `Wasm.Build.Tests`, which we can use here too.
## Trimming:
- Since, AOT can be so expensive, we use `EnableAggressiveTrimming=true`(EAT), but
that means that we could have issues due to trimming.
- And it can sometimes be unclear whether the build/test failures are due to trimming
or AOT.
- Because these builds+test runs are different from other builds, owing to the
"build partially on helix" step, a normal EAT build would not be the same as
- to help with testing this, we add two lanes to `runtime-staging`:
- `*_Mono_AOT`: builds AOT+EAT on helix
- `*_Mono_EAT`: builds EAT, on helix
- this is required because we want to run almost the same kinda
build: 1. build test assembly; *2. send to helix; 3. build wasm app;* 4. run tests
- This should effectively mean that we can see which errors might be due to EAT, and
which are clearly because of EAT+AOT.
Co-authored-by: Steve Pfister <steve.pfister@microsoft.com>
Co-authored-by: Marek Safar <marek.safar@gmail.com>
Co-authored-by: Mitchell Hwang <mitchell.hwang@microsoft.com>
Co-authored-by: Ankit Jain <radical@gmail.com>
We have a bunch of test assemblies that don't make sense on some platforms, e.g. Browser.
Right now we're skipping them via `[SkipOnMono("reason", TestPlatforms.Browser)]` but there's nothing that inherently ties this to Mono other than the current implementation.
The more generic `SkipOnPlatform` attribute can be used instead.
We can also use it in places where we do `[PlatformSpecific(~TestPlatforms....)]` to avoid the double inversion.
* Add documentation on running tests using different Browsers other than chrome.
- XHARNESS_COMMAND="test-browser --browser=safari"
- Add ability to run against different browsers from the make file target `run-browser-tests` by adding `XHARNESS_BROWSER= ` to command line.
At the moment supported values are:
- `chrome`
- `safari`
- `firefox`
* Add gecko firefox requirements to testing docs.
* Improve generated dump debugging instructions
* Update eng/testing/debug-dump-template.md
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* Update eng/testing/debug-dump-template.md
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* Update eng/testing/debug-dump-template.md
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* Update debug-dump-template.md
* Fix ```
* fix some ```cmd
* Reverse .cmd to .bat
* Open only for reading
* Improve foldername
* Install only one SOS
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
The current framework relays on CoreCLR to create the Core_Root and then patch it with mono runtime. So building CoreCLR is required before building runtime tests for desktop Mono
* Fix various typos
* `occurence`, `occurance` -> `occurrence`
* `accross` -> `across`
* `adddress`, `addresss` -> `address`
* `alignement`, `aligment` -> `alignment`
* Remove `c` from triple c.
* Remove `s` from triple s.
* In palsuite, `Successs` (with triple s) was in commented out
`Trace()` message, removed all commented out `Trace()` calls
from such files.
* Fix invalid markdown in `unix-test-instructions.md`.
* Revert event source change
* Move the script runtesttilstable.sh under src/tests
* Update most references to build-test / runtest in the runtime repo
With this change I can't find any remaining orphaned references to
the old script names except for a few superpmi-related scripts
I need to follow up on separately as I'm not too familiar with
their functionality.
Thanks
Tomas
Refactor the PAL test framework to be useable in our CI system, and enable it for testing
The existing PAL tests are problematic in a few ways for our CI.
1. They are large in terms of disk space used per test, this interferes with building them properly, especially as they are part of the product build, and not of the test build
2. While part of the product build, the option to build them was well hidden.
3. The tests are not related to our existing tests, and in particular are not driven from managed xunit wrappers.
The change here has 4 components
1. Reduce the volume of the binaries such that they can be passed from a product build to a test job effectively. This is done by merging all tests which are not strictly dependent on secondary libraries.
- This was done via a tool which parsed and replaced all of the CMakeFiles.txt
- And then manual editing to remove all duplicate global symbols
2. The build flag for building the tests has been exposed as a first class option to build-runtime, as well as through the subset mechanism. As of this change, developers who wish to build the tests should use the clr.paltests subset to build the tests.
3. As the tests do not have normal xunit wrappers, but they *do* have a script which will generate xunit output, they cannot quite be run in the existing helix pathway. Add a separate path for launch the appropriate helix workitem as part of outerloop test runs.
4. A new issue exclusion mechanism has been built. Add exclusions by modifying src/coreclr/src/pal/tests/palsuite/issues.targets
- There are a number of failures in the current test suite that should probably be investigated, but making meaningful changes to the tests should not be part of this change.
Add a subset for compiling only the wasm runtime: `mono.wasmruntime`
This allows us to rebuild the wasm runtime without building the full libraries subset.
Updated documentation with new guidance.
Updated how we build the dotnet.timezones.blat.
* Trim trailing whitespaces
* Match raw with rendered
* Delete extra asterisks and |
* Update ELT Hooks - tail calls.md
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
I am not very familiar with Unix so I was not aware why I was having below error:
```
/home/jozky/runtime/.dotnet/sdk/5.0.100-preview.8.20362.3/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(379,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.browser-wasm was not downloaded. Try running a NuGet restore with the RuntimeIdentifier 'browser-wasm'. [/home/jozky/runtime/src/libraries/System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj]
```
I tried running `dotnet restore --runtime browser-wasm` as suggested by the error but that didn't help.
The correct thing to do was to pass `Release` (with upper case R) to `Configuration` parameter.
cc @safern
* Build all managed coreclr tests on OSX in CI (#36253)
Remove concept of targetGeneric and targetSpecific
Add OSX corelcr managed tests builds to all pipes
Remove all other platform coreclr managed test build
Remove property `TestUnsupportedOutsideWindows`
Rename conditional `DisableProjectBuild` to `CLRTestTargetUnsupported`
Refactor tests to allow all managed tests to be built on OSX.
Split managed tests which based on target properties conditionally:
+ `<Compile Include/>`
+ `<DefineConstant/>`
This creates a separate test for each target configuration permutation.
Add `<CLRTestTargetUnsupported/>` conditions to select these at build and/or
run time
Append split tests with `_Target*` to identify intended test target
For tests which depend on target specific internal details od System.Private.Corlib,
expose a dummy implementation for unsupported targets.
Clean up
Remove <TraitTags/>
Remove <CLRTestNeedTarget/>
Remove managedOSXBuild
Remove managedTestBuildOsGroup
Remove managedTestBuildOsSubGroup
Use issues.target to disable expected failures
Some tests are configured to run on a targets with
specific charecteristics.
Use issues.targets to conditionally disable tests
not intended to run on all targets.
* Setup pr and batch runs for dev/infrsatructure (#36218) (#36299)
Co-authored-by: Jarret Shook <jashoo@microsoft.com>
* Add alias CoreClrTestBuildHost (#36256)
* Add alias CoreClrTestBuildHost
Intended to allow easy switching of test build host between OSX_x64 & Linux_x64
depending on availablility and reliability.
* Add missing issues.targets from #36253 (#36351)
* Add missing issues.targets from #36253
PR #36253 enabled building all tests on OSX. It was intended to include this list of
tests to disable on unsupported platforms. This was dropped as part of rebasing for
dev/infra branch.
Lack of CI on dev/infra allowed this to be missed.
* Revise issues.targets
* Add src/coreclr/tests/src/baseservices/typeequivalence/simple
* Remove #if Windows from DllImportPathTest
* Build scripts while copying native binaries (#36482)
* Build scripts while copying native binaries
Stop building test scripts during the managed build phase
Build after copying the native targets
Fixes issues relative to building scripts on OSX
* Fix superpmi script generation
* Cleanup issues.targets
* Keep 6 failing IJW tests
* Fix IJW when managed tests are built on OSX (#36711)
* Fix IJW when managed tests are built on OSX
- Refactor Interop.setting.targets to Directory.Build.{targets,props}
- Remove include Interop.setting.targets from interop projects rely on SDK including
Directory.Build.* automatically
- Add new CopyInteropNativeRuntimeDependencies target for copying IJW dependencies
* Add <Copy/> task
- Delete obsolete instruction in Interop/ReadMe.md
* Cleanup Readme.md
* Optional stress dependencies & don't populate CORE_ROOT twice (#36851)
Bruce noticed that we're downloading and installing stress
runtime dependencies several times during build. I have added a new
option to suppress this behavior in build-test and I fixed the
invocations of build-test based on Bruce's analysis. I have also
patched the build scripts to avoid populating CORE_ROOT in
"copynativeonly" mode.
Thanks
Tomas
Fixes: #36797
* Build test scripts whenever CopyNativeProjectBinaries builds (#37028)
* Fix pri0 test build.
* Clean pri1 test build (#37266)
* Clean pri1 test build
* Undo testing comment
* Make sure weakreferencetest does not run on unix
* Fixup WinRT proj
* Fix path
Co-authored-by: Steve MacLean <Steve.MacLean@microsoft.com>
Co-authored-by: Tomáš Rylek <trylek@microsoft.com>
Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com>
Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Andy Ayers <andya@microsoft.com>
Co-authored-by: Adeel Mujahid <adeelbm@outlook.com>
Co-authored-by: Ganbarukamo41 <ganbarukamo@gmail.com>
Co-authored-by: monojenkins <jo.shields+jenkins@xamarin.com>
Co-authored-by: thaystg <thaystg@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
Co-authored-by: Kevin Jones <kevin@vcsjones.com>
Co-authored-by: Jeremy Barton <jbarton@microsoft.com>
Co-authored-by: Egor Bogatov <egorbo@gmail.com>
Co-authored-by: Simon Nattress <nattress@gmail.com>
Co-authored-by: Aleksey Kliger (λgeek) <alklig@microsoft.com>
Co-authored-by: Krzysztof Wicher <kwicher@microsoft.com>
Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>
Co-authored-by: Carol Eidt <carol.eidt@microsoft.com>
Co-authored-by: Shimmy <2716316+weitzhandler@users.noreply.github.com>
Co-authored-by: Stephen Toub <stoub@microsoft.com>
Co-authored-by: Layomi Akinrinade <laakinri@microsoft.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>
Co-authored-by: Jeremy Koritzinsky <jekoritz@microsoft.com>
Co-authored-by: Jose Perez Rodriguez <joperezr@microsoft.com>
Co-authored-by: Alexander Chermyanin <flamencist@mail.ru>
Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com>
Co-authored-by: Sergey Andreenko <seandree@microsoft.com>
Co-authored-by: Nikola Milosavljevic <nikolam@microsoft.com>
Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com>
Co-authored-by: David Wrighton <davidwr@microsoft.com>
Co-authored-by: buyaa-n <bunamnan@microsoft.com>
Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com>
Co-authored-by: Vladimir Sadov <vsadov@microsoft.com>
Co-authored-by: Leandro Pereira <leandro.pereira@microsoft.com>
Co-authored-by: Swaroop Sridhar <swaroop.sridhar@microsoft.com>
Co-authored-by: Marie Píchová <11718369+ManickaP@users.noreply.github.com>
Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>
Co-authored-by: Manish Godse <61718172+mangod9@users.noreply.github.com>
Co-authored-by: Bruce Forstall <brucefo@microsoft.com>
Co-authored-by: Alexis Christoforides <alexis@thenull.net>
Co-authored-by: BrzVlad <BrzVlad@users.noreply.github.com>
Co-authored-by: Alfred Myers <git@alfredmyers.com>
Co-authored-by: Nick Craver <nrcraver@gmail.com>
Co-authored-by: Nathan Ricci <naricc@microsoft.com>
Co-authored-by: Thays Grazia <thaystg@gmail.com>
Co-authored-by: Yoh Deadfall <yoh.deadfall@hotmail.com>
Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
Co-authored-by: Jo Shields <directhex@apebox.org>
Co-authored-by: Jan Vorlicek <janvorli@microsoft.com>
Co-authored-by: Anirudh Agnihotry <anirudhagnihotry098@gmail.com>
Co-authored-by: Roman Marusyk <Marusyk@users.noreply.github.com>
Co-authored-by: Zoltan Varga <vargaz@gmail.com>
Co-authored-by: Youssef Victor <31348972+Youssef1313@users.noreply.github.com>
Co-authored-by: Ben Adams <thundercat@illyriad.co.uk>
Co-authored-by: Marek Safar <marek.safar@gmail.com>
Co-authored-by: Anton Lapounov <antonl@microsoft.com>
Co-authored-by: AraHaan <15173749+AraHaan@users.noreply.github.com>
Co-authored-by: Tom Deseyn <tom.deseyn@gmail.com>
Co-authored-by: Tomas Weinfurt <tweinfurt@yahoo.com>
Co-authored-by: Tomas Weinfurt <furt@Shining.local>
Co-authored-by: David Cantu <dacantu@microsoft.com>
Co-authored-by: Sung Yoon Whang <suwhang@microsoft.com>
Co-authored-by: Levi Broderick <GrabYourPitchforks@users.noreply.github.com>
Co-authored-by: vargaz <vargaz@users.noreply.github.com>
Co-authored-by: David Mason <davmason@microsoft.com>
Co-authored-by: UnityAlex <UnityAlex@users.noreply.github.com>
Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com>
Co-authored-by: Tanner Gooding <tagoo@outlook.com>
Co-authored-by: Egor Chesakov <Egor.Chesakov@microsoft.com>
Co-authored-by: radical <radical@users.noreply.github.com>
Co-authored-by: Ivan Diaz Sanchez <ivdiazsa@microsoft.com>
Co-authored-by: Gleb Balykov <g.balykov@samsung.com>
Co-authored-by: Honza Rameš <ramejan@gmail.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Ivan <ivanbuha@outlook.com>
Co-authored-by: Ryan Lucia <rylucia@microsoft.com>
Co-authored-by: SRV <roman.sakno@gmail.com>
Co-authored-by: Clinton Ingram <clinton.ingram@outlook.com>
Co-authored-by: Mikhail Pilin <ww898@users.noreply.github.com>
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: w-flo <w-flo@users.noreply.github.com>
Co-authored-by: Peter Sollich <petersol@microsoft.com>
Co-authored-by: grendello <grendello@users.noreply.github.com>
Co-authored-by: Johan Lorensson <lateralusx.github@gmail.com>
Co-authored-by: Erhan Atesoglu <47518605+eanova@users.noreply.github.com>
Co-authored-by: Matt Mitchell <mmitche@microsoft.com>
Co-authored-by: Josh Schreuder <joshschreuder@gmail.com>
Co-authored-by: Josh Schreuder <josh.schreuder@gmail.com>
Co-authored-by: Andrew Au <andrewau@microsoft.com>
Co-authored-by: Eugene Rozenfeld <erozen@microsoft.com>
Co-authored-by: Vlad Brezae <brezaevlad@gmail.com>
Co-authored-by: Kenneth Pouncey <kjpou@pt.lu>
Co-authored-by: Kirill Frolov <k.frolov@samsung.com>
Co-authored-by: Tomas Weinfurt <furt@DESKTOP-0J9757E.corp.microsoft.com>
