Define a new container format for .NET assemblies that looks less like a Windows PE file. Use it for bundling assemblies in wasm projects.
* Implement WebCIL loader
It will try to look for WebCIL formatted images instread of normal .dll files
* Checkpoint works on wasm sample; add design doc
* Push .dll->.webcil probing lower in the bundle logic
* Also convert satellite assemblies and implement satellite matching
* [wasm] don't leak .webcil image names to the debugger
In particular this will make source and breakpoint URLs look like `dotnet://foo.dll/Foo.cs` which means that grabbing PDBs via source link will work, etc.
* Add PE DebugTableDirectory to webcil
This is used to retrieve the PPDB data and/or the PDB checksum from an image.
Refactor mono_has_pdb_checksum to support webcil in addition to PE images
* Implement a WebcilReader for BorwserDebugProxy like PEReader
This needs some improvements:
- add support for reading CodeView and EmbeddedPDB data
- copy/paste less from the WebcilWriter task
- copy/paste less from PEReader (will require moving WebcilReader to SRM)
* [debug] Match bundled pdbs if we're looking up .webcil files
The pdbs are registered by wasm with a notional .dll filename. if the debugger does a lookup using a .webcil name instead, allow the match
* Adjust debug directory entries when writing webcil files
the PE COFF debug directory entries contain a 'pointer' field which is an offset from the start of the file.
When writing the webcil file, the header is typically smaller than a PE file, so the offsets are wrong. Adjust the offsets by the size of the file.
We assume (and assert) the debug directory entries actually point at some PE COFF sections in the PE file (as opposed to somewhere past the end of the known PE data).
When writing, we initially just copy all the sections directly, then seek to where the debug directory entries are, and overwrite them with updated entries that have the correct 'pointer'
* Fix bug in WebcilWriter
Stream.CopyTo takes a buffer size, not the number of bytes to copy.
* bugfix: the debug directory is at pe_debug_rva not at the CLI header
* skip debug fixups if there's no debug directory
* WebcilReader: implement CodeView and Emebedded PPDB support
* [WBT] Add UseWebcil option (default to true)
* rename WebcilWriter -> WebcilConverter [NFC]
* fixup AssemblyLoadedEventTest
* hack: no extension on assembly for breakpoint
* pass normal .dll name for MainAssemblyName in config
let the runtime deal with it - bundle matching will resolve it to the .webcil file
* Wasm.Debugger.Tests: give CI 10 more minutes
* Add Microsoft.NET.WebAssembly.Webcil assembly project
Mark it as shipping, but not shipping a nuget package.
The idea is that it will be shipped along with the WasmAppBuilder msbuild task, and with the BrowserDebugProxy tool.
* Move WebcilConverter to Microsoft.NET.WebAssembly.Webcil
* Move WebcilReader to Microsoft.NET.WebAssembly.Webcil
delete the duplicated utility classes
* make the webcil magic and version longer
* Code style improvements from review
* Improve some exception messages, when possible
* Suggestings from code review
* Add WasmEnableWebcil msbuild property. Off by default
* Build non-wasm runtimes without .webcil support
* Run WBT twice: with and without webcil
This is a total of 4 runs: with and without workloads x with and without webcil
* do the cartesian product correctly in msbuild
* also add webcil to template projects
* environment variable has to be non-null and "true"
We set it to "false" sometimes
* Fix wasm work items
They should be the same whether or not webcil is used. Just the WorkloadItemPrefix should be used to change the name.
* Update src/libraries/sendtohelix-wasm.targets
* PInvokeTableGeneratorTests: don't try to use the net472 WasmAppBuilder
Look for the default target framework subdirectory under the tasks directory in the runtime pack when trying to find the tasks dll. In particular don't try to load the net472 version on modern .NET
* PInvokeTableGeneratorTests: Add more diagnostic output if tasksDir is not found
* simplify prefix comparison in bundled_assembly_match
* WasmAppBuilder improve logging
Just emit a single Normal importance message about webcil; details as Low importance.
* Add missing using
Co-authored-by: Ankit Jain <radical@gmail.com>
Co-authored-by: Larry Ewing <lewing@microsoft.com>
Change github illink links to point from dotnet/linker to dotnet/runtime
Delete scripts that clone illink into runtime since illink now lives in runtime
Create a forward link for error 1012 in illink, that leads users to open an issue in runtime
* Replace the last two SIMDIntrinsic in LIR with NamedIntrinsic and delete GT_SIMD
* Applying formatting patch
* Ensure SIMD_UpperRestore/Save is handled in gtSetEvalOrder
* Handle some asserts on Arm64
* Created memory-model.md
* addresses some comments
* More details and samples.
* Fix trailing whitespaces.
* More trailing whitespace
* More trailing whitespace.
* Apply suggestions from code review (typos)
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
* replaced references to CLR with ".NET runtime"
* Addressed some PR review feedback
* Moved to specs folder
* More addressing PR feedback
* Volatile/Interlocked methods are atomic
* Better notes about atomicity of pointers
* Apply suggestions from code review
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* Addressing more PR feedback
* Updated singleton sample for more clarity.
* Trailing whitespace.
* Move data dependent reads to general section
* Compat disambiguation note on object assignments.
* Apply suggestions from code review
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* No dots at title ends
* "Data-dependent" spelled with dash consistently
* Apply suggestions from code review
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* Apply suggestions from code review
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
* Apply suggestions from code review
Co-authored-by: Bruce Forstall <brucefo@microsoft.com>
* order of object assignment and data-dependent memory accesses
* Listed primitive types.
* Briefly explained motivations for the treatment of memory access sideeffects.
* Update docs/design/specs/Memory-model.md
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
* Link to the data-dependent accesses and compiler optimizations followup issue.
* removed unnecessary `[`
* Update docs/design/specs/Memory-model.md
* Trailing whitespace
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
Co-authored-by: Bruce Forstall <brucefo@microsoft.com>
* [mono] Add LoaderAllocator type, whose instances are used to
detect whenever a collectible ALC has managed references.
* [mono] Add an implicit GC reference between objects allocated from a collectible ALC and its LoaderAllocator object.
* [mono] Add a new hash table which is similar to MonoGHashTable, but it doesn't keep the key/value objects alive by itself.
* [mono] Add a keepalive field to some reflection objects to keep the alc alive if user code
holds a reference to them. Change the reflection hashes in MonoMemoryManager to weak hashes
so the hashes themselves don't keep the alc alive.
* Fix reflection hashes.
* [mono] Optimize the case when mono_method_get_signature_checked () is called with a non-zero context and a non-generic signature.
* Free memory manager caches.
* [mono] Store static variables with GC references in collectible alcs
on the GC heap.
Normally, static variables are stored in an array inside MonoVTable
which is registered as a GC root. For collectible alcs, this would
not work, since GC references in these arrays would keep the alc alive.
Instead, store them in arrays referenced by the LoaderAllocator object.
This assumes the static variables will no longer be accessed after
the LoaderAllocator object dies.
* Add basic unload functionality.
* Fix weak hashes.
* Free MonoJitInfos belonging to unloaded memory managers.
* Avoid returning collectible types from mono_metadata_get_shared_type ().
* Add docs.
* Fix the build.
* Fix the build.
* Disable unloading for now.
* Design to support ByRefLike types in Generics
* Trim whitespace
* Review feedback
* Remove words
* Update based on feedback
* Add F# as being impacted.
* Feedback and clarify wording for constrained calls.
* Add well-known IL sequences for boxing
Record APIs where compiler constraint analysis should be suppressed.
Defined analysis suppression attribute.
* Minor nits.
* Update docs/design/features/byreflike-generics.md
Co-authored-by: Jared Parsons <jaredpparsons@gmail.com>
* Review feedback
* Update to design doc
* Misspelling
* Update byreflike-generics.md
Co-authored-by: Jared Parsons <jaredpparsons@gmail.com>
This changes how the JIT matches method names and signatures for method
sets (e.g. JitDisasm). It also starts printing method instantiations for full method names
and makes references to types consistent in generic instantiations and the signature.
In addition it starts supporting generic instantiations in release too.
To do this, most of the type printing is moved to the JIT, which also aligns the output
between crossgen2 and the VM (there were subtle differences here, like spaces between generic type arguments).
More importantly, we (for the most part) stop relying on JIT-EE methods that are documented to only be for debug purposes.
The new behavior of the matching is the following:
* The matching behavior is always string based.
* The JitDisasm string is a space-separated list of patterns. Patterns can arbitrarily
contain both '*' (match any characters) and '?' (match any 1 character).
* The string matched against depends on characters in the pattern:
+ If the pattern contains a ':' character, the string matched against is prefixed by the class name and a colon
+ If the pattern contains a '(' character, the string matched against is suffixed by the signature
+ If the class name (part before colon) contains a '[', the class contains its generic instantiation
+ If the method name (part between colon and '(') contains a '[', the method contains its generic instantiation
For example, consider
```
namespace MyNamespace
{
public class C<T1, T2>
{
[MethodImpl(MethodImplOptions.NoInlining)]
public void M<T3, T4>(T1 arg1, T2 arg2, T3 arg3, T4 arg4)
{
}
}
}
new C<sbyte, string>().M<int, object>(default, default, default, default); // compilation 1
new C<int, int>().M<int, int>(default, default, default, default); // compilation 2
```
The full strings are:
Before the change:
```
MyNamespace.C`2[SByte,__Canon][System.SByte,System.__Canon]:M(byte,System.__Canon,int,System.__Canon)
MyNamespace.C`2[Int32,Int32][System.Int32,System.Int32]:M(int,int,int,int)
```
Notice no method instantiation and the double class instantiation, which seems like an EE bug. Also two different names are used for sbyte: System.SByte and byte.
After the change the strings are:
```
MyNamespace.C`2[byte,System.__Canon]:M[int,System.__Canon](byte,System.__Canon,int,System.__Canon)
MyNamespace.C`2[int,int]:M[int,int](int,int,int,int)
```
The following strings will match both compilations:
```
M
*C`2:M
*C`2[*]:M[*](*)
MyNamespace.C`2:M
```
The following will match only the first one:
```
M[int,*Canon]
MyNamespace.C`2[byte,*]:M
M(*Canon)
```
There is one significant change in behavior here, which is that I have removed the special case that allows matching class names without namespaces. In particular, today Console:WriteLine would match all overloads of System.Console.WriteLine, while after this change it will not match. However, with generalized wild cards the replacement is simple in *Console:WriteLine.
Add --crashreportonly command line option that doesn't generated a dump.
Add matching DOTNET_EnableCrashReportOnly env var.
Add LoadModule error logging for MacOS.
Make DAC validate EEClass and MethodDesc functions more robust on Linux/MacOS so SOS's eestack and dumpstack commands don't segfault.
Update createdump doc.
* Describe the validity of null managed pointers
- Declare that it is valid to have a null managed pointer, but declare it invalid to actually read from such a pointer
- In practice this has always been legal, as it has been legal to managed pointer locals for years, and they are included in the list of values that are zeroinitialized on method start
- Also clarify the rules to permit a managed pointer to the location directly following a managed object.
- This is a new capability in the spec that will likely be useful for accessing fixed size data buffers held in objects of the GC heap. However, the GC has been able to tolerate this behavior for many years, so there is no code change necessary.
Fixes#69690
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
The initial implementation of this did not handle the fact that retbuf
can point to GC heap during reflection invoke. It was fixed in #39815,
but the way it was fixed was by copying it into a local. This changes
the fix so that we simply report the return value pointer as a byref
throughout the mechanism, which simplifies the JIT's handling and is a
perf improvement as well.
* Fix typos
* Cleanup trailing whitespaces in committed files
* Revert a macro for win32 compat
* Disambiguate test data method
* Revert XMLPath test which rely on external assets
* Revert whitespace change in Xml tests
* Revert ClrEtwAl and ILLink.Shared
* Revert crossgen2 props/targets and *.wxl
- Refactor Module into ModuleBase and Module
- The goal is to have allow a subset version of Module which can only hold refs, this is to be used by the manifest module in an R2R image to allow for version resilient cross module references.
- Update handling of ModuleBase so that its used everywhere that tokens are parsed from R2R
- Remove ENCODE_MODULE_ID_FOR_STATICS and ENCODE_ACTIVE_DEPENDENCY
- These were only used for NGEN, and conflict with easy impelmentation for the ModuleBase concept
- Remove locking in ENCODE_STRING_HANDLE processing, and tweak comments. Comments applied to the removed ngen based code, and the lock was only necessary for the old ngen thing.
- Adjust ComputeLoaderModuleWorker for locating loader module
- Follow comment more accurately, to avoid putting every generic into its definition module. This will make R2R function lookup able to find compiled instantiations in some cases. This may be what we want long term, it may not.
- Remove MemberRefToDesc map and replace with LookupMap like the other token types. We no longer make use of the hot table, so this is more efficient
- Also reduces complexity of implementation of ModuleBase
- Build fixup to describe a single method as a standalone blob of data
- There are parallel implementations in Crossgen2 and in the runtime
- They produce binary identical output
- Basic R2RDump support for new fixup
- Adjust module indices used within the R2R format to support a module index which refers to the R2R manifest metadata. This requires bumping the R2R version to 6.2
- Add a module index between the set of assembly refs in the index 0 module and the set of assembly refs in the R2R manifest metadata
- Adjust compilation dependency rules to include a few critical AsyncStateMachineBox methods
- Remove PEImage handling of native metadata which was duplicative
- Do not enable any more devirtualization than was already in use, even in the cross module compilation scenario. In particular, do not enable devirtualization of methods where the decl method isn't within the version bubble, even if the decl method could be represented with a cross module reference token. (This could be fixed, but is out of scope for this initial investigation)
Make the compilation deterministic in this new model, even though we are generating new tokens on demand
- Implement this by detecting when we need new tokens during a compile, and recompiling with new tokens when necessary
- This may result in compiling the same code as much as twice
Compile the right set of methods with cross module inlining enabled
- Add support for compiling the called virtual methods on generic types
- This catches the List<T> and Dictionary<TKey,TValue> scenarios
- Support input of PGO data to control the set of methods
- Enable new `READYTORUN_FLAG_UNRELATED_R2R_CODE` flag on R2R images which is used to indicate which modules may have generic code not directly related to the metadata of the image
- Lookup R2R methods in an `alternate` location as well as the metadata defining module. This allows for many generics to be embedded without needing to use the new `READYTORUN_FLAG_UNRELATED_R2R_CODE` flag, which has global effects on performance.
- Add command line switches to enable/disable the new behavior
- Enhance the version resilience test to cover this new behavior
Basic stateless linear collection marshalling for blittable elements
Not handled:
- caller-allocated buffer
- guaranteed unmarshal
- pinnable reference
- non-blittable element marshalling
- element scenarios on custom marshallers
* Add support for cross module inlining and cross module generic compilation to Crossgen2
- Refactor Module into ModuleBase and Module
- The goal is to have allow a subset version of Module which can only hold refs, this is to be used by the manifest module in an R2R image to allow for version resilient cross module references.
- Update handling of ModuleBase so that its used everywhere that tokens are parsed from R2R
- Remove ENCODE_MODULE_ID_FOR_STATICS and ENCODE_ACTIVE_DEPENDENCY
- These were only used for NGEN, and conflict with easy impelmentation for the ModuleBase concept
- Remove locking in ENCODE_STRING_HANDLE processing, and tweak comments. Comments applied to the removed ngen based code, and the lock was only necessary for the old ngen thing.
- Adjust ComputeLoaderModuleWorker for locating loader module
- Follow comment more accurately, to avoid putting every generic into its definition module. This will make R2R function lookup able to find compiled instantiations in some cases. This may be what we want long term, it may not.
- Remove MemberRefToDesc map and replace with LookupMap like the other token types. We no longer make use of the hot table, so this is more efficient
- Also reduces complexity of implementation of ModuleBase
- Build fixup to describe a single method as a standalone blob of data
- There are parallel implementations in Crossgen2 and in the runtime
- They produce binary identical output
- Basic R2RDump support for new fixup
- Adjust module indices used within the R2R format to support a module index which refers to the R2R manifest metadata. This requires bumping the R2R version to 6.2
- Add a module index between the set of assembly refs in the index 0 module and the set of assembly refs in the R2R manifest metadata
- Adjust compilation dependency rules to include a few critical AsyncStateMachineBox methods
- Remove PEImage handling of native metadata which was duplicative
- Do not enable any more devirtualization than was already in use, even in the cross module compilation scenario. In particular, do not enable devirtualization of methods where the decl method isn't within the version bubble, even if the decl method could be represented with a cross-module reference token. (This could be fixed, but is out of scope for this initial investigation)
Make the compilation deterministic in this new model, even though we are generating new tokens on demand
- Implement this by detecting when we need new tokens during a compile, and recompiling with new tokens when necessary
- This may result in compiling the same code as much as twice
Compile the right set of methods with cross module inlining enabled
- Add support for compiling the called virtual methods on generic types
- This catches the List<T> and Dictionary<TKey,TValue> scenarios
- Add command line switches to enable/disable the new behavior
- By default the new behavior is not enabled
* Do not set NO_CSE on ARR_ADDRs
It is effectively no-CSE already because of how "optIsCSECandidate" works.
* Delete GT_INDEX
Instead:
1) For "ldelem", import "IND/OBJ(INDEX_ADDR)".
2) For "ldelema", import "INDEX_ADDR".
This deletes two usages of "ADDR":
1) "OBJ(ADDR(INDEX))" from "ldelem<struct>".
2) "ADDR(INDEX)" from "ldelema".
* Add a zero-diff quirk
* Update the first class structs document
Remove references to things that no longer exist.
This adds support for EnC on arm64. A couple of notes on the
implementation compared to x64:
- On x64 we get the fixed stack size from unwind info. However, for the
frames we set up on arm64 for EnC it is not possible to extract the
frame size from there because their prologs generally look like
stp fp, lr, [sp,#-16]!
mov fp, sp
sub sp, sp, #144
with unwind codes like the following:
set_fp; mov fp, sp
save_fplr_x #1 (0x01); tp fp, lr, [sp, #-16]!
As can be seen, it is not possible to get the fixed stack size from
unwind info in this case. Instead we pass it through the GC info that
already has a section for EnC data.
- On arm64 the JIT is required to place the PSPSym at the same offset
from caller-SP for both the main function and for funclets. Due to
this we try to allocate the PSPSym as early as possible in the main
function and we must take some care in funclets. However, this
conflicts with the EnC frame header that the JIT uses to place values
that must be preserved on EnC transitions. This is currently
callee-saved registers and the MonitorAcquired boolean.
Before this change we were allocating PSPSym above (before) the
monitor acquired boolean, but we now have to allocate MonitorAcquired
first, particularly because the size of the preserved header cannot
change on EnC transitions, while the PSPSym can disappear or appear.
This changes frame allocation slightly for synchronized functions.
