Move clang-format download into dotnet/runtime and add docs for setting up auto-formatting in the repository (#59374)

.clang-format

@@ -0,0 +1 @@
+DisableFormat: true

.config/dotnet-tools.json

@@ -25,6 +25,12 @@
       "commands": [
         "slngen"
       ]
+    },
+    "dotnet-format": {
+      "version": "6.0.240501",
+      "commands": [
+        "dotnet-format"
+      ]
     }
   }
 }

docs/coding-guidelines/code-formatting-tools.md

@@ -0,0 +1,93 @@
+# Code Formatting Tools in dotnet/runtime
+
+In this repository, we use various different formatting conventions depending on who owns the code. Additionally, we use different formatting tools depending on if the code is managed C# or VB code or if the code is native C or C++ code.
+
+To help enable an easy workflow and reduce the number of formatting changes requested, this document provides steps to download and enable the various different tools in different development environments to enable a seamless workflow for ensuring that keeping code formatted is a pleasant experience.
+
+## Downloading formatting tools
+
+### C#/VB
+
+C# and VB code in the repository use the built-in Roslyn support for EditorConfig to enable auto-formatting in many IDEs. As a result, no additional tools are required to enable keeping your code formatted. If you want to use `dotnet format` to do formatting or are using the git pre-commit hook mentioned later in this document, you can run `./dotnet.cmd tool restore` or `./dotnet.sh tool restore` from the root of the repository to enable the `dotnet format` command.
+
+### C/C++
+
+To download the formatting tools for C++, run the `download-tools.ps1/sh` script in the `eng/formatting` folder. Specifically, run the `download-tools.ps1` script if you're on Windows, or the `download-tools.sh` script otherwise.
+
+This script will download the `clang-format` and `clang-tidy` tools to the `artifacts/tools` folder in your clone of the repository. Only specific parts of the repository use `clang-tidy`, so this document primarily focuses on setting up `clang-format` for C and C++ scenarios.
+
+#### CoreCLR JIT
+
+The JIT team uses a special `jit-format` tool that runs `clang-format` and `clang-tidy` on their directory with their settings. You can get the `jit-format` tool from the [dotnet/jitutils repository](https://github.com/dotnet/jitutils).
+
+## Setting up automatic formatting
+
+To make the formatting workflow more seamless for contributors, instructions are included below to help enable "format-on-save" or other forms of automatic formatting in various different IDEs or as a Git pre-commit hook for IDEs that do not support "format-on-save" scenarios.
+
+This section is open to contributions for instructions on how to enable "format-on-save"-like semantics in IDEs and editors not mentioned.
+
+### Visual Studio Code
+
+Enabling a "format-on-save" experience in VSCode is quite easy. Add the following setting to your `.vscode/settings.json` file to enable "format-on-save":
+
+```json
+    "editor.formatOnSave": true,
+```
+
+The sections below include any additional instructions to configure the experience for different languages.
+
+#### C#/VB
+
+There are currently some limitations in OmniSharp that causes issues with the auto-formatting. Once a VSCode C# extension releases with both https://github.com/OmniSharp/omnisharp-roslyn/pull/2227 and https://github.com/OmniSharp/omnisharp-vscode/pull/4738, then format-on-save should be possible to enable in VSCode for C# and Visual Basic sources.
+
+#### C/C++
+
+VSCode ships with a different version of clang-format than we use in this repo, so you will need to add a few more settings to get VSCode to run the correct clang-format for "format-on-save".
+
+You can add the following setting on Windows to your `.vscode/settings.json` file to configure clang-format for the repository:
+
+```json
+    "C_Cpp.clang_format_path": "./artifacts/tools/clang-format.exe"
+```
+
+On non-Windows, you can add the following setting instead:
+
+```json
+    "C_Cpp.clang_format_path": "./artifacts/tools/clang-format"
+```
+
+### Visual Studio
+
+Visual Studio does not have a "format-on-save" feature but it does have settings for "format on end of statement" or "format on end of block" that can provide some auto-formatting features.
+
+Using these features in combination with the steps specified in the [Git Hooks](#git-hooks) section will enable a seamless formatting experience.
+
+The settings for C#, VB, C, and C++ are all in a dialog under the `Tools > Options` menu item.
+
+#### C#/VB
+
+In the options view, go to `Text Editor > C# > Code Style > Formatting > General` for C# settings. Make sure to check the various options for whenever you want Visual Studio to auto-format your code.
+
+![C# formatting settings in Visual Studio](./vs-csharp-formatting.png)
+
+#### C/C++
+
+In the options view, go to `Text Editor > C/C++ > Code Style > Formatting > General` for C# settings. Make sure to check the various options for whenever you want Visual Studio to auto-format your code.
+
+![C/C++ formatting settings in Visual Studio](./vs-cpp-formatting.png)
+
+### Git Hooks
+
+Git provides a number of hooks to enable running scripts before commit, push, pull, etc. This section describes adding a pre-commit hook to automatically format code before committing to make formatting seamless even when your development environment doesn't support "format-on-save" or similar functionality with the formatting tools this repository uses.
+
+#### Auto-format before committing
+
+To enable auto-formatting before committing, you can create a `.git/hooks/pre-commit` file in your local `dotnet/runtime` clone and add a call to the script located at `eng/formatting/format.sh` to auto-format your code before committing. Since Git for Windows also installs Git Bash, this script will work for both Windows and non-Windows platforms.
+
+The following code block can be used as the contents of the `pre-commit` file to enable the auto-formatting hook:
+
+```sh
+#!/bin/sh
+./eng/formatting/format.sh
+
+```

docs/coding-guidelines/coding-style.md

@@ -1,7 +1,7 @@
 C# Coding Style
 ===============
 
-For C++ files (*.cpp and *.h), we use clang-format (version 3.6+) to ensure code styling. After changing any Cpp or H file and before merging, src/Native/format-code.sh must be run. This script will ensure that all native code files adhere to the coding style guidelines.
+For C++ files (*.cpp and *.h), we use clang-format (version 3.8) to ensure code styling. After changing any Cpp or H file and before merging, src/Native/format-code.sh must be run. This script will ensure that all native code files adhere to the coding style guidelines.
 
 For other types of files (xml, bat, sh, etc), our current best guidance is consistency. When editing files, keep new code and changes consistent with the style in the files. For new files, it should conform to the style for that component. If there is a completely new component, anything that is reasonably broadly accepted is fine. For script files, please refer to the scripting blog for [tips](https://devblogs.microsoft.com/scripting/tag/powertip) and [best practices](https://devblogs.microsoft.com/scripting/tag/best-practices).
 

eng/formatting/download-tools.ps1

@@ -0,0 +1,46 @@
+
+function DownloadClangTool
+{
+    param (
+        [string]
+        $toolName,
+        [string]
+        $downloadOutputPath
+    )
+
+    $baseUri = "https://clrjit.blob.core.windows.net/clang-tools/windows"
+
+    if (-not $(ls $downloadOutputPath | Where-Object {$_.Name -eq "$toolName.exe"}))
+    {
+        $baseBackoffSeconds = 2;
+
+        $success = $false
+        for ($i = 0; $i -lt 5; $i++) {
+            echo "Attempting download of '$baseUri/$toolName.exe'"
+            $status = Invoke-WebRequest -Uri "$baseUri/$toolName.exe" -OutFile $(Join-Path $downloadOutputPath -ChildPath "$toolName.exe")
+            if ($status.StatusCode -lt 400)
+            {
+                $success = $true
+                break
+            } else {
+                echo "Download attempt $($i+1) failed. Trying again in $($baseBackoffSeconds + $baseBackoffSeconds * $i) seconds"
+                Start-Sleep -Seconds $($baseBackoffSeconds + $baseBackoffSeconds * $i)
+            }
+        }
+        if (-not $success)
+        {
+            Write-Output "Failed to download clang-format"
+            return 1
+        }
+    }
+}
+
+$downloadPathFolder = Split-Path $PSScriptRoot -Parent | Split-Path -Parent | Join-Path -ChildPath "artifacts" | Join-Path -ChildPath "tools"
+
+mkdir $downloadPathFolder -ErrorAction SilentlyContinue
+
+DownloadClangTool "clang-format" "$downloadPathFolder"
+DownloadClangTool "clang-tidy" "$downloadPathFolder"
+
+# Add to path to enable scripts to skip additional downloading steps since the tools will already be on the path.
+$env:PATH = "$downloadPathFolder;$env:PATH"

eng/formatting/download-tools.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+set -ue
+
+source="${BASH_SOURCE[0]}"
+
+# resolve $source until the file is no longer a symlink
+while [[ -h "$source" ]]; do
+  scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
+  source="$(readlink "$source")"
+  # if $source was a relative symlink, we need to resolve it relative to the path where the
+  # symlink file was located
+  [[ $source != /* ]] && source="$scriptroot/$source"
+done
+scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
+
+function DownloadClangTool {
+    targetPlatform=$(dotnet --info |grep RID:)
+    targetPlatform=${targetPlatform##*RID:* }
+
+    toolUrl=https://clrjit.blob.core.windows.net/clang-tools/${targetPlatform}/$1
+    toolOutput=$2/$1
+
+    if [[ ! -x "$toolOutput" ]]; then
+        curl --retry 5 -o "${toolOutput}" "$toolUrl"
+        chmod 751 $toolOutput
+    fi
+
+    if [[ ! -x "$toolOutput" ]]; then
+        echo "Failed to download $1"
+        exit 1
+    fi
+}
+
+
+engFolder="$(cd -P "$( dirname "$scriptroot" )" && pwd )"
+downloadPathFolder="$(cd -P "$( dirname "$engFolder" )" && pwd )/artifacts/tools"
+
+mkdir -p "$downloadPathFolder"
+
+. "$scriptroot/../common/tools.sh"
+
+InitializeDotNetCli true
+
+DownloadClangTool "clang-format" "$downloadPathFolder"
+DownloadClangTool "clang-tidy" "$downloadPathFolder"
+
+export PATH=$downloadPathFolder:$PATH

eng/formatting/format.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+
+LC_ALL=C
+# Select files to format
+NATIVE_FILES=$(git diff --cached --name-only --diff-filter=ACM "*.h" "*.hpp" "*.c" "*.cpp" "*.inl" | sed 's| |\\ |g')
+MANAGED_FILES=$(git diff --cached --name-only --diff-filter=ACM "*.cs" "*.vb" | sed 's| |\\ |g')
+
+exec 1>&2
+
+if [[ -n "$NATIVE_FILES" ]]; then
+    # Format all selected files
+    echo "$NATIVE_FILES" | cat | xargs | sed -e 's/ /,/g' | xargs "./artifacts/tools/clang-format" -style=file -i
+
+    # Add back the modified files to staging
+    echo "$NATIVE_FILES" | xargs git add
+fi
+if [[ -n "$MANAGED_FILES" ]]; then
+    # Format all selected files
+    echo "$MANAGED_FILES" | cat | xargs | sed -e 's/ /,/g' | dotnet format --include
+
+    # Add back the modified files to staging
+    echo "$MANAGED_FILES" | xargs git add
+fi
+
+
+exit 0

src/tests/Common/scripts/format.py

@@ -105,6 +105,23 @@ def main(argv):
 
     my_env = os.environ
 
+    # Download formatting tools
+    repoRoot = os.path.dirname(os.path.dirname(coreclr))
+    formattingScriptFolder = os.path.join(repoRoot, "eng", "formatting")
+    formattingDownloadScriptCommand = []
+    if platform == 'Linux' or platform == 'OSX':
+        formattingDownloadScriptCommand = [os.path.join(formattingScriptFolder, "download-tools.sh")]
+    elif platform == 'windows':
+        formattingDownloadScriptCommand = ["powershell", os.path.join(formattingScriptFolder, "download-tools.ps1")]
+
+    proc = subprocess.Popen(formattingDownloadScriptCommand)
+
+    if proc.wait() != 0:
+        print("Formatting tool download failed")
+        return -1
+
+    my_env["PATH"] = os.path.join(repoRoot, "artifacts", "tools") + os.pathsep + my_env["PATH"]
+
     # Download bootstrap
 
     bootstrapFilename = ""