Initial commit of jitutils

Repo contains:

* jit-dasm - a tool to generate *.dasm file from the JIT code generator.
* jit-analyze - a tool to analyze *.dasm output to extract a summary of code size deltas.
* cijobs - a tool to list/copy down build artifacts from the dotnet Jenkins CI.
* jit-diff - a tool that orchestrates the above tools in to a convienient end to end asmdiff.

See ./doc/getstarted.md for more details.

.gitignore

@@ -0,0 +1,15 @@
+##  Hides dotnet cli build artifacts as well as project system cookies.
+##
+
+# ignore default binary/obj locations
+**/bin
+**/obj
+/fx
+.vscode
+
+# project system lock files
+project.lock.json
+
+# vs files
+.vs
+browse.VC.db

LICENSE.TXT

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) .NET Foundation and Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

NuGet.Config

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="utf-8"?>
+<configuration>
+  <packageSources>
+    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
+    <clear />
+    <add key="dotnet-cli" value="https://dotnet.myget.org/F/dotnet-cli/api/v3/index.json" />
+    <add key="dotnet-corefxlab" value="https://dotnet.myget.org/F/dotnet-corefxlab/api/v3/index.json" />
+    <add key="dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />
+    <add key="cli-deps" value="https://dotnet.myget.org/F/cli-deps/api/v3/index.json" />
+    <add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
+  </packageSources>
+  <packageRestore>
+    <add key="enabled" value="False" />
+    <add key="automatic" value="False" />
+  </packageRestore>
+</configuration>

README.md

@@ -0,0 +1,137 @@
+# Dotnet JIT code gen utilities - jitutils
+
+This repo holds a collection of utilities used by RyuJIT developers to 
+automate tasks when working on CoreCLR.  Initial utilities are around 
+producing diffs of codegen changes.
+
+## Summary
+
+Copy down the appropraite `bootstrap.{cmd|sh}` for your platform and run it.  The script 
+will enlist in the jitutils project and perform the basics listed in the details section 
+following.
+
+### The details
+
+To prepare the tools for use, from the root of the cloned jitutils repo, do the following: 
+ 1. Download required package dependencies: `dotnet restore`
+    * NOTE: On Mac, you need to first use `ulimit -n 2048` or the `dotnet restore` will fail.
+ 2. Build and publish the tools: `build.{cmd|sh} -f -p`
+
+This will create the following directories in the repo root:
+ 1. `bin` - contains one directory for each built tool.
+ 2. `fx` - contains a set of frameworks assemblies that can be used for asm diffs.
+
+Add each tool subdirectory to your path so you can easily invoke them, e.g.:
+```
+set jitutils=<path to root of jitutils clone>
+set PATH=%PATH%;%jitutils%\bin\jit-dasm;%jitutils%\bin\jit-diff;%jitutils%\bin\jit-analyze
+```
+
+For a more complete introduction look at the [getting started guide](doc/getstarted.md).
+
+## jit-dasm
+
+This is a general tool to produce diffs for compiled MSIL assemblies.  The 
+tool relies on a base and diff crossgen.exe as inputs, either by using a
+prebuilt base from the CI builds and a local experimental build, or 
+building both a base and diff locally.
+
+Sample help commandline:
+```
+    $ jit-dasm --help
+    usage: jit-dasm [-b <arg>] [-d <arg>] [-o <arg>] [-t <arg>] [-f <arg>]
+                    [--gcinfo] [-v] [-r] [-p <arg>...] [--] <assembly>...
+
+        -b, --base <arg>           The base compiler exe.
+        -d, --diff <arg>           The diff compiler exe.
+        -o, --output <arg>         The output path.
+        -t, --tag <arg>            Name of root in output directory.  Allows
+                                   for many sets of output.
+        -f, --file <arg>           Name of file to take list of assemblies
+                                   from. Both a file and assembly list can
+                                   be used.
+        --gcinfo                   Add GC info to the disasm output.
+        -v, --verbose              Enable verbose output.
+        -r, --recursive            Scan directories recursively.
+        -p, --platform <arg>...    Path to platform assemblies
+        <assembly>...              The list of assemblies or directories to
+                                   scan for assemblies.
+```
+
+## jit-diff
+
+jit-diff is a tool specifically targeting CoreCLR.  It has a prebaked list of interesting
+assemblies to use for generating assembly diffs and understands enough of the structure
+to make it more streamlined.  jit-diff uses jit-dasm under the covers to produce the diffs so 
+for other projects a new utility could be produced that works in a similar way.
+
+Sample help commandlines:
+```
+    $ jit-diff --help
+    usage: jit-diff <command> [<args>]
+
+        diff       Run asm diff of base/diff.
+        list       List defaults and available tools asmdiff.json.
+        install    Install tool in config.
+```
+```
+    $ jit-diff diff --help
+    usage: jit-diff diff [-b <arg>] [-d <arg>] [-o <arg>] [-a] [-t <arg>]
+                    [-m] [-f] [-v] [--core_root <arg>] [--test_root <arg>]
+
+        -b, --base <arg>        The base compiler exe or tag.
+        -d, --diff <arg>        The diff compiler exe or tag.
+        -o, --output <arg>      The output path.
+        -a, --analyze           Analyze resulting base, diff dasm
+                                directories.
+        -t, --tag <arg>         Name of root in output directory.  Allows
+                                for many sets of output.
+        -c, --corlibonly        Disasm *corlib only
+        -f, --frameworksonly    Disasm frameworks only
+        -v, --verbose           Enable verbose output
+        --core_root <arg>       Path to test CORE_ROOT.
+        --test_root <arg>       Path to test tree.
+```
+
+## jit-analyze
+
+The jit-analyze tool understand the format of the *.dasm files in a diff and can extract
+this data to produce a summary and/or dump the info to a unified data file (CSV, JSON)
+The common usage of the tool is to extract interesting diffs, if any, from a diff run as
+part of the development progresses.
+
+Sample help commandline:
+```
+    $ jit-analyze --help
+    usage: jit-analyze [-b <arg>] [-d <arg>] [-r] [-c <arg>] [-w]
+                    [--reconcile] [--json <arg>] [--tsv <arg>]
+
+        -b, --base <arg>    Base file or directory.
+        -d, --diff <arg>    Diff file or directory.
+        -r, --recursive     Search directories recursively.
+        -c, --count <arg>   Count of files and methods (at most) to output
+                            in the summary. (count) improvements and
+                            (count) regressions of each will be included.
+                            (default 5)
+        -w, --warn           Generate warning output for files/methods that
+                            only exists in one dataset or the other (only
+                            in base or only in diff).
+        --reconcile         If there are methods that exist only in base or
+                            diff, create zero-sized counterparts in diff,
+                            and vice-versa. Update size deltas accordingly.
+        --json <arg>        Dump analysis data to specified file in JSON
+                            format.
+        --tsv <arg>         Dump analysis data to specified file in
+                            tab-separated format.
+```
+
+## packages
+
+This is a skeleton project that exists to pull down a predictable set of framework 
+assemblies and publish them in the root in the subdirectory './fx'.  Today this is 
+set to the RC2 version of the NetCoreApp1.0 frameworks.  When this package is installed 
+via the `build.{cmd|sh}` script this set can be used on any supported platform for 
+diffing.  Note: The RC2 mscorlib.dll is removed, as this assembly should be updated from 
+the selected base runtime that is under test for consistency. To add particular packages 
+to the set you diff, add their dependencies to the project.json in this project and 
+they will be pulled in and published in the standalone directory './fx'.

bootstrap.cmd

@@ -0,0 +1,29 @@
+:: Quick and dirty bootstrap. 
+
+where /q dotnet.exe || echo Can't find dotnet.exe! Please add to PATH && goto :EOF
+where /q git.exe || echo Can't find git.exe! Please add to PATH && goto :EOF
+set root=%~dp0
+
+:: Clone the mcgutils repo
+
+git clone https://github.com/dotnet/jitutils.git
+
+pushd .\jitutils
+
+:: Pull in needed packages.  This works globally. (due to global.json)
+
+dotnet restore
+
+:: Build and publish all the utilties and frameworks
+
+call .\build.cmd -p -f
+
+popd
+
+:: set utilites in the current path
+
+set PATH=%PATH%;%root%\jitutils\bin\jit-dasm;%root%\jitutils\bin\jit-diff;%root%\jitutils\bin\jit-analyze;%root%\jitutils\bin\cijobs
+
+:: lunch getstarted.md doc
+
+start https://github.com/dotnet/jitutils/blob/master/doc/getstarted.md

bootstrap.sh

@@ -0,0 +1,39 @@
+#Quick and dirty bootstrap. 
+
+if ! dotnet --info; then 
+    echo "Can't find dotnet! Please add to PATH."
+    return 1
+fi
+
+if ! git --version; then 
+    echo "Can't find git! Please add to PATH."
+    return 1
+fi
+
+root=$(pwd)
+
+# Clone the mcgutils repo
+
+git clone https://github.com/dotnet/jitutils.git
+
+pushd ./jitutils
+
+# Pull in needed packages.  This works globally. (due to global.json)
+
+dotnet restore
+
+# Build and publish all the utilties and frameworks
+
+./build.sh -p -f
+
+popd
+
+# set utilites in the current path
+
+export PATH=$PATH:$root/jitutils/bin/jit-dasm:$root/jitutils/bin/jit-diff:$root/jitutils/bin/jit-analyze:$root/jitutils/bin/cijobs
+
+# launch getstarted.md doc
+
+open https://github.com/dotnet/jitutils/blob/master/doc/getstarted.md
+
+echo "Done setting up!"

build.cmd

@@ -0,0 +1,83 @@
+@echo off
+setlocal
+
+REM Build and optionally publish sub projects
+REM
+REM This script will by default build release versions of the tools.
+REM If publish (-p) is requested it will create standalone versions of the
+REM tools in <root>/src/<project>/<buildType>/netcoreapp1.0/<platform>/Publish/.
+REM These tools can be installed via the install script (install.{sh|cmd}) in
+REM this directory.
+
+set scriptDir=%~dp0
+set appInstallDir=%scriptDir%bin
+set fxInstallDir=%scriptDir%fx
+set buildType=Release
+set publish=false
+set fx=false
+
+REM REVIEW: 'platform' is never used
+for /f "usebackq tokens=1,2" %%a in (`dotnet --info`) do (
+    if "%%a"=="RID:" set platform=%%b
+)
+
+:argLoop
+if "%1"=="" goto :build
+
+if /i "%1"=="-b" (
+    set buildType=%2
+    shift
+    goto :nextArg
+)
+if /i "%1"=="-f" (
+    set fx=true
+    goto :nextArg
+)
+if /i "%1"=="-p" (
+    set publish=true
+    goto :nextArg
+)
+if /i "%1" == "-h" (
+    goto :usage
+)
+echo ERROR: unknown argument %1
+goto :usage
+
+:nextArg
+shift
+goto :argLoop
+
+:build
+
+REM Declare the list of projects
+set projects=jit-diff jit-dasm jit-analyze cijobs
+
+REM Build each project
+for %%p in (%projects%) do (
+    if %publish%==true (
+        dotnet publish -c %buildType% -o %appInstallDir%\%%p .\src\%%p
+    ) else (
+        dotnet build  -c %buildType% .\src\%%p
+    )
+)
+
+if %fx%==true (
+    dotnet publish -c %buildType% -o %fxInstallDir% .\src\packages
+
+    @REM remove package version of mscorlib* - refer to core root version for diff testing
+    if exist %fxInstallDir%\mscorlib* del /q %fxInstallDir%\mscorlib*
+)
+
+REM Done
+exit /b 0
+
+:usage
+echo.
+echo  build.cmd [-b ^<BUILD TYPE^>] [-f] [-h] [-p]
+echo.
+echo      -b ^<BUILD TYPE^> : Build type, can be Debug or Release.
+echo      -h                : Show this message.
+echo      -f                : Publish default framework directory in ^<script_root^>\fx.
+echo      -p                : Publish utilities.
+echo. 
+exit /b 1