Unreal Engine Crash Reporter

Overview Crash report system allows collecting crash information for developer analysis. Crash report system requires a server to process, aggregate and display crash data. This document describes ...

Updated over 4 years ago Edit Page Revisions
Legacy

Overview

Crash report system allows collecting crash information for developer analysis. Crash report system requires a server to process, aggregate and display crash data. This document describes the procedure for setting up the system on your own server.

The document describes the setup for running the system on a single machine. Epic is moving the system to the cloud to handle the increasing load. At the moment of writing, latest version of Unreal engine 4.19 doesn’t have source code for the CrashReportServer. The process is based on source code for 4.17.2 version.

Here is a typical timeline for a single crash:

  • An exception (or an assert) is triggered while running game application on the target system.
  • A crash folder is generated with log and dump files
  • CrashReportClient(CRC) is launched
  • CRC sends HTTP request to the crash report server with crash files
  • The crash files are received by CrashReportReceiver(CRR) on the server
  • The crash files are written to the LandingZone folder
  • CrashReportProcess(CRP) scans Landing Zone Folder and discover *new* crash folders
  • For every crash CRP:
    • If callstack is not symbolized, CRP executes the request to MinidumpDiagnostics (MDD)
    • Make an HTTP request to CrashReportWebsite(CRW) to add the crash
    • CRW add the crash to the CrashReportDatabase (CRD)
    • Move the crash report files to Processed Folder

When the crash has gone through this chain, developer could go to the crash report website in the browser and find the crash.
Note that for MDD to symbolize the callstack, a process need to upload the symbols to the server.

File:CrashReporter-SystemOverview.png

Edit diagram

File:CrashReporter-SystemOverview-Source.zip

Dependencies

CRW is written in ASP.NET with use of Entity Framework. That requires the server to run Windows OS + IIS with MS SQL database.
Before proceeding forward choose a database license that suits your demands:

The fork for Unreal Engine with crash report server sources files and fixes:

In CrashReporter branch

The system has been testing on Windows and Linux. Other platforms haven’t been tested.

Configuration

The configuration process has the following steps:
# Server Machine Setup

  1. Crash Report Database
  2. Crash Report Website
  3. Crash Report Client
  4. Crash Report Receiver
  5. Crash Report Process
  6. Versioning
  7. Minidump Diagnostics
  8. Symbol Uploader
  9. Crash Reporting For Editor

Server Machine Setup

To start come up with the web address for the crash reporter website. Note that system is capable to aggregate crashes from different games made with Unreal Engine if that applies to you.

Verify that web address is reachable by pinging it.

Crash Report Database

Download and install MS SQL server with MS SQL Management Studio.
Modify and execute sql script:

UnrealEngine\Engine\Source\Programs\CrashReporter\CrashReportWebSite\Models\CreateDatabase.sql
  • In CREATE DATABASE CrashReport statement modify the filename to a folder that exists.
  • Note that database max size depends on the license and might require editing.
  • If the following error occurs: cannot create more than one clustered index on table …
    • Modify Statement to create NONCLUSTERED index

When the database is created, execute the following script:

INSERT INTO dbo.UserGroups(Name) VALUES('Group1')
INSERT INTO dbo.UserGroups(Name) VALUES('Group2')
INSERT INTO dbo.UserGroups(Name) VALUES('Coders')
INSERT INTO dbo.UserGroups(Name) VALUES('Group4')
INSERT INTO dbo.UserGroups(Name) VALUES('General')
INSERT INTO dbo.UserGroups(Name) VALUES('Group6')
INSERT INTO dbo.UserGroups(Name) VALUES('Group7')
INSERT INTO dbo.UserGroups(Name) VALUES('Group8')
INSERT INTO dbo.UserGroups(Name) VALUES('Group9')
INSERT INTO dbo.UserGroups(Name) VALUES('Group10')
INSERT INTO dbo.UserGroups(Name) VALUES('Group11')
INSERT INTO dbo.UserGroups(Name) VALUES('Group12')
INSERT INTO dbo.UserGroups(Name) VALUES('Group13')
INSERT INTO dbo.UserGroups(Name) VALUES('Group14')
INSERT INTO dbo.UserGroups(Name) VALUES('Group15')
INSERT INTO dbo.UserGroups(Name) VALUES('Group16')
INSERT INTO dbo.UserGroups(Name) VALUES('Group17')
INSERT INTO dbo.UserGroups(Name) VALUES('Group18')
INSERT INTO dbo.UserGroups(Name) VALUES('Group19')
INSERT INTO dbo.UserGroups(Name) VALUES('Group20')
INSERT INTO dbo.UserGroups(Name) VALUES('Group21')
INSERT INTO dbo.UserGroups(Name) VALUES('Group22')
INSERT INTO dbo.UserGroups(Name) VALUES('Group23')
INSERT INTO dbo.UserGroups(Name) VALUES('Group24')
INSERT INTO dbo.UserGroups(Name) VALUES('Anonymous')
INSERT INTO dbo.Users(UserName, UserGroupId) VALUES('Anonymous', 25)

Add CrashReporter user with SQL authentication:
* In MS SQL Management Studio go to Security -> Logins -> Add New Login

  • Add SQL Server authentication
    • General
      • Uncheck password policy
      • Uncheck password expiration
      • Default database: CrashReport
    • Server Roles
      • Check sysadmin
    • Check CrashReport with Default schema ‘dbo’
  • Right click on the server in Object explorer
    • In security tab change to SQL Server and Windows Authentication mode
    • From Windows command line go to services and restart the SQL Server service.
  • Try to login with CrashReporter user

Crash Report Website

Open CRW solution with Visual Studio 2015:

UnrealEngine\Engine\Source\Programs\CrashReporter\CrashReportWebSite\CrashReportWebSite.sln
  1. Build the solution
  2. Publish the website to a folder.
  3. Copy published folder to the server.
  4. On the server:
  • Turn IIS
    • Go to Windows Features and enable:
      • Internet Information Services -> Web Management Tools -> IIS Management Console
      • Internet Information Services -> WWW Services -> App Developement -> ASP.NET 3.5 / 4.7
      • Internet Information Services -> WWW Services -> Health and Diagnostics
      • Internet Information Services -> WWW Services -> Performance Features
      • Internet Information Services -> Common HTTP Features -> Static Content
  • Modify Web.config:
    • DownloadFromS3 = false
  • Launch IIS Manager
    • Add a new website
    • Select physical path to published folder
    • Double click on connection strings and set the credentials to CrashReporter user for CrashReportEntities and HangFire
    • In IIS add mime types: 
      • .dmp : application/octet-stream

      • .log : text/plain

      • .runtime-xml : text/xml

In the browser go to crash reporter website. The page should be loaded with no crashes to present.

Crash Report Client

To test a crash, include a crash function into the code:

UFUNCTION(Exec)
void Crash()
{
checkf(false, TEXT("Crashing on purpose via Crash command.));
}

A good place for this function could be a project-specific cheat manager.
To package CRC with the game add

-crashreporter

switch to cooking the game

Package the game, launch and trigger a crash. If game runs with user interface a CRC window should appear.
Navigate to crashes folder and open the minidump with VisualStudio.
If Visual Studio is unable to match the PDBs that probably indicate the bug on the systems with Windows Fall Creators Update.
Consult this UDN question to resolve the issue:

(Please update to answerhub when it's available)

Crash Report Receiver

By default CRC is wired to Epic services.
Modify

UnrealEngine\Engine\Programs\CrashReportClient\Config\DefaultEngine.ini

to submit crashes to your server.

DataRouterUrl=
CrashReportReceiverIP=webaddress.com:57005

  • Build solution:

    UnrealEngine\Engine\Source\Programs\CrashReporter\CrashReporterServer.sln

  • Copy files to the server:

UnrealEngine\Engine\Binaries\DotNET\CrashReportCommon.dll
UnrealEngine\Engine\Binaries\DotNET\CrashReportReceiver.exe
UnrealEngine\Engine\Binaries\DotNET\CrashReportReceiver.exe.config

Create Landing Zone Folder and specify it in CrashReportRepository value inside the CRR config file.

Launch CRR as administartor.
The application will print few lines into console:

CrashReportReceiver, Copyright 2014-2017 Epic Games, Inc.
[STATUS] Saved to ***\ReceivedReports.xml
Landing zone: ***

Launch browser on the server and go to this page
The reply must be an XML similar to this one:

Now crash the game and inspect the output for CRR. At some point the following log messages will appear:

[STATUS] A new report has been received into ***
[STATUS] Saved to ***\ReceivedReports.xml
[PERF  ] CPU Usage: 08%, Received data:     0MB/00.05 of 10.00 MB, Uploads:  0, Files:     1,     0 : 00%

Inspect Landing Zone Folder and find a new crash folder been created.
To finish this step setup a windows service to run CRR. For details consult:

The logs can be accessed at Logs folder near the CRR executable.

Crash Report Process

  • Copy following files to the server:
UnrealEngine\Engine\Binaries\DotNET\AWSSDK.dll 
UnrealEngine\Engine\Binaries\DotNET\CompressionHelper.dll 
UnrealEngine\Engine\Binaries\DotNET\CrashReportCommon.dll 
UnrealEngine\Engine\Binaries\DotNET\CrashReportProcess.exe 
UnrealEngine\Engine\Binaries\DotNET\CrashReportProcess.config
  • Modify CrashReportProcess.config
REPORTS FOLDER  
WEB ADDRESS 
VIDEO FOLDER 
A dummy folder 
LANDING ZONE FOLDER (SEE CRR config) 
true 
false 
Path to index folder

Note that all folders must exist!
Open IIS Manager:
Edit the website and add virtual folders with name CrashReporterFiles and CrashReporterVideos pointing to processed reports and videos folders
Install CRP as service.
Trigger a crash from the game.
The crash will appear in Landing Zone Folder first and will be transferred to Processed Folder.
Goto CRW and check that crash listed in the crash list

Versioning

To distinguish two builds version file needs to be updated.
There is a build command with name UpdateLocalVersion that suppose to automate that. However the command requires P4 access and that doesn't work if code is stored in Git.
The idea is simple thought. Something have to update

UnrealEngine\Engine\Source\Runtime\Launch\Resources\Version.h

And specify branch and changelist. A small script could handle this job.

Be aware that the file change adds up to the build time and generate lots cooking warning due to version mismatch.

MinidumpDiagnostics

The configuration above will work but requires PDBs published with executable. MDD allows to process minidumps on the server where all PDBs will have to be stored. Not that this process is not required for Linux platform.

  • Open UE4 solution and build MinidumpDiagnostics (MDD)
  • Copy executable 
    UnrealEngine\Engine\Binaries\Win64\MinidumpDiagnostics.exe
    to a server folder 
    \Program\Binaries\Win64\
  • Modify CrashReportReceiver.config:
Path To MDD exe 
Path to symbols Root
  • Create Engine.ini in folder 
    \Engine\Programs\MinidumpDiagnostics\Saved\Config\Windows\Engine.ini
  • Edit Engine.ini:
PDBCache_0_Branch=BranchName 
PDBCache_0_ExecutablePathPattern=\%BUILD_VERSION%\%PLATFORM_NAME% 
PDBCache_0_SymbolPathPattern=\%BUILD_VERSION%\%PLATFORM_NAME%

Symbol Uploader

To automate the upload a small script can be created that uploads files to crash report server for every packaged game. The script transfers all files that match file mask *.pdb and maintains the folder structure.
The uploader writs to symbol folder of this structure:

\Branch-CL-BuildNumber\PlatformName (In UE terms - WindowsClient, WindowsServer, etc)

Crash Reporting For Editor

The process for editor crashes is similar to packaged game with few exceptions.
CrashReportClient settings need to be set in BaseEngine.ini:

[CrashReportClient]
CrashReportReceiverIP="http://webaddress:57005"

When uploading symbols these folders need to be considered:

UnrealEngine/Engine/Binaries/Win64
UnrealEngine/Engine/Plugins
UnrealProject/Binaries/Win64
UnrealProject/Plugins