Jenkins Example with Perforce
This page provides information on setting up a basic starter Jenkins CI environment.
Introduction
This page provides information on setting up a basic Jenkins environment (running on Windows) which will automatically sync from a Perforce server (running on Linux), build binaries, and submit binaries back to Perforce. This page assumes that you:
-
Have no experience with Jenkins,
-
Have a running Perforce server on a Linux VPS,
-
Are experienced with Perforce and Linux, and
-
Are using an EGS Launcher version of Unreal Engine.
Goal
This example setup uses Stream depots with Perforce. The intended workflow is for general users to work in the regular mainline stream, coders will work in a virtual stream of mainline which excludes all Binaries folders in your project and Jenkins will work in a virtual stream of mainline which excludes any Content folders in your project. The result is that coders can "just work", creating code changes and submitting them. Jenkins will be notified by Perforce when a source code file is changed in the mainline stream and automatically start running. Jenkins will sync to the latest changes from Perforce, initiate a build, and (if successful) add generated DLLs to a new Changelist and submit it. Artists/designers will receive the DLLs when they run Get Latest from Perforce.
The machine running Jenkins must be capable of syncing from Perforce and building Unreal projects. This means the machine must:
-
Have Perforce installed,
-
Have Visual Studio Build Tools installed (you can install Visual Studio Community Edition, or you can install the basic CLI Build Tools), and
-
Have Unreal Engine/Editor installed (or plan to build it from source, possibly with a separate Jenkins Job, which is beyond the scope of this example),
This setup is intended to be used on a computer which is not actively used for UE4 development. Live Coding may prevent MSVS builds from running while the editor is open, even if you set up Jenkins on a separate user login. It may be possible to install and run everything within a Virtual Machine (untested, and beyond the scope of this example).
Perforce Setup
Jenkins Stream
Create a virtual stream based off of your project's Mainline stream named Jenkins. In the Advanced tab of the Stream settings, under Paths, enter:
share Binaries/...
share Plugins/SomePluginName/Binaries/... (Specify any Plugin's Binaries folders like this)
exclude Content/... (Prevent Jenkins from downloading Content since it won't need it)
exclude ContentSource/... (Specify folders that won't be needed for DLL building like this)
isolate ... (Ensure Jenkins can't submit anything except Binaries)
Coders Stream
Create a second virtual stream also based off of the Mainline stream named MainlineCoding. In the Advanced tab, under Paths, enter:
share ...
exclude Binaries/...
exclude Plugins/SomePluginName/Binaries/... (Specify any Plugin Binaries folders like this)
After creating both virtual streams you should have a Stream Graph with the three grey streams below:
Workspaces
Ensure general developers create their workspaces using the Mainline stream and coders create their workspaces using the MainlineCoding stream.
Jenkins Setup
Download and install from Jenkins Website.
Jenkins may prompt you whether to run as LocalSystem or under a User Account. It is recommended to run Jenkins under a User Account. LocalSystem is considered an insecure global user and you may run into permissions issues when trying to make Jenkins perform certain actions such as use SSH.
When setup completes, Jenkins should automatically launch. After following the instructions to Unlock Jenkins, it will prompt you about plugins. Choose Select plugins to install. Ensure the following plugins are selected (others are optional). We will add more plugins later from within Jenkins.
-
P4
-
Matrix Authorization Strategy
When prompted, specify login info for an Admin User.
When prompted, specify the URL from which Jenkins will be accessible by all users. This should be something like http://JenkinsServerPublicIPAddress:8080/. Jenkins may insert this URL into page data so it may be necessary to specify it to avoid users from getting broken links.
Installation should now be complete. Jenkins will open up to its front page.
Jenkins - Creating a build user and enabling anonymous access
We will create a separate Jenkins "build user" which only has permissions to control builds, and we will enable public read-only access which is useful for allowing team members to look at build logs.
Create a build user
From the main page, select Manage Jenkins. Select Manage Users. Select Create User. Enter in user data and press OK. You are returned to the Users page. Below I've created user buildbot.
Set up security permissions
Go to Manage Jenkins. Select Configure Global Security. Under Authorization, choose Matrix-based security. Press Add user or group... and enter your build user's name. Set up your security matrix as desired, see example below.
Create an API Token for your build user
A token will be used later to authenticate remote requests. Open a second browser window in incognito/private mode. Log in to Jenkins again (e.g. go to localhost:8080), this time logging in as the build user you created. Click on your username on the top bar to bring you to the user page. Click on Configure. Under API Token press Add new token. Give it a name (e.g. perforce) and press Generate. Copy this token somewhere safe for now. Press Save and close this browser window.
Jenkins - Getting necessary plugins
This example setup will use the following additional plugins:
-
PostBuildScript Plugin
-
Pre SCM BuildStep Plugin
-
SCM Skip Plugin
-
Environment Injector Plugin
Go to Manage Jenkins. Go to Manage Plugins. Click on the Available tab. Search for and install each of the above plugins (search for a plugin and check it on, then search for the next one and check it on), press Download now and install after restart. After the screen changes, select Restart Jenkins when installation is complete and no jobs are running. Wait for it to complete, and log into Jenkins again.
Jenkins - Setting up global settings
Environment Variables
Go to Manage Jenkins. Select Configure System. Under Global Properties turn on Environment Variables. Add variables PROJECTNAME_UE4_VERSIONSELECTOR and PROJECTNAME_UE4_REBUILDBAT. Our Job will use these variables to find UE4 utilities. By creating these as global variables we can use them in other Jobs in the future and easily update them if we change engine versions.
Set up Perforce Credentials
Go to Manage Jenkins. Select Manage Credentials. Click into the Global store (beside the Jenkins domain). Select Add Credentials from the left pane. Change the Kind to a Perforce Password Credential and specify your desired Perforce login info, which Jenkins will use to connect to Perforce.
Jenkins - Creating the Job
At the front page, select New Item from the left pane. Select Freestyle project, give the Job a name, and press OK. The configuration window for a new Job appears.
Source Code Management
Select Perforce Software'. Beside Perforce Credentials, select the credential you created before. Under Workspace Behaviour, select Streams and type your stream name into the Stream Codeline (e.g.
//StreamDepotName/MainlineStream
). Under Populate Options, select Auto cleanup and sync.
Build Triggers
You can leave everything here unchecked. For information only, temporarily check on Trigger builds remotely (e.g., from scripts). Note that it states we can use a URL to trigger builds remotely - take note of this URL, we will use it later. Jenkins does not actually need this option turned on for remote triggered builds to work, unless you want to use an Authentication Token. We will skip use of a Token for this example; uncheck Trigger builds remotely.
Build Environment
Select Inject environment variables to the build process. Under Properties Content add:
Build
Click Add build step and select Execute Windows batch command. Enter:
CALL "%PROJECTNAME_UE4_REBUILDBAT%" "%BUILDCONFIG%" "%PLATFORM%" "%CONFIG%" "%WORKSPACE%\%PROJECTNAME%.uproject" -WaitMutex -FromMsBuild
IF %ERRORLEVEL% NEQ 0 (EXIT %ERRORLEVEL%)
Post-build Actions
Click Add post-build action and select Perforce: Publish assets. Set it up similarly as the Source Code Management section above. Under Publish options, select Submit change. Enter a suitable description (e.g. "Binaries updated by Jenkins. Build: ${BUILD_TAG}"). Activate Only publish on build success. Click on Advanced, and in Publish paths enter any folders in the depot which will contain binary files that are being updated. This should include your main game project Binaries folder as well as any Plugins\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\PluginName\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Binaries folders.
Press Save. You will be returned to the front page for the Job.
Test it
At this point we should have a working Job. Press Build Now from the left pane. A build should appear running under the Build History. Click on the dropdown arrow beside the build entry and click on Console Output to watch the build progress.
The Jenkins Perforce plugin may not add "new" files with this setup. It may be necessary to first add and submit the binaries manually with P4V once. After this, Jenkins will detect its built binaries as changed files and should submit them.
Perforce - Creating a Build-trigger Script
Pending - create a script that initiates a build with cURL
Perforce - Setting up Triggers to Launch the Build-trigger Script
Pending - set up p4 triggers to run the script