This page is entirely outdated, automated builds on Windows are performed by Jenkins as described in JenkinsSetup.

(Related to ticket #224.)

NOTE: This document is partly planning from before implementation, partly experience from after - not everything here is actually implemented.

Rough overview:

  • The server runs on Amazon EC2.
  • The server has Windows Server 2003, VC++ 2008, SVN, Perl, and little else installed.
  • The server has a persistent EBS disk with the SVN checkout.
  • On startup, the server runs a script. The script updates SVN, builds, saves the log output to S3, and if it was successful then it commits the new binaries.
  • A permanent non-Windows server runs some scripts that provide an authenticated web interface to (re)start the build server, and terminate it after some period of time when it's done.

All code is available in SVN. (Only configuration files with private data are excluded.)

Virtual machine image installation

The software versions listed below are the latest when I set this up - more recent versions ought to work fine too.

  • Create a 2GB EBS disk in us-east-1c. (...or any other availability zone, but it needs to match the instance's zone, and I can only make Windows instances in us-east-1c.)
  • Start with basic Win2003 image (ami-0456b26d, ec2-public-windows-images/Server2003r2-i386-Win-v1.03.manifest.xml), on a Small Instance in us-east-1c.
  • Change the Administrator password, and write it down somewhere.
  • Install VC++ 2008 Express SP1. (Skip the Silverlight and SQL Server options). Register it.
  • Install command-line SVN 1.5.5.
  • Install ActivePerl.
  • Install Net::Amazon::EC2, Amazon::S3, DateTime::Format::ISO8601 using the Perl Package Manager.
  • Download SDelete, extract into c:\bin
  • Download Sync, extract into c:\bin
  • Create the registry key HKEY_USERS\S-1-5-18\Software\Sysinternals\Sync, set DWORD EulaAccepted = 1
  • Attach the EBS disk.
  • If the disk hasn't been set up already: Administrative Tools -> Computer Management -> Disk Management -> Disk 2 -> Initialize Disk, New Partition -> drive letter E.
  • svn co http://svn.wildfiregames.com/svn/ps/trunk/ e:\svn
  • Follow the wxWidgets installation instructions in e:\svn\libraries\wxwidgets\README.txt
  • mkdir c:\0ad\autobuild\
  • copy e:\svn\source\tools\autobuild2\startup.pl c:\0ad\autobuild\
  • copy e:\svn\source\tools\autobuild2\aws.conf.example c:\0ad\autobuild\aws.conf
  • Set up c:\0ad\autobuild\aws.conf:
    • aws_access_key_id, aws_secret_access_key are from your Amazon account.
  • Set up the build-runner server's run.conf (which will be copied to the build server on startup):
    • ebs_volume_id is from the EBS volume you created.
    • svn_username, svn_password are the SVN login details.
  • copy e:\svn\build\autobuild2\startupservice.exe c:\0ad\autobuild\
  • sc create Autobuilder binPath= c:\0ad\autobuild\startupservice.exe start= auto depend= Ec2Config
  • Start -> Programs -> Ec2Service Setting -> disable "Set Computer Name". (That feature changes the computer name then reboots, which is annoying.)
  • Bundle the instance:
    • To save some space, delete c:\windows\$*Uninstall*$, c:\windows\temp\*.
    • Defragment drive C.
    • c:\bin\sdelete -c c:
    • Bundle into s3://wfg-autobuild-images/vc2008
    • Register the bundle as an AMI.
  • Set up manage.conf on the build-runner server:
    • image_id is from the bundled AMI.

Build script

The startup.pl script (stored on the image on S3) does the following:

  • Create d:\0ad. (The script might run before the data disk has been mounted on d:, so it repeats until it's successful.)
  • Download the EC2 user data resource, then unzip it and execute its run.pl.

The user data contains run.pl and run.conf.example, uploaded by the build-runner script. startup.pl is hard to modify (since it's frozen into the bundle), so it does as little as possible and defers to run.pl for most of the work.

run.pl does something like:

  • Start writing logs on S3. (We use S3 just because it's a convenient web-accessible hosting service.)
  • Attach the EBS disk to the machine.
  • svn up on the EBS disk.
  • Run the rest of the build script (build.pl) from the SVN directory.
  • Shut down the machine.

build.pl does:

  • Copy SVN (excluding the binaries directory) to the local disk.
  • Store the SVN revision in build\svn_revision\svn_revision.txt
  • update-workspaces.bat (optionally with --atlas and/or --collada depending on the build-runner's configuration)
  • vcbuild /time /M2 /logfile:d:\0ad\buildlogs\build_vcbuild.txt vc2008\pyrogenesis.sln "Release|Win32"
  • Copy compiled binaries back into the SVN directory.
  • svn commit -m 'Automated build.'

Build runner

  • Runs on same server as SVN.
  • Uses SVN auth info to only allow registered developers to trigger builds (because they cost money).
  • The server also runs a cron job every 5 minutes, that checks if a build was started more than 50 minutes ago, and calls TerminateInstances if so. (That should be enough time for a legitimate build to end, and prevents a runaway machine costing lots of money.)
  • The web interface also provides an admin-only 'down for maintenance' mode that prevents any builds and pauses that cron job, so I can manually run the build server without interference.

EC2

Handy tools: ElasticFox, s3://.

Alternatives to EC2

Currently (December 2008) the only suitable alternatives to EC2 I've seen are FlexiScale and GoGrid. Since all the services provide a plain Windows environment, it should be fairly easy to transition between them if necessary.

EC2 appears to offer the best value. Comparing the lower-end offerings from each:

ServiceRAM (GB)CPU (EC2 units)Machine per hourStorage per GB-monthTotal per month
EC21.71$0.125$0.15$4.55
EC21.75$0.30$0.15$9.88
FlexiScale0.50.5 (?)£0.08£0.30£3.93
FlexiScale0.51 (?)£0.11£0.30£4.85
FlexiScale1.00.5 (?)£0.13£0.30£5.45
GoGrid0.50.125 (?)$0.095$0$2.90
GoGrid1.00.25 (?)$0.190$0$5.78

(This assumes 5GB of storage, and 1 hour usage per day.)

(£1 UKP = $1.50 USD right now. Tax (15%) is not included for EC2; unsure about others.)

Regarding CPU: "One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor". FlexiScale says "I would expect a typical 2gb virtual server with us to get more CPU resources compared to Amazon", implying that their "2 CPU" offering is roughly comparable to 1 EC2 Compute Unit. GoGrid says "we will always maintain a ratio of 1 Xeon core (equivalent to a P4 2.0 chip) to 4 GB of RAM across our entire grid", which sounds like about 1 EC2 Compute Unit per 4GB.

Builds take somewhere around 20 minutes on 1 EC2 Compute Unit, which is barely adequate. Anything less than that is rubbish.

Turns out that GoGrid actually charges even when the machine is switched off, since you pay for reserved resources, so that's not acceptable. But they claim they're looking into changing that, in which case they'd be more suitable.

Last modified 3 months ago Last modified on Jan 23, 2017, 1:07:36 PM