[Git] [Step 2] Create update.bat to sync windows binaries

[Kieran P]

With our move to git, we will be removing pyrogenesis.exe, pyrogenesis.pdb, and AtlasUI.dll (these three get generated by the autobuilder). As a result, we need an update.bat, or update.exe (whichever is easier) for windows users which can pull the latest binaries.

(quick background, the autobuilder will generate binaries for different code branches, and when its done, it will commit an updated manifest.txt file into the branch the binaries relate to, which points to the latest files)

• Use the manifest.txt file in the 0 A.D. code repo (probably stored at binaries/system), which contains a list of files, an md5 checksum for each file, and a url to download them from. Here is an example manifest:

AtlasUI.dll 52fe3f8c2725445b488f646dca28021f http://.../AtlasUI.dll
pyrogenesis.exe 89158e5ea74d5987ed8be026e7f090ce http://.../pyrogenesis.exe
pyrogenesis.pdb 44889d0fbf2f48972992c2a4d9e629c2 http://.../pyrogenesis.pdb

• Compare the manifest with the md5 of the current files. If no current file, download it. If existing file exists, find out if checksum has changed, and download the file if different.

• Once a file is downloaded, check file downloaded properly by comparing the files checksum with the checksum of the manifest.

This will allow us to easily add files in the future if need be (i.e. additional windows binary/dlls) and allows artists quick access to pull the latest binaries.

[zoot]

I wonder if it would be useful to make some kind of launcher (perhaps separately from the updater), which would automatically update the binaries and then, once the update is finished, launch the game.

Then the artists would just have to get into the habit of starting the launcher instead of pyrogenesis.exe when they want to open the game - the rest of the process would be as seamless as it is now.

[historic_bruno]

Latest plan is to have autobuilt binaries stored in SVN (see #1819), which is easy on Linux and OS X; we specify that as part of the build process and make an e.g. Python script to automate grabbing dependencies, if applicable.

It's a bit more complicated on Windows, as the required command line tools aren't included by default or "easy" for people to acquire and install. Instead, we'll have a dev package that people download, that contains win32 binaries for svn, git, python, and their dependencies. They extract the package where we tell them, then run the update script to retrieve autobuilt and library win32 binaries from the SVN repo (this part will work the same as on Linux/OS X).

[historic_bruno]

​NSClient++ (an open source server monitoring client) has an interesting ​build process on Windows, and may contain some bits we find useful. I recently built it to debug an issue I encountered. The process is more painful than ours, because no prebuilt win32 binaries are provided by the project, instead the dependency source code is pulled in via ​Python scripts (very similar to our build-osx-libs.sh on OS X) and everything is then built with msbuild (MSVC command line tool). But things break, and it requires some fiddling to adjust the script and the options aren't well documented.

However, we might be able to use their build scripts to automate rebuilding of our Windows dependencies, it has some nice features like conversion of old VC projects to the newer format when a newer compiler is used, supports 32- and 64-bit builds, and creates debug and release builds.

There may be other things we can learn from the project, for instance they switched from SVN to git a few years ago. They use ​Travis as continuous integration platform.

[mimo]

moved to backlog as no progress since ages

[Christopher Buteau]

Hello,

I just found your project. This looks like a great starter project.

I script at work using node js. I think I can build something with node js and configuration to do the work but I will need help debugging the other platforms.

Lets start with a simple command line utility using yargs. ​http://yargs.js.org/

See we build a tool that does the trick and then you integrate with your autobuild. We just get the build creation of manifest and upload working. Which where is the manifest to be stored? Is it a separate manifest for each target MacOS, Windows, Linux Or one manifest for all three?

So working on node js we can use things like these to get shit done. ​https://www.npmjs.com/package/prebuilt-download

Lets start talking the format of the manifest. I am assuming json ...if not be convincing.

I wrote a JSON parser generator.

{
  "ver": {
    "major": 0,
    "minor": 13
  },
  "targets": {
    "AtlasUI.dll": {
      "md5": "52fe3f8c2725445b488f646dca28021f",
      "get": "!http://.../AtlasUI.dll"
  },
    "pyrogenesis.exe": {
      "md5": "89158e5ea74d5987ed8be026e7f090ce",
      "get": "!http://.../pyrogenesis.exe"
  },
    "pyrogenesis.pdb": {
      "md5": "44889d0fbf2f48972992c2a4d9e629c2",
      "get": "!http://.../pyrogenesis.pdb"
    }
  }
} 

I think we can use this or another lib to do the download. ​https://www.npmjs.com/package/prebuilt-download

So first we get the generation of the manifest and its upload to whatever target.

Next we put together the thing that downloads processes the manifest and downloads the target.

Since you guys are batch experts.
Start with a name.
like tlz or something short.
make a tlz.bat and dtlz.bat that execute a main.bat that executes a node js script.

one will launch it like this...
node mainScript.js
and the other
node --inspect mainScript.js 
and I can take it from there building a command structure.

See we can feed yargs a bunch of command objects.

​http://yargs.js.org/docs/#api-reference-commanddirdirectory-opts

It will be a series of command in the style of git...
tlz manifest platform
tlz digest platform

Hopefully I set up my TRAC notifications so I get a message if anyone thinks this is a good topic to revive.

[hamletmun]

If you create ​https://trac.wildfiregames.com/export/HEAD/ps/trunk/binaries/system/SHA256SUMS

510612bace4951e3e7678c36b01cdbeb45404200185cbcb07bede15ee4e4224f *AtlasUI.dll
f2acb29c3c47f8ebfdd7d61da32cb278508845094f8c75191da81974a387efc2 *pyrogenesis.exe
e3af819ee8eceea566227ac2eecc3d7f6ff6541ef3f10cad4ad0604cb36c7e09 *pyrogenesis.pdb

you could use update.bat​ or update.ps1​