Opened 10 years ago

Last modified 2 weeks ago

#1814 new task

[Git] [Step 2] Create update.bat to sync windows binaries

Reported by: Kieran P Owned by:
Priority: Must Have Milestone: Backlog
Component: Build & Packages Keywords:
Cc: Patch:

Description (last modified by Christopher Buteau)

With our move to git, we will be removing pyrogenesis.exe, pyrogenesis.pdb, and AtlasUI.dll (these three get generated by the autobuilder). As a result, we need an update.bat, or update.exe (whichever is easier) for windows users which can pull the latest binaries.

(quick background, the autobuilder will generate binaries for different code branches, and when its done, it will commit an updated manifest.txt file into the branch the binaries relate to, which points to the latest files)

  • Use the manifest.txt file in the 0 A.D. code repo (probably stored at binaries/system), which contains a list of files, an md5 checksum for each file, and a url to download them from. Here is an example manifest:
AtlasUI.dll 52fe3f8c2725445b488f646dca28021f http://.../AtlasUI.dll
pyrogenesis.exe 89158e5ea74d5987ed8be026e7f090ce http://.../pyrogenesis.exe
pyrogenesis.pdb 44889d0fbf2f48972992c2a4d9e629c2 http://.../pyrogenesis.pdb
  • Compare the manifest with the md5 of the current files. If no current file, download it. If existing file exists, find out if checksum has changed, and download the file if different.
  • Once a file is downloaded, check file downloaded properly by comparing the files checksum with the checksum of the manifest.

This will allow us to easily add files in the future if need be (i.e. additional windows binary/dlls) and allows artists quick access to pull the latest binaries.

Change History (19)

comment:1 by Kieran P, 10 years ago

Description: modified (diff)

comment:2 by zoot, 10 years ago

I wonder if it would be useful to make some kind of launcher (perhaps separately from the updater), which would automatically update the binaries and then, once the update is finished, launch the game.

Then the artists would just have to get into the habit of starting the launcher instead of pyrogenesis.exe when they want to open the game - the rest of the process would be as seamless as it is now.

comment:3 by Kieran P, 10 years ago

Summary: Create update.bat to sync windows binaries[Git] [Step 2] Create update.bat to sync windows binaries

comment:4 by Kieran P, 10 years ago

Milestone: Alpha 13Alpha 14

comment:5 by historic_bruno, 10 years ago

Owner: set to Philip Taylor

comment:6 by Beau, 9 years ago

Owner: changed from Philip Taylor to Beau

comment:7 by Beau, 9 years ago

Owner: changed from Beau to Philip Taylor

comment:8 by historic_bruno, 9 years ago

Latest plan is to have autobuilt binaries stored in SVN (see #1819), which is easy on Linux and OS X; we specify that as part of the build process and make an e.g. Python script to automate grabbing dependencies, if applicable.

It's a bit more complicated on Windows, as the required command line tools aren't included by default or "easy" for people to acquire and install. Instead, we'll have a dev package that people download, that contains win32 binaries for svn, git, python, and their dependencies. They extract the package where we tell them, then run the update script to retrieve autobuilt and library win32 binaries from the SVN repo (this part will work the same as on Linux/OS X).

comment:9 by Kieran P, 9 years ago

Milestone: Alpha 14Alpha 15

comment:10 by leper, 9 years ago

Milestone: Alpha 15Alpha 16

comment:11 by leper, 9 years ago

Milestone: Alpha 16Alpha 17

comment:12 by historic_bruno, 9 years ago

Owner: Philip Taylor removed

comment:13 by historic_bruno, 8 years ago

NSClient++ (an open source server monitoring client) has an interesting build process on Windows, and may contain some bits we find useful. I recently built it to debug an issue I encountered. The process is more painful than ours, because no prebuilt win32 binaries are provided by the project, instead the dependency source code is pulled in via Python scripts (very similar to our build-osx-libs.sh on OS X) and everything is then built with msbuild (MSVC command line tool). But things break, and it requires some fiddling to adjust the script and the options aren't well documented.

However, we might be able to use their build scripts to automate rebuilding of our Windows dependencies, it has some nice features like conversion of old VC projects to the newer format when a newer compiler is used, supports 32- and 64-bit builds, and creates debug and release builds.

There may be other things we can learn from the project, for instance they switched from SVN to git a few years ago. They use Travis as continuous integration platform.

Last edited 8 years ago by historic_bruno (previous) (diff)

comment:14 by leper, 8 years ago

Milestone: Alpha 17Alpha 18

comment:15 by Itms, 8 years ago

Milestone: Alpha 18Alpha 19

comment:16 by mimo, 7 years ago

Milestone: Alpha 19Alpha 20
Priority: Release BlockerMust Have

comment:17 by mimo, 7 years ago

moved to backlog as no progress since ages

comment:18 by mimo, 7 years ago

Milestone: Alpha 20Backlog

comment:19 by Christopher Buteau, 4 weeks ago

Description: modified (diff)

Hello,

I just found your project. This looks like a great starter project.

I script at work using node js. I think I can build something with node js and configuration to do the work but I will need help debugging the other platforms.

Lets start with a simple command line utility using yargs. http://yargs.js.org/

See we build a tool that does the trick and then you integrate with your autobuild. We just get the build creation of manifest and upload working. Which where is the manifest to be stored? Is it a separate manifest for each target MacOS, Windows, Linux Or one manifest for all three?

So working on node js we can use things like these to get shit done. https://www.npmjs.com/package/prebuilt-download

Lets start talking the format of the manifest. I am assuming json ...if not be convincing.

I wrote a JSON parser generator.

{
  "ver": {
    "major": 0,
    "minor": 13
  },
  "targets": {
    "AtlasUI.dll": {
      "md5": "52fe3f8c2725445b488f646dca28021f",
      "get": "!http://.../AtlasUI.dll"
  },
    "pyrogenesis.exe": {
      "md5": "89158e5ea74d5987ed8be026e7f090ce",
      "get": "!http://.../pyrogenesis.exe"
  },
    "pyrogenesis.pdb": {
      "md5": "44889d0fbf2f48972992c2a4d9e629c2",
      "get": "!http://.../pyrogenesis.pdb"
    }
  }
} 

I think we can use this or another lib to do the download. https://www.npmjs.com/package/prebuilt-download

So first we get the generation of the manifest and its upload to whatever target.

Next we put together the thing that downloads processes the manifest and downloads the target.

Since you guys are batch experts.
Start with a name.
like tlz or something short.
make a tlz.bat and dtlz.bat that execute a main.bat that executes a node js script.

one will launch it like this...
node mainScript.js
and the other
node --inspect mainScript.js 
and I can take it from there building a command structure.

See we can feed yargs a bunch of command objects.

http://yargs.js.org/docs/#api-reference-commanddirdirectory-opts

It will be a series of command in the style of git...
tlz manifest platform
tlz digest platform

Hopefully I set up my TRAC notifications so I get a message if anyone thinks this is a good topic to revive.

Last edited 2 weeks ago by Stan‘ (previous) (diff)
Note: See TracTickets for help on using tickets.