Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#3499 closed defect (fixed)

OOS without rejoin - Pathfinder on OS X

Reported by: Ben Brian Owned by: ben
Priority: Release Blocker Milestone: Alpha 19
Component: Core engine Keywords:
Cc: Patch:

Description

While testing my patch for #3108, I have encountered a different OOS. It seems to occur the first time a player moves a unit and the diff is all in the pathfinder data:

diff --git "a/C:\\Users\\Ben\\Desktop\\oos_dump_client_osx.txt" "b/C:\\Users\\Ben\\Desktop\\oos_dump_host_win.txt"
index 64d7605..3559730 100644
--- "a/C:\\Users\\Ben\\Desktop\\oos_dump_client_osx.txt"
+++ "b/C:\\Users\\Ben\\Desktop\\oos_dump_host_win.txt"
@@ -35402,8 +35402,8 @@ entities:
     group: 6667
     key: 23
     entity: 6668
-    x: 291.37151
-    z: 1239.88567
+    x: 291.09096
+    z: 1238.9258
     clearance: 1
     flags: 21
     group: 6668
@@ -163352,8 +163352,8 @@ entities:
     flags: 1
     scripted visibility: 0
     key: 6668
-    x: 291.37151
-    z: 1239.88567
+    x: 291.09096
+    z: 1238.9258
     vision: 0
     visibilities: 0
     size: 1
@@ -796328,7 +796328,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -798710,8 +798710,8 @@ entities:
   Footprint:
   Minimap:
     active: true
-    x: 291.37151
-    z: 1239.88567
+    x: 291.09096
+    z: 1238.9258
   Obstruction:
     active: true
     moving: true
@@ -798725,9 +798725,9 @@ entities:
     owner: 0
   Position:
     in world: true
-    x: 291.37151
+    x: 291.09096
     y: 0
-    z: 1239.88567
+    z: 1238.9258
     last x: 290.76642
     last y diff: 0
     last z: 1237.8154
@@ -798759,20 +798759,20 @@ entities:
     facePointAfterMove: false
     length: 0
     length: 1
-    waypoint x: 291.53164
-    waypoint z: 1239.62586
+    waypoint x: 291.25109
+    waypoint z: 1238.666
     passability map changed recently: false
     length: 0
     expected path ticket: 0
     next step clean: true
     type: 2
-    goal x: 291.53164
-    goal z: 1239.62586
+    goal x: 291.25109
+    goal z: 1238.666
     goal u x: 0
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -800348,7 +800348,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -815058,7 +815058,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1234873,7 +1234873,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1237120,7 +1237120,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1237387,7 +1237387,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1241336,11 +1241336,9 @@ entities:
     moving: true
     facePointAfterMove: false
     length: 0
-    length: 2
-    waypoint x: 1089.55315
-    waypoint z: 573.57074
-    waypoint x: 1089.8196
-    waypoint z: 574.31116
+    length: 1
+    waypoint x: 1089.42556
+    waypoint z: 574.69596
     passability map changed recently: false
     length: 0
     expected path ticket: 0
@@ -1241352,7 +1241350,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1241487,7 +1241485,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1243074,7 +1243072,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:
@@ -1246113,7 +1246111,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 9
+    goal hw: 8
     goal hh: 0
     maxdist: 0
   Vision:

Attachments (2)

oos_dumps.7z (1.4 MB) - added by Ben Brian 4 years ago.
oos_dumps_r17108.7z (156.1 KB) - added by elexis 4 years ago.
Stan and me could reproduce the issue. He was using OSX 10.9.5. (Mavericks) and I'm on linux. We moved some units and got that OOS without rejoining. There are some LOS changes in the diff, but it seems all of them could be caused by the different waypoints. Notice we had some svn games using windows and linux and never noticed an OOS without rejoin there.

Download all attachments as: .zip

Change History (11)

Changed 4 years ago by Ben Brian

Attachment: oos_dumps.7z added

comment:1 Changed 4 years ago by Ben Brian

Seems to be an OS X issue: with the same map and settings but an Ubuntu 12.02 client and Windows 7 host, there is no OOS.

comment:2 Changed 4 years ago by elexis

Priority: Should HaveMust Have
Summary: OOS with Windows host and OS X (10.10) clientPathfinder OOS with Windows host and OS X (10.10) client

So this is different from #3292 because it happens without rejoin?

comment:3 Changed 4 years ago by Ben Brian

Really I don't have time to compare the two, but that was my assumption. This is a guaranteed OOS between OS X 10.10 and Windows, it happens instantly with an AI (presumably they move units more quickly than I do).

Changed 4 years ago by elexis

Attachment: oos_dumps_r17108.7z added

Stan and me could reproduce the issue. He was using OSX 10.9.5. (Mavericks) and I'm on linux. We moved some units and got that OOS without rejoining. There are some LOS changes in the diff, but it seems all of them could be caused by the different waypoints. Notice we had some svn games using windows and linux and never noticed an OOS without rejoin there.

comment:4 Changed 4 years ago by elexis

Priority: Must HaveRelease Blocker
Summary: Pathfinder OOS with Windows host and OS X (10.10) clientOOS without rejoin - Pathfinder on OS X (10.9.5 and 10.10)

comment:5 Changed 4 years ago by Ben Brian

Summary: OOS without rejoin - Pathfinder on OS X (10.9.5 and 10.10)OOS without rejoin - Pathfinder on OS X

Also confirmed on 10.11. r16751 produces the OOS, r16750 does not. My current thinking is it's something related to libc++, if anyone on Linux could test build with clang and libc++ it might help.

comment:6 Changed 4 years ago by Ben Brian

I made a minimal test map, empty except for two enemy infantry. The units start isolated and can move without causing an OOS. However, an OOS occurs as soon as one of the units sees the other and initiates an attack order.

To reduce the possible cases, I made player 2's unit stand ground while player 1 approached, here is the resulting oos_dump diff (only player 1's unit differs):

--- windows-oos_dump.txt
+++ osx-oos_dump.txt
@@ -76153,7 +76153,7 @@ entities:
     facePointAfterMove: true
     length: 2
     waypoint x: 519.5
-    waypoint z: 322.5
+    waypoint z: 323.5
     waypoint x: 519.5
     waypoint z: 276.5
     length: 0
@@ -76168,7 +76168,7 @@ entities:
     goal u z: 0
     goal v x: 0
     goal v z: 0
-    goal hw: 29.4142
+    goal hw: 28.4142
     goal hh: 0
     maxdist: 0
   Vision:

Notice both numbers differ by 1. In more complicated tests, they differ by some fractional amount.

Full entity states on the host:

- id: 11
  Decay:
  Footprint:
  Minimap:
    r: 46
    g: 46
    b: 200
    active: true
    x: 519.51343
    z: 275.29194
  Obstruction:
    active: true
    moving: true
    control group: 11
    control group 2: 0
    tag: 2
    flags: 5
    clearance: 1
  OverlayRenderer:
  Ownership:
    owner: 1
  Position:
    in world: true
    x: 519.51343
    y: 0
    z: 275.29194
    last x: 519.51104
    last y diff: 0
    last z: 271.42331
    territory: 0
    rot x: 0
    rot y: -0.07943
    rot z: 0
    altitude: 0
    relative: true
    floating: false
    constructionprogress: 0
    anchor: "upright"
    turret parent: 0
  Selectable:
  UnitMotion:
    state: 3
    path state: 3
    pass class: "default"
    ticket: 0
    target entity: 12
    target pos x: 0
    target pos y: 0
    target offset x: 0
    target offset y: 0
    target min range: 0
    target max range: 28
    speed: 14
    moving: true
    facePointAfterMove: true
    length: 2
    waypoint x: 519.5
    waypoint z: 322.5
    waypoint x: 519.5
    waypoint z: 276.5
    length: 0
    passability map changed recently: false
    length: 0
    expected path ticket: 0
    next step clean: true
    type: 1
    goal x: 515.45728
    goal z: 351.47248
    goal u x: 0
    goal u z: 0
    goal v x: 0
    goal v z: 0
    goal hw: 29.4142
    goal hh: 0
    maxdist: 0
  Vision:
  VisualActor:
    base actor: "units/athenians/infantry_javelinist_a.xml"
    r: 1
    g: 1
    b: 1
    anim run threshold: 19
    anim name: "idle"
    anim once: false
    anim speed: 1
    sound group: ""
    anim desync: 0.05
    anim sync repeat time: 0
    seed: 20690
    actor: "units/athenians/infantry_javelinist_a.xml"
  AIProxy:
  Armour:
    object: {
  "invulnerable": false
}
  Attack:
  Builder:
  Cost:
    object: {
  "populationCost": 1,
  "populationBonus": 0
}
  Guard:
    object: {
  "entities": []
}
  Health:
    object: {
  "maxHitpoints": 60,
  "hitpoints": 60,
  "regenRate": 0
}
  Identity:
  Loot:
  Looter:
    object: {}
  Promotion:
    object: {
  "currentXp": 0
}
  ResourceGatherer:
    object: {
  "carrying": {}
}
  Sound:
  Stamina:
    object: {}
  StatusBars:
    object: {
  "auraSources": {}
}
  UnitAI:
    object: {
  "orderQueue": [
    {
      "type": "Attack",
      "data": {
        "target": 12,
        "force": false,
        "forceResponse": true,
        "allowCapture": true,
        "attackType": "Ranged"
      }
    }
  ],
  "order": {
    "type": "Attack",
    "data": {
      "target": 12,
      "force": false,
      "forceResponse": true,
      "allowCapture": true,
      "attackType": "Ranged"
    }
  },
  "formationController": 0,
  "isGarrisoned": false,
  "isIdle": false,
  "lastFormationTemplate": "",
  "finishedOrder": false,
  "heldPosition": {
    "x": 519.513427734375,
    "z": 275.29193115234375
  },
  "workOrders": [],
  "stance": "aggressive",
  "fsmStateName": "INDIVIDUAL.COMBAT.APPROACHING",
  "timer": 9,
  "losRangeQuery": 9,
  "fsmReenter": false
}
  Visibility:
    object: {
  "retainInFog": false,
  "alwaysVisible": false,
  "corpse": false,
  "preview": false,
  "activated": false
}

- id: 12
  Decay:
  Footprint:
  Minimap:
    r: 150
    g: 20
    b: 20
    active: true
    x: 515.45728
    z: 351.47248
  Obstruction:
    active: true
    moving: false
    control group: 12
    control group 2: 0
    tag: 4
    flags: 5
    clearance: 1
  OverlayRenderer:
  Ownership:
    owner: 2
  Position:
    in world: true
    x: 515.45728
    y: 0
    z: 351.47248
    last x: 515.45728
    last y diff: 0
    last z: 351.47248
    territory: 0
    rot x: 0
    rot y: 2.35621
    rot z: 0
    altitude: 0
    relative: true
    floating: false
    constructionprogress: 0
    anchor: "upright"
    turret parent: 0
  Selectable:
  UnitMotion:
    state: 0
    path state: 0
    pass class: "default"
    ticket: 0
    target entity: 0
    target pos x: 0
    target pos y: 0
    target offset x: 0
    target offset y: 0
    target min range: 0
    target max range: 0
    speed: 14
    moving: false
    facePointAfterMove: true
    length: 0
    length: 0
    passability map changed recently: false
    length: 0
    expected path ticket: 0
    next step clean: true
    type: 0
    goal x: 0
    goal z: 0
    goal u x: 0
    goal u z: 0
    goal v x: 0
    goal v z: 0
    goal hw: 0
    goal hh: 0
    maxdist: 0
  Vision:
  VisualActor:
    base actor: "units/athenians/infantry_javelinist_a.xml"
    r: 1
    g: 1
    b: 1
    anim run threshold: 0
    anim name: "idle"
    anim once: false
    anim speed: 1
    sound group: ""
    anim desync: 0.05
    anim sync repeat time: 0
    seed: 63480
    actor: "units/athenians/infantry_javelinist_a.xml"
  AIProxy:
  Armour:
    object: {
  "invulnerable": false
}
  Attack:
  Builder:
  Cost:
    object: {
  "populationCost": 1,
  "populationBonus": 0
}
  Guard:
    object: {
  "entities": []
}
  Health:
    object: {
  "maxHitpoints": 60,
  "hitpoints": 60,
  "regenRate": 0
}
  Identity:
  Loot:
  Looter:
    object: {}
  Promotion:
    object: {
  "currentXp": 0
}
  ResourceGatherer:
    object: {
  "carrying": {}
}
  Sound:
  Stamina:
    object: {}
  StatusBars:
    object: {
  "auraSources": {}
}
  UnitAI:
    object: {
  "orderQueue": [],
  "formationController": 0,
  "isGarrisoned": false,
  "isIdle": true,
  "lastFormationTemplate": "",
  "finishedOrder": false,
  "heldPosition": {
    "x": 515.457275390625,
    "z": 351.47247314453125
  },
  "workOrders": [],
  "stance": "standground",
  "fsmStateName": "INDIVIDUAL.IDLE",
  "losRangeQuery": 11
}
  Visibility:
    object: {
  "retainInFog": false,
  "alwaysVisible": false,
  "corpse": false,
  "preview": false,
  "activated": false
}

comment:7 Changed 4 years ago by Ben Brian

This turned out to be an interesting problem. I noticed "goal hw" differed and there are only 5 places in the code where that value is set. I placed printfs after each one to see which was hit and if they differed before an OOS in the simple test map. Sure enough, one of them differed by exactly 1. Then it was a matter of inspecting each of the variables used to calculate that result.

That revealed the culprit: g_goalDelta in CCmpUnitMotion.cpp. It is a static const assigned the value of a const variable (Pathfinding::NAVCELL_SIZE) from another source file (Terrain.cpp). The problem is the init order of these variables relative to eachother is undefined! On MSVC and GCC, g_goalDelta was assigned the uninitialized value of 0, while on clang/libc++ for whatever reason it had the correct value of 1 (result of 4/4).

Not only does this cause an OOS between these platforms, it results in unintended behavior of the pathfinder.

To fix this, I am moving both consts to Pathfinding.h, which I think is a little tidier anyway.

Last edited 4 years ago by Ben Brian (previous) (diff)

comment:8 Changed 4 years ago by ben

Owner: set to ben
Resolution: fixed
Status: newclosed

In 17132:

Fixes global init order bug that caused OOS between OS X and other platforms, fixes #3499. May improve pathfinding behavior in some cases, please test!

comment:9 Changed 4 years ago by elexis

Testing revealed #3530. If you apply this changeset to previous revisions (or use OSX accordingly), the bug also appears. This commit should not be the cause of the bug, it only revealed it.

Note: See TracTickets for help on using tickets.