Opened 3 years ago

Last modified 2 months ago

#5874 new task

Threading improvements

Reported by: wraitii Owned by:
Priority: Must Have Milestone: Work In Progress
Component: Core engine Keywords:
Cc: Patch: Phab:D3848

Description (last modified by phosit)

There are issues with our implementation: we use "naked" std::thread instead of cleverer solutions. This, notably, means we fail to detect exceptions happening in the threads (see https://code.wildfiregames.com/rP22649#45341 )

[25656] implements a task manager. Now not every task wich is run concurently does need to create it's own std::thread.

Phab:D4956 does implement exception handling for tasks in the task manager.

These tasks use the task manager:

  • map generator
  • pathfinder
  • texture converter

These tasks should be adapted to use the task manager:

  • upnp

A task in the task manager should never 'wait' (for a condition_variable; for a mutex; for some time...) that would put the whole thread to sleep. Instead it would be better to run another task in the meantime and resume the original task later. Phab:D3877 or Phab:D4907 does extend the task manager to make that possible. These tasks rely on such a feature:

Change History (17)

comment:2 by wraitii, 3 years ago

Description: modified (diff)
Milestone: Alpha 24Alpha 25

Net client was threaded in A24, A25 will hopefully thread the pathfinder.

comment:3 by Imarok, 3 years ago

Patch: Phab:D3848

comment:4 by wraitii, 3 years ago

In 25656:

Implement a global task manager using a pool of worker threads

Tasks are simple callables (e.g. lambdas), and can be pushed with 2 priority levels. Pushing a task returns a future.
Futures can be waited on, can return results, and can be cancelled deterministically. Futures can also not be waited on.

This gives 'hardware concurrency - 1' threads to maximize CPU usage in a work-stealing workflow.

Reviewed by: vladislavbelov

Refs #5874

Differential Revision: https://code.wildfiregames.com/D3848

comment:5 by Stan, 3 years ago

Description: modified (diff)

comment:6 by Stan, 3 years ago

Milestone: Alpha 25Alpha 26

comment:7 by Stan, 2 years ago

Milestone: Alpha 26Alpha 27

Unlikely to get done for A26

comment:8 by jprahman, 21 months ago

@wraitii @stanislas69 what sorts of remaining improvements do we see here? Have some interest in looking at this. Have a fair bit of experience with system programming/threading from my day job.

in reply to:  8 comment:9 by wraitii, 21 months ago

Replying to jprahman:

@wraitii @stanislas69 what sorts of remaining improvements do we see here? Have some interest in looking at this. Have a fair bit of experience with system programming/threading from my day job.

Well, plenty of systems aren't threaded and _could_ be threaded.If you check the last commits of https://github.com/wraitii/0ad/commits/__THREADPOOL, you'll see some ways to thread some systems, such as sound.

More generally, threading is interesting in general as a speedup, though there are limitations on what you can do in 0 A.D.

comment:10 by phosit, 18 months ago

Description: modified (diff)

Add link to differentials

comment:11 by Freagarach, 14 months ago

Milestone: Alpha 27Backlog

Pushing back.

comment:12 by phosit, 12 months ago

Description: modified (diff)
Milestone: BacklogWork In Progress

Rewrite the ticket. Task manager wasn't mentioned and i described the problem some tasks have wich are not jet converted. I also added a link two new diffs.

comment:13 by phosit, 11 months ago

Description: modified (diff)

user reporter aslo relyes on a repeating task.

comment:14 by phosit, 10 months ago

Description: modified (diff)

The wrong diff was linked

comment:15 by phosit, 7 months ago

Description: modified (diff)

Add missing upnp.

comment:16 by phosit, 3 months ago

In 27944:

Put the CMapGeneratorWorker completely inside the task

The return-slot provided by the Future is used for synchronisation.

Refs: #5874

Comments By: @Stan, @vladislavbelov, @wraitii

Differential Revision: https://code.wildfiregames.com/D5001

comment:17 by phosit, 2 months ago

Description: modified (diff)

I forgot to refference this ticket in the commit [27984].

Note: See TracTickets for help on using tickets.