Question about cuda parallelism

[trsh]

So in raw cuda, when we run kernels, we define blocks and thread size.
How does it work for example for thrust:for_each? What are those parameters? Is it auto adjusted for max parallelism? Is it configurable?
I googled madly, but did not find anything on the subject, so any help is highly appreciated.

[jrhemstad]

What are those parameters? Is it auto adjusted for max parallelism?

We generally refer to things like block size, items per thread, etc. as "tuning parameters". They are not currently exposed for users to control.

Internally within Thrust and CUB we statically dispatch to a particular set of tuning parameters based on the algorithm and the architecture you're running on. The tuning parameters we select are necessarily a compromise based on what our benchmarking has found to be the "best" in the most cases.

This can often be unsatisfying if your use case happens to not perform well with the tuning parameters we have selected. As a result, it is on our roadmap for this year to design extended APIs that allow you to control these tuning parameters. I don't have a more specific timeline than that. Just know that we know this is important and it is on our radar.

[jrhemstad]

It looks like the current tuning parameters for for_each use a block size of 256 and each thread processes 2 items: https://github.com/NVIDIA/thrust/blob/202a8b098fe134c423ee4551220ff5e25b9d539c/thrust/system/cuda/detail/parallel_for.h#L61-L65

At the moment, this policy is used universally for all architectures.

In addition to introducing user tuning APIs, this year we are also investing a significant amount of effort in going through all of the Thrust/CUB algorithms and updating the tuning policies for the latest architectures. You can read more about that effort in CUB here: NVIDIA/cub#631

[trsh]

It looks like the current tuning parameters for for_each use a block size of 256 and each thread processes 2 items:

https://github.com/NVIDIA/thrust/blob/202a8b098fe134c423ee4551220ff5e25b9d539c/thrust/system/cuda/detail/parallel_for.h#L61-L65

At the moment, this policy is used universally for all architectures.

In addition to introducing user tuning APIs, this year we are also investing a significant amount of effort in going through all of the Thrust/CUB algorithms and updating the tuning policies for the latest architectures. You can read more about that effort in CUB here: NVIDIA/cub#631

So 256 * 2 means 512 items per kernel? And then the kernel is run in a loop it self? Is this more efficient then dynamically scaling the block size, depending on item count and GPU thread capabilities?

[jrhemstad]

512 items processed per block. We will launch a kernel with as many blocks as needed to cover the input. So in this case, ceil(N/512) blocks.

[trsh]

512 items processed per block. We will launch a kernel with as many blocks as needed to cover the input. So in this case, ceil(N/512) blocks.

So in cuda it would look like <<< ceil(N/512), 256 * 2 >>> f() ?

[jrhemstad]

More precisely:

int block_size = 256;
int items_per_thread = 2;
<<< ceil(N/(block_size * items_per_thread)), block_size>>> f()

There are 256 threads per block, and each thread processes 2 input elements. So we need half as many blocks as we would otherwise need if each thread processed a single element.