Difference between the batch size of an environment and using ParallelEnv

[YannPourcenoux]

Hello,
I am new to Reinforcement Learning and TorchRL (not that it is very old, but still), so if my question is inappropriate here, please direct me to where I should ask it.

In my free time, I am playing with reinforcement learning and finance. I wrote an environment to be able to change the composition of an investor's portfolio. I have two questions regarding the use of Parallel Environments that are unrelated to my use case, but I will use it to illustrate.
I see the benefit of using different environments if, for example, I want one environment where the traded shares differ from another but still want to train simultaneously.
However, in the case of hyperparameters that can be set for each batch dimension, I am struggling to see the difference between using one environment or parallel ones.
An example is if I only want to have different starting positions (different starting portfolio composition or different starting value of the portfolio). This can be done alongside the batch dimension.
I have two questions.

1. In this last scenario, can I expect the same training behavior between having different environments and a single environment where I change these parameters along the batch dimension? If not, why?
2. If it would yield the same agent performance, which one would you choose? I think that having only one environment and relying on batching may have a faster training speed. Is that a correct hypothesis?

I can't find my answer online regarding this. I would gladly appreciate it if somebody could help me with these two questions. Don't hesitate to ask me to be more specific if you need more details to provide an answer.

Thanks!

[vmoens]

Hello!

In this last scenario, can I expect the same training behavior between having different environments and a single environment where I change these parameters along the batch dimension? If not, why?

I think both options can lead to the same result. Without looking carefully at what your specs look like it's a bit hard to give a general guidance but feel free to reach out with more details (even through a private message on our discord channel, see link on top of readme).

If it would yield the same agent performance, which one would you choose? I think that having only one environment and relying on batching may have a faster training speed. Is that a correct hypothesis?

I would choose the parallel setting if you think those are really different envs and you will be using them independently from time to time. If they're just subsets of a big env I would do a single one with a batch size.

Again happy to help if you have more details to share

[yann-pourcenoux]

Hi Vincent,

Thanks for the fast answer!

I would choose the parallel setting if you think those are really different envs and you will be using them independently from time to time. If they're just subsets of a big env I would do a single one with a batch size.

This answers my questions for now.

I may come to you later when I am looking into improvements. I am just at the baseline step for now.