Incorrect default value of CUBLAS_WORKSPACE_CONFIG

[afennelly-mitre]

ray/rllib/utils/torch_utils.py

Line 702 in 168ae2e

os.environ["CUBLAS_WORKSPACE_CONFIG"] = "4096:8"

and

ray/rllib/utils/debug/deterministic.py

Line 38 in 575d6af

os.environ["CUBLAS_WORKSPACE_CONFIG"] = "4096:8"

The above lines manually set the value of the environment variable CUBLAS_WORKSPACE_CONFIG. There is a comment above the line which points to this pytorch issue. However, I suspect this is an error and the intended default value should be preceded with a colon, ie. :4096:8 rather than the current value of 4096:8.

This intuition is taken from this comment on a pytorch issue.

This minor difference makes my application logs extremely difficult to read, as they become cluttered with:

2024-09-16 11:11:26,220 INFO multi_agent_fire_harness.py:222 -- Agent agent_5 is now moving right.
2024-09-16 11:11:26,220 INFO agent.py:178 -- Agent agent_6 has completed a control line of length 2953 ft.
2024-09-16 11:11:26,221 INFO multi_agent_fire_harness.py:209 -- Agent agent_6 has completed the current mitigation. The current mitigation count for this agent is: 1
2024-09-16 11:11:26,221 INFO multi_agent_fire_harness.py:222 -- Agent agent_6 is now moving down.
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
2024-09-16 11:11:26,316 INFO fire_harness.py:324 -- Agent agent_4 is starting to set a control line on fuel model 'Brush' at position (0, 63).
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
2024-09-16 11:11:32,693 INFO fire_harness.py:324 -- Agent agent_5 is starting to set a control line on fuel model 'Brush' at position (126, 75).
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
  return F.linear(input, self.weight, self.bias)

and so on. Please make this update ASAP !!!