pip install fairseq fails due to invalid metadata for PyYAML dependency in omegaconf

[Gan-Xing]

Title

pip install fairseq fails due to invalid metadata for PyYAML dependency in omegaconf

Issue Description

Environment:

• Ubuntu 22.04
• Conda environment: streamspeech
• Python 3.10
• GCC version: 12.3.0

Steps to Reproduce:

1. Create and activate conda environment:

   conda create -n streamspeech python=3.10 -y
   conda activate streamspeech

2. Install dependencies:

   conda install pytorch=2.0.1 torchvision torchaudio cudatoolkit=11.7 -c pytorch
   sudo apt-get update
   sudo apt-get install -y build-essential python3-dev
   pip install cython numpy
   pip install --upgrade pip setuptools
   pip install PyYAML

3. Check PyYAML version:

   python -c "import yaml; print(yaml.__version__)"
   # Output: 6.0.1

4. Try to install fairseq:

   pip install --editable ./ --no-build-isolation

Observed Behavior:

• The installation process fails with the following errors:

  WARNING: Error parsing dependencies of omegaconf: .* suffix can only be used with `==` or `!=` operators
      PyYAML (>=5.1.*)
              ~~~~~~^
  
  ERROR: Exception:
  Traceback (most recent call last):
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3070, in _dep_map
      return self.__dep_map
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2863, in __getattr__
      raise AttributeError(attr)
  AttributeError: _DistInfoDistribution__dep_map
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/requirements.py", line 36, in __init__
      parsed = _parse_requirement(requirement_string)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
      return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
      url, specifier, marker = _parse_requirement_details(tokenizer)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/_parser.py", line 118, in _parse_requirement_details
      specifier = _parse_specifier(tokenizer)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/_parser.py", line 214, in _parse_specifier
      parsed_specifiers = _parse_version_many(tokenizer)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/_parser.py", line 229, in _parse_version_many
      tokenizer.raise_syntax_error(
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
      raise ParserSyntaxError(
  pip._vendor.packaging._tokenizer.ParserSyntaxError: .* suffix can only be used with `==` or `!=` operators
      PyYAML (>=5.1.*)
              ~~~~~~^
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
      status = run_func(*args)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
      return func(self, options, args)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 377, in run
      requirement_set = resolver.resolve(
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
      result = self._result = resolver.resolve(
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
      state = resolution.resolve(requirements, max_rounds=max_rounds)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
      failure_causes = self._attempt_to_pin_criterion(name)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
      criteria = self._get_updated_criteria(candidate)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 247, in get_dependencies
      return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 247, in <listcomp>
      return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 401, in iter_dependencies
      for r in self.dist.iter_dependencies():
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_internal/metadata/pkg_resources.py", line 247, in iter_dependencies
      return self._dist.requires(extras)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2786, in requires
      dm = self._dep_map
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3072, in _dep_map
      self.__dep_map = self._compute_dependencies()
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3082, in _compute_dependencies
      reqs.extend(parse_requirements(req))
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3135, in __init__
      super().__init__(requirement_string)
    File "/root/exit/envs/streamspeech/lib/python3.10/site-packages/pip/_vendor/packaging/requirements.py", line 38, in __init__
      raise InvalidRequirement(str(e)) from e
  pip._vendor.packaging.requirements.InvalidRequirement: .* suffix can only be used with `==` or `!=` operators
      PyYAML (>=5.1.*)
              ~~~~~~^
  

Expected Behavior:
fairseq should be installed without errors.

Additional Information:

• PyYAML version: 6.0.1
• omegaconf version: 2.0.5
• hydra-core version: 1.0.7
• torch version: 2.3.1
• pip version: 23.1

Notes:

• The PyYAML dependency in omegaconf has an invalid metadata format that causes the error.
• Running as root might cause permissions and conflicts, but switching to a non-root user does not resolve the dependency issues.

Steps Tried:

1. Upgrading/downgrading pip, setuptools, and wheel.
2. Installing specific versions of PyYAML and omegaconf.
3. Setting up a new conda environment with specific versions of dependencies.
4. Verifying GCC installation and setting default versions.

Request for Help:
I need assistance in resolving this installation failure. Any guidance or solutions would be appreciated.

Contact Information:
[email protected]

Labels

• needs triage
• bug

[maxjkwang]

I believe it's due to a newer version of pip. I'm using pip 24.0, and it gives me the following warning but works fine:

DEPRECATION: omegaconf 2.0.6 has a non-standard dependency specifier PyYAML>=5.1.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of omegaconf or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at pypa/pip#12063

I'd say downgrading pip is probably the easiest way, or try upgrading/editing omegaconf to see if they fixed the issue in a newer version, as is mentioned in the warning.

I see that you've tried downgrading pip, but I believe it should work as it's pip that is unhappy about the dependencies.