Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add macos-arm64.yml for setup #1121

Draft
wants to merge 8 commits into
base: tf_2_9
Choose a base branch
from

Conversation

BrianHenryIE
Copy link
Contributor

@BrianHenryIE BrianHenryIE commented Apr 9, 2023

This is a draft PR because I don't really know what is/is not working yet.

Pytorch and Tensorflow are both installing!

Edit: This post was compared to main. The tf_2_9 branch has fewer differences and is imminent. See below

Presumed problems:

  • python=3.8, should be 3.7, 3.8 is the oldest version available
  • numpy=1.23.5, should be 1.19
  • protobuf=4.21.12, should be 3.2.0
  • torchvision=0.16.0.dev, should be 0.12
  • pytorch-lightning=2.0.1, should be >=1.9,<2.0
  • kivy=2.1.0, should be 2.0.0
  • tensorflow-macos=2.12.0, should be 2.2.0, 2.5.0 is the oldest version available

pytest output:

=============================== short test summary info ===============================
FAILED donkeycar/tests/test_actuator.py::test_PCA9685 - ModuleNotFoundError: No module named 'Adafruit_PCA9685'
FAILED donkeycar/tests/test_actuator.py::test_PWMSteering - ModuleNotFoundError: No module named 'Adafruit_PCA9685'
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasLinear] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmp9b0qqryx/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasCategorical] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmp3p95_t_q/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasInferred] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmpothpx9jf/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasLocalizer] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmp_legew87/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasIMU] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmpmg3ainen/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[Keras3D_CNN] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmpkmr1ps15/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasMemory] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmphmgcapo8/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_keras.py::test_keras_vs_tflite_and_tensorrt[KerasBehavioral] - OSError: SavedModel file does not exist at: /var/folders/sh/cygymmqn36714790jj3r33200000gn/T/tmputy4i9dc/model.trt/{saved_model.pbtxt|saved_model.pb}
FAILED donkeycar/tests/test_torch.py::test_train[data0] - TypeError: __init__() got an unexpected keyword argument 'gpus'
FAILED donkeycar/tests/test_torch.py::test_training_pipeline[resnet18] - TypeError: __init__() got an unexpected keyword argument 'gpus'
FAILED donkeycar/tests/test_train.py::test_train[data10] - ModuleNotFoundError: No module named 'fastai'
========== 13 failed, 155 passed, 11 skipped, 21 warnings in 313.90s (0:05:13) ==========

Slightly related to #1061

TODO: Rename from macosm1.yml to macos-arm64.yml.

Maybe this entire file will be redundant eventually. Hopefully it's useful in the short term for figuring out compatibility.

@BrianHenryIE BrianHenryIE changed the base branch from main to tf_2_9 April 10, 2023 03:14
@BrianHenryIE
Copy link
Contributor Author

Current differences between mac.yml and this mac-arm64.yml are now only:

  • kivy is installed through pip
  • tensorflow==2.9 becomes tensorflow-macos==2.9
  • tensorflow-metal==0.5.0 is added. If only tensorflow-metal is specified the installed version does not match the Tensorflow version

pytest output:

============================== short test summary info ==============================
FAILED donkeycar/tests/test_actuator.py::test_PCA9685 - ModuleNotFoundError: No module named 'Adafruit_PCA9685'
FAILED donkeycar/tests/test_actuator.py::test_PWMSteering - ModuleNotFoundError: No module named 'Adafruit_PCA9685'
FAILED donkeycar/tests/test_train.py::test_train[data2] - AssertionError: assert 0.15446822345256805 < (0.17121504247188568 * 0.9)
FAILED donkeycar/tests/test_web_socket.py::WebSocketCalibrateTest::test_calibrate_servo_esc_1b - AssertionError: assert <Mock name='mock.left_pulse' id='14266780688'> == 444
FAILED donkeycar/tests/test_web_socket.py::WebSocketCalibrateTest::test_calibrate_servo_esc_2a - AssertionError: assert <Mock name='mock.right_pulse' id='14256777248'> == 555
========== 5 failed, 163 passed, 11 skipped, 126 warnings in 726.81s (0:12:06) ==========

@BrianHenryIE BrianHenryIE changed the title Add macosm1.yml for setup Add macos-arm64.yml for setup Apr 10, 2023
- albumentations # 1.3.0
# - imgaug # 0.4.0 # This _was_ in mac.yml for an older Donkey version
- pip:
- kivy==2.1.0 # 2.1.0 # Moved from macos.yml dependencies section.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would kivy fail in the conda install, do you know?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried moving it back but got ResolvePackageNotFound:

$ conda env create -f mac-arm64.yml 
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - kivy=2.1.0

With verbose output:

$ conda env create -f mac-arm64.yml -v      
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Traceback (most recent call last):
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/exception_handler.py", line 16, in __call__
    return func(*args, **kwargs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda_env/cli/main.py", line 49, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/notices/core.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda_env/cli/main_create.py", line 163, in execute
    result[installer_type] = installer.install(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda_env/installers/conda.py", line 50, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 153, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 214, in solve_for_diff
    final_precs = self.solve_final_state(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 379, in solve_final_state
    ssc = self._add_specs(ssc)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 732, in _add_specs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/resolve.py", line 649, in _get_package_pool
    pool = self.get_reduced_index(specs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/common/io.py", line 83, in decorated
    return f(*args, **kwds)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/resolve.py", line 674, in get_reduced_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/resolve.py", line 329, in verify_specs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound: 
  - kivy=2.1.0

With no version specified:

$ conda env create -f mac-arm64.yml         
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - kivy

I moved it back under the pip section and it worked again:

$ conda env create -f mac-arm64.yml
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
                                                                                                                                                                    
Preparing transaction: done                                                                                                                                         
Verifying transaction: done                                                                                                                                         
Executing transaction: done                                                                                                                                         
Installing pip dependencies: / Ran pip subprocess with arguments:                                                                                                   
['/Users/brianhenry/anaconda3/envs/donkey-test/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/brianhenry/Sites/donkey-m1-test/condaenv.yqqhctpo.requirements.txt', '--exists-action=b']                                                                                                                                       
Pip subprocess output:                                                                                                                                              

... cut lots of downloads etc ...

Installing collected packages: tensorboard-plugin-wit, libclang, keras, flatbuffers, werkzeug, termcolor, tensorflow-estimator, tensorboard-data-server, six, simple-pid, pygments, pyasn1, protobuf, proglog, opt-einsum, oauthlib, imageio-ffmpeg, grpcio, gast, docutils, decorator, cachetools, absl-py, tensorflow-metal, rsa, requests-oauthlib, pyasn1-modules, moviepy, markdown, Kivy-Garden, keras-preprocessing, google-pasta, astunparse, kivy, google-auth, keras-vis, google-auth-oauthlib, tensorboard, tensorflow-macos
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
Successfully installed Kivy-Garden-0.1.5 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.1 decorator-4.4.2 docutils-0.20.1 flatbuffers-1.12 gast-0.4.0 google-auth-2.21.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.56.0 imageio-ffmpeg-0.4.8 keras-2.9.0 keras-preprocessing-1.1.2 keras-vis-0.5.0 kivy-2.1.0 libclang-16.0.0 markdown-3.4.3 moviepy-1.0.3 oauthlib-3.2.2 opt-einsum-3.3.0 proglog-0.1.10 protobuf-3.19.6 pyasn1-0.5.0 pyasn1-modules-0.3.0 pygments-2.15.1 requests-oauthlib-1.3.1 rsa-4.9 simple-pid-2.0.0 six-1.15.0 tensorboard-2.9.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-estimator-2.9.0 tensorflow-macos-2.9.0 tensorflow-metal-0.5.0 termcolor-2.3.0 werkzeug-2.3.6

done
#
# To activate this environment, use
#
#     $ conda activate donkey-test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Then after activating:

$ conda list | grep kivy
kivy                      2.1.0                    pypi_0    pypi
kivy-garden               0.1.5                    pypi_0    pypi

Regardless, should it be removed since it looks like it will be installed during pip install -e .[pc] in setup.py:

          'pc': [
              'matplotlib',
              'kivy',
              'pandas',
              'plotly',
              'albumentations'
          ],

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @BrianHenryIE - it seems that for apple silicon the conda install of kivy is not working for whatever reasons, so what you suggest makes sense (to install it w/ pip). In general, the pip install using setup.py is mimicking the conda install, all packages are listed there too. It means you could (theoretically) install DC w/ pip only, but we don't recommend doing that. If you do the conda install first, then the pip install -e .[pc] will not do anything, except setting the entry point. All packages are already found by pip and nothing should be installed at that step any more.

To give you a quick heads-up, we are currently working on retiring the conda install and will be going w/ pip only very soon (I hope by this month otherwise next). So merging what you have done here, would be short lived. Once we are on pip only, it would be good to have apple silicon support too. It would be great if you could help testing that, as it looks like the only changes are the tf packages as you outlined in your yaml file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants