Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Could not compile on 16.04.5 LTS Xenial Xerus (Was: How to transcompile python code that have logging or psutil libraries imported.) #112

Open
nutant-h opened this issue Oct 23, 2018 · 12 comments
Open

Could not compile on 16.04.5 LTS Xenial Xerus (Was: How to transcompile python code that have logging or psutil libraries imported.) #112

nutant-h opened this issue Oct 23, 2018 · 12 comments

Comments

@nutant-h
Copy link

I am trying to transcompile a Python file into Go.

Following the README file I installed grumpy, then tested it;
#echo 'import sys; print sys.version' | grumprun
2.7.13
works fine.

My python code needs logging and psutil libraries imported;
but executing echo 'import logging, psutil, sys; print sys.version' | grumprun throws the following error.

# echo 'import logging, psutil, sys; print sys.version' | grumprun
Traceback (most recent call last):
  File "/root/grumpy/grumpy-runtime-src/build/bin/grumprun", line 8, in <module>
    sys.exit(cli.main())
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/cli.py", line 84, in run
    result = grumprun.main(stream=stream, modname=modname, pep3147=pep3147, clean_tempfolder=(not keep_main))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumprun.py", line 114, in main
    transpiled = grumpc.main(stream, modname=modname, pep3147=True, recursive=True)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 156, in main
    deps, import_objects = _collect_deps(script, modname, pep3147_folders, from_cache=(not will_refresh))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 77, in _collect_deps
    deps, import_objects = pydeps.main(script, modname, with_imports=True) #, script, gopath)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/pydeps.py", line 35, in main
    imports = imputil.collect_imports(modname, script, gopath, package_dir=package_dir)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 216, in collect_imports
    collector.visit(mod)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 205, in visit_ImportFrom
    self.imports.extend(self.importer.visit(node))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 119, in visit_ImportFrom
    imp = self._resolve_import(node, node.module)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 170, in _resolve_import
    raise util.ImportError(node, 'no such module: {} (script: {})'.format(modname, self.script))
grumpy_tools.compiler.util.ImportError: line 15: no such module: _codecs (script: /usr/lib/python2.7/codecs.py)

I am sure there must be a way to use other libraries. How can I use logging library?
I am just not able to find the documentations, need help.

Thank you.
@nutant-h
Copy link
Author

Found out this https://github.com/google/grumpy/wiki/Standard-libraries:-guidelines-for-integration

@nutant-h nutant-h reopened this Oct 23, 2018
@nutant-h
Copy link
Author

Followed the above mentioned wiki.
though getting the same error.

root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# make
find: ‘compiler’: No such file or directory
find: ‘compiler’: No such file or directory
find: ‘compiler’: No such file or directory
make: *** No rule to make target 'build/pkg/linux_amd64/__python__/codecs.a', needed by 'build/pkg/linux_amd64/__python__/logging.a'.  Stop.
root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# echo 'import sys; print sys.version' | grumprun
2.7.13
root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# echo 'import logging, sys; print sys.version' | grumprun
Traceback (most recent call last):
  File "/root/grumpy/grumpy-runtime-src/build/bin/grumprun", line 8, in <module>
    sys.exit(cli.main())
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/cli.py", line 84, in run
    result = grumprun.main(stream=stream, modname=modname, pep3147=pep3147, clean_tempfolder=(not keep_main))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumprun.py", line 114, in main
    transpiled = grumpc.main(stream, modname=modname, pep3147=True, recursive=True)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 156, in main
    deps, import_objects = _collect_deps(script, modname, pep3147_folders, from_cache=(not will_refresh))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 77, in _collect_deps
    deps, import_objects = pydeps.main(script, modname, with_imports=True) #, script, gopath)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/pydeps.py", line 35, in main
    imports = imputil.collect_imports(modname, script, gopath, package_dir=package_dir)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 216, in collect_imports
    collector.visit(mod)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 205, in visit_ImportFrom
    self.imports.extend(self.importer.visit(node))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 119, in visit_ImportFrom
    imp = self._resolve_import(node, node.module)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 170, in _resolve_import
    raise util.ImportError(node, 'no such module: {} (script: {})'.format(modname, self.script))
grumpy_tools.compiler.util.ImportError: line 15: no such module: _codecs (script: /usr/lib/python2.7/codecs.py)

@alanjds
Copy link

alanjds commented Oct 23, 2018

Hi, @nutant-h

The README is outdated. I should update it. Sorry for that.

Please follow the following (copied from #79 (comment)):

  • Install Python 2.7
  • Install Go >= 1.10
  • Install Grumpy packages for development:
cd grumpy-tools-src/
python setup.py develop

cd grumpy-runtime-src/
python setup.py develop
  • Run your tests via the grumpy command line tool:
$ grumpy -c 'import sys; print sys.version'
2.7.13

This should be working.

@alanjds
Copy link

alanjds commented Oct 23, 2018

I created an issue to track the update of README on #113. Thanks for reporting.

Please feed back if my instructions had worked or not. Are the instructions clear enough?

@alanjds alanjds added the waiting feedback Waiting confirmation of changes or fixes label Oct 23, 2018
@nutant-h
Copy link
Author

Hi @alanjds
Followed your instructions;

I got error while doing

cd grumpy-runtime-src/
python setup.py develop

This is the error;

root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# python setup.py develop
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)

Installed /root/grumpy/grumpy-runtime-src/.eggs/setuptools-40.4.3-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    setup(**GRUMPY_RUNTIME_OPTIONS)
  File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 272, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 327, in finalize_options
    ep.load()(self, ep.name, value)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2239, in resolve
    raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'check_specifier'

@alanjds
Copy link

alanjds commented Oct 24, 2018

This is strange, as we have automated tests for Linux. A couple of questions:

  • What version/distribution of Linux you are using?
  • What version of Python (python --version)? Installed via apt/yum?
  • The error persists if trying in a virtualenv?

I am trying to reproduce here, but it works for me

@nutant-h
Copy link
Author

Hi

What version/distribution of Linux you are using?

root@dev-vm-charlie:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

What version of Python (python --version)? Installed via apt/yum?

root@dev-vm-charlie:~# python --version
Python 2.7.12

The error persists if trying in a virtualenv?
Have not tried that but how would it make any difference?

@nutant-h
Copy link
Author

nutant-h commented Oct 24, 2018 •
edited
Loading

I tried in virtualenv and it worked till the point where I did

cd grumpy-runtime-src/
python setup.py develop

and at the end got this;

    File 'build/pkg/linux_amd64/__python__/logging.a' does not exist.
       File 'build/src/__python__/logging/module.go' does not exist.
      Must remake target 'build/src/__python__/logging/module.go'.
      Successfully remade target file 'build/src/__python__/logging/module.go'.
       File 'build/pkg/linux_amd64/__python__/codecs.a' does not exist.
      Must remake target 'build/pkg/linux_amd64/__python__/codecs.a'.
make: *** No rule to make target 'build/pkg/linux_amd64/__python__/codecs.a', needed by 'build/pkg/linux_amd64/__python__/logging.a'.  Stop.
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    setup(**GRUMPY_RUNTIME_OPTIONS)
  File "/root/grumpy2/local/lib/python2.7/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/root/grumpy2/local/lib/python2.7/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/root/grumpy2/local/lib/python2.7/site-packages/setuptools/command/develop.py", line 138, in install_for_development
    self.run_command('build_ext')
  File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "setup.py", line 132, in run
    _run_make(self, *args, **kwargs)
  File "setup.py", line 75, in _run_make
    subprocess.check_call(["""echo "print 'Make Runtime Success'" | make run --debug=bjm -r"""], shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['echo "print \'Make Runtime Success\'" | make run --debug=bjm -r']' returned non-zero exit status 2

I did copied the logging Lib from CPython 2.7 repo into third_party/stdlib.

@alanjds
Copy link

alanjds commented Oct 24, 2018

So, it is not compiling codecs.py on your env.

Last thing: please try with an standard user, not root.

I will try to create a docker machine matching yours and reproduce it.

Thanks for the effort.

@nutant-h
Copy link
Author

Tried with ubuntu user. Got the same codecs error while doing

cd grumpy-runtime-src/
python setup.py develop

@alanjds alanjds removed the waiting feedback Waiting confirmation of changes or fixes label Oct 25, 2018
@alanjds alanjds changed the title How to transcompile python code that have logging or psutil libraries imported. Could not compile on 16.04.5 LTS Xenial Xerus (Was: How to transcompile python code that have logging or psutil libraries imported.) Oct 25, 2018
@alanjds
Copy link

alanjds commented Oct 27, 2018

There is two issues:

  1. ImportError: grumpy_tools.compiler.util.ImportError: line 15: no such module: _codecs (script: /usr/lib/python2.7/codecs.py)
  2. Build Error: make: *** No rule to make target 'build/pkg/linux_amd64/__python__/codecs.a', needed by 'build/pkg/linux_amd64/__python__/logging.a'. Stop.

The 1st issue is because the parser on master is wrongly resolving some ImportErrors on compile time instead of run time. This should be addressed on the PR #118, when merged.

The 2nd issue should be a collateral of the 1st one.

If you want, please try the branch fix-importerror-star and see if it fixes the 2nd issue. If so, I will mark such PR to be fixing this issue.

@quark-zju
Copy link

I also hit ImportError: 'module' object has no attribute 'check_specifier' while running pip2 install -U grumpy-runtime -I --no-cache. Upgrading setuptools using pip2 install --upgrade setuptools fixed it for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alanjds @quark-zju @nutant-h