Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KMeans on 4 gpus #815

Open
sh1ng opened this issue Nov 25, 2019 · 1 comment
Open

KMeans on 4 gpus #815

sh1ng opened this issue Nov 25, 2019 · 1 comment
Assignees
@sh1ng

Comments

@sh1ng
Copy link
Contributor

sh1ng commented Nov 25, 2019

mr-0xp2 has 4 gpus and 2 tests are constantly failing

16:05:50  =================================== FAILURES ===================================
16:05:50  _________________________ TestKmeans.test_fit_iris[C] __________________________
16:05:50  [gw1] linux -- Python 3.6.4 /opt/h2oai/h2o4gpu/python/bin/python
16:05:50  
16:05:50  self = <CallInfo when='call' exception: assert False
16:05:50   +  where False = <function allclose at 0x7fffb80e87b8>(array([[5.005999...4286  ,\n        1.2285714285714289 ]]) = <h2o4gpu.solvers.kmeans.KMeansH2O object at 0x7fff870e5668>.cluster_centers_>
16:05:50  func = <function call_runtest_hook.<locals>.<lambda> at 0x7fff86fec378>
16:05:50  when = 'call', treat_keyboard_interrupt_as_exception = False
16:05:50  
16:05:50      def __init__(self, func, when, treat_keyboard_interrupt_as_exception=False):
16:05:50          #: context of invocation: one of "setup", "call",
16:05:50          #: "teardown", "memocollect"
16:05:50          self.when = when
16:05:50          self.start = time()
16:05:50          try:
16:05:50  >           self.result = func()
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/runner.py:212: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  >       lambda: ihook(item=item, **kwds),
16:05:50          when=when,
16:05:50          treat_keyboard_interrupt_as_exception=item.config.getvalue("usepdb"),
16:05:50      )
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/runner.py:194: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_HookCaller 'pytest_runtest_call'>, args = ()
16:05:50  kwargs = {'item': <Function 'test_fit_iris[C]'>}, notincall = set()
16:05:50  
16:05:50      def __call__(self, *args, **kwargs):
16:05:50          if args:
16:05:50              raise TypeError("hook calling supports only keyword arguments")
16:05:50          assert not self.is_historic()
16:05:50          if self.spec and self.spec.argnames:
16:05:50              notincall = (
16:05:50                  set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
16:05:50              )
16:05:50              if notincall:
16:05:50                  warnings.warn(
16:05:50                      "Argument(s) {} which are declared in the hookspec "
16:05:50                      "can not be found in this hook call".format(tuple(notincall)),
16:05:50                      stacklevel=2,
16:05:50                  )
16:05:50  >       return self._hookexec(self, self.get_hookimpls(), kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/hooks.py:286: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_pytest.config.PytestPluginManager object at 0x7fffb98cd5c0>
16:05:50  hook = <_HookCaller 'pytest_runtest_call'>
16:05:50  methods = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa...ffb89a1d30>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fffb8456898>>]
16:05:50  kwargs = {'item': <Function 'test_fit_iris[C]'>}
16:05:50  
16:05:50      def _hookexec(self, hook, methods, kwargs):
16:05:50          # called from all hookcaller instances.
16:05:50          # enable_tracing will set its own wrapping function at self._inner_hookexec
16:05:50  >       return self._inner_hookexec(hook, methods, kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:92: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  hook = <_HookCaller 'pytest_runtest_call'>
16:05:50  methods = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa...ffb89a1d30>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fffb8456898>>]
16:05:50  kwargs = {'item': <Function 'test_fit_iris[C]'>}
16:05:50  
16:05:50      self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
16:05:50          methods,
16:05:50          kwargs,
16:05:50  >       firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
16:05:50      )
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:86: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  item = <Function 'test_fit_iris[C]'>
16:05:50  
16:05:50      def pytest_runtest_call(item):
16:05:50          _update_current_test_var(item, "call")
16:05:50          sys.last_type, sys.last_value, sys.last_traceback = (None, None, None)
16:05:50          try:
16:05:50  >           item.runtest()
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/runner.py:122: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <Function 'test_fit_iris[C]'>
16:05:50  
16:05:50      def runtest(self):
16:05:50          """ execute the underlying test function. """
16:05:50  >       self.ihook.pytest_pyfunc_call(pyfuncitem=self)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/python.py:1438: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_HookCaller 'pytest_pyfunc_call'>, args = ()
16:05:50  kwargs = {'pyfuncitem': <Function 'test_fit_iris[C]'>}, notincall = set()
16:05:50  
16:05:50      def __call__(self, *args, **kwargs):
16:05:50          if args:
16:05:50              raise TypeError("hook calling supports only keyword arguments")
16:05:50          assert not self.is_historic()
16:05:50          if self.spec and self.spec.argnames:
16:05:50              notincall = (
16:05:50                  set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
16:05:50              )
16:05:50              if notincall:
16:05:50                  warnings.warn(
16:05:50                      "Argument(s) {} which are declared in the hookspec "
16:05:50                      "can not be found in this hook call".format(tuple(notincall)),
16:05:50                      stacklevel=2,
16:05:50                  )
16:05:50  >       return self._hookexec(self, self.get_hookimpls(), kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/hooks.py:286: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_pytest.config.PytestPluginManager object at 0x7fffb98cd5c0>
16:05:50  hook = <_HookCaller 'pytest_pyfunc_call'>
16:05:50  methods = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa..., plugin=<module '_pytest.skipping' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/skipping.py'>>]
16:05:50  kwargs = {'pyfuncitem': <Function 'test_fit_iris[C]'>}
16:05:50  
16:05:50      def _hookexec(self, hook, methods, kwargs):
16:05:50          # called from all hookcaller instances.
16:05:50          # enable_tracing will set its own wrapping function at self._inner_hookexec
16:05:50  >       return self._inner_hookexec(hook, methods, kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:92: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  hook = <_HookCaller 'pytest_pyfunc_call'>
16:05:50  methods = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa..., plugin=<module '_pytest.skipping' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/skipping.py'>>]
16:05:50  kwargs = {'pyfuncitem': <Function 'test_fit_iris[C]'>}
16:05:50  
16:05:50      self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
16:05:50          methods,
16:05:50          kwargs,
16:05:50  >       firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
16:05:50      )
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:86: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  pyfuncitem = <Function 'test_fit_iris[C]'>
16:05:50  
16:05:50      @hookimpl(trylast=True)
16:05:50      def pytest_pyfunc_call(pyfuncitem):
16:05:50          testfunction = pyfuncitem.obj
16:05:50          if pyfuncitem._isyieldedfunction():
16:05:50              testfunction(*pyfuncitem._args)
16:05:50          else:
16:05:50              funcargs = pyfuncitem.funcargs
16:05:50              testargs = {}
16:05:50              for arg in pyfuncitem._fixtureinfo.argnames:
16:05:50                  testargs[arg] = funcargs[arg]
16:05:50  >           testfunction(**testargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/python.py:166: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <test_kmeans.TestKmeans object at 0x7fff909fc320>, order = 'C'
16:05:50  
16:05:50      @pytest.mark.parametrize("order", ['C', 'F'])
16:05:50      def test_fit_iris(self, order):
16:05:50          X = np.asanyarray(load_iris().data, order=order)
16:05:50          clusters = 4
16:05:50          model = KMeans(n_gpus=1, n_clusters=clusters, random_state=123).fit(X)
16:05:50      
16:05:50          assert model.cluster_centers_.shape == (X.shape[1], clusters)
16:05:50      
16:05:50          model_rerun = KMeans(n_gpus=1, n_clusters=clusters,
16:05:50                               random_state=123).fit(X)
16:05:50      
16:05:50          # Same random_state should yield same results
16:05:50          assert np.allclose(
16:05:50              np.sort(model.cluster_centers_, axis=0), np.sort(
16:05:50                  model_rerun.cluster_centers_, axis=0)
16:05:50          )
16:05:50      
16:05:50          model_rerun2 = model_rerun.fit(X)
16:05:50      
16:05:50          # Multiple invocations of fit with the same random_state
16:05:50          # also should produce the same result
16:05:50          assert np.allclose(
16:05:50              np.sort(model_rerun.cluster_centers_, axis=0), np.sort(
16:05:50                  model_rerun2.cluster_centers_, axis=0)
16:05:50          )
16:05:50      
16:05:50          model_all = KMeans(n_clusters=clusters, random_state=123).fit(X)
16:05:50      
16:05:50          # Multi GPU should yield same result as single GPU
16:05:50  >       assert np.allclose(
16:05:50              np.sort(model.cluster_centers_, axis=0), np.sort(
16:05:50                  model_all.cluster_centers_, axis=0)
16:05:50          )
16:05:50  E       assert False
16:05:50  E        +  where False = <function allclose at 0x7fffb80e87b8>(array([[5.005999999999999 , 2.622222222222222 , 1.462             ,\n        0.246             ],\n       [5.52962962962... 1.6219512195121948],\n       [6.912499999999999 , 3.428000000000001 , 5.846874999999998 ,\n        2.131249999999999 ]]), array([[5.005999999999999  , 2.6357142857142857 , 1.4620000000000002 ,\n        0.24600000000000005],\n       [5.5321428...05128205128205 ],\n       [6.935483870967742  , 3.428000000000001  , 5.861290322580645  ,\n        2.125806451612903  ]]))
16:05:50  E        +    where <function allclose at 0x7fffb80e87b8> = np.allclose
16:05:50  E        +    and   array([[5.005999999999999 , 2.622222222222222 , 1.462             ,\n        0.246             ],\n       [5.52962962962... 1.6219512195121948],\n       [6.912499999999999 , 3.428000000000001 , 5.846874999999998 ,\n        2.131249999999999 ]]) = <function sort at 0x7fffb80fa488>(array([[6.912499999999999 , 3.099999999999999 , 5.846874999999998 ,\n        2.131249999999999 ],\n       [5.00599999999... 1.2185185185185186],\n       [6.236585365853657 , 2.858536585365854 , 4.80731707317073  ,\n        1.6219512195121948]]), axis=0)
16:05:50  E        +      where <function sort at 0x7fffb80fa488> = np.sort
16:05:50  E        +      and   array([[6.912499999999999 , 3.099999999999999 , 5.846874999999998 ,\n        2.131249999999999 ],\n       [5.00599999999... 1.2185185185185186],\n       [6.236585365853657 , 2.858536585365854 , 4.80731707317073  ,\n        1.6219512195121948]]) = <h2o4gpu.solvers.kmeans.KMeansH2O object at 0x7fff870e5320>.cluster_centers_
16:05:50  E        +    and   array([[5.005999999999999  , 2.6357142857142857 , 1.4620000000000002 ,\n        0.24600000000000005],\n       [5.5321428...05128205128205 ],\n       [6.935483870967742  , 3.428000000000001  , 5.861290322580645  ,\n        2.125806451612903  ]]) = <function sort at 0x7fffb80fa488>(array([[6.261538461538462  , 2.8512820512820514 , 4.8076923076923075 ,\n        1.6205128205128205 ],\n       [5.0059999...5806451612903  ],\n       [5.5321428571428575 , 2.6357142857142857 , 3.960714285714286  ,\n        1.2285714285714289 ]]), axis=0)
16:05:50  E        +      where <function sort at 0x7fffb80fa488> = np.sort
16:05:50  E        +      and   array([[6.261538461538462  , 2.8512820512820514 , 4.8076923076923075 ,\n        1.6205128205128205 ],\n       [5.0059999...5806451612903  ],\n       [5.5321428571428575 , 2.6357142857142857 , 3.960714285714286  ,\n        1.2285714285714289 ]]) = <h2o4gpu.solvers.kmeans.KMeansH2O object at 0x7fff870e5668>.cluster_centers_
16:05:50  
16:05:50  tests/python/open_data/kmeans/test_kmeans.py:52: AssertionError
16:05:50  _________________________ TestKmeans.test_fit_iris[F] __________________________
16:05:50  [gw1] linux -- Python 3.6.4 /opt/h2oai/h2o4gpu/python/bin/python
16:05:50  
16:05:50  self = <CallInfo when='call' exception: assert False
16:05:50   +  where False = <function allclose at 0x7fffb80e87b8>(array([[5.006000...4286  ,\n        1.2285714285714284 ]]) = <h2o4gpu.solvers.kmeans.KMeansH2O object at 0x7fff897e28d0>.cluster_centers_>
16:05:50  func = <function call_runtest_hook.<locals>.<lambda> at 0x7fff86fec2f0>
16:05:50  when = 'call', treat_keyboard_interrupt_as_exception = False
16:05:50  
16:05:50      def __init__(self, func, when, treat_keyboard_interrupt_as_exception=False):
16:05:50          #: context of invocation: one of "setup", "call",
16:05:50          #: "teardown", "memocollect"
16:05:50          self.when = when
16:05:50          self.start = time()
16:05:50          try:
16:05:50  >           self.result = func()
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/runner.py:212: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  >       lambda: ihook(item=item, **kwds),
16:05:50          when=when,
16:05:50          treat_keyboard_interrupt_as_exception=item.config.getvalue("usepdb"),
16:05:50      )
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/runner.py:194: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_HookCaller 'pytest_runtest_call'>, args = ()
16:05:50  kwargs = {'item': <Function 'test_fit_iris[F]'>}, notincall = set()
16:05:50  
16:05:50      def __call__(self, *args, **kwargs):
16:05:50          if args:
16:05:50              raise TypeError("hook calling supports only keyword arguments")
16:05:50          assert not self.is_historic()
16:05:50          if self.spec and self.spec.argnames:
16:05:50              notincall = (
16:05:50                  set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
16:05:50              )
16:05:50              if notincall:
16:05:50                  warnings.warn(
16:05:50                      "Argument(s) {} which are declared in the hookspec "
16:05:50                      "can not be found in this hook call".format(tuple(notincall)),
16:05:50                      stacklevel=2,
16:05:50                  )
16:05:50  >       return self._hookexec(self, self.get_hookimpls(), kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/hooks.py:286: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_pytest.config.PytestPluginManager object at 0x7fffb98cd5c0>
16:05:50  hook = <_HookCaller 'pytest_runtest_call'>
16:05:50  methods = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa...ffb89a1d30>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fffb8456898>>]
16:05:50  kwargs = {'item': <Function 'test_fit_iris[F]'>}
16:05:50  
16:05:50      def _hookexec(self, hook, methods, kwargs):
16:05:50          # called from all hookcaller instances.
16:05:50          # enable_tracing will set its own wrapping function at self._inner_hookexec
16:05:50  >       return self._inner_hookexec(hook, methods, kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:92: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  hook = <_HookCaller 'pytest_runtest_call'>
16:05:50  methods = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa...ffb89a1d30>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fffb8456898>>]
16:05:50  kwargs = {'item': <Function 'test_fit_iris[F]'>}
16:05:50  
16:05:50      self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
16:05:50          methods,
16:05:50          kwargs,
16:05:50  >       firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
16:05:50      )
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:86: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  item = <Function 'test_fit_iris[F]'>
16:05:50  
16:05:50      def pytest_runtest_call(item):
16:05:50          _update_current_test_var(item, "call")
16:05:50          sys.last_type, sys.last_value, sys.last_traceback = (None, None, None)
16:05:50          try:
16:05:50  >           item.runtest()
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/runner.py:122: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <Function 'test_fit_iris[F]'>
16:05:50  
16:05:50      def runtest(self):
16:05:50          """ execute the underlying test function. """
16:05:50  >       self.ihook.pytest_pyfunc_call(pyfuncitem=self)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/python.py:1438: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_HookCaller 'pytest_pyfunc_call'>, args = ()
16:05:50  kwargs = {'pyfuncitem': <Function 'test_fit_iris[F]'>}, notincall = set()
16:05:50  
16:05:50      def __call__(self, *args, **kwargs):
16:05:50          if args:
16:05:50              raise TypeError("hook calling supports only keyword arguments")
16:05:50          assert not self.is_historic()
16:05:50          if self.spec and self.spec.argnames:
16:05:50              notincall = (
16:05:50                  set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
16:05:50              )
16:05:50              if notincall:
16:05:50                  warnings.warn(
16:05:50                      "Argument(s) {} which are declared in the hookspec "
16:05:50                      "can not be found in this hook call".format(tuple(notincall)),
16:05:50                      stacklevel=2,
16:05:50                  )
16:05:50  >       return self._hookexec(self, self.get_hookimpls(), kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/hooks.py:286: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <_pytest.config.PytestPluginManager object at 0x7fffb98cd5c0>
16:05:50  hook = <_HookCaller 'pytest_pyfunc_call'>
16:05:50  methods = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa..., plugin=<module '_pytest.skipping' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/skipping.py'>>]
16:05:50  kwargs = {'pyfuncitem': <Function 'test_fit_iris[F]'>}
16:05:50  
16:05:50      def _hookexec(self, hook, methods, kwargs):
16:05:50          # called from all hookcaller instances.
16:05:50          # enable_tracing will set its own wrapping function at self._inner_hookexec
16:05:50  >       return self._inner_hookexec(hook, methods, kwargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:92: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  hook = <_HookCaller 'pytest_pyfunc_call'>
16:05:50  methods = [<HookImpl plugin_name='python', plugin=<module '_pytest.python' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-pa..., plugin=<module '_pytest.skipping' from '/opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/skipping.py'>>]
16:05:50  kwargs = {'pyfuncitem': <Function 'test_fit_iris[F]'>}
16:05:50  
16:05:50      self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
16:05:50          methods,
16:05:50          kwargs,
16:05:50  >       firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
16:05:50      )
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/pluggy/manager.py:86: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  pyfuncitem = <Function 'test_fit_iris[F]'>
16:05:50  
16:05:50      @hookimpl(trylast=True)
16:05:50      def pytest_pyfunc_call(pyfuncitem):
16:05:50          testfunction = pyfuncitem.obj
16:05:50          if pyfuncitem._isyieldedfunction():
16:05:50              testfunction(*pyfuncitem._args)
16:05:50          else:
16:05:50              funcargs = pyfuncitem.funcargs
16:05:50              testargs = {}
16:05:50              for arg in pyfuncitem._fixtureinfo.argnames:
16:05:50                  testargs[arg] = funcargs[arg]
16:05:50  >           testfunction(**testargs)
16:05:50  
16:05:50  /opt/h2oai/h2o4gpu/python/lib/python3.6/site-packages/_pytest/python.py:166: 
16:05:50  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
16:05:50  
16:05:50  self = <test_kmeans.TestKmeans object at 0x7fff897e2710>, order = 'F'
16:05:50  
16:05:50      @pytest.mark.parametrize("order", ['C', 'F'])
16:05:50      def test_fit_iris(self, order):
16:05:50          X = np.asanyarray(load_iris().data, order=order)
16:05:50          clusters = 4
16:05:50          model = KMeans(n_gpus=1, n_clusters=clusters, random_state=123).fit(X)
16:05:50      
16:05:50          assert model.cluster_centers_.shape == (X.shape[1], clusters)
16:05:50      
16:05:50          model_rerun = KMeans(n_gpus=1, n_clusters=clusters,
16:05:50                               random_state=123).fit(X)
16:05:50      
16:05:50          # Same random_state should yield same results
16:05:50          assert np.allclose(
16:05:50              np.sort(model.cluster_centers_, axis=0), np.sort(
16:05:50                  model_rerun.cluster_centers_, axis=0)
16:05:50          )
16:05:50      
16:05:50          model_rerun2 = model_rerun.fit(X)
16:05:50      
16:05:50          # Multiple invocations of fit with the same random_state
16:05:50          # also should produce the same result
16:05:50          assert np.allclose(
16:05:50              np.sort(model_rerun.cluster_centers_, axis=0), np.sort(
16:05:50                  model_rerun2.cluster_centers_, axis=0)
16:05:50          )
16:05:50      
16:05:50          model_all = KMeans(n_clusters=clusters, random_state=123).fit(X)
16:05:50      
16:05:50          # Multi GPU should yield same result as single GPU
16:05:50  >       assert np.allclose(
16:05:50              np.sort(model.cluster_centers_, axis=0), np.sort(
16:05:50                  model_all.cluster_centers_, axis=0)
16:05:50          )
16:05:50  E       assert False
16:05:50  E        +  where False = <function allclose at 0x7fffb80e87b8>(array([[5.006000000000001  , 2.622222222222222  , 1.4620000000000004 ,\n        0.24599999999999989],\n       [5.5296296...19512195121946 ],\n       [6.9125             , 3.4280000000000004 , 5.846875           ,\n        2.1312499999999996 ]]), array([[5.005999999999999  , 2.6357142857142857 , 1.462              ,\n        0.24600000000000008],\n       [5.5321428...05128205128205 ],\n       [6.935483870967742  , 3.4279999999999995 , 5.861290322580645  ,\n        2.1258064516129034 ]]))
16:05:50  E        +    where <function allclose at 0x7fffb80e87b8> = np.allclose
16:05:50  E        +    and   array([[5.006000000000001  , 2.622222222222222  , 1.4620000000000004 ,\n        0.24599999999999989],\n       [5.5296296...19512195121946 ],\n       [6.9125             , 3.4280000000000004 , 5.846875           ,\n        2.1312499999999996 ]]) = <function sort at 0x7fffb80fa488>(array([[6.9125             , 3.0999999999999996 , 5.846875           ,\n        2.1312499999999996 ],\n       [5.0060000...85185185185183 ],\n       [6.236585365853658  , 2.858536585365854  , 4.807317073170732  ,\n        1.6219512195121946 ]]), axis=0)
16:05:50  E        +      where <function sort at 0x7fffb80fa488> = np.sort
16:05:50  E        +      and   array([[6.9125             , 3.0999999999999996 , 5.846875           ,\n        2.1312499999999996 ],\n       [5.0060000...85185185185183 ],\n       [6.236585365853658  , 2.858536585365854  , 4.807317073170732  ,\n        1.6219512195121946 ]]) = <h2o4gpu.solvers.kmeans.KMeansH2O object at 0x7fff897e22e8>.cluster_centers_
16:05:50  E        +    and   array([[5.005999999999999  , 2.6357142857142857 , 1.462              ,\n        0.24600000000000008],\n       [5.5321428...05128205128205 ],\n       [6.935483870967742  , 3.4279999999999995 , 5.861290322580645  ,\n        2.1258064516129034 ]]) = <function sort at 0x7fffb80fa488>(array([[6.2615384615384615 , 2.8512820512820514 , 4.8076923076923075 ,\n        1.6205128205128205 ],\n       [5.0059999...58064516129034 ],\n       [5.5321428571428575 , 2.6357142857142857 , 3.960714285714286  ,\n        1.2285714285714284 ]]), axis=0)
16:05:50  E        +      where <function sort at 0x7fffb80fa488> = np.sort
16:05:50  E        +      and   array([[6.2615384615384615 , 2.8512820512820514 , 4.8076923076923075 ,\n        1.6205128205128205 ],\n       [5.0059999...58064516129034 ],\n       [5.5321428571428575 , 2.6357142857142857 , 3.960714285714286  ,\n        1.2285714285714284 ]]) = <h2o4gpu.solvers.kmeans.KMeansH2O object at 0x7fff897e28d0>.cluster_centers_
16:05:50  
16:05:50  tests/python/open_data/kmeans/test_kmeans.py:52: AssertionError
@sh1ng sh1ng self-assigned this Dec 17, 2019
@monk1337
Copy link

monk1337 commented Sep 8, 2020

@sh1ng Can you please take a look into this,
#860

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sh1ng sh1ng

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sh1ng @monk1337