StackStorm Quick Install Failure due to RabbitMQ (new error)

[lm-ydubler]

I've installed StackStorm numerous times on my Linux Centos7 Virtual Machine (VMWARE) before using the exact same steps without issue. Now, when trying to do a Quick Install, I get the following error due to a RabbitMQ failure:

20211108T134036-0500 Resolving Dependencies
20211108T134036-0500 --> Running transaction check
20211108T134036-0500 ---> Package rabbitmq-server.noarch 0:3.9.8-1.el7 will be installed
20211108T134037-0500 --> Finished Dependency Resolution
20211108T134037-0500 
20211108T134037-0500 Dependencies Resolved
20211108T134037-0500 
20211108T134037-0500 ================================================================================
20211108T134037-0500  Package            Arch      Version         Repository                   Size
20211108T134037-0500 ================================================================================
20211108T134037-0500 Installing:
20211108T134037-0500  rabbitmq-server    noarch    3.9.8-1.el7     rabbitmq_rabbitmq-server     14 M
20211108T134037-0500 
20211108T134037-0500 Transaction Summary
20211108T134037-0500 ================================================================================
20211108T134037-0500 Install  1 Package
20211108T134037-0500 
20211108T134037-0500 Total download size: 14 M
20211108T134037-0500 Installed size: 19 M
20211108T134037-0500 Downloading packages:
20211108T134040-0500 Running transaction check
20211108T134040-0500 Running transaction test
20211108T134040-0500 Transaction test succeeded
20211108T134040-0500 Running transaction
20211108T134041-0500   Installing : rabbitmq-server-3.9.8-1.el7.noarch                           1/1 
20211108T134041-0500   Verifying  : rabbitmq-server-3.9.8-1.el7.noarch                           1/1 
20211108T134041-0500 
20211108T134041-0500 Installed:
20211108T134041-0500   rabbitmq-server.noarch 0:3.9.8-1.el7                                          
20211108T134041-0500 
20211108T134041-0500 Complete!
20211108T134129-0500 Created symlink from /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service to /usr/lib/systemd/system/rabbitmq-server.service.
20211108T134213-0500 Error: unable to perform an operation on node 'rabbit@ip-192-168-63-137'. Please see diagnostics information and suggestions below.
20211108T134213-0500 
20211108T134213-0500 Most common reasons for this are:
20211108T134213-0500 
20211108T134213-0500  * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
20211108T134213-0500  * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
20211108T134213-0500  * Target node is not running
20211108T134213-0500 
20211108T134213-0500 In addition to the diagnostics info below:
20211108T134213-0500 
20211108T134213-0500  * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
20211108T134213-0500  * Consult server logs on node rabbit@ip-192-168-63-137
20211108T134213-0500  * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
20211108T134213-0500 
20211108T134213-0500 DIAGNOSTICS
20211108T134213-0500 ===========
20211108T134213-0500 
20211108T134213-0500 attempted to contact: ['rabbit@ip-192-168-63-137']
20211108T134213-0500 
20211108T134213-0500 rabbit@ip-192-168-63-137:
20211108T134213-0500   * connected to epmd (port 4369) on ip-192-168-63-137
20211108T134213-0500   * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
20211108T134213-0500   * can't establish TCP connection to the target node, reason: timeout (timed out)
20211108T134213-0500   * suggestion: check if host 'ip-192-168-63-137' resolves, is reachable and ports 25672, 4369 are not blocked by firewall
20211108T134213-0500 
20211108T134213-0500 Current node details:
20211108T134213-0500  * node name: 'rabbitmqcli-938-rabbit@ip-192-168-63-137'
20211108T134213-0500  * effective user's home directory: /var/lib/rabbitmq
20211108T134213-0500  * Erlang cookie hash: oD7i3mI6r9JthBecmSClQQ==
20211108T134213-0500 
20211108T134213-0500 ############### ERROR ###############
20211108T134213-0500 # Failed on Install st2 dependencies (RabbitMQ) #
20211108T134213-0500 #####################################

I followed the exact same steps which led to successful installations in the past but now suddenly it broke. Any thoughts?

[amanda11]

I presume you are installing from the unstable repo? Can you confirm?

The bash installer was updated for the upcoming 3.6 release to use the latest rabbitmq - so I'm not sure if this is the first time you have installed since that update? That may be the difference between previous installs, as up until fairly recently it would have installed the rabbitMQ that is packaged with EL7 (which is old).

Can you check the ports as detailed in the message, check they aren't in use or blocked by firewall?
It may also be worth checking if any SELinux errors are reported.

[lm-ydubler]

Yes, both ports indicated (25672, 4369) are open (the command being run after the installation failed

)

[lm-ydubler]

So this was using the Quick Install method -- https://docs.stackstorm.com/install/index.html - which uses the following command:
bash <(curl -sSL https://stackstorm.com/packages/install.sh) --user=st2admin --password=Ch@ngeMe
I don't know what repo this method is installing from.

And yes, I agree. It is likely because the installer is using a new RabbitMQ version. That said, it's probably good that StackStorm knows their Quick Install method doesn't work on a fresh install of a Linux Centos7 VM in it's current state (I did install GNOME Desktop).

I will take a look at the ports and report back. Thanks so much.

[nzlosh]

As a side note: The RabbitMQ version installed by default from the CentOS7 repository is out of date and has known security vulnerabilities. Our intention is to update the documentation to use the official RabbitMQ provided packages, but in the case of CentOS7, the OS is not supported by them due to the age of openssl. https://www.rabbitmq.com/install-rpm.html

[lm-ydubler]

Good to know. Can you confirm a work around?

I'm thinking I'll pre-install the latest version of RabbitMQ up on Centos7 (if that's even possible?) using those length instructions then try the Quick Install method? Would you have a reccomended route for getting StackStorm up on Centos7 with this RabbitMQ obstacle in mind?

[lm-ydubler]

The Quick Install method also failed in the same way (RabbitMQ) for VMWare Linux Centos8.

[amanda11]

That is odd, it is working on our CI systems, and CentOS 8 has always used the latest RabbitMQ. So that hasn't changed, and we've had no problems before reported.

I would suspect that if you install RabbitMQ manually you will get the same error, and then would need to debug with RabbitMQ instructions - but you could go down the detailed instruction route.

If you want to install the older version then you can use --version 3.5.0 added to the command line, so that it installs ST2 3.5 instead.

[nzlosh]

I'm unable to reproduce this issue. I tested the script installation on a clean installation of CentOS8, it worked flawlessly.

[root@localhost ~]# systemctl restart rabbitmq-server.service 
[root@localhost ~]# systemctl status rabbitmq-server.service 
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-11-11 07:57:29 EST; 4s ago
  Process: 47234 ExecStop=/usr/sbin/rabbitmqctl shutdown (code=exited, status=0/SUCCESS)
 Main PID: 47278 (beam.smp)
    Tasks: 22 (limit: 11472)
   Memory: 100.7M
   CGroup: /system.slice/rabbitmq-server.service
           ├─47278 /usr/lib64/erlang/erts-12.1/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t >
           ├─47293 erl_child_setup 32768
           ├─47317 /usr/lib64/erlang/erts-12.1/bin/epmd -daemon
           ├─47334 inet_gethost 4
           └─47335 inet_gethost 4

Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Doc guides:  https://rabbitmq.com/documentation.html
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Support:     https://rabbitmq.com/contact.html
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Tutorials:   https://rabbitmq.com/getstarted.html
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Monitoring:  https://rabbitmq.com/monitoring.html
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Logs: /var/log/rabbitmq/[email protected]
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:         /var/log/rabbitmq/rabbit@localhost_upgrade.log
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:         <stdout>
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Config file(s): (none)
Nov 11 07:57:29 localhost.localdomain rabbitmq-server[47278]:   Starting broker... completed with 0 plugins.
Nov 11 07:57:29 localhost.localdomain systemd[1]: Started RabbitMQ broker.

It appears your VM is not in a good state for rabbitmq. My guess would there are residual configuration in /etc/rabbimq or you need to purge /var/lib/rabbitmq

[lm-ydubler]

Interesting, ok. I installed Centos8 on VMWare using an official ISO (pretty sure it was an official ISO...). I did the absolute minimum to get it installed and then installed GNOME Desktop. And it failed for me with the RabbitMQ issue... so... it's likely you might install Centos8 with a different set of initial conditions. In any case, I switched to Ubuntu 20.04 and it's installing perfectly so I will just work with that environment instead to save time. That said, thanks for all of your help from both of you, it's much appreciated !