Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2013-11-19

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:16 TSCHAKMac joined #fuel
01:31 xarses joined #fuel
01:57 rmoe joined #fuel
02:47 _ilbot joined #fuel
02:47 Topic for #fuel is now Fuel for Openstack: http://fuel.mirantis.com/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
02:47 aleksandr_null joined #fuel
03:51 ArminderS joined #fuel
04:20 TSCHAKMac joined #fuel
04:35 IlyaE joined #fuel
04:46 fandikurnia01 joined #fuel
04:55 IlyaE joined #fuel
05:05 IlyaE joined #fuel
05:06 mihgen joined #fuel
05:53 mihgen joined #fuel
06:15 e0ne joined #fuel
06:44 e0ne joined #fuel
07:16 TSCHAKMac joined #fuel
07:33 mrasskazov joined #fuel
07:49 tatyana joined #fuel
07:54 e0ne joined #fuel
07:57 dnikishov left #fuel
08:22 amartellone joined #fuel
08:24 amartellone @MiroslavAnashkin: Do you have (good) news about the volume problem that we have seen yesterday?
08:33 xdeller joined #fuel
08:35 r0mikiam joined #fuel
08:40 mihgen joined #fuel
08:45 tatyana joined #fuel
08:50 ArminderS- joined #fuel
09:03 vkozhukalov joined #fuel
09:04 anotchenko joined #fuel
09:05 tatyana joined #fuel
09:24 e0ne joined #fuel
09:25 e0ne_ joined #fuel
09:29 r0mikiam joined #fuel
10:03 teran joined #fuel
10:09 teran_ joined #fuel
10:09 b-zone joined #fuel
10:12 teran joined #fuel
10:14 steale joined #fuel
10:16 alex_stepanchuk joined #fuel
10:23 ArminderS joined #fuel
10:34 Nikolay amartllone:  I have analysed your logs. It seems that you have problem with mysql.
10:34 Nikolay here it is the excerpt from your logs with error messages related to volume creation and mysql:   http://paste.openstack.org/show/53538/
10:35 Nikolay check the state of corosync first
10:35 Nikolay with command "# service corosync status "
10:35 Nikolay on controller
10:36 Nikolay try to run the command  " crm resource  restart  clone_p_mysql "  on one of your controllers to rebuild mysql
10:36 Nikolay You can get more info on troubleshooting on our FAQ:  http://docs.mirantis.com/fuel/fuel-3.2/frequently-asked-questions.html
11:03 IlyaE joined #fuel
11:19 amartellone @Nikolay : Hi...corosync is running...
11:22 amartellone I 've done crm resource restart clone_p_mysql
11:22 amartellone now do I retry to create a new volume?
11:24 amartellone but now I can't enter in dashboard!
11:24 amartellone login failed
11:29 anotchenko joined #fuel
11:31 Nikolay You should wait up to 5 minutes for mysql to rebuild
11:36 amartellone ok...but now I've restarted corosync...
11:36 amartellone how is the procedure? I should run "crm resource  restart  clone_p_mysql"
11:36 amartellone ?
11:39 aabashkin joined #fuel
11:41 r0mikiam joined #fuel
11:48 jkirnosova joined #fuel
11:53 IlyaE joined #fuel
11:53 vk_ joined #fuel
11:57 e0ne joined #fuel
11:58 Nikolay Yes.  You should login to controller as root and run this command.
12:03 r0mikiam joined #fuel
12:03 amartellone I've done corosync restart and crm resource  restart  clone_p_mysql... but now I have a 504 error when I try to access to dashboard
12:05 amartellone but the corosync stop I must to do on all controller nodes?
12:06 amartellone and after restart them on all nodes...and after on a controller crm resource restart clone_p_mysql?
12:07 amartellone in corosync log I see err:    [MAIN  ] Another Corosync instance is already running.
12:07 amartellone 2013-11-19T19:08:26.708079+00:00 err:    [MAIN  ] Corosync Cluster Engine exiting with status 18 at main.c:1756.
12:08 ruhe joined #fuel
12:13 vk_ joined #fuel
12:18 MiroslavAnashkin Please stop corosync service `/etc/init.d/corosync stop`
12:19 MiroslavAnashkin Repeate this on all controllers
12:21 MiroslavAnashkin It may take up to 20 minutes to stop last instance of corosync.
12:22 MiroslavAnashkin Then check processes, and kill the remained corosync processes, if exist .
12:24 MiroslavAnashkin After that, start corosync on the first controller `/etc/init.d/corosync start` and wait about 5 minutes
12:25 MiroslavAnashkin Then, restart all Openstack services on the controller with started corosync.
12:27 MiroslavAnashkin You may restart whole Openstack with the following command:
12:27 MiroslavAnashkin `chkconfig --list | grep 3:on | grep -e openstack -e quantum -e neutron -e libvirt | awk '//{print $1}' | xargs -i service {} restart \;`
12:28 MiroslavAnashkin Repeat the corosync start procedure on the remained controller nodes.
12:29 MiroslavAnashkin Repeat Openstack services restart procedure on the remained controller nodes and on the compute+cinder ones
12:30 MiroslavAnashkin Openstack restart command works on any Openstack role nodes.
12:32 MiroslavAnashkin And we suggest such corosync behavior (lost nodes) comes from slow network speed in GRE mode. GRE greatly depends on underlying hardware and settings.
12:39 vk_ joined #fuel
12:45 rshon joined #fuel
12:59 amartellone @miroslav: thanks...I'm doing..
13:08 aabashkin joined #fuel
13:17 amartellone @Miroslav; I haven't chkconfig...Can I use another tool?
13:18 vk_ joined #fuel
13:22 MiroslavAnashkin Ah, sorry, you use Ubuntu.
13:23 MiroslavAnashkin Then, correct command should be
13:24 amartellone ok Miroslav, I've downloaded .deb and I've runned chkconfig ....
13:29 rmoe joined #fuel
13:31 MiroslavAnashkin `for i in /etc/init.d/openstack* /etc/init.d/swift* /etc/init.d/quantum* /etc/init.d/neutron* /etc/init.d/libvirt*; do $i restart ; done`
13:34 TSCHAKMac joined #fuel
14:04 ruhe joined #fuel
14:31 anotchenko joined #fuel
15:07 amartellone Help...how can I restart mysql on controller nodes (in HA)?
15:08 amartellone I don't found in service mysql or in /etc/init.d
15:09 MiroslavAnashkin you don't need to restart mysql in HA mode directly
15:09 MiroslavAnashkin Pacemaker/Corosync does it
15:10 amartellone I've done it
15:10 amartellone after that I've stopped corosync
15:11 MiroslavAnashkin Manual mysql restart may force corosync to run mysql restart procedure as well.
15:12 MiroslavAnashkin Please keep in mind - there is no MySQL in HA mode, instead, there is MySQL cluster.
15:12 amartellone oki
15:13 amartellone Now, if I restart corosync service?
15:14 amartellone Mysql will restart?
15:19 TSCHAKMac joined #fuel
15:21 amartellone ok...I've restarded corosync and the service are on
15:37 amartellone what clone_p_mysql does?
15:39 MiroslavAnashkin It points to MySQL cluster related actions - start, stop, restart etc
15:42 amartellone now I have running corosync on only one controller..I see the openstack dashboard but I have some error on quotas(Unable to retrieve quota information.) and I can't create new volume (Unable to create volume)
15:45 amartellone and I always the same error on compute node...cinder-cinder.db.sqlalchemy.session.log:  File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/session.py", line 104, in _wrap_db_error
15:49 MiroslavAnashkin please run `crm status` first and paste its output somewhere to http://paste.openstack.org/
15:50 MiroslavAnashkin Normally cluster should stop operations if there is only one cluster node is up and running.
15:51 amartellone yes, i would try with one controller and after with all 3 controllers
15:51 amartellone the output of crm status is http://paste.openstack.org/show/53604/
15:58 Nikolay joined #fuel
15:58 IlyaE joined #fuel
16:04 MiroslavAnashkin Then you have to reconfigure vote number in corosync first - to set it to 1.
16:05 MiroslavAnashkin Corosync docs http://clusterlabs.org/doc/
16:06 MiroslavAnashkin Please also consult http://docs.mirantis.com/fuel/fuel-3.2/frequently-asked-questions.html#id8
16:17 IlyaE joined #fuel
16:22 ruhe joined #fuel
16:38 ruhe joined #fuel
16:39 IlyaE joined #fuel
16:53 tatyana joined #fuel
16:55 Fecn joined #fuel
16:57 Fecn Hi Folks - we're seeing extremely poor network performance (kilobits per second) on tenant networks when using a Neutron+GRE setup, but we're still getting blistering perfomance on the net04_ext network - anyone got any idea what might be the problem?
17:06 xarses joined #fuel
17:08 amartellone @Miroslav: I've done all steps writes in the faq page....but I have 3 controller, where corosync running, mysqld process running, but I can't connect to mysql
17:08 MiroslavAnashkin GRE greatly depends on underlying hardware and settings.
17:08 MiroslavAnashkin It may work out of the box and may not.
17:09 Fecn underlying hardware is suitably kickass.. 10gbit networks, 192gb ram, 12cores etc
17:09 MiroslavAnashkin I mean compatibility.
17:09 Fecn broadcom nics
17:09 Fecn but yes.. there have been driver issues with those fixed in more recent kernels
17:10 rmoe joined #fuel
17:16 MiroslavAnashkin <Fecn>: You may try set promisc mode=on for the NICS, which serves GRE traffic, you may try to set Generic receive offload OFF and generic Segmentation offload off (with `ethtool -K ethx gro off` and `ethtool -K ethx gso off` )
17:17 MiroslavAnashkin <Fecn> Please also check the following for GRE troubleshooting:
17:17 MiroslavAnashkin http://www.cisco.com/en/US/tech/tk827/tk369/technologies_tech_note09186a0080093f1f.shtml
17:19 MiroslavAnashkin <amartellone>: please check if mysqld and mysqld_safe processes are running
17:22 IlyaE joined #fuel
17:31 amartellone yes, they're running
17:32 MiroslavAnashkin <amartellone>: please try `service haproxy restart`
17:33 amartellone ok...I've done
17:34 amartellone it is necessary do it on all controllers?
17:35 amartellone is it necessary does it on all controllers? (sorry)
17:36 vkozhukalov joined #fuel
17:36 MiroslavAnashkin For haproxy - yes.
17:38 amartellone I've done
17:40 r0mikiam joined #fuel
17:41 amartellone If I try, on controller node, to connect to db ...but it refuses connection
17:42 amartellone I remember that a few of days ago I try connection by root
17:46 MiroslavAnashkin Please check /var/log/messages for mysql-related errors. You may also do it from master node, on the Logs tab.
17:50 MiroslavAnashkin <Fecn>: Did you deployed Openstack under Ubuntu?
17:51 Fecn MiroslavAnashkin: Hi
17:51 MiroslavAnashkin <amartellone>: please check corosync with `crm status` one more time
17:51 Fecn MiroslavAnashkin: No - on CentOS - we have an issue whereby our interfaces on ubuntu aren't detected in the same order as the centos bootstrap gets them
17:52 Fecn for some reason, ubuntu finds the two 10gbit interfaces as eth0 and eth1, whereas centos finds the four onboard 1gbit nics as eth0-eth3
17:53 MiroslavAnashkin <Fecn>: Thank you! Actually it is first time we hear about GRE issues under Centos, all the previous cases were with Ubuntu.
17:53 Fecn MiroslavAnashkin: I'm happy to test any/all fixes - we have two racks of gear just for playing with Fuel right now
18:03 Fecn MiroslavAnashkin: OK - I tried running 'ethtool -K eth8 gro off; ethtool -K eth8 gso off; ifconfig eth8 promisc' across all the nodes, but that doesn't seem to have made any noticable difference
18:03 Fecn Performance is bursty... does 14KB/sec for a few dozen secs... then burst up to a few MB/sec... then drops back down to kb/sec
18:05 jsergent joined #fuel
18:06 xdeller Fecn: may I suggest to play with tx queue length of the nic driver? it requires module reload but can be helpful
18:06 Fecn xdeller: If you can point me in the right direction, I'll give it a try
18:08 xdeller is it bnx2x?
18:08 Fecn It is
18:12 xdeller ok, can you try to reload it with num_queues=1 at first?
18:13 Fecn OK.. one moment please (got 10 machines to do it on)
18:15 Fecn I'm going to need to restart openvswitch or somesuch after this too
18:16 xdeller no, you`ll just lost connectivity for a moment, so do it in screen session or in physical console, no other actions should be taken
18:16 xdeller just check nova-compute health after
18:17 xdeller by issuing source openrc ; nova-manage service list on one of controllers
18:17 Fecn The interface I access from is not the bnx2x one... so I don't loose connectivity... but after rmmod + modprobe, I lost the ability to ping the defgw
18:17 xdeller ok, just re-add default route via ip r add default via <ip>
18:17 Fecn defgw is on a tagged vlan.. hence my thinking it is openvswitch
18:18 xdeller generally you should not restart it
18:33 teran_ joined #fuel
18:36 teran__ joined #fuel
18:48 Fecn xdeller: restarting openvswitch did indeed retsore things back to normal... so now I'm going for it with... for host in `grep node /etc/dnsmasq.conf | awk -F, {'print $3'}`; do echo $host; ssh $host 'rmmod bnx2x && sleep 5 && modprobe bnx2x num_queues=1 && sleep 5 && /etc/init.d/openvswitch restart' ; done
18:48 xdeller ok
18:49 mihgen joined #fuel
18:54 * Fecn has broken it all
19:00 angdraug joined #fuel
19:03 ruhe joined #fuel
19:42 IlyaE joined #fuel
20:07 IlyaE joined #fuel
20:45 IlyaE joined #fuel
20:49 teran joined #fuel
21:07 IlyaE joined #fuel
21:32 IlyaE joined #fuel
21:34 rmoe_ joined #fuel
21:51 xarses joined #fuel
22:16 rmoe joined #fuel
22:23 rmoe joined #fuel
22:25 xarses_ joined #fuel
22:53 tatyana joined #fuel
23:13 * Fecn has fixed it all
23:24 anoopl joined #fuel
23:25 rmoe joined #fuel
23:26 anoopl hi
23:27 anoopl I got installed Fuel 3.2 as a VM on a KVM host
23:27 anoopl I have a couple of fresh blade servers as nodes
23:28 anoopl the problem is
23:28 anoopl i have already running dhcp server for the network range i used for the Fuel master
23:29 anoopl i am trying to make the blades/nodes installed without disturbing/stopping the DHCP server on this network
23:29 anoopl any suggestions on this
23:29 anoopl I am really confused
23:30 anoopl any help would be appreciated
23:30 anoopl ty
23:43 anoopl is it not possible?
23:48 xarses anoopl: the Fuel node needs to be able to be the DHCP server for the hosts you want to use
23:50 xarses anoopl: fuel will also test the public, mangement, and storage network for DHCP servers aswell as they can cause issues having a usable openstack environment
23:52 xarses the Fuel DHCP PXE network (sometimes refereed to as the admin network) and the management, and storage networks can be private networks that only the fuel and nodes use, the public network is the only network that needs to be accessible to your actual network
23:54 anoopl Thanks
23:54 anoopl I guess I got it
23:54 anoopl Thanks Xarses:
23:55 rmoe joined #fuel
23:55 anoopl Let me try what I would try
23:56 anoopl say i  use 192.168.5.0 as Fuel Pxe/Admin network
23:57 anoopl which does not exist in our network
23:58 anoopl then use 192.168.1.xx (say range of IPs that we already used in our network)
23:58 anoopl as public IP for the instances/VMs in compute node
23:58 anoopl is that right?

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary