Camelia, the Perl 6 bug

IRC log for #fuel, 2013-10-28

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
16:00 _ilbot joined #fuel
16:00 Topic for #fuel is now Fuel for Openstack: http://fuel.mirantis.com/ | Paste here http://paste.openstack.org/
16:01 Topic for #fuel is now Fuel for Openstack: http://fuel.mirantis.com/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
16:02 mattymo guys, heads up that we are now logging this channel
16:07 mihgen mattymo: where can I see logs?
16:10 mattymo look at topic
16:10 mattymo mihgen, ^
16:33 xdeller joined #fuel
16:33 xdeller dhblaz, please plug in http://download.mirantis.com/elrepo-mirror/ and install kernel-ml
16:34 xdeller or you can use elrepo with current kernel, but this one was tested on couple of our production deployments
16:35 dhblaz Unfortunately that kernel and kernel-firmware combination doesn't work
16:35 dhblaz And I was hoping to modify my the ks so a deployment would put that kernel on all machines
16:36 xdeller do you use firmware-ml?
16:37 xdeller as i remember it may conflict with regular one
16:37 xdeller and for ceph, what problem do you have exactly? I see your configs and compute one does not include any mentions of rbd driver
16:43 dhblaz It doesn't matter if I use firmware-ml or the firmware included in the nailgun repo on my master. 3.10 kernels boot without working network because of missing firmware file.
16:44 dhblaz I'm working on a workaround using a third party kernel-firmware that includes the firmwares I would need to boot 2.6 or 3.10 kernels
16:45 xdeller heh, what hardware in use?
16:46 dhblaz HP blades
16:46 dhblaz bl460c and bl460c g6
16:46 dhblaz using bnx2 and bnx2x drivers
16:47 xdeller hm. bcm drivers should be presented in the firmware pkg
16:47 xdeller may be wrong paths?
16:48 dhblaz no, wrong versions
16:48 dhblaz 3.10 wants bnx2x/bnx2x-e1h-7.8.17.0.fw
16:48 xdeller wow, that`s strange
16:48 xdeller not mips-xxx?
16:48 xdeller and bnx?
16:49 dhblaz I'll be back in about 15 minutes
16:49 dhblaz sorry for the delay
16:49 xdeller ok
17:22 dhblaz joined #fuel
17:36 dhblaz I'm not sure what is going on with cinder; I get a different error message than I was getting on Friday
17:36 dhblaz It looks like a problem with the database connection now
17:36 xdeller does regular client work with same endpoint?
17:36 dhblaz And I don't see mysql running on the controller
17:36 xdeller eh..
17:38 dhblaz http://paste.openstack.org/​show/ZqUAlb3X04JEhFzPyyZQ/
17:38 dhblaz node-29 is the first controller
17:38 xdeller seemingly galera is broken
17:38 angdraug joined #fuel
17:38 xdeller it can respond with no such command when cluster is broken
17:39 dhblaz Looks like the startup script may be hung:
17:39 dhblaz [root@node-29 ~]# ps ax | fgrep mysql
17:39 dhblaz 5160 ?        S      0:00 /bin/bash /usr/lib/ocf/resource.d/mirantis/mysql start
17:39 dhblaz 5867 pts/0    S+     0:00 fgrep mysql
17:39 xdeller what`s total execution time?
17:40 xdeller i wrote it in kinda-self-healing way but it may not handle any possible wreck
17:40 VladNaboychenko joined #fuel
17:40 dhblaz ps -elf shows 00:00:00
17:41 dhblaz If there is another way I should check let me know
17:41 xdeller ps axu
17:41 xdeller should give start time
17:42 xdeller wall time should be almost zero yeah
17:42 dhblaz [root@node-29 ~]# ps axu | fgrep 5160
17:42 dhblaz root      5160  0.0  0.0 108692  2052 ?        S    17:39   0:00 /bin/bash /usr/lib/ocf/resource.d/mirantis/mysql start
17:42 dhblaz root     16636  0.0  0.0 100976   764 pts/0    S+   17:42   0:00 fgrep 5160
17:42 dhblaz [root@node-29 ~]# date
17:42 dhblaz Mon Oct 28 17:42:37 UTC 2013
17:43 xdeller can you check the logs for mysql output?
17:44 dhblaz Which logs do you suggest?
17:44 albionandrew joined #fuel
17:46 xdeller ehm, messages/syslog + mysql*
17:50 dhblaz Okay, not used to looking in /var/log/syslog on CentOS
17:50 dhblaz So I found what the logs describe as a serious problem
17:51 dhblaz It is too big for my scroll back buffer
17:51 dhblaz so I am going to attach that log segment to the ticket
17:53 MiroslavAnashkin Hmm, what is the output from  `mysql -e 'show status like "wsrep_local_state%";'` from any of controllers?
17:55 MiroslavAnashkin And `mysql -e 'show status like "wsrep_incoming%";'` ?
17:55 dhblaz output supports the assumption that mysql isn't running:
17:55 dhblaz [root@node-29 ~]# mysql -e 'show status like "wsrep_local_state%";'
17:55 dhblaz ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
17:55 dhblaz I don't see mysql in my process table
18:00 mihgen joined #fuel
18:03 MiroslavAnashkin Then, please run `crm status` command and check everything related to p_mysql.
18:05 dhblaz http://paste.openstack.org/​show/cLaasvkzYskLSNN9clBa/
18:07 MiroslavAnashkin Please also check if you have at least one running mysql instance on all controllers - or no one at all.
18:08 dhblaz Restarting the network on node-29 made it so it can now ping mysql vip
18:08 dhblaz I think that this particular problem (today, not friday) is related to network connectivity dying
18:09 dhblaz Suggestion is to move to kernel 3.10
18:09 dhblaz I have not been able to do this yet
18:09 MiroslavAnashkin There is database corruption possible.
18:09 dhblaz I'm working on changing the kickstart script so nodes are built with that kernel then I will try another deployment
18:10 MiroslavAnashkin In case you have controller with non-corrupted DB - galera still may sync with this controller.
18:11 MiroslavAnashkin unstable network connection may be the root cause as well
18:21 albionandrew Hi, I work with dhblaz we were asked by xarses to "test a machine as an ubuntu deployment (ubuntu, multi-node, single node with controller role) if possible and comment on the pull request if it works." The machine pxe boots into ubuntu and then says it can't get an IP. It clearly gets an ip because it pxe boots, can you suggest how to proceed? We have the first NIC in the PXE vlan the rest into an unused vlan so as not to conflict with
18:21 albionandrew current deployment.
18:53 mihgen joined #fuel
20:09 dhblaz joined #fuel
20:28 alex_null joined #fuel
20:31 alex_null joined #fuel
20:45 alex_null joined #fuel
20:57 alex_null joined #fuel
21:55 mihgen joined #fuel
22:30 rmoe joined #fuel
22:39 mihgen joined #fuel
22:45 xarses joined #fuel
22:48 dhblaz joined #fuel
23:11 mihgen joined #fuel
23:29 mihgen joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary