Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-12-10

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:59 Longgeek joined #fuel
01:08 rongze joined #fuel
01:09 rmoe joined #fuel
01:40 rongze joined #fuel
02:02 coryc joined #fuel
02:21 xarses joined #fuel
02:21 rongze joined #fuel
02:48 Longgeek joined #fuel
03:24 fandi joined #fuel
04:14 dpyzhov joined #fuel
04:24 SergK joined #fuel
04:24 mattymo joined #fuel
04:24 xarses joined #fuel
04:37 Longgeek joined #fuel
05:16 rongze_ joined #fuel
05:38 Longgeek joined #fuel
06:20 clsRobNeff joined #fuel
06:20 coryc joined #fuel
06:45 Longgeek joined #fuel
06:46 teran joined #fuel
07:07 dklepikov joined #fuel
07:08 e0ne joined #fuel
07:10 IlyaE joined #fuel
07:34 ntt joined #fuel
07:40 alexbh joined #fuel
07:45 teran_ joined #fuel
07:59 adanin joined #fuel
08:05 Philipp_ joined #fuel
08:06 RobN-Flex joined #fuel
08:09 kaliya joined #fuel
08:38 sc-rm kaliya: Now after some normal use at our place of openstack/swift part, I get this message from ceph: “HEALTH_WARN pool .rgw.buckets has too few pgs”
08:40 sc-rm kaliya: http://paste.openstack.org/show/148515/ is what we have of health details
09:00 rtau joined #fuel
09:01 rtau Hi, just tried to update Mirantis Fuel master from 5.1 to 5.1.1 then update the Openstack installation as well. Now, the update failed due to timeout.
09:02 rtau Anyone have luck with that?
09:09 stamak joined #fuel
09:10 taj joined #fuel
09:21 iscsi joined #fuel
09:21 iscsi morning
09:22 iscsi fresh mos 5.1.1 get "Please insert CentOS disk 1 to continue" while setup
09:24 kaliya hi all, here for a short while
09:25 [HeOS] joined #fuel
09:27 kaliya sc-rm: `ceph osd lspools` ?
09:27 kaliya hi rtau, the master upgrade to 5.1.1 has completed succesfully?
09:30 iscsi okay, probably this setup issue is virtual drive problem (remote drive)
09:30 rtau @kaliya, yes! Without any problem.
09:32 kaliya rtau: which update packages did you select?
09:32 rtau kaliya, it is the ubuntu one.
09:32 rtau I could see the puppet apply on the primary controller never end with error due to timeout on changing the state of p_neutron_metadata-agent to running state, further checking the CRM status shows that there are some location rule which prevent the resource from running.
09:33 rtau So, I played by deleting those location rule and deploy the change again..
09:33 rtau The deployment is still running now.
09:33 kaliya rtau: so is it failing on controllers?
09:34 rtau kaliya: Yes, on the primary controller....
09:34 kaliya rtau: can you inspect in nailgun logs and find the exact timeout error? I will look into bugs or file a new one
09:35 kaliya rtau: moreover, prior, it's possible you have to apply this https://review.openstack.org/#/c/139130/
09:36 rtau kaliya: There are several logs in the nailgun docker: api, app, receiverd, assassind, uwsgi. Which one do you need?
09:36 kaliya app
09:37 sc-rm kaliya: 0 data,1 metadata,2 rbd,3 .rgw.root,4 images,5 volumes,6 compute,7 .rgw.control,8 .rgw,9 .rgw.gc,10 .users.uid,11 .rgw.buckets.index,12 .rgw.buckets
09:38 kaliya rtau: the patch above is highly suspected to fix your error
09:38 rtau kaliya: Can't find the word 'timeout' in the app.log.
09:39 Philipp_ is it possible to change anything after a deployment using fuel? all fields seem to be write protected
09:39 kaliya Philipp_: no. You have to reset the environment to.
09:40 kaliya rtau: is your deployment-upgrade still running?
09:40 Philipp_ this would lead to data loss wouldn't it?
09:41 kaliya Philipp_: yes
09:41 e0ne joined #fuel
09:41 rtau kaliya: Yes, clicked on 'Deploy Changes' several times.
09:41 kaliya Philipp_: you cannot modify things after a deployment, e.g. neutron subnets or adding ceilometer, without destroying things. That's why we don't allow.
09:42 kaliya rtau: several times? :D
09:42 Philipp_ kaliya what about upgrades do they lead to data loss as well?
09:42 kaliya Philipp_: they upgrade packages, leaving confs and data untouched
09:42 kaliya rtau: before redeploying again the upgrade, please try to apply the patch
09:43 Philipp_ kalya I can extend the openstack cluster at anytime without data loss?
09:44 rtau kaliya: Yes, that's kinda of toys for me. Just applied the patch. if the deploy still fail, I will click on it again.
09:44 kaliya Philipp_: you can add/remove nodes under the current configuration, yes
09:45 kaliya thanks rtau, it's important to me know if this patch will solve your issue. Are you online later in case (it won't work)?
09:45 Philipp_ kaliya how can I see all used ip addresses? I would like to add some other vms into the public network without having ip conflicts
09:46 dklepikov <sc-rm>: hello
09:46 kaliya Philipp_: `nova list`
09:46 rtau kaliya: I may leaving it running tonight, will be back tomorrow 10am in HKT (UTC+8).
09:46 kaliya rtau: that's nice. Please join us again here. If it won't work, we'll file some bug. Thanks!
09:47 rtau kaliya: The primary controller just finished deployment successfully, and now deploying on another controller (for a total of 3).
09:47 rtau I didn't click on 'deploy change' again after applying the patch through.
09:47 kaliya dklepikov: sc-rm has some problem here http://paste.openstack.org/show/148515/ is what we have of health details
09:47 teran joined #fuel
09:49 dklepikov please run 'ceph health detail'
09:50 dklepikov <kaliya>: Also please provide us with the output of next two commands:
09:50 dklepikov <kaliya> for i in $(rados df | awk '{print $1}' | grep -v pool | grep -v total); do echo -n $i " : " && ceph osd pool get $i pg_num; done
09:50 dklepikov and 'for i in $(rados df | awk '{print $1}' | grep -v pool | grep -v total); do echo -n $i " : " && ceph osd pool get $i pgp_num; done'
09:50 kaliya sc-rm: can you run those commands?
09:51 Philipp_ kaliya where can I download the openrc file for my installation?
09:51 dklepikov also we neet to know 'cep osd tree', 'ceph health detail', 'rados df'
09:51 dklepikov need
09:52 rtau Philipp_: I could find it on the root user's home directory on controller node.
09:52 Philipp_ but the ip of the controller node is unkown to me... I can do some try and error to find out which ip is the controller?
09:52 sc-rm dklepikov: kaliya: http://paste.openstack.org/show/148561/
09:54 kaliya Philipp_: on the master, `fuel --env X nodes` where X is your environment ID
09:54 rtau Philipp_: From my experience, if you are in on fuel master, `fuel node list` will show you the node id, name, ip, and role. Then ssh to the controller node you like by ssh node-<node_id>
09:54 kaliya I will be away for some hours
09:54 sc-rm dklepikov: kaliya: http://paste.openstack.org/show/148562/
09:54 dklepikov 'ceph osd tree', 'ceph health detail', 'rados df' ['df -h' on ceph nodes]
09:56 subscope joined #fuel
09:58 sc-rm dklepikov: df -h on all ceph nodes: http://paste.openstack.org/show/148563/
10:01 dklepikov It looks like too many objects are stored in .rgw.buckets
10:02 dklepikov <kaliya>: You can increase pg_num and pgp_num for this poll, but it needs a re-balance of ceph
10:05 dklepikov <kaliya>: but for remove warn it should be 4096 pg and pgp in pool .rgw.buckets (too big number)
10:05 sc-rm dklepikov: we have like 40.000 objects in the storage right now
10:06 dklepikov i see (37305)
10:06 sc-rm dklepikov: And we expect that number to increase to 10-20 times more
10:06 dklepikov ceph told you that "cluster average (7) object placed per PG "
10:07 dklepikov <sc-rm>: and into .rgw.buckets  poll (72) objects per PG
10:07 sc-rm dklepikov: I don’t understand what that message means?
10:12 dklepikov sc-rm: It means that in the specified pool (.rgw.buckets) number of objects in a placement  group (objects per pg (72)) is much higher (more than 10.2857 times ) than the average for the cluster (cluster average (7) 7 objects in PG)
10:12 alexbh joined #fuel
10:13 sc-rm dklepikov: so, will it require som reconfiguration of ceph to make it more balanced?
10:22 dklepikov sc-rm: Let me check please.
10:23 SergK_ joined #fuel
10:23 sc-rm dklepikov: I’ll wait for something we can try out
10:27 clsRobNeff joined #fuel
10:28 vtzan joined #fuel
10:29 clsRobNeff joined #fuel
10:35 geekinutah joined #fuel
10:37 vtzan joined #fuel
10:39 sc-rm dklepikov: will it give any performance issues? As i read it it’s only some avg number calculated for the entire ceph cluster.
11:11 teran joined #fuel
11:15 dklepikov sc-rm: please provide us with the output "ceph osd dump"
11:37 clsRobNeff joined #fuel
11:38 dklepikov sc-rm: please provide us with the output "ceph osd dump"
11:41 sc-rm dklepikov: http://paste.openstack.org/show/148605/
11:46 clsRobNeff joined #fuel
11:54 saibarspeis joined #fuel
12:05 clsRobNeff joined #fuel
12:09 clsRobNeff joined #fuel
12:14 dklepikov sc-rm: our ceph experts tell us that you can delete data metadata and rbd pools
12:16 dklepikov sc-rm: with the command  rados rmpoll see "GLOBAL COMMANDS" in http://ceph.com/docs/master/man/8/rados/
12:18 sc-rm dklepikov: then ceph will rebalance it self?
12:20 dklepikov sc-rm: our ceph experts tell us that no re-balance needed
12:22 sc-rm dklepikov: but if I do rados rmpool it will delete the .rgw.buckets pool and it’s data, but I’m not interested in deleting any data.
12:23 sc-rm dklepikov: if ceph will run fine with this warning and it looks so for me right now. Then I guess I can ignore this warning completely
12:24 dklepikov sc-rm: http://paste.openstack.org/show/148562/ look on 'rados df'  poll names "data", "rbd", "metadata"
12:25 dklepikov sc-rm: yes you can ignore this warning
12:25 clsRobNeff joined #fuel
12:25 sc-rm dklepikov: so what you are saying is that data, metadata and rbd are not used for anything and therefore they can be deleted as you suggest
12:52 e0ne joined #fuel
13:00 rongze joined #fuel
13:01 dklepikov sc-rm: it's a suggestion of our ceph experts, but I think that it's better not to remove these pools
13:01 sc-rm dklepikov: Then I leave them as is right now :-)
13:04 vt102 joined #fuel
13:16 clsRobNeff joined #fuel
13:23 clsRobNeff joined #fuel
13:32 teran joined #fuel
13:37 teran_ joined #fuel
13:52 acca joined #fuel
13:53 acca left #fuel
14:00 clsRobNeff joined #fuel
14:01 Longgeek joined #fuel
14:04 sressot joined #fuel
14:22 subscope joined #fuel
14:27 getup joined #fuel
14:29 jaypipes joined #fuel
14:57 swordz joined #fuel
14:58 swordz Hi. I'm trying to get SR-IOV working on a VM using 6.0 Fuel Tech Preview. Currently all my attempts have ended with this error:
14:58 swordz internal error: missing IFLA_VF_INFO in netlink response
14:58 coryc joined #fuel
14:59 swordz I've tried Ubuntu 12.04, currently trying CentOS with the 3.10 kernel. Same error on both.
14:59 swordz Anyone seen this before and know what I'm doing wrong? Anything I find online points to it being a number of VFs (i.e. > ~30) error, but I've got 8, and that's meant to be fixed.
15:01 mattgriffin joined #fuel
15:07 merdoc swordz: you have infiniband switch/cards?
15:07 swordz You're asking for my NIC info?
15:07 merdoc yep
15:07 swordz The switch is Cisco
15:07 merdoc ah. I see
15:07 swordz Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
15:07 swordz Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
15:07 swordz They both give the same VF:
15:08 swordz Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
15:08 merdoc I'm currently trying to start infiniband fabric, and have several questions (%
15:08 swordz Lost everything after the % there...
15:08 swordz Or is that a smiley I don't recognise?
15:09 merdoc yes (%
15:09 getup joined #fuel
15:14 Longgeek joined #fuel
15:14 fandi joined #fuel
15:16 blahRus joined #fuel
15:23 fandi joined #fuel
15:26 emagana joined #fuel
15:41 jobewan joined #fuel
15:45 teran joined #fuel
15:45 teran joined #fuel
15:46 clsRobNeff joined #fuel
16:04 emagana_ joined #fuel
16:05 IlyaE joined #fuel
16:16 adanin joined #fuel
16:18 e0ne_ joined #fuel
16:18 rongze joined #fuel
16:21 clsRobNeff joined #fuel
16:35 fandi hi all,
16:35 fandi i'm using fule5.1
16:35 fandi also already open ticket, but i still problem when try on ice house
16:35 fandi http://rbgeek.files.wordpress.com/2014/04/modify-vpc.png
16:36 fandi i try implemented as from that configuration
16:37 adanin joined #fuel
16:38 fandi thanks
16:38 kaliya fandi: what's the question?
16:38 fandi hi kaliya, do you see my pic
16:39 fandi http://rbgeek.files.wordpress.com/2014/04/modify-vpc.png
16:39 fandi i try as build vm NAT instances
16:39 fandi and the second vm as webserver
16:40 fandi but it's not working in grizzly :)
16:40 fandi *sorry in icehouse i mean
16:41 fandi kaliya, do you ever test this or is there i miss for this configuration? because in grizzly it's working ..
16:41 kaliya I have no idea where this configuration is from. But I guess on the NAT instance you have to set ip forwarding = 1, and some other iptables rules
16:42 fandi i'm sure already do the same things as you said .. but it's not working
16:42 kaliya how did you troubleshoot?
16:42 fandi i have tcpdump .. see google as reply from my request but it's not through
16:42 kaliya can webserver ping nat instance at first?
16:42 fandi to web
16:43 fandi it can
16:43 kaliya and nat instance can ping igw?
16:43 kaliya what's VPC for?
16:43 fandi nat instances i associate to floating ip
16:44 fandi it router :) to reach public ip
16:44 kaliya in neutron how many networks?
16:44 kaliya and are you using ovs? do you have the patch ports and all ok?
16:44 fandi yups i using ovs
16:45 kaliya this doesn't look much related to Fuel, more to some neutron conf
16:46 fandi i'm only have 2 network, add to router and the other is not connect to anywhere
16:46 fandi but i build vm on private subnet
16:47 fandi i'm still using ovs version 1.10
16:47 kaliya maybe it's some routing problems in q routers or in the instance itself
16:47 fandi on nat instance i can reach internet
16:47 kaliya can your nat instance reach internet yes?
16:47 fandi yups
16:48 kaliya what's the routing table on your webserver
16:48 fandi i'm using windows
16:48 fandi i put gateway  on windows server using ip nat instance
16:49 kaliya yes but those two nodes are on different subnets :D
16:49 kaliya evg: do you think we can help fandi ?
16:51 fandi and on ip nat instance i already configure
16:51 fandi root@lb1:/home/ubuntu# sysctl -p
16:51 fandi net.ipv4.conf.default.forwarding = 1
16:51 fandi net.ipv4.conf.all.forwarding = 1
16:51 fandi net.ipv4.ip_forward = 1
16:51 fandi sorry paste here ;)
16:51 subscope joined #fuel
16:51 fandi because it's weird
16:51 fandi lb1 is my -- router / nat istance :)
16:52 fandi kaliya, do you have idea.. because all forwarding is not working
16:52 kaliya fandi: which routers do you have there? I mean neutron ones
16:54 fandi i'm using root@node-1:~# neutron --version
16:54 fandi 2.3.4
16:55 fandi because i'm also find problem when try vrrp
16:55 fandi it's not working also on icehouse
16:56 fandi but it's already solve with --allowed-address-pairs type=dict list=true
16:56 kaliya fandi: print the neutron routers alongside with their ports?
16:57 fandi from network topology or cli ?
16:59 kaliya up to you
17:05 swordz Hi. I'm trying to get SR-IOV working on a VM using 6.0 Fuel Tech Preview, Ubuntu or CentOS. Currently all my attempts have ended with this error: internal error: missing IFLA_VF_INFO in netlink response. Any idea what I'm doing wrong?
17:06 swordz Anything I find online points to it being a number of VFs (i.e. > ~30) error, but I've got 8 VFs, and that error is meant to be fixed anyway.
17:07 rmoe joined #fuel
17:08 emagana joined #fuel
17:08 fandi hi kaliya http://paste.openstack.org/show/TQyW0A275BhI5zzDMckH/
17:08 fandi that's my router configuration from cli
17:11 fandi and this my picture
17:11 fandi https://drive.google.com/a/cbncloud.co.id/file/d/0B6cLwK5pBvzVOHNvbWw1MWxUUWs/view?usp=sharing
17:11 taj joined #fuel
17:21 fandi hi kaliya, may you check it, please ? thanks
17:22 kupo24z joined #fuel
17:23 rongze joined #fuel
17:23 emagana_ joined #fuel
17:24 kaliya fandi: is traffic passing from web to the outside of nat instance? Did you troubleshoot with tcpdump this?
17:27 fandi kaliya, i think so.. let me show you
17:30 mattgriffin joined #fuel
17:50 mattgriffin joined #fuel
18:08 emagana joined #fuel
18:08 justif joined #fuel
18:11 emagana_ joined #fuel
18:16 mattgriffin joined #fuel
18:25 e0ne joined #fuel
18:25 MiroslavAnashkin swordz: Mellanox NICs export virtual functions. What is your NIC model(s)?
18:28 rongze joined #fuel
18:28 IlyaE joined #fuel
18:31 IlyaE joined #fuel
18:33 MiroslavAnashkin swordz: BTW, for some NICs you may need both, correct firmware version flashed into NIC and correct loadable firmware version in your OS distro.
18:34 mattgriffin joined #fuel
18:34 emagana joined #fuel
18:38 clsRobNeff joined #fuel
18:42 miroslav_ joined #fuel
18:45 angdraug joined #fuel
18:45 lordd joined #fuel
18:45 teran joined #fuel
18:53 xarses joined #fuel
19:02 tatyana joined #fuel
19:07 jaypipes joined #fuel
19:11 vogelc joined #fuel
19:12 emagana joined #fuel
19:12 vogelc I have an existing MOS 5.1 envronment that I just deployed the zabbix server into.  During the installation process, no agents were installed on the cluster nodes.  Do I have run puppet locally on each one?
19:17 miroslav_ joined #fuel
19:26 [HeOS] joined #fuel
19:26 emagana joined #fuel
19:30 vogelc akislitsky: is there a process to add existing MOS nodes into a new Zabbix deployment?
19:33 rongze joined #fuel
19:39 fandi hi kaliya sorry for late reply
19:40 fandi kaliya, here my test lab http://paste.openstack.org/show/148901/
19:41 fandi kaliya, i put here, success there is success test using grizzly and the failed is using icehouse
19:42 tatyana joined #fuel
19:42 emagana_ joined #fuel
19:44 emagana joined #fuel
19:45 miroslav_ vogelc: No such process. Unfortunately, existing nodes already have generated configuration onboard. So, running pupet locally will not change existing node configuration.
19:46 vogelc miroslav_: What if we upgraded the fuel master node to 5.1.1 and pushed out an upgrade to the environment?
19:48 clsRobNeff joined #fuel
20:01 miroslav_ Hmm. It is experimental feature. You may have to hack /etc/astute.yaml on every node where you are going to add zabbix agents
20:06 coryc I upgraded our fuel node from 5.1 to 5.1.1 last night. Do I have to enable experimental mode before the OpenStack upgrade option shows up in the cluster Actions tab
20:07 teran joined #fuel
20:07 coryc And is it safe to do an ostack upgrade on an environment that is running instances
20:07 miroslav_ yes. experimental features are turned off by default
20:08 miroslav_ Sinde openstack upgrade is still experimental - it definitely not safe.
20:08 miroslav_ s/sinde/since/
20:08 coryc ok, sticking with what we have then
20:10 IlyaE joined #fuel
20:27 kozhukalov joined #fuel
20:41 rongze joined #fuel
20:47 clsRobNeff joined #fuel
20:55 adanin joined #fuel
20:57 teran_ joined #fuel
21:13 teran joined #fuel
21:15 emagana joined #fuel
21:16 IlyaE joined #fuel
21:17 clsRobNeff joined #fuel
21:21 clsRobNeff joined #fuel
21:25 clsRobNeff joined #fuel
21:25 teran_ joined #fuel
21:27 clsRobNeff left #fuel
21:27 clsRobNeff joined #fuel
21:32 teran joined #fuel
21:43 rongze joined #fuel
21:53 IlyaE joined #fuel
21:55 IlyaE joined #fuel
21:58 Longgeek joined #fuel
22:04 emagana joined #fuel
22:11 apalkina joined #fuel
22:17 tatyana_ joined #fuel
22:20 IlyaE joined #fuel
22:22 emagana joined #fuel
22:26 RobN-Flex joined #fuel
22:26 teran_ joined #fuel
22:29 emagana joined #fuel
22:43 emagana joined #fuel
22:45 IlyaE joined #fuel
22:48 rongze joined #fuel
22:50 jobewan joined #fuel
22:53 coryc1 joined #fuel
23:02 mattgriffin joined #fuel
23:11 emagana joined #fuel
23:46 nurla joined #fuel
23:57 rongze joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary