Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-10-14

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:22 rmoe joined #fuel
00:46 Rajbir joined #fuel
01:08 Rajbir joined #fuel
01:10 xarses joined #fuel
01:29 kupo24z xarses: got a new error.. when the 'Assign Public IP' box is unchecked it still checks for the public avail space even though it doesnt use it
01:29 kupo24z Not enough IP addresses. Public network must have at least 22 IP addresses for the current environment.
01:31 kupo24z Pretty easy to reproduce it seems
02:00 jpf_ joined #fuel
02:14 Kupo24z1 joined #fuel
02:47 sc-rm_ joined #fuel
02:50 sanek_ joined #fuel
02:52 mihgen_ joined #fuel
02:54 mrasskazov joined #fuel
02:54 holser joined #fuel
02:55 xarses joined #fuel
02:58 dkaigarodsev joined #fuel
03:00 dshulyak joined #fuel
03:00 evg joined #fuel
03:01 meow-nofer joined #fuel
03:01 harybahh joined #fuel
03:01 izinovik joined #fuel
03:03 jaypipes joined #fuel
03:07 Samos123 joined #fuel
03:21 aglarendil joined #fuel
03:21 bdudko joined #fuel
03:40 AKirilochkin joined #fuel
04:05 Rajbir Kaliya : Any updates from the developers ?
04:07 ArminderS joined #fuel
04:08 ArminderS- joined #fuel
04:10 AKirilochkin joined #fuel
04:12 ArminderS joined #fuel
04:14 ArminderS- joined #fuel
04:17 kio joined #fuel
04:17 ArminderS joined #fuel
04:24 ArminderS joined #fuel
05:02 harybahh joined #fuel
05:06 AKirilochkin joined #fuel
05:14 emagana joined #fuel
05:16 Longgeek joined #fuel
05:35 emagana_ joined #fuel
05:46 teran joined #fuel
05:46 mutex joined #fuel
05:56 e0ne joined #fuel
06:16 dancn joined #fuel
06:44 Longgeek joined #fuel
06:50 Longgeek_ joined #fuel
06:55 saibarspeis joined #fuel
06:57 saibarspeis Hi, I have a fuel 5.0 installed into vmware and 5 environemnts with openstack deployed with it. Because of bug or incomplete implementation I ran out of space in my /var due to logs. Now i've cleared the logs and the nevironments came back into fuel interface.
06:57 saibarspeis I have performed a reset on one of the environments and the nodes rebooted but never came back online.
06:58 saibarspeis I can confirm it's on the same layer 2 segment, dnsmasq is running and I can see in fuel the dhcp requests.
06:58 saibarspeis Any ideas why this might be happening and how to fix it ?
06:59 syt joined #fuel
07:00 pasquier-s joined #fuel
07:05 kaliya saibarspeis: how did you clear the logs?
07:06 saibarspeis I've deleted them :)
07:06 kaliya saibarspeis: better to stop rsyslog container before
07:06 kaliya are your containers all up and running? `dockerctl check`
07:06 saibarspeis I've restarted ALL the environment after
07:07 kaliya saibarspeis: sure about your nodes boot order? ;)
07:07 saibarspeis dockerctl check shows no errors
07:07 saibarspeis All is up
07:07 saibarspeis kaliya: Very sure, I've manually selected the boot method and the interface
07:07 saibarspeis And I already said I see the bootp requests in fuel
07:07 saibarspeis dnsmasq is running but there is no response from it
07:08 kaliya please look if dhcpcheck discover --ifaces eth0 is up
07:09 harybahh joined #fuel
07:10 saibarspeis Run this on fuel ?
07:10 kaliya yep on master node
07:10 saibarspeis Says ERROR: (app)
07:12 kaliya saibarspeis: on which master's interface is dhcp running?
07:12 kaliya are you running a default installation?
07:12 saibarspeis eth0 should be pxe
07:13 saibarspeis Let me check
07:13 saibarspeis Yep
07:13 saibarspeis So running this command shows that error
07:13 kaliya so `dhcpcheck discover --ifaces eth0` gives error?
07:13 saibarspeis Yes
07:13 saibarspeis ERROR: (app)
07:13 kaliya try `dockerctl restart cobbler`
07:14 saibarspeis fuel-core-5.0-cobbler
07:14 kaliya retry the dhcpcheck?
07:14 saibarspeis Same result
07:15 hyperbaba joined #fuel
07:15 saibarspeis ERROR: (app)
07:18 saibarspeis Any ideas ?
07:18 kaliya saibarspeis: try `dockerctl destroy cobbler; dockerctl build cobbler`
07:18 azemlyanov joined #fuel
07:18 saibarspeis In cobbler.log at the end it sais: starting dhcprelay [ OK ]
07:19 saibarspeis Ok
07:20 saibarspeis Done it
07:20 saibarspeis It gave me a warning because cobbler.tar.xz does not exist. Skipping
07:20 saibarspeis Then Shutting down dhcprelay and starting up dhcprelay
07:21 saibarspeis Ran dhcpcheck discover --ifaces eth0
07:21 kaliya yes wait a moment and then retry dhcpcheck
07:21 saibarspeis Same ERROR (app)
07:21 saibarspeis Ahh
07:21 saibarspeis Ok I ran it right after
07:21 saibarspeis Whoa
07:21 saibarspeis Works
07:21 saibarspeis Let me try with a node
07:21 kaliya well, it's not immediate as the cobbler container comes up
07:22 saibarspeis THANK YOU
07:22 saibarspeis Geniouses :)
07:22 kaliya you're always welcome
07:22 saibarspeis Thank you very much, that was the thing
07:22 saibarspeis Ok, now I learned something new today :)
07:22 saibarspeis Thank you!
07:22 kaliya saibarspeis: feel free to come often to visit us here in #fuel
07:23 saibarspeis Thanks kaliya
07:46 stamak joined #fuel
07:57 vtzan joined #fuel
08:01 Longgeek joined #fuel
08:03 sc-rm kaliya: After having 80 instances running for 16 hours, I terminated them all and try to run the healt check, and then the same problem with “Create volume and attach it to instance
08:03 sc-rm Volume couldn`t be attached. Please refer to OpenStack logs for more details.”
08:03 sc-rm kaliya: but which log should I look into, because cinder-volume does not giva any errors, just warnings
08:16 vtzan joined #fuel
08:27 adanin joined #fuel
08:27 HeOS joined #fuel
08:44 Longgeek joined #fuel
08:45 syt1 joined #fuel
08:52 syt joined #fuel
09:04 akupko joined #fuel
09:05 akupko joined #fuel
09:07 kaliya sc-rm: sorry again, 5.1?
09:07 e0ne joined #fuel
09:07 sc-rm kaliya: 5.1
09:09 syt joined #fuel
09:09 sc-rm kaliya: How do I remove a controller node, so the remaning controller nodes do not try to sync with the removed one?
09:09 kaliya sc-rm: are you operating in HA or simple multinode?
09:10 sc-rm kaliya: In HA mde
09:10 kaliya if your environment allows, you go to the Fuel UI and delete node and deploy again
09:11 sc-rm kaliya: We did so, but the rabbitmq still tries to connect to the removed node, even after a reboot of the respective controller nodes
09:12 kaliya sc-rm: couldn't be, rabbit should connect to the vip
09:12 kaliya let me check
09:13 sc-rm We had 3 controller nodes with 192.168.0.4 192.168.0.5 and 192.168.0.6 pr node, which is on the br-mgmt interface
09:13 sc-rm then we removed the node with 192.168.0.5
09:13 sc-rm but the other two controllers tries to connect to 192.168.0.5
09:15 kaliya sc-rm: what is giving you rabbitmqctl cluster_status
09:17 sc-rm kaliya: now it only includes the node-28 and node-30 as expected. Could it that rabbitmq just needs to figure that node-29 has gone completely
09:17 kaliya yep, but services don't know that node-29 has gone, they connect to a vip
09:17 kaliya I'm looking if I eventually can find recent related bugs, cause sounds like one
09:18 sc-rm kaliya: To me it’s okay that removing a node has some timedelay before the env stabilizes, as long as it stabilizes
09:20 kaliya sc-rm: yep, but how long is now?
09:20 sc-rm kaliya: it was like between 5-10 minutes
09:20 kaliya ahh
09:54 adanin joined #fuel
10:07 Longgeek joined #fuel
10:33 harybahh joined #fuel
10:33 Longgeek joined #fuel
10:37 teran joined #fuel
10:44 merdoc anyone knows working centos image for cloud?
10:47 kaliya merdoc: don't the cloud images from centos work for you?
10:47 merdoc kaliya: dunno (%
10:49 merdoc kaliya: link 'live cd image' on their website tells 404 %(
10:49 kaliya ops
10:49 kaliya merdoc: better ask in their channel why that :)
10:52 merdoc kaliya: I can't. It's to complicated - #centos Cannot join channel (+r) - you need to be identified with services %(
10:52 merdoc I'd rather google for it
10:52 kaliya maybe a bug to centos might be filed
10:54 kaliya merdoc: these don't work? http://cloud.centos.org/centos/7/devel/
10:55 merdoc kaliya: hm. thx. I will test it
10:57 teran joined #fuel
11:01 teran_ joined #fuel
11:02 teran joined #fuel
11:33 sc-rm when deploying zabbix after an env have been deployed, then it looks like ona have to do a manual addition of all the nodes or how do one make zabbix discover all the nodes?
11:38 harybahh joined #fuel
11:45 Gue______ joined #fuel
11:49 e0ne_ joined #fuel
11:50 evg sc-rm: I haven't try tried it yet but as I've got configured zabbix agents should be installed on each nodes as well as entries in zabbix db
11:53 sc-rm If I can not even ping the openstack router, which log/services should I try to restart? Tried to reboot both node-controllers, but no luck there
11:59 anand_ts joined #fuel
12:01 anand_ts hi all, is it possible to change the settings after fuel master node is installed, for eg I want to change DHCP range, password of fuel node etc
12:05 Dr_Drache anand_ts; "fuel menu" on the fuel node
12:07 anand_ts Dr_Drache: From Fuel node command line?
12:07 Dr_Drache anand_ts yes
12:07 anand_ts Dr_Drache: Thanks
12:08 evg anand_ts: you can change password from ui since 5.1
12:09 anand_ts evg: using 5.1 now
12:09 Dr_Drache anand_ts, evg I didn't see that working, but he needs acccess to everything else
12:10 anand_ts Dr_Drache: Yes. just want to confirm everything before starting to install controller and compute node.
12:11 anand_ts Dr_Drache: If I deploy with 3 servers, one Fuel node, 1 controller + 1 compute setup, I can access horizon with controller IP right?
12:11 Dr_Drache anand_ts, yes
12:12 Dr_Drache as long as your external IPs are routable from your network
12:12 Dr_Drache fuel will tell you what the IP is
12:13 anand_ts Dr_Drache: I Created a dedicated network for Fuel administration and NAT Fuel IP to access that from my local machine
12:18 teran_ joined #fuel
12:19 pasquier-s joined #fuel
12:21 HeOS joined #fuel
12:27 Dr_Drache anand_ts, ahh.. I did that, but it was diffucult once my devs needed/wanted in
12:28 anand_ts Dr_Drache: oh
12:29 Dr_Drache anand_ts, because I had difficulty routing anyone else into that network.
12:29 Dr_Drache but, just thoughts, not trying to steer you.
12:29 anand_ts Dr_Drache: Yes, Thinking that too.
12:57 aarefiev_ joined #fuel
13:00 aarefiev__ joined #fuel
13:00 f13o_f13o joined #fuel
13:01 aarefiev_ joined #fuel
13:04 merdoc if I changed motherboard on my controllers/compute, what should I need to do for bringing it back to fuel? or they should work after mac changing?
13:13 kaliya merdoc: nodes are identified by the MAC. So if you change your NICs, nodes will be recognized as new nodes
13:13 stamak joined #fuel
13:13 sc-rm kaliya: merdoc: would it not be theoritcally possible just to replace the mac address in respective tables on the fuel-master?
13:14 syt joined #fuel
13:14 merdoc kaliya: is it possible change MACs in configs? I found it in cobbler, where it can be else?
13:16 kaliya sc-rm: would be, probably
13:16 sc-rm That could actually be a really nice feature to have in fuel-web, because sometimes we just replace a nic, but not the rest of the node
13:17 sc-rm or as merdoc just replace the mb
13:19 tdubyk joined #fuel
13:21 sc-rm kaliya: When I removed the 3rd controller node, it made the network break. How do I force the controllers to repair the network, I can not ping the router inside openstack
13:25 sc-rm kaliya: ip net show does not list an qrouter on any of the controllers, but is it not suppose to do so?
13:30 mattgriffin joined #fuel
13:30 azemlyanov joined #fuel
13:33 sc-rm kaliya: for some reason neutron-l3-agent was not running on both controller nodes even after reboot.
13:34 kaliya sc-rm: I don't remember how did you remove the controller?
13:35 sc-rm kaliya: the node was too slow and causing the rabbitmq to have problems, so we physically removed it to also see how openstack will handle a permanent lack removal in the HA setup
13:36 sc-rm kaliya: when it was removed the remaning controllers needed to be restartet to have the change take effect, but the network never came up.
13:37 kaliya sc-rm: thanks, I'm checking
13:38 sc-rm kaliya: then I restartet the controllers several times, but no recovering from the missing controller even after removal from the nova db. Network was first recovered after a manual service neutron-l3-agent stop; service neutron-l3-agent start
13:38 sc-rm kaliya: on both controllers
13:40 kaliya sc-rm: thanks, I'm checki
13:42 kaliya sc-rm: so your issue is in neutron or in the node's network?
13:42 jobewan joined #fuel
13:43 sc-rm kaliya: I’ll write it all down as a scenario on paste - just a second ;-)
13:43 kaliya thanks sc-rm
13:51 kaliya sc-rm: in the meanwhile, do you notice something relevant to haproxy in /var/log/messages ?
13:52 sc-rm kaliya: http://paste.openstack.org/show/121007/
13:53 sc-rm kaliya: if I do cat /var/log/messages | grep haproxy | grep error
13:53 sc-rm kaliya: I get nothing...
13:54 merdoc yay! i broke fuel.
13:56 Dr_Drache merdoc, again?
13:56 merdoc Dr_Drache: yes (%
13:56 merdoc it's because I changed motherboards on several nodes. and then try to reset env (%
13:57 Dr_Drache lol
13:59 merdoc I'm accidently Fuel (c) (%
13:59 kaliya merdoc: you're part of the QA team :)
14:00 kaliya seriously, we should have a blueprint https://blueprints.launchpad.net/fuel/+spec/rejoin-provisioned-nodes
14:00 merdoc kaliya: I so wanted to be developer. crap (((%
14:00 evg sc-rm: you've replaced the 3rd NIC. Are you sure your new NIC is exactly the 3rd (not 4th)?
14:01 sc-rm evg: I did not replace a nic ;-)
14:02 evg sc-rm: oh, i've misread, sorry
14:02 kaliya evg: it's merdoc, who changed the mb
14:02 merdoc kaliya: how can I delete fuel env from command line?
14:02 merdoc ya, it's me (%
14:03 kaliya merdoc: fuel --env <ID> env delete
14:03 MiroslavAnashkin fuel --env 1 env delete
14:04 merdoc HTTP Error 400: Bad Request (Environment removal already started)
14:04 merdoc it stuck
14:05 kaliya merdoc: `fuel nodes` ? error?
14:06 merdoc kaliya: no. http://paste.openstack.org/show/121013/
14:07 MiroslavAnashkin What does `fuel env` without additional parameters says?
14:07 merdoc and fuel task http://paste.openstack.org/show/121014/
14:07 merdoc 6  | remove | vit-o-cloud | multinode | 3          | []      | None
14:28 merdoc kaliya: MiroslavAnashkin: http://paste.openstack.org/show/121027/
14:31 mpetason joined #fuel
14:37 angdraug joined #fuel
14:37 dhblaz joined #fuel
14:42 merdoc kaliya: when releas of 5.1.1 scheduled?
14:46 AKirilochkin joined #fuel
14:48 HeOS joined #fuel
14:56 kaliya merdoc: don't know yet
15:01 dhblaz I’m about to add a bunch of OSDs to a 4.0 deployment; does anyone have any suggstions for if I should use a partition table and any arguments to pass to  mkfs?
15:01 kaliya dhblaz: it's reccomended you format them as XFS
15:02 dhblaz kaliya: thanks, I plan to
15:02 kaliya or BTRFS, but I wouldn't work in production with BTRFS :)
15:03 kaliya dhblaz: are you running Swift on them?
15:03 dhblaz I will be
15:04 AKirilochkin joined #fuel
15:06 dhblaz Based on this it looks like fdisk for partition then use mkfs.xfs -i size=1024 /dev/sd??
15:07 dhblaz But in production I see I have gpt partitions
15:07 dhblaz so I guess I will go that route
15:07 angdraug dhblaz: we use sgdisk to give them gpt labels that ceph automount expects
15:09 angdraug https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cobbler/templates/scripts/pmanager.py#L256
15:09 angdraug etc
15:15 dhblaz So the big steps are 1) disable journal on existing osd, chop up journal dev into new partitions, reenable jounral for existing osd
15:15 dhblaz 2) parted new osd
15:16 BillTheKat joined #fuel
15:16 dhblaz s/osd/device/
15:16 dhblaz 3) sgdisk new device
15:16 angdraug sounds right
15:17 blahRus joined #fuel
15:17 dhblaz 4) mkfs.xfs -i size=1024 new device
15:18 dhblaz then add the new osd to ceph
15:18 angdraug not sure size=1024 is necessary, depends on your disks
15:18 angdraug afair we added that for compatibility
15:18 AKirilochkin joined #fuel
15:19 dhblaz I don’t see how to make the partition_guid and effectively use it so the ceph automount uses it.
15:19 mattgriffin joined #fuel
15:21 Dr_Drache I thought fuel can just add new nodes?
15:23 angdraug dhblaz: https://github.com/ceph/ceph/blob/master/src/ceph-disk#L76
15:23 angdraug these are the guids you're looking for
15:23 angdraug https://github.com/ceph/ceph/blob/master/src/ceph-disk-udev
15:24 dhblaz Oh, how interesting thanks for pointing that out.  I hadn’t stumbled across that in the docs yet and I never would have guessed it a good idea to have several devices with the same guid
15:24 dhblaz something about the “gu” in guid through me off ;)
15:28 angdraug it's a type guid, not device guid
15:29 angdraug I guess you figured that already )
15:32 adanin joined #fuel
15:34 emagana joined #fuel
15:36 dhblaz It looks like /usr/sbin/ceph-disk already does a lot of this; is there some reason fuel does them manually?
15:48 angdraug last time I checked ceph-disk was too picky about zapping existing partitions
15:49 angdraug rmoe: is ^ still the reason we don't use ceph-disk?
15:58 rmoe yes, also the zap command in ceph-disk only does a subset of the zapping we do in pmanager
15:59 thehybridtech joined #fuel
16:04 emagana joined #fuel
16:06 HeOS joined #fuel
16:16 dhblaz joined #fuel
16:20 AKirilochkin_ joined #fuel
16:28 rmoe joined #fuel
16:37 Kupo24z1 joined #fuel
16:50 BillTheKat joined #fuel
16:51 jobewan joined #fuel
16:52 mutex joined #fuel
16:57 AKirilochkin joined #fuel
16:59 syt joined #fuel
17:11 sressot joined #fuel
17:18 dhblaz I’m not sure if it is still true but in fuel 4.0 the ceph journal refers to the block device by the drive “letter” and partition number.  (i.e. /dev/sda1)  But the ceph tools refer by uuid.  I feel that the ceph way is superior.
17:27 AKirilochkin joined #fuel
17:28 bookwar angdraug: btw, I've checked with Fedora people, and it seems I was wrong,  we actually never trigger automatic rebuilds. It is maintainer's responsibility to inform others who depend on him for all compatibility issues, and to watch for updates related to his package.
17:28 bookwar http://fedoraproject.org/wiki/Package_maintainer_responsibilities#Notify_others_of_changes_that_may_affect_their_packages
17:30 kupo24z Is there a way to comment out the public ip check for a workaround in this bug? https://bugs.launchpad.net/fuel/+bug/1376426
17:34 xarses joined #fuel
17:38 Dr_Drache bookwar, that seems backwards.
17:43 MiroslavAnashkin kupo24z: Yes. First, run `dockerctl shell nailgun` on master node.
17:43 MiroslavAnashkin Then `yum install vim`.
17:44 MiroslavAnashkin Then modify `/usr/lib/python2.6/site-packages/nailgun/task/task.py`
17:46 kupo24z MiroslavAnashkin: I've commented 692-694 out and i still get the error, do i need to delete my env?
17:46 MiroslavAnashkin Find `def _check_network(cls, task):` method (L684 or nearby)
17:47 kupo24z comment 683-694?
17:47 MiroslavAnashkin After the file modification - run `rm -rf /usr/lib/python2.6/site-packages/nailgun/*.pyc` and `rm -rf /usr/lib/python2.6/site-packages/nailgun/*.pyo`
17:48 MiroslavAnashkin Then exit Nailgun container and restart nailgun with `doclkerctl restart nailgun; dockerctl restart nginx`
17:54 e0ne joined #fuel
17:55 kupo24z MiroslavAnashkin: looks like that did it, thanks
17:58 BillTheKat joined #fuel
17:58 jpf joined #fuel
18:07 emagana joined #fuel
18:13 AKirilochkin joined #fuel
18:17 AKirilochkin joined #fuel
18:23 emagana joined #fuel
18:26 emagana joined #fuel
18:44 HeOS joined #fuel
18:48 syt joined #fuel
18:55 harybahh joined #fuel
19:34 e0ne joined #fuel
19:38 AKirilochkin joined #fuel
19:42 stamak joined #fuel
19:55 BillTheKat joined #fuel
20:09 hyperbaba joined #fuel
20:10 designated joined #fuel
20:11 designated How long does it take for the docker image to finish loading on a fuel 5.1 install?
20:11 designated it's been going for 30+ minutes so far.
20:12 Dr_Drache designated, that +
20:13 MiroslavAnashkin Up to one hour with HDD. You may notice, docker loads different images, there are 10+ images.
20:13 designated thank you.
20:14 Dr_Drache designated, with raid-0 SAS mine is about 45 min from first boot
20:14 designated I evaluated 5.0 and had nothing but problems, never could get a deployment working.  Have there been significant enough improvements in 5.1 that alleviate such concerns?
20:15 Dr_Drache well, in general, yes.
20:16 designated Dr_Drache, are there any show stoppers?  any outstanding issues that would prevent a production deployment using fuel 5.1?
20:17 MiroslavAnashkin 5.1 has improved HA mode and now assembles cluster after cold shutdowns almost everytime. So, it is more disaster-proof, especially if you install all the critical hot-fixes.
20:17 Dr_Drache qcow2 and ceph = bad mojo; there is a api patch for ceph to work "more correctly" but that's my only stoppers.
20:17 MiroslavAnashkin I'd recommend upcoming 5.1.1 for production.
20:17 Dr_Drache I'd have to agree.
20:18 designated When is 5.1.1 scheduled for release?  We have a ceph cluster that must be used.
20:18 Dr_Drache but i'm running production level testing on my tiny cluster.
20:18 MiroslavAnashkin For 5.1 you have to install 4-5 critical fixes before rolling it to production.
20:21 MiroslavAnashkin Currently 5.1.1 is scheduled somewhere in early December
20:22 designated unfortunately we can't wait until December.  if fuel in it's current state isn't recommended for production, I'll have to find another solution.
20:24 MiroslavAnashkin Vanilla Fuel 5.1 is not recommended for production. We are currently preparing patch with all the critical hot-fixes and hope to release this patch next week.
20:25 designated is there an easier way to patch fuel now or is it still a manual patching process?
20:27 MiroslavAnashkin It depends. I'd like to automate the whole patching process, but I am pretty sure, some of these patches require OpenStack restart and may not be safe if automated.
20:28 kupo24z You can try the community ISO
20:30 MiroslavAnashkin Yes, daily community ISO already have these fixes.
20:32 kupo24z almost, ceilometer not commited yet
20:33 kupo24z https://bugs.launchpad.net/fuel/+bug/1373569
20:57 e0ne joined #fuel
21:40 syt joined #fuel
21:50 emagana joined #fuel
22:04 emagana joined #fuel
22:43 jetole joined #fuel
22:44 jetole Hey guys. I have just finished deploying a multi-node fuel environment and I am having some issues with networking and was wondering if anyone could help me out?
22:47 xarses jetole: just go ahead and ask, and anyone that can will attempt to help
22:50 jetole Well I have set up a environment using "Neutron with VLAN segmentation". In horizon, I see one router. It shows status of down for the public subnet. It shows a status of up for the private subnet. I can't reach the a instance on either subnet. It seems to have the wrong subnet selected for private
22:51 xarses have you changed the secuity group to allow icmp to the instance?
22:51 xarses the net04_ext network doesn't have DHCP, its not intended to be consumed by instances directly
22:52 jetole I have not changed the security group however I ran arping from the fuel server
22:53 jetole In the case of net04_ext, what or how should it be used as?
22:53 xarses left #fuel
22:53 xarses joined #fuel
22:54 jetole arping should not be affected by firewalls afaik. ARP is below IP and a requisite for Ethernet networks to function
22:55 xarses it won't work from the fuel node since you are crossing the vlan boundry
22:55 jetole Well I have also tried from the edge firewalls which are a pair of Linux servers with vlan interfaces, etc
22:55 xarses the gateway route is maintained by the neutron router in the controller
22:56 jetole I just added a ICMP ingress allow from 0.0.0.0/0 but I ran arp-scan on the firewall on bond0.190 and bond0.230 which are supposed to represent the two subnets
22:57 jetole I have a feeling I may need to redeploy
22:58 xarses for net04_ext, you typically will consume floating address from the network. You could use it directly, but if you don't intend to use the neutron router, then you should reconfigure for provider networks
22:59 jetole xarses: general question, if I assign a vlan VID pool to all ports on the switch and set that pool for the neutron config, that will ensure that instances are isolated to that VLAN VID but outside of mirantis/fuel/openstack environment, I don't need to deploy those VID's to switch ports for hardware firewalls and routers or the firewalls and routers themselves. Is that right or did I misunderstand
22:59 xarses if you logged in as another tenant besides admin, you would not see the net04_ext network to allocate to the instance
22:59 jetole I am admin atm
23:01 jetole xarses: i.e. if I set the VID pool 1200-1800 then I need to set the switch ports to allow tagged packets on those ports only for fuel/openstack ports but I don't need to configure them on the non-openstack router. Right?
23:01 xarses jetole: for how fuel deploys Neutron VLAN, the VLAN range, only your computes, and controllers need access to the ranges the one of the controllers will maintain the software router to translate the requests from the vlan
23:01 jetole Ok. Good
23:02 xarses jetole: correct
23:02 xarses putting it on to the hardware routers is referred to as 'provider networks'
23:02 xarses Fuel doesn't set that up out of the box, but it could be reconfigured by hand
23:02 jetole I'll be back in 10, I was wondering, if I want to redeploy, can I do Q&A with you as I go on the network deployment to make sure I have selected the proper subnets and vlan's for what I intend to accomplish?
23:03 jetole no and I don't want to do that
23:03 jetole so that's fine
23:03 xarses jetole: that's fine, I'll be around for at least another hour
23:11 jetole xarses: Thank you
23:12 jetole I have a /24 public subnet that all client instances will want to be able to reach remotely. On the network settings page, I define this CIDR in the public CIDR block?
23:14 xarses public would become what ever network your users / customers would use to access their instances from locations remote of the cloud. The floating-ip network comes out of the CIDR for the public range as well
23:14 jetole OK
23:14 xarses for 5.1, you only need enough addresses in the public range for controllers + 1 (If HA mode)
23:14 jetole xarses: now the start/end range under public, what am I defining there?
23:15 xarses for older versions, or if you re-enable it from the settings page on 5.1, you need one public address for each node
23:15 xarses jetole: the last two responses
23:16 jetole xarses: wait. public net, btw, yes for 5.1, the public net is for remotely accessible IP addresses i.e. if I want ssh into a instance while sitting at starbucks without VPN or anything then that would be public. Right?
23:17 xarses it doesn't have to be, it's what ever network you want your users to be able to access the nodes from. If you are creating a public cloud, this would typically be real routable public ip addresses
23:17 jetole xarses: oh so the start/end is sum(controllers)+1 and the rest of the subnet is for remote access as defined by floating IP's?
23:17 jetole right
23:17 jetole ok
23:17 jetole and sum(controllers)+1 is the start/end range?
23:17 jetole that's what the start/end range is for?
23:18 xarses its the range available to assign to the nodes that need addresses on that network
23:18 xarses and can't overlap with the floating range
23:18 jetole what are nodes?
23:18 jetole do my ceph ods servers count as nodes?
23:18 jetole or my computer servers?
23:19 xarses the [likely] physical servers you are deploying openstack to
23:19 jetole ok
23:19 xarses in the case of 5.1, and neutron the only roles that need address on public are the controllers +1 (for the HA vip)
23:20 jetole I am using HA via 5.1. I have 3 controllers. 4 OSD and 4 compute. So I need 3+1 in that range?
23:20 jetole or 11+1?
23:20 xarses with neutron vlan or gre, 3+1
23:20 jetole neutron vlan
23:20 jetole ok
23:20 jetole ok
23:20 kupo24z xarses: Having a weird issue with no-public IP's on the compute nodes, the nova client cant talk to the controllers after getting the initial token since there is no route, even if i specifiy a private address to keystone
23:20 jetole one sec while I note this
23:21 xarses kupo24z: what network provider?
23:21 xarses and OS
23:21 kupo24z neutron
23:21 jetole now for neutron vlan with what I described, I do not set up my hardware firewall as the router for this public net?
23:21 kupo24z ubuntu
23:21 xarses HA?
23:21 kupo24z yeah
23:22 kupo24z xarses: http://pastebin.mozilla.org/6789322
23:22 kupo24z for some reason the curl is the public ip
23:22 xarses kupo24z: I'll give it a look, will probably have to get back to you tomorrow. You can try tracing the API call, it Sounds like it's trying to access the public endpoint, which it cant
23:23 xarses jetole: you will need to provide a router for the public network
23:23 xarses just not the private ones
23:23 jetole xarses: actually I set up my hardware firewall as the router and horizon was assigned to this subnet and I can access it offsite so it seems... oh ok
23:23 jetole OK
23:24 jetole xarses: the management vlan. I set this up as 10.2.8.0/24 on vlan250. I'm not 100% sure what this subnet is for. Do I need to provide hw routing for this subnet? I did not originally
23:26 jetole xarses: I believe management, storage and neutron L2 are set up correctly. They are private nets, vlan tagged, etc with no hw router set up
23:26 xarses jetole: correct
23:26 adanin joined #fuel
23:27 xarses management is used by the Openstack API's and the nodes in the cluster will use it to communicate with each other
23:27 jetole OK. Now I think I may have made a mistake with Neutron L3 regarding internal CIDR and gateway. I put down the subnet used for fuel PXE admin
23:27 jetole and the gateway is a hardware router
23:27 xarses in the case of GRE the traffic will also go over this network.
23:27 jetole no GRE. Neutron w/ VLAN
23:28 xarses just letting you know
23:28 jetole ok. Thank you
23:28 jetole xarses: for Neutron L3, I put down the subnet that fuel uses for PXE admin and the gateway of a hardware router. Is that incorrect?
23:28 xarses also the ceph client uses the network for initial data between the computes, monitors and OSD's
23:29 xarses for ceph replication the storage network is used
23:29 jetole understood
23:29 xarses the Neutron L3, its a private network so the values can be whatever you would like
23:29 xarses it will be used to create net04
23:30 jetole ok
23:30 jetole should the internal network gateway in L3 be a hw router or would this be an arbitrary openstack software router?
23:30 xarses its arbitrary, and neutron will create a soft router for it
23:31 xarses so you likely don't want it to collide with the other networks for logical sense, but the only place it cant collide is the floating network
23:32 xarses kupo24z: what is the request that is attempting this call?
23:32 jetole xarses: so I can pick a subnet out of thin air and any address I like and neither the subnet or the gateway need to correlate with anything outside of the fuel/openstack deployment for L# CIDR and gateway
23:33 xarses yep
23:33 jetole OK. now if I pick 10.20.30.0/24 for this subnet, how do I set up routing to it?
23:34 jetole on the firewall I would say something like ip route add 10.20.30.0/24 via x.x.x.x
23:34 xarses you dont, it will exist as a nat network. When you want to talk to an instance inside from outside it you attach a floating-ip
23:34 jetole xarses: by the way, thanks again for the help so far and I only have a couple more questions
23:35 kupo24z xarses: just nova --debug list
23:35 jetole xarses: I see
23:35 kupo24z i can get around it by doing --bypass-url
23:35 jetole so what size would I select for this network?
23:35 kupo24z but it seems like that shouldnt be needed
23:35 xarses kupo24z: from the compute node?
23:35 kupo24z Yeah
23:36 jetole xarses: do I pick size based on estimated number of instances or instances per project or does every instance get a copy of this net as it's own or how does sizing work?
23:36 xarses jetole: however large you would like it, it limits only the number of addresses on the network before you have to create another
23:36 jetole cool. So I start with /24 and another later?
23:36 xarses you attach the network to an instance. with neutron, you can have multiple networks per tenant/project
23:37 jetole if I need to?
23:37 jetole Ok
23:37 xarses whatever you are comfortable for a broadcast domain
23:38 jetole floating IP range. I can essentially pick the unused range from public i.e. if I picked 10.0.0.2-10.0.0.10 for public then I can select 10.0.0.11-10.0.0.254 for floating?
23:38 xarses jetole: correct
23:39 jetole OK
23:39 jetole :-)
23:39 xarses kupo24z: in openrc change *ENDPOINT_TYPE to 'internalURL'
23:40 jetole now for the router, router04, it shows the interface down that has a public IP on it now. This is pre-redeployment but should it be like that or should it be up?
23:41 kupo24z xarses: that fixed it thanks, that should probably be default
23:41 kupo24z as internalURL will always work
23:43 xarses jetole: it's likely to not be fully configured until the first instance is booted so I wouldn't worry about the state until then. Even then i've seen it report wrong
23:43 jetole OK. I'm running verify network now from fuel dash
23:44 xarses jetole: you would want to manually inspect the network namespace on whoever is the router for the network
23:44 jetole ok
23:44 xarses ip netns
23:45 xarses you will see qrouter-<UUID> on one of the nodes where uuid matches the routers UUID
23:45 jetole ok
23:45 jetole I'll write that down but I am redeploying now so I would need to run that after
23:45 xarses it will typically be on the lowest numbered node with the controller role
23:45 jetole Is there any reason not to use ceph for ephemeral of swift?
23:46 xarses ephemeral, there could be some latency, but I love it. For swift, its not 100% matching the swift API as radosgw follows the S3 api first
23:46 kupo24z xarses: looks like export NEUTRON_ENDPOINT_TYPE='internalURL' doesnt work as it should
23:46 kupo24z neutron client still uses the public url
23:47 xarses the copy on write for ephemeral is fantastic
23:47 jetole OK. I read something similar to that the other night regarding swift. It doesn't seem like a large concern at the moment
23:47 xarses kupo24z: you can try OS_ENDPOINT_TYPE
23:47 kupo24z that fixed it
23:47 kupo24z thx
23:48 jetole What is ephemeral? I'm mostly migrating from a VPS set up so some of the cloud terminology is new. I know what ephemeral means but not what ephemeral means in regards to openstack
23:48 xarses the main difference is that RadowsGW uses a global namespace, where swift has them scoped per tenant
23:48 xarses ephemeral would be any root disk not backed by a cinder volume (EBS)
23:49 xarses in that it is destroyed when you destroy (terminate) the instance
23:49 jetole which are ... I was just about to say that
23:49 jetole ok
23:49 jetole xarses: I want to step out of the office for a smoke while I wait on verify network to complete. Will you still be around in <10 minutes?
23:50 jetole scratch that
23:50 jetole it just failed
23:50 xarses you will also need to store your images in glance as raw (exploded, large) in glance to take advantage of copy on write
23:50 jetole only for untagged but on all nodes
23:50 xarses (for ceph)
23:51 xarses kupo24z: I'll poke some people about it. I don't think we expected copying the openrc to a compute node, although it is silly we set OS_AUTH_URL to internal, be ENDPOINT_TYPPE is publicURL
23:51 jetole for network verify, for untagged on every node, what exactly is it saying failed? Each node except one controller has 4 ports. One is set as a access port on the vlan for fuel pxe and the others are set to vlan tag all other options defined on the networking page, I believe
23:52 jetole xarses: untagged isn't a very specific error
23:52 xarses can you paste the text it returned?
23:52 jetole oh the other three ports that are tagged are LACP, forgot to mention
23:52 xarses jetole: did you bond them then?
23:53 jetole yes
23:53 jetole the text it returned is
23:53 jetole Node NameNode MAC addressNode InterfaceExpected VLAN (not received)
23:53 jetole hv-100:21:9b:9a:0e:77eth0untagged
23:53 jetole except one line there for every node
23:54 jetole but all of the vlan tagged ones passed. The vlan tagged ones failed a few times on the old set up and I reconfig'd the switch as I was troubleshooting.
23:55 jetole on hv-1 I see one nic with admin(pxe) and then I see 3 nic's as ovs-bond0, the option "lacp balance tcp" and the four groups on the bond of "private: vlan ids: 1200-1499", "Public: vlan 230", "Storage: vlan 240" and "Management: vlan 250"
23:56 jetole all nodes are configured consistently with that except one controller which only has two nic's so no bonding
23:58 xarses is eth0 your admin(pxe) nic?
23:58 jetole eth0 I believe is non-bonded... yes
23:59 jetole it says eth0 for every node
23:59 xarses odd that its reporting the error, since pxe is used to report the node status and run the tests
23:59 xarses so I'm guessing its a false error

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary