Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2016-07-26

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:37 johnavp19893 joined #fuel
00:47 code-R joined #fuel
00:48 code-R_ joined #fuel
00:59 fritchie joined #fuel
01:00 fritchie joined #fuel
01:06 code-R joined #fuel
01:10 aneliubin joined #fuel
01:21 Zer0Byte__ joined #fuel
01:34 aneliubin joined #fuel
02:22 code-R joined #fuel
02:33 Liuqing joined #fuel
02:52 Liuqing_ joined #fuel
05:14 code-R joined #fuel
05:58 Liuqing joined #fuel
05:58 Miouge joined #fuel
06:04 Liuqing joined #fuel
06:11 ub joined #fuel
06:11 javeriak joined #fuel
06:12 code-R joined #fuel
06:22 javeriak joined #fuel
07:07 javeriak joined #fuel
07:33 toMeloos joined #fuel
07:34 javeriak joined #fuel
07:35 code-R_ joined #fuel
07:38 Zer0Byte__ joined #fuel
07:50 code-R joined #fuel
08:00 yassine joined #fuel
08:39 vkulanov joined #fuel
09:02 javeriak joined #fuel
09:10 priya joined #fuel
09:11 priya joined #fuel
09:15 ekosareva joined #fuel
09:19 maniram477 joined #fuel
09:20 maniram477 Hi everyone
09:31 tatyana joined #fuel
09:33 DaveJ__ joined #fuel
09:37 ekosareva joined #fuel
09:38 sfilatov joined #fuel
09:38 sfilatov left #fuel
09:38 romcheg vkulanov: ping
09:39 romcheg Since all the present core reviewers voted I put the nomination of vkulanov for python-fuelclient-core group to power
09:39 romcheg Congratulations, well done!
09:48 xarses_ joined #fuel
09:48 DavidRama joined #fuel
09:49 xarses_ joined #fuel
09:50 xarses_ joined #fuel
10:02 yassine joined #fuel
10:40 maniram477 Is it possible to add new resources to pacemaker cluster  on openstack deployed by fuel 7.0?
10:50 mskalski maniram477: I would say that nothing prevent you from adding your custom configuration, just refer to pacemaker documentation
10:52 javeriak joined #fuel
10:57 maniram477 mskalski: Hi, I even tried adding resource like ocf:heartbeat:IPaddr2. But resource is always in stopped state even after crm resource start
11:34 vkulanov romcheg, thank you
12:17 Liuqing joined #fuel
12:37 javeriak_ joined #fuel
12:46 maniram477 joined #fuel
12:58 javeriak joined #fuel
13:00 severion joined #fuel
13:01 flerfb0rt1 joined #fuel
13:01 Sketch hmm, how does dnsmasq get launched on the controllers?  there is no obvious way to restart it
13:03 Sketch of course, the real problem is why fuel generated a broken configuration for it...
13:05 Sketch hmm, i think i was looking at the wrong dnsmasq config
13:05 Sketch i think the actual problem is that --interface=tap3953c4f1-cf doesn't exist
13:06 Sketch ah, unless it's a netns in...which it is
13:06 asvechnikov ikutukov, ikalnitsky, akasatkin core-review please https://review.openstack.org/#/c/341366/
13:08 Liuqing joined #fuel
13:13 Sketch ok, so it appears that there is one dnsmasq that runs for resolving dns, and 2 that run for dhcp.  the one that runs on the qdhcp netns for dhcp on admin_floating_net points to it's own IP for resolving DNS.  but the dnsmasq that resolves DNS doesn't listen on that IP.
13:13 Sketch if i understand this correctly
13:14 Liuqing joined #fuel
13:17 Sketch ok, i'm wrong, it does listen on that port.  it just doesn't answer queries for some reaosn.
13:18 Sketch s/port/IP/
13:35 tatyana joined #fuel
13:35 venkat joined #fuel
13:36 johnavp19891 joined #fuel
13:49 fritchie joined #fuel
14:02 code-R joined #fuel
14:04 DavidRama1 joined #fuel
14:11 fritchie joined #fuel
14:34 Liuqing joined #fuel
14:40 Sketch apparently it picks the routers as DNS resolvers if you don't specify them for the admin_floating_net.  so that at least half fixes the problem
14:41 Sketch i don't see a way to change the search path, though... but maybe setting the fqdn will fix that
14:43 xarses_ joined #fuel
14:47 Sketch hmm, nope. and that doesn't persist, it gets overwritten
14:48 Sketch is it better to disable dhcp and statically define an IP once an instance is up?
14:49 mskalski Sketch: by default admin_floating_net has no dhcp server enabled, it used for floating ips
14:53 aderyugin joined #fuel
14:53 HeOS joined #fuel
14:54 aderyugin joined #fuel
14:57 Sketch the instances get their IP via dhcp, though
14:57 Sketch and there is a dnsmasq server running which appears to serve dhcp...
14:59 aderyugin joined #fuel
15:00 mskalski Sketch: when you look at the process in ps aux do you see that --dhcp-range is for your public network?
15:01 Sketch yeah
15:02 akscram akasatkin: Could you pleave review and merge if possible this fix? https://review.openstack.org/#/c/341366/
15:09 Sketch i guess my problem may actually be that i shouldn't be using the default domain
15:09 Sketch assuming the openstack domains map to dns domains
15:11 Sketch i guess that's probably not true
15:12 ub joined #fuel
15:18 Sketch i guess my real problem is i don't know what i'm doing ;)
15:18 Sketch i was setting the FQDN for the instance name, but it appends .openstacklocal as the actual domain
15:19 Sketch there is no obvious way in horizon to set a dns name anywhere that i can find
15:19 Sketch s/dns/domain/
15:20 dguryanov2 asvechnikov, akislitsky, gkibardin, ikutukov, could you, please, re-review https://review.openstack.org/#/c/340976/ ?
15:24 aderyugin joined #fuel
15:25 code-R_ joined #fuel
15:29 ikutukov done
15:33 fatdragon joined #fuel
15:34 mskalski Sketch: https://ask.openstack.org/en/question/57611/how-to-run-vm-within-its-own-dns-domain/ dns_domain is actually now defined in neutron.conf file
15:36 Sketch i did come across that, but i would think that, although I am, most people would not be deploying uniform dns names across all instances, so it doesn't seem like that would be the "right" way to do it
15:44 mskalski Sketch: http://docs.openstack.org/mitaka/networking-guide/adv-config-dns.html
15:53 javeriak joined #fuel
15:58 fritchie joined #fuel
16:05 venkat joined #fuel
16:10 Sketch dns_domain did not affect the server's actual domain name
16:10 Sketch (as provided by dhcp)
16:17 tatyana joined #fuel
16:33 code-R joined #fuel
16:45 fritchie joined #fuel
16:53 fritchie joined #fuel
17:12 code-R joined #fuel
17:13 code-R_ joined #fuel
17:17 fritchie joined #fuel
17:19 Miouge joined #fuel
17:22 fatdragon joined #fuel
17:31 cr0wrx joined #fuel
17:33 cr0wrx anyone know if vlans need to be added to fuel master node for connectivity checks (and if so, would it be over the management/public network if separate from pxe in two NIC setup)?
17:33 cr0wrx I'm failing connectivity checks but switch and ports all look fine, links show as up, not sure why it's failing and also causing deploy to fail (it stops deploy half-way through stating network errors)
17:34 Sketch yeah, the fuel master has to be able to reach the same networks on the nodes
17:35 cr0wrx out of which nic? The pxe one or non-pxe?
17:37 Sketch the physical setup doesn't matter, it just has to be able to reach the same networks on the install nodes. (IE, you can have vlans on the fuel master talking to multiple physical MICs on the clients)
17:39 AlexAvadanii cr0wrx, Sketch: no, fuel master only requires the admin network; the nodes have to be able to communicate between them over the configured VLANs, but not with fuel master
17:39 Sketch oh, i thought the network tests failed when i tried it without having the vlans on the master
17:41 AlexAvadanii our current setups (in opnfv) for fuel master only have the admin network going to the POD, and a public one for internet access
17:42 cr0wrx hrm, which could it be :) Network dude here unavailable to verify what has access currently but will try. I know that the network connectivity failed, but I tried to deploy anyways thinking it was just a test issue - pretty sure all actual slave nodes are set up correctly. It started deploy, but then stopped halfway through complaining about network issues - not sure if they were actually network issues on slave nodes, or because the
17:42 cr0wrx test failed it just stopped deploy anyways
17:42 AlexAvadanii one minor thing I would add is that, afaik, netcheck uses promiscous mode for listening to test traffic, so if your eth drivers have wierd default for offloading modes (one of our boards had "rx-vlan-filter=on"), traffic might be dropped
17:43 cr0wrx so that's another thing that may or may not be related....When the nodes bootstrap, the PXE interface comes up, but the other interfaces are all down. They show this way both in fuel UI and if I ssh and check which interfaces are up/down
17:44 AlexAvadanii cr0wrx: make sure all nodes have the right network to interface assignments: if all the nodes are identical, you can select all and "configure interfaces" in fuel ui
17:44 cr0wrx I wasn't sure if it was a driver issue or not, but I wouldn't think so because it's a quad port card and PXE port comes up fine....
17:46 Miouge joined #fuel
17:47 cr0wrx AlexAvadanil: I'm pretty confident all the slave nodes have interfaces configured correctly both in fuel and on the switch. Triple checked. Each has 3 NICS (PXE, Storage, and then the other openstack networks)
17:48 mskalski cr0wrx: by default interfaces on bootstaped nodes are in down state, except the one in admin network
17:49 mskalski but during network check there are set up
17:49 cr0wrx and on switch everything seems ok. However in fuel UI and on each node if I SSH after bootstrap NICs 2 and 3 are down in OS (including admin network), but first is up (PXE only). All three show links as up on physical nodes, though
17:49 AlexAvadanii cr0wrx: I would test it manually, while in bootstrap, by creating vlan interfaces on a few nodes and pinging between them
17:50 AlexAvadanii as for the down/up state during bootstrap, like Michal said, it is misleading, imo, as netcheck brings them up, and so does the deploy procedure
17:50 AlexAvadanii sry, got to go
17:50 cr0wrx thanks. I'll poke around some more. So 1) interfaces may be down when bootstrapped (not a concern) and 2) Fuel master needs access to all vlans maybe (or at least admin network + pxe network)
17:51 mskalski fuel master only have access to admin(pxe) network
17:52 cr0wrx well pxe it has for sure. admin (openstack admin, vlan tagged for me) not sure, double checking
17:53 mskalski cr0wrx: what exact error you see in networking check, that expected VLAN was not received?
17:53 cr0wrx if it's just pxe needed, it had that. It installed OS no issue
17:53 cr0wrx yes, pretty much for all vlans and untagged (storage NICs are untagged)
17:53 mskalski just for clarification in fuel admin network is used for pxe
17:54 mskalski when you say openstack admin you mean management network?
17:54 cr0wrx er, yes that's correct
17:55 mskalski cr0wrx: what segmentation type do you use vlan or vxlan?
17:55 cr0wrx pxe network worked fine. All nodes were discovered fine no issue, and OS installed as well I believe (at least on controller before it broke during provisioning)
17:55 cr0wrx vlan
17:56 mskalski and when you do network check does it complain about not receiving vlan from this private network vlan range or also for different networks?
18:00 cr0wrx actually, re-ran and it's tossing "repo availability verification failed on controller01 in red, and then the table shows each node (expected vlan not received) and lists pretty much every vlan for every node
18:01 cr0wrx gonna look into that first issue, controller deployed with interfaces but not able to route externally, not sure what's going on
18:02 Sketch it has to be able to route externally to verify repo availability
18:03 Sketch i think you only need it on admin and/or public
18:03 cr0wrx yea gonna get that added, would that cause all the vlan checks to fail below as well? or unrelated
18:03 Sketch i don't think so, but not certain
18:04 mskalski cr0wrx: unrelated, you need to be sure that you assign private network to good intarface which is connected to port on switch which allow to trunk all defined vlans
18:05 cr0wrx yes, I have openstack management, private, and public all on separate vlans on one interface, all trunked to switch. Storage is on a separate interface on each node
18:06 cr0wrx although one thing I do wonder about - in the past with fuel 8.0 we couldn't re-assign interfaces after deploy with fuel - is it possible in fuel 9.0?
18:07 mskalski cr0wrx: but for private network you have defined range of vlans which will be used for tenants network, does this vlans are also trunked on switch?
18:07 cr0wrx If later I want to move storage on nodes to a different interface, without replacing entire node? If I recall fuel 9.0 has some better post-deploy stuff
18:07 cr0wrx mskalski: yes, it's a defined range of vlans, all trunked
18:07 Sketch fuel won't let you change interfaces on a node after deployment
18:07 Sketch you'd have to delete it and re-add it
18:08 cr0wrx bummer, was hoping it would in fuel 9 since it said it handled more post-deploy stuff without downtime.
18:09 Sketch it does allow more than it used to, but not that, unfortunately
18:09 cr0wrx I'd probably just swap interface manually on nodes that were already deployed rather than redeploy and hope for the best :D Right now it's a 1G network but will want it to be 10G, just didn't have the hardware initially for these nodes
18:10 cr0wrx and for now we won't have a lot of extra capacity or an HA setup for controllers and whatnot, so blowing away a node would be hard. Still in the early stages and PoC type build out I guess
18:12 mskalski if you are 100% sure that all nodes are wired exactly the same then select are nodes in fuel web ui and check interface configuration to be sure that network assignment  is the same on all nodes
18:13 mskalski *all nodes
18:15 mskalski cr0wrx: do you  use http_proxy in your env?
18:15 cr0wrx nope
18:16 mskalski cr0wrx: when you ssh to bootstrapped node are you able to ping external host through dns name?
18:17 cr0wrx I can't ping through IP, let alone DNS :) Working through that currently
18:17 cr0wrx I think our router needs adjustment for the public netblock setup in fuel
18:18 mskalski you will have to check connectivity to external network both through admin network and public network
18:18 cr0wrx btw, are any of you mirantis guys or familiar in what the difference is in MOS 9.0 (Ubuntu+UCA deploy) vs fuel community 9.0? My understanding is MOS 9.0 uses mirantis repos and packages over community, but if selecting UCA then that sounds like it doesn't use the mirantis packages
18:21 AlexAvadanii cr0wrx: most of the packages will come from UCA, but there is still 1-2 packages from Mirantis, afaik
18:21 mskalski uca use ubuntu cloud archive as a source of packages for openstack components
18:22 mwhahaha it still uses the mirantis haproxy, rabbitmq, ceph and openvswitch packages
18:22 mskalski but it is configurable
18:22 cr0wrx ok, just curious. Thanks
18:22 mskalski at least for haproxy, rabbit and ceph
18:23 AlexAvadanii mwhahaha: thank you, I have testing UCA on my todo list for some time, nice to  know what comes with it beforehand
18:23 mskalski you may choose to use upstream version of this package
18:23 mwhahaha so the mirantis packages are still in the list, we just put the UCA packages at a higher priority in apt. we had to pin the above mentioned packages due to deployment issues with the ubuntu versions of the packages
18:24 mwhahaha haproxy is because fuel requires a custom patch that isn't in upstream to support the haproxy conf.d/* configs
18:24 mwhahaha ceph because the ubuntu ceph is broken or somehting to that effect (also is older i believe)
18:24 mwhahaha we are also providing a newer version of rabbitmq and openvswitch so we need those versions as well
18:25 mskalski so why we allow to choose upstream ceph when you say it is broken?
18:26 mwhahaha just because it's a bad idea doesn't mean we should prevent someone from doing something
18:26 Miouge joined #fuel
18:26 mwhahaha so if the end user created their own packages or knows how to fix the upstream one then they should be allowed to configure it as such
18:26 mwhahaha i also don't remember the specific issue we ran into around the ubuntu ceph package
18:27 mwhahaha oh it's just really old
18:27 mwhahaha 0.79
18:27 mwhahaha MOS has 0.94
18:27 mwhahaha so the setup is not the same
18:27 mwhahaha also if you aren't deploying with ceph, there's no need to pin it
18:28 mskalski if you not deploying with then probably you don't care if it pinned or not ;)
18:28 mwhahaha or if you provide your own ceph repo, you wouldn't want to pin it
18:29 cr0wrx are other UCA packages alright outside of these as far as you are aware? The reason I ask is for some plugins (designate, etc) in the past I recall some having dependency issues when trying to set up because of the MOS packages
18:29 mwhahaha no idea we don't test it beyond some basic deployment tests
18:30 mwhahaha i'd assume it's ok but you'll end up with different bugs and beholden to canonical on updates for the openstack stuff
18:30 cr0wrx sure
18:32 cr0wrx decisions decisions. I assume that if choosing UCA it probably isn't covered under the mirantis enterprise support either since it starts falling outside of MOS packages and whatnot
18:33 mwhahaha yea UCA is not supported by mirantis support
18:33 mwhahaha as far as i know
18:34 cr0wrx seems odd it's included as option in MOS iso then since that is geared towards mirantis, but maybe that's just because it's in upstream fuel or something
18:35 mwhahaha it's upstream fuel
18:36 cr0wrx thanks. Helpful to know differences! Remind your boss again you earned another gold star
18:38 AlexAvadanii if bosses hang around here, I'd give mskalskiand and  mwhahaha at least one goldstar each ;)
18:38 Sketch so it turns out i had to change dhcp_domain in nova.conf rather than dns_domain in neutron.conf.  and spawn a new vm, as i guess those are cached somewhere.
18:38 cr0wrx yes mwhahaha gets one every time I come in here I think
18:45 fritchie joined #fuel
18:52 Miouge joined #fuel
18:55 fatdragon joined #fuel
19:13 Miouge joined #fuel
19:23 fatdragon joined #fuel
20:36 thiagolib joined #fuel
21:15 flerfb0rt1 joined #fuel
21:26 dburmistrov joined #fuel
21:26 izinovik_ joined #fuel
21:27 MiroslavAnashkin joined #fuel
21:28 akislitsky joined #fuel
21:28 kat_pimenova joined #fuel
21:28 mrasskazov joined #fuel
21:29 apalkina joined #fuel
21:30 aglarendil_ joined #fuel
21:30 nurla joined #fuel
21:31 vkramskikh joined #fuel
21:31 sbog joined #fuel
21:34 meow-nofer joined #fuel
21:35 MorAle joined #fuel
21:39 cr0wrx mskalski: I return, much later than planned, but still return! So the problem seems to be interfaces during connectivity check don't seem to come up? I have a bunch of vlans tied to enp7sf02 for example (enp7sf02.*). Astute logs show a bunch of errors "Spawning listener for enp7sf01 failed....Network for iface enp7s0f1 is down"
21:40 cr0wrx er, both enp7sf01 and enp7sf02, the two interfaces that are failing...they don't seem to be getting brought up for whatever reason....Later in astute logs there are more errors when 'ip link show enpfs0f1.40' is executed, for example, because that device does not exist...Probably related to earlier errors
21:47 mskalski cr0wrx: did you try maybe go to boostrap node and set link up on this interface: ip link set dev  up enp7sf02   can you check if after that: ethtool enp7sf02 shows that link is detected ?
21:50 cr0wrx yes, that works on a bootstrap node
21:50 cr0wrx link detected, interface is up
21:51 mskalski what is the nic model ?
21:52 cr0wrx bcm5719
21:53 cr0wrx but what's odd is the fuel management/pxe network is on enp7sf0, the first port on that quad-port card...and pxe works fine, it's just these others that fuel needs to bring up
21:54 mskalski can we then check if we have communication between two nodes if we setup on the same interfaces the same vlan and common subnet: https://wiki.ubuntu.com/vlan
22:00 fritchie joined #fuel
22:03 fritchie joined #fuel
22:05 fritchie joined #fuel
22:13 cr0wrx Nope, it's not working even though network guys swore it was accurate. Thanks got another batch of debugging to do.

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary