Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-06-03

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 themistymay joined #fuel
00:21 IlyaE joined #fuel
00:30 rmoe joined #fuel
01:00 jaypipes joined #fuel
01:12 designated joined #fuel
01:30 IlyaE joined #fuel
01:38 xarses joined #fuel
02:06 zhiyan joined #fuel
02:11 zhiyan hi, anybody here can gives a help on fuel 5.0 basic usage? I met an issue on ostf testing after my first successful deployment, all tests say "Keystone client is not available. Please, refer to OpenStack logs to fix this problem", and in ostf-stdout log I can see an exception "Authorization Failed: Service Unavailable (HTTP 503)". I'm using Neutron+GRE+CentOS deployment. Thanks.
04:10 rmoe joined #fuel
04:45 mutex joined #fuel
05:32 IlyaE joined #fuel
06:36 IlyaE joined #fuel
06:37 al_ex joined #fuel
06:44 e0ne joined #fuel
07:00 IlyaE joined #fuel
07:02 taj joined #fuel
07:04 artem_panchenko1 joined #fuel
07:05 Kbee joined #fuel
07:05 Kbee in Fuel UI, Is there a way to find out ip addresses of deployed nodes
07:14 ilbot3 joined #fuel
07:14 Topic for #fuel is now Fuel 4.1 for Openstack: http://fuel.mirantis.com/ | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
07:14 e0ne joined #fuel
07:20 tim|mint joined #fuel
07:20 Kbee in Fuel UI, Is there a way to find out ip addresses of deployed nodes ??
07:21 tim|mint hi all, has anyone actually gotten the 'make iso' to work in a fresh ubuntu 12.04 install? I keep running into problems :S
07:21 tim|mint I'd love to get a machine image in which I can actually build the stuff
07:21 tim|mint or a Vagrant script or what-you-will
08:03 e0ne joined #fuel
08:45 e0ne joined #fuel
09:03 alex_didenko joined #fuel
09:26 bogdando joined #fuel
09:44 aglarendil tim|mint: which Ubuntu do you have ? 12.04 or 12.04.4
09:45 tim|mint 12.04.4
09:46 tim|mint fresh install (in virtualbox)
09:46 tim|mint tried both desktop and server version
09:46 tim|mint the ubuntu bootstrap process is broken, which I was able to fix with some work, but no I'm getting RPC errors
09:46 holser What issue do you have when you do “make iso”
09:47 tim|mint just a sec, starting my vm again to get it :)
09:49 alex_didenko 12.04 is too old, http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso The following steps are required to build the Fuel ISO images on Ubuntu 12.10 or newer (excluding newest 14.04)
09:50 tim|mint ah
09:51 tim|mint tried it on 13.10 as well, but got the same errors
09:51 tim|mint so i reverted to 12.04 (thought it mentioned the LTS release, but apparantly I wasn't reading correctly)
09:53 tim|mint npm ERR! Error: EACCES, mkdir '/home/tim/tmp/npm-3503-D8IpLycE'
09:53 tim|mint [stacktrace]
09:53 tim|mint npm ERR! Please try running this command again as root/Administrator.
09:54 tim|mint unless you really want me to do sudo make iso, which is fine within a VM, but probably not really what you want to promote?
09:54 alex_didenko make iso does not require sudo
09:56 alex_didenko do you have /home/tim/tmp dir and is it writable for the user you're running make iso under?
09:57 tim|mint nope, it's root:root, but apparantly that's done by the Makefile... I'll chown it and see if that helps
09:58 tim|mint it's continuing now... let's see where we end up
09:59 mattymo yeah don't run sudo make iso. you need sudo to create chroots and so on, but not only for a few build tasks
10:02 tim|mint agreed, but apparantly the Makefile does at some point create the ~/tmp directory as root :)
10:02 tim|mint unless the ubuntu installer does that (didn't check after the install) but that would be weird
10:10 e0ne_ joined #fuel
10:13 tim|mint still running now
10:39 themistymay joined #fuel
11:09 tim|mint heh... no space left on device :)
11:12 mattymo yes you need a LOT of space to build ISO
11:12 mattymo approximately 15gb
11:13 tim|mint I think more... was using a 20GB virtual disk with only ubuntu installed on it
11:13 tim|mint but got lvm underneath, so no problem
11:38 e0ne joined #fuel
11:41 e0ne_ joined #fuel
11:43 artem_panchenko joined #fuel
11:52 tim|mint there were changes in some package in the fuel-repository, right? as in, not ubuntu default packages? are those changes published somewhere as well?
11:54 odyssey4me joined #fuel
11:56 odyssey4me I was thinking earlier today that Fuel is excellent for deployment tooling, but doesn't help much with the actual configuration of Openstack. In other words, it has no exposure to the settings in the Openstack configuration files to allow them to be changed - all it exposes is some basics, like which back-end to use for Glance, etc.
11:57 odyssey4me Would it not make sense to start exposing some settings - at least those commonly used, and ideally already instrumented for within the underlying puppet manifests?
11:57 odyssey4me If one were to want to make changes - what is the paradigm? Can we simply change config files on nodes - or do we have a way of using puppet to deploy config changes that's considered standard with a Fuel deployment?
12:00 odyssey4me (by the way - the topic needs to be updated)
12:04 mattymo odyssey4me, we wanted the tool to be easy enough that anyone could use
12:04 mattymo if you start putting hundreds of options in front a user, it gets confusing
12:05 mattymo if you care about some particular options, we can take suggestions (or patches)
12:06 odyssey4me mattymo - sure, I'm just thinking about a few things here... some examples may be
12:07 odyssey4me the ability to, through the GUI, add the extra repo's you'd like included when deploying a node
12:07 mattymo so the logic is you add to openstack.yaml and it creates a UI element in settings
12:07 mattymo wipe your nailgun db, upload new fixtures, resync db, create new env, etc..
12:08 mattymo then this file gets added to astute.yaml that gets distributed to nodes. it becomes a new variable available to puppet in $::fuel_settings
12:08 odyssey4me yeah, I noticed that - as an aside, are there instructions to setting up a dev environment with a live running Fuel server? I'm thinking about the need we have to have an environment where we can make UI changes, then test immediately on a set of nodes
12:09 mattymo it's a good recommendation
12:09 mattymo but in practice, we just take a patchset and build a new iso and run automated tests
12:09 mattymo it works for us
12:11 odyssey4me sure, but that doesn't really work when you're in the early stages of figuring out how everythiong fits together - the workflow is too slow
12:11 odyssey4me that workflow is great for the finalisation of a patch to be submitted
12:12 odyssey4me making an ISO on my laptop takes quite a bit of time, then of course I have to actually do a Fuel server deployment afterwards, which also takes ages
12:12 odyssey4me I'm looking for a way to quickly prepare an environment that I can do rapid local tests and observations with
12:15 odyssey4me hah, I knew I'd seen it in a bug/blueprint: https://bugs.launchpad.net/fuel/+bug/1279303
12:17 mattymo akislitsky, ping
12:18 mattymo he's assigned to that bug report
12:19 odyssey4me I see so, however I also see that xarses noted that he had some samples to share which would help use a running Fuel node but to pull patches in and run them.
12:29 odyssey4me mattymo - how plausible would it be to plug chef or another deployment tool behind the rest of the framework? I realise that there may be a lot of work involved, but once a little decoupling is done I would think it's quite achievable?
12:32 mattymo it's not ready yet, but it would be a new mcollective plugin that runs chef, and possibly 1 more to transport the files from fuel master
12:32 mattymo but I believe we have some interest in doing some orchestration tasks via salt or ansible in the future
12:55 odyssey4me one of those settings which may be useful to edit in fuel (perhaps introduce an 'advanced settings'?) would be the overcommit ratio's
12:55 odyssey4me also the default quotas
13:03 mattymo these are 2 good spots, yes
13:03 jaypipes joined #fuel
13:34 vogelc Has anyone seen or heard that iptables is causing issue's with deploying nodes?  If we disable iptables our dhcp requests get back to the guest, but so far that is the only way we can get it to work.
13:39 vogelc I should note this only happens on Fuel 5.0,  4.1 works perfectly.
13:43 akislitsky odyssey4me, unfortunately we have no fuse-sshfs support in containers on master node. so we copy files via mounted directories into containers. it should be fixed, but now we have such ugly workflow.
13:52 MiroslavAnashkin Have you had a chance to check what iptables rule is causing issue?
13:55 vogelc I'll be honest,  I havent had much exposure to iptables.  We tried some changes but nothing seemed to work.  Is there a way to shut them down one by one?
14:05 jobewan joined #fuel
14:15 MiroslavAnashkin Could you please run iptables-save on master node and share the output somewhere on http://paste.openstack.org/ or similar service?
14:18 BillTheKat joined #fuel
14:22 BillTheKat So I am trying to install fuel 5.0 -after initial install from cdrom on reboot it ask to hit a key to enter fuel setup. I have done this twice now an both times it did not see my key hit and continued without entering fuel setup. Am I missing something? Anyone else see this?
14:28 MiroslavAnashkin BillTheKat: Such behavior may happen with specific USB hardware. Does your keyboard work after master node completely installed?
14:29 BillTheKat yes - actually on console I see the hit hits after it continues..... Ugh......
14:30 MiroslavAnashkin You may also press the <TAB> key on the very first installation screen which says "Welcome to Fuel Installer!" and update the kernel option showmenu=no to showmenu=yes.
14:30 BillTheKat ok
14:30 BillTheKat will try that
14:34 BillTheKat so i will automatically enter me into fuel menu at the same point if showmenu=yes?
14:39 MiroslavAnashkin Yes
14:40 BillTheKat ok - thanks for the info - greatly appreciated.......
14:40 odyssey4me MiroslavAnashkin - I noticed earlier that the .img image (not the ISO) actually doesn't have the 'showmenu' in the kernel command line. Obviously adding it still works, but I was interested to see that it's missing.
14:42 MiroslavAnashkin Hmm - .img includes the very same ISO plus bootloader.
14:42 odyssey4me akislitsky - ugly it may be, but that method saves more time than waiting for the only available workflow I have now, which is to rebuild the ISO and redeploy from scratch
14:45 odyssey4me MiroslavAnashkin - when I noticed it, I was using the .img file from the Mirantis website, not one build from source
14:46 akislitsky odyssey4me, you are right. it saves time, but when we will have support of fuse-sshfs it will be easy to mount required files direct into container without passing through other directories
14:47 odyssey4me akislitsky - what would the workflow be if I'm trying to do the following:
14:47 odyssey4me 1) edit the .yaml file to add some new options into the Environment GUI
14:48 odyssey4me 2) Do whatever needs to be done in puppet in order to use those settings
14:48 odyssey4me eg: say I want to split the roles up a bit more - so I create the new roles I want, then I want to make use of them
14:50 odyssey4me I don't see that requiring much copying/pasting - surely I'd be able to do some editing in line to figure things out, maybe I'll need to kill database tables and rebuild them or something like that, but not a lot of copying, surely?
14:50 odyssey4me Once I've got it all figured out, then I'd go back to the git repo's and put together the actual code changes
14:54 akislitsky odyssey4me, if you have mounted git repo into container you can work directly in it and with your favorite development tools. it will more comfortable, isn't it?
14:56 vogelc I have pasted the iptables ouput to http://paste.openstack.org/show/82613/
14:59 odyssey4me akislitsky - sure, but for now I'm happy to make do... I'd just like to know what the workflow would be when doing the development on a working node, even if the workflow is messy. The stuff I'm interested in is:
14:59 IlyaE joined #fuel
14:59 odyssey4me 1) setup of the dev fuel master
15:00 odyssey4me 2) where to find things
15:00 odyssey4me 3) how to make things I edit go live on the dev fuel master
15:01 odyssey4me 4) how to debug using local logs and/or unit tests
15:02 odyssey4me MiroslavAnashkin - I think vogelc's msg was meant for you. ;)
15:22 tim|mint meh... No Match for argument curl-7.19.7-37.el6_4
15:25 tim|mint ah well, trying again tomorrow :)
15:25 tim|mint bye all and thanks for the tips!
15:32 akislitsky odyssey4me, thanks a lot for structured questions. we will include howto steps into documentation.
15:34 odyssey4me akislitsky - thanks, I'll be happy to test and feedback on the documentation
15:42 odyssey4me odd - my dhcp packets don't appear to be received by anything, although they were a day ago. I can see them leave the source (through tcpdump on the vm host) and see them arrive on the fuel master (also through tcpdump)
15:43 odyssey4me from there I expect they should be passed through to the cobbler container? how do I confirm whether it's receiving or processing anything?
15:46 mattymo odyssey4me, tcpdump from inside cobbler container as well
15:47 odyssey4me mattymo - trying to figure out how to actually get an interactive shell in the docker container... running attach does nothing
15:48 mattymo dockerctl shell cobbler
15:51 odyssey4me mattymo - great, thanks... now how would I have figured out that 'cobbler' was a suitable name?
15:53 odyssey4me no tcpdump there, bother
15:54 odyssey4me haha, nvm - used yum to get it
15:55 odyssey4me ah, so it's definitely not getting through to the container
15:58 odyssey4me I see that the Fuel master is missing the port forward to the docker container.
16:00 odyssey4me It doesn't appear to even be in the container's definition!
16:01 alex_didenko what port?
16:03 e0ne joined #fuel
16:03 alex_didenko we forward dhcp packets via 'dhcrelay' which is running on the host system
16:04 alex_didenko dockerctl takes care of 'dhcrelay' during cobbler docker container start/restart
16:05 alex_didenko udp        0      0 0.0.0.0:67                  0.0.0.0:*                               17213/dhcrelay
16:05 rmoe joined #fuel
16:07 mattymo odyssey4me, dockerctl is still really a work in progess. I am adding more features to it, like listing container names, allowing dockerctl to work with arbitrary names instead of the fixed ones I set up
16:08 odyssey4me mattymo - ah, so it's a convenience script. maybe a 'list' command would be useful, although I see that the 'check' command provides some reasonable guesses
16:09 alex_didenko mattymo, how about "wipe" command? :) to do "docker rm" and reset container to the original state (from the image)
16:10 mattymo I would rather call it reset
16:10 mattymo but reset sounds like reboot
16:10 mattymo maybe revert?
16:11 odyssey4me alex_didenko - that seems to be in, nothing listening on that port on the host
16:11 odyssey4me mattymo - revert sounds like it'll revert to last snapshot, which if there is none is a revert to base image - sounds good to me
16:11 mattymo yes, that's the idea
16:12 alex_didenko we can elaborate in usage output :)
16:17 odyssey4me interesting - dhcrelay was actually running, but not listening on port 67
16:17 odyssey4me restarting the cotainer with dockerctl didn't work
16:17 odyssey4me I had to restart dhcrelay itself
16:22 mattymo yes always good to be clear
16:22 mattymo odyssey4me, anything interesting in /var/log/messages?
16:25 odyssey4me mattymo: Jun  3 11:16:48 fuel dhcrelay: Error getting hardware address for "docker0": No such device
16:25 odyssey4me it would appear that when I tried restarting the host earlier, the container wasn't up in time
16:27 mattymo odyssey4me, that should be okay. dhcrelay starts before docker
16:27 mattymo then when cobbler container starts, it will relaunch dhcrelay
16:29 odyssey4me mattymo - in this case, it didn't seem to happen
16:30 IlyaE joined #fuel
16:30 odyssey4me I've got to run - chat tomorrow.
16:31 odyssey4me Thanks for all the assistance. I'm slowly learning my way through it. :)
16:36 xarses joined #fuel
16:48 angdraug joined #fuel
17:23 Kupo24z1 joined #fuel
17:54 BillTheKat joined #fuel
17:55 Kupo24z1 xarses: you around?
17:58 BillTheKat I am trying fuel 5.0 - I have it up and set up network/pxe setting etc much like I did with fuel 4.1 - however the fuel server is not responding to dhcp pxe boot requests - any idea where I can  look to see why?  Also, I notice there is a docker0 in the fuelmenu - I set up PXE on the eth0 interface addresses? Should I have used the docker0?
17:58 angdraug Kupo24z1: he's around, but away from his desk
18:03 BillTheKat I also notice there is no dnsmasq running on fuel 5.0 like there is on fuel 4.1....  This is probably the problem - but I am a bit confused about setting it up
18:05 xarses Kupo24z1: wha'ts up?
18:06 xarses BillTheKat: there should be a neutron-dhcp-agent running, dnsmasq will only be running if there is a instance running on a network configured for DHCP
18:06 Kupo24z1 xarses: with 5.0 is the MongoDB used in the HA cluster or is it useless to setup 3 Nodes with it?
18:07 xarses Kupo24z1: it's supposed to work with 1 or 3 nodes, I cant remember if there is a constraint active in HA mode or not.
18:08 xarses but there is all kinds of logic in there to set up replicas and the like. Honestly I haven't tried it myself. but as far as I understand it works correctly with three nodes
18:08 xarses and is HA
18:08 BillTheKat xarses:  I am referring to the fuel server itself - I have no enviroments defined yet - I am trying to discover nodes on the fuel server and it is not reponding to dhcp requests
18:09 xarses BillTheKat: Oh, we are using docker on the master node, dnsmasq runs in a container
18:09 xarses I think with cobbler, but I'm not sure
18:10 BillTheKat that is what I figured - so in the fuel menu I need to set PXE addresses using the docker0 address as defined in fuelmenu
18:11 BillTheKat FYI - the user guide does not explain this  - it has what looks like fuel 4.1 fuelmenu example
18:11 vogelc BillTheKat:  I am having the exact same problem.
18:11 Kupo24z1 xarses: okay cool, so recommended to run on the same nodes as the 3x controllers?
18:13 MiroslavAnashkin BillTheKat: vogelc: Please try `service dhcrelay restart' on master node and then check its parameters with `ps aux | grep dhcrelay`
18:16 BillTheKat MiroslavAnashkin: dhcrelay running on 172.17.0.8
18:17 MiroslavAnashkin Yes, its command line should be like `/usr/sbin/dhcrelay -i eth0 -i docker0 172.17.0.3`
18:17 vogelc `/usr/sbin/dhcrelay -i eth0 -i docker0 172.17.0.3`
18:18 xarses left #fuel
18:18 BillTheKat MiroslavAnashkin: Yes - but it is .8 not .3
18:18 xarses joined #fuel
18:18 vogelc BillTheKat:  What kind of network switches are you using?  We have found if we turn off iptables on the fuel master that dhcp does work.
18:18 BillTheKat vogelc: arista
18:19 BillTheKat vogelc: this worked for fuel 4.1 - just not on 5.0
18:19 vogelc BillTheKat: Cisco here.  Yeah I can load 4.1 one and deploy clouds all day.
18:21 xarses Kupo24z1: it would depend on your load, and server layout. the disk layout for the mongo role will at least attempt to consume one whole disk. beyond that ceilometer can get quite noisy. For a production cluster I'd never co-habitat them
18:23 MiroslavAnashkin vogelc: BillTheKat: Please share the output from `ip r` command on master node
18:24 BillTheKat joined #fuel
18:25 BillTheKat 10.20.1.0/24 dev eth0  proto kernel  scope link  src 10.20.1.4
18:25 BillTheKat 169.254.0.0/16 dev eth0  scope link  metric 1002
18:25 BillTheKat 172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1
18:25 BillTheKat default via 10.20.1.1 dev eth0
18:25 vogelc '10.209.56.0/24 dev eth0  proto kernel  scope link  src 10.209.56.1'
18:25 vogelc '169.254.0.0/16 dev eth0  scope link  metric 1002'
18:25 vogelc '169.254.0.0/16 dev docker0  scope link  metric 1004'
18:25 vogelc '172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1'
18:26 vogelc 'default via 10.209.56.254 dev eth0'
18:28 vogelc gotta step away for a few minutes.
18:34 BillTheKat xarses: I provide the ip r output in case you missed it
18:44 xarses MiroslavAnashkin: wanted it
18:44 xarses something about dhcrelay and dnsmasq I see
18:45 BillTheKat yes
18:46 BillTheKat but I do not know who to set this up - the docs do not explain this new docker0 relay etc.
18:46 MiroslavAnashkin yes. I see non-default master node addresses in both cases and search where we may have forgotten to update some settings
18:46 BillTheKat Not sure if I need to rebuild from scratch.....
18:48 BillTheKat MirslavAnashki: So if my eth0 is 10.20.1.0/24 how do I set the setting in fuel menu?  I can not use the default 10.20.0.0/24 becuase I already have my fuel 4.1 on that subnet
18:49 BillTheKat MirslavAnashki: I am a tad confused about the docker0 and eth0 when it comes to setting up PXE
18:51 MiroslavAnashkin docker0 is just the bridge to the intercontainer's internal network
18:54 BillTheKat MiroslavAnashkin: ok - I am going to try to rebuild from scratch and use defaults to see if this works - the only change is I am going to set eth0 to 10.20.1.0/24 as opposed to 10.20.0.0/24
18:54 BillTheKat MiroslavAnashkin: Then that should work - I hope?
18:55 MiroslavAnashkin yes, please check if possible. And I am going otherwise, rebuild with non-default settings
18:56 designated Has anyone successfully deployed a HA environment with neutron+vlans on fuel 5.0?
18:58 xarses designated: with ceph, all the time was what I used during the whole dev cycle
18:58 MiroslavAnashkin designated: Our automated tests usually deploy this config successfully:-)
18:59 BillTheKat MiroslavAnashkin: Also I need to start on 0.4 becuase .1 is gateway and .2 and .3 is arista ports - do I need to change anything in docker0 becuase I am starting on.4
19:01 BillTheK_ joined #fuel
19:01 designated I tried fuel 5.0 the day it came out, it would get past OS load, and fail during openstack install.  I posted a tech support snapshot but no one ever provided a solution to the problem.
19:02 odyssey4me joined #fuel
19:03 BillTheK_ joined #fuel
19:03 e0ne joined #fuel
19:04 MiroslavAnashkin BillTheKat: It is exactly what I am searching now - the lost default settings. So far I don't know.
19:07 BillTheK_ ok - should I wait to hear back from you?
19:09 BillTheK_ MiroslavAnashkin: should I wait to hear back from you
19:09 vogelc BillTheKat:  I have rebuilt our environement probably 10 times.  Before you rebuild,  can you try turning off iptables on the fuel master?
19:09 IlyaE joined #fuel
19:10 MiroslavAnashkin Yes, I believe I'll get new master node in ~30 minutes
19:10 vogelc BillTheKat:  If you turn off iptables,  DHCP should work but tftp will not.
19:10 BillTheK_ vogelc: exactly how do  I turn off iptables
19:11 vogelc service iptables stop
19:11 BillTheK_ MiroslavAnashkin: OK I will wait in the mean time I will stop iptables and vogelc suiggests
19:12 odyssey4me BillTheKat: On the initial boot menu, change the IP details there already on the kernel command line, then add 'showmenu=yes' at the end to allow yourself to verify. Generally I've found if you add the correct IP's on the kernel load line, there's no need to edit them later and everything comes up correctly.
19:13 odyssey4me vogelc, check if dhcrelay is actually listening on port 67 on your fuel master?
19:14 vogelc dhcrelay is listening and I can see the requests make it to cobbler.  Cobbler responds, but the dhcp packet never makes it back to the client.  it does if I turn off iptables though on the fuel master.
19:15 MiroslavAnashkin Oh, great. Our master node PXE configuration section in the docs is completely outdated.
19:16 BillTheK_ MiroslavAnashkin: correct :)   I need help :)
19:19 odyssey4me vogelc - ah, different to the issue I was having earlier then, that's odd. I wonder what puts the iptables rules in place - docker, or something else. If it's docker, then maybe a restart of the container will help?
19:19 MiroslavAnashkin OK, BillTheK_ and vogelc, how did you managed to change your master node IP address if there is no such description in the docs?
19:21 BillTheK_ MiroslavAnashkin: I built from cd and set startmenu=yes and used the fuel menu to change addresses and PXE settings
19:22 MiroslavAnashkin There is no more such feature in fuelmenu
19:23 BillTheK_ MiroslavAnashkin: yes there is under the network section - not fuel command but the "fuelmenu" command
19:26 BillTheK_ vogelc: tried turning off iptables - same result.....
19:30 vogelc BillTheK:  we got it working
19:30 BillTheK_ vogelc: and what did you do
19:30 vogelc BillTheK: there are two types of dhcp relays on the network switches.  broadcast and unicast.  We turned off broadcast relay for dhcp and it started working
19:31 vogelc BillTheK:  turns out the hosts dont like getting a broadcast relay from the switch and another broadcast relay from the dhcrelay on the fuel master
19:32 odyssey4me Is it possible to remove a node entirely from the list if it's offline?
19:34 xarses odyssey4me: I think you should be able to curl -X DELETE http://fuel-url/api/v1/nodes/<number> (syntax from memory)  where number is the id of the node in the first column of 'fuel node' output
19:36 odyssey4me xarses - good memory :) that really should be in the cli and/or gui
19:38 xarses odyssey4me: it would be awesome if you can create a bug https://bugs.launchpad.net/fuel/+filebug for that =)
19:39 e0ne joined #fuel
19:39 odyssey4me zarses - yup, will do - doing them as I go along
19:40 MiroslavAnashkin That's interesting, but the correct way to set master node PXE IP address is via kernel parameters, in the same place where you set showmenu=yes...
19:42 Kupo24z1 Is VNC console currently broken with HA?
19:43 BillTheK_ MiroslavAnashkin: the documentation says to change the setting using fuel menu like I am - specifically I need to change the network addresses and PXE ranges
19:43 MiroslavAnashkin It is the greatest misguide in the docs
19:44 odyssey4me xarses - done: https://bugs.launchpad.net/fuel/+bug/1326116
19:45 xarses Kupo24z1: yes, if you refresh it up to 3 times, it will work
19:45 BillTheK_ Then what should I do to set this up - I have a /24 I want to use and maximise the nodes I can fit on a /24 and I want to change the subnet and ensure the PXE rnages start on .5
19:45 xarses Kupo24z1: its a problem accessing the token correctly
19:46 xarses Kupo24z1: there is a bug somewhere about it
19:47 BillTheK_ vogelc: according to my network person dhcp relay is not on in my switches - they are arista switches
19:48 odyssey4me xarses - that's an old bug, generally resolved by sharing memcache between all nodes running nova-consoleauth as I recall
19:49 BillTheK_ MiroslavAnashkin: Any suggestions on how I can proceed..... This looks like a common problem?
19:49 xarses odyssey4me: yep, I'm not sure why it surfaced all the sudden
19:49 odyssey4me alternatively, disable the nova-consoleauth service (through the nova-manage cli) on the nodes that don't have the virtual ip on it
19:51 odyssey4me BillTheK_ - set the fuel master's address/mask/gateway/etc in the kernel command-line when booting the ISO, then set the discovery/static addresses when you setup your environment
19:52 MiroslavAnashkin BillTheK_: To set up custom master node address you have to change related default values in the kernel options on the Fuel ISO Welcome screen. And set showmenu=yes in the same place to be able to configure the remained settings.
19:52 Kupo24z1 I'm trying to setup the Ceph GRE network on 9000MTU, what interface do I change in /etc/sysconfig/network-scripts on each node?
19:52 Kupo24z1 im guessing ifcfg-br-storage but im not positive
19:53 MiroslavAnashkin All PXE-related settings in fuemenu in 5.0 defaults to docker0 bridge - it is correct.
19:54 BillTheK_ ok - I will node IP to .4 at boot time  and set startmenu=yes anf see what things look like. Hopefully if I set the node IP to .4 it will set the PXE range to start at .5?
19:58 pbrooko joined #fuel
19:59 MiroslavAnashkin No, you still have to set PXE range start in fuelmenu - by default is starts from .3
20:02 BillTheK_ MiroslavAnashkin: ok - do I need change anything on docker0 in fuelmenu?
20:02 BillTheK_ MiroslavAnashkin: Can I assume the you guys know about the documentation issues in use guide?
20:04 BillTheK_ MiroslavAnashkin: I am rebuilding now - will let you know how it goes - doing it via virtual cdrom from home - so it takes a few hours - will get back to you
20:05 MiroslavAnashkin No, you may leave everything related to docker0 as is.
20:11 BillTheKat joined #fuel
20:11 MiroslavAnashkin You have to tune PXE range only, in the PXE settings section
20:12 BillTheKat MiroslavAnashkin: ok
20:16 Kupo24z1 Hey all quick question, on a standard ceph RBD pool with replication of 2 using all SSD's is it worth it to assign a single SSD on a storage node as a journal or just not have any journals as they are all the same drive? These nodes have 3 OSD's and 1 OS disk/Journal
20:20 odyssey4me Kupo24z1 - as far as I've researched, using an ssd to journal an ssd is a waste. The only way it would not be is if the journaling ssd has higher throughput/IOPs than the ones acting as OSD's
20:20 Kupo24z1 odyssey4me: thats what I figured too, doesnt fuel/ceph create journals on each OSD if you dont create a journal in Fuel before deployment?
20:21 odyssey4me Kup24z1 - as far as I see, the default is not to journal unless you configure the nodes to use journal space
20:23 odyssey4me In native Ceph, the journal is zero-sized by default as I recall. I see you're in #ceph on OFTC - they'll give you better advice on how best to configure Ceph and the hardware underneath.
20:23 odyssey4me There are some papers publiched by Inktank on the matter which are pretty useful.
20:29 odyssey4me MiroslavAnashkin - you know, the menu to change the settings should actually be reduced for v5.0 onwards... to take away the ability to change the settings that should not be editable
20:52 IlyaE joined #fuel
21:11 Kupo24z1 xarses MiroslavAnashkin do you guys know of any current 5.0 bugs with Swift w/ Ceph? getting 'Error: Unable to retrieve container list.'
21:12 xarses Kupo24z1: not that I've seen, you used ceph for object?
21:14 xarses Kupo24z1: using a separate journal device on top of a ssd will only serve to create a bottleneck. Ceph will still create a journal file (unless you use btrfs) on the device and use that
21:15 Kupo24z1 xarses: yes, using ceph for swift/object and getting that error.
21:16 xarses Kupo24z1: check that radosgw (ceph-radosgw on ubunut) is running on the controllers
21:16 Kupo24z1 xarses: so just create each node with no journals then, is 'Virtual Storage' still needed?
21:17 xarses if you are using ephemeral ceph, and this is a compute node, then no virtual storage is not needed, the disk allocator will still force at least 2gb to be assigned to it.
21:18 Kupo24z1 Looks like its stopped, tried starting and then status, not running
21:19 Kupo24z1 and /var/log/ceph/radosgw.log is empty
21:29 xarses Kupo24z1: can you paste the /etc/ceph/ceph.conf file?
21:29 xarses and I assume that 'ceph -s' returns 'HEALTH_OK'?
21:31 Kupo24z1 Yes, health is OK
21:32 Kupo24z1 xarses: http://pastebin.mozilla.org/5334922
21:32 tim|mint joined #fuel
21:42 xarses /etc/ceph/keyring.radosgw.gateway exists and is accessable by apache?
21:43 xarses and rgw_keystone_url is the address of your public_vip?
21:45 Kupo24z1 -rw-r----- 1 apache apache 74 Jun  3 20:42 /etc/ceph//keyring.radosgw.gateway
21:45 Kupo24z1 xarses: yes that ip is the address of horizon
21:50 xarses check that host, and 'hostname ' (command) and host in the config are exactly the same
21:51 xarses sorry, typed that twice, hostname command, and the value of host in [client.radosgw.gateway]
21:51 Kupo24z1 change rdw_dns_name to our correct domain as well?
21:51 * Kupo24z1 thinks vim should be installed on fuel deployed nodes in 5.1
21:52 xarses not as important, but it's best if it matches whatever the domainname the box is
21:52 xarses Kupo24z1: https://bugs.launchpad.net/fuel/+bug/1325804  << comment on that
21:52 Kupo24z1 sweet it runs now
21:53 Kupo24z1 thanks!
21:53 xarses what was wrong?
21:53 Kupo24z1 changed domain.tld to our correct domain
21:53 Kupo24z1 then restarted
21:53 xarses hmm
21:54 xarses did you pass fuel the same domain name?
21:54 xarses (that you needed to set it to)
21:54 blahRus joined #fuel
21:54 xarses and it just left domain.tld?
21:55 Kupo24z1 I never saw a domain configuration option in fuel
21:55 Kupo24z1 all our nodes are untitled
21:56 xarses I'm trying to figure out how to stop this from occurring again. I know it happens on occasion
21:56 Kupo24z1 I can verify it has happened in the last few installs of 5.0 ive done
21:57 xarses It would have been in kernel options, or fuelmenu (domain)
21:58 BillTheKat joined #fuel
21:59 xarses if you can upload a support bundle to a bug, it would be helpful. I'd like to get rid of the issue.
21:59 BillTheKat MiroslavAnashkin: OK I did wha you said - it worked - discovered 4 nodes....  Man the documentation needs to be updated :)   THANK-YOU!!!!!!
22:00 xarses I can make the bug private if you desire as well if you want to control access to the support bundle.
22:00 xarses BillTheKat: MiroslavAnashkin Please create a bug for that =))
22:02 xarses BillTheKat: is this similar to your issue https://bugs.launchpad.net/fuel/+bug/1324152?
22:02 designated as patches get commited, does the ISO get rebuilt or do those patches have to be manually applied?
22:03 xarses designated: They will not be applied to the released iso, they will have to be patched in.
22:03 Kupo24z1 Do patches get downloaded with the deployement process or will they have to be manually added?
22:04 designated i thought fuel 5.0 was supposed to have some automated upgrade process?
22:07 xarses Kupo24z1: designated: Fuel 5.0 can be upgraded, to say 5.0.1 or 5.1, but we don't handle the incremental patches at this time
22:07 IlyaE joined #fuel
22:08 BillTheKat xarses: maybe - I did seem to have issues with dhcrelay - but  MiroslavAnashkin observed that I was making changes to eth0 ip addresses only in fuel menu at build/boot time. I since changed the eth0 subnet in initial boot string and set startmenu=yes to allow me to update PXE range during initial boot time as well as some other paramters etc. This seemed to work. I have not interrogated logs yet - but on my screen I saw dhcre
22:08 designated xarses, i tried fuel 5.0 the day it was released and ran into that issue where i couldn't even add nodes, every once in a while it would fail to add a node and hang gui.  submitted a technical snapshot and never got a complete response on how to solve the problem.  i was able to work around the issue using the cli but once os load completed, hit another issue trying to install openstack.
22:13 Kupo24z1 designated: I had to clear browser cache and temp files otherwise the 5.0 GUI bugged out on me
22:16 designated Kupo24z1, it was actually complaining about adding a node, something to do with eth0.  I was talking to someone about it, submitted the snapshot he asked for and he left me hanging.
22:17 designated it was random and all of the nodes have the exact same hardware.
22:24 IlyaE joined #fuel
22:26 mpetason joined #fuel
22:28 xarses designated: this error came up when the nodes discovered and registered them selves ? or when you added discovered nodes into a cluster ?
22:43 xarses designated: I'm sorry to hear you had a bad experience with someone however, this is a community support channel. While some of us are Mirantis employees (like my self) we are only able to help as we are able and sometimes things fall off. The best way to ensure that you get support with the community side is to file a bug at https://bugs.launchpad.net/fuel/+filebug and attach / link to a support snapshot. If you are looking for SLA based support then you s
22:43 xarses hould be engaging with our paid support team. That said, I'd like to see you not have an issues and would like to help get this resolved.
22:45 Kupo24z1 xarses: do you know off the top what interfaces i need to change on the nodes to switch to 9000 MTU for Ceph?
22:45 xarses you can do a manual deployment and modify the mtu in the serialized yaml
22:45 Kupo24z1 im guessing br-storage but unaware of any others i should change, I've already enabled jumbo frames on the switch
22:46 Kupo24z1 anything i can do post-install?
22:46 xarses ubuntu/centos?
22:46 Kupo24z1 CentOS
22:49 xarses I think you can add MTU=9000 in the ifcfg.br-storage file and restart the interface. Ceph Replication will use the storage network, Compute to Ceph will use management network IIRC
22:51 designated xarses, I requested a quote for paid support last week and the lady told me on the phone she would have someone contact me.  I never heard anything else.
22:52 xarses designated: I'm sorry to hear that, If you know who you spoke to and want to provide some contact information I can see that you are taken care of.
22:53 designated xarses, her name is Erica Pace
22:57 Kupo24z1 xarses: Do i need to change the management interface as well than ? its all on the same interface in this setup
22:57 designated xarses, I sent you a private message with my email address.
22:58 xarses designated: I saw, It looks like the person you spoke with Isn't currently in the office. I will send them a message and cc you and their manager.
22:58 designated xarses, thank you.
23:03 ykotko joined #fuel
23:04 xarses Kupo24z1: I would consider changing the mgmt interface aswell
23:08 Kupo24z1 xarses: is there any documentation to creating a yaml file for fuel deployments?
23:11 xarses yes there should a bit of it, search for "fuel deployment default" the actual command you need to start with is 'fuel --env <num>|1 deployment default'
23:11 xarses I can look in a couple of min
23:12 Kupo24z1 cool thanks
23:35 xarses designated: you say that it's completely random if the node will have the error one time, and not another?
23:37 xarses designated: Also, I need some clarification as to when the trace occurs. Assigning the public interface should only occur when adding the node to the cluster, updating the interface layout, or starting the deployment. Which cases your issue?
23:38 IlyaE joined #fuel
23:39 xarses concurrency should't be an issue there was load testing on the agent handlers to validate the rewrite
23:39 designated xarses, it is random and occurs on different nodes.
23:40 designated xarses, after I loaded 5.0, I create an environment and go to add nodes.  after I begin adding nodes, randomly one of the nodes will present the error, CanNotFindInterface: Cannot find interface by name "public" for node: Untitled
23:40 designated xarses, which locks up the gui
23:41 designated i was able to get around the error, using the cli ( i don't remember exactly how) and get to the point where I was able to move the troublesome interface off the pxe interface.  after configuration and trying to deploy, I got through OS load and when it began installing openstack it failed.
23:43 designated this was the error I received on deployment:
23:43 designated MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
23:43 designated ID: 164 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://192.168.168.8:/puppet/modules/ /etc/puppet/modules/.
23:43 designated Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
23:43 designated rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]
23:43 designated ID: 160 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://192.168.168.8:/puppet/modules/ /etc/puppet/modules/.
23:46 Kupo24z1 xarses: I can see I can download the config via https://wiki.openstack.org/wiki/Fuel_CLI however I dont see any MTU settings
23:48 Kupo24z1 'fuel --env 5 deployment default' shows:There is no deployment info for this environment!
23:52 xarses designated: what OS are you deploying?
23:53 xarses Kupo24z1: you can only deployment default if there are pending changes
23:53 xarses Kupo24z1: I'd have to look exactly, but there should be a mtu value in the yaml data already and set to 1500

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary