Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-09-14

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:01 zhangjn joined #fuel
01:48 ilbot3 joined #fuel
01:48 Topic for #fuel is now Fuel 6.1 (Juno) https://software.mirantis.com | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
01:49 sergmelikyan joined #fuel
02:09 serg_melikyan joined #fuel
02:21 gongysh joined #fuel
03:01 fedexo joined #fuel
03:09 gongysh joined #fuel
03:21 gongysh joined #fuel
03:26 fedexo joined #fuel
03:30 Samos123 joined #fuel
04:16 RageLtMan joined #fuel
04:17 RageLtMan I'm trying to figure out how to disable HA in Fuel 7, can anyone lend a hand?
05:00 vsedelnik joined #fuel
05:59 monester joined #fuel
06:15 RageLtMan joined #fuel
06:43 mkwiek1 joined #fuel
08:07 hyperbaba joined #fuel
08:07 nihilifer RageLtMan: there's no such possibility
08:11 RageLtMan one of the developers was telling me a few days ago and that evg would know how to do so via the CLI by editing the YML. I dont even mind having multiple controllers, the idea is that in the event of a power outage or something, that the cluster could come back up
08:11 RageLtMan we tried building a single controller setup and galera refused to resuscitate itself after a reboot.
08:12 monester joined #fuel
08:25 zhangjn joined #fuel
08:29 zhangjn joined #fuel
08:29 td_ joined #fuel
08:30 td_ hi
08:31 adanin joined #fuel
08:35 zhangjn joined #fuel
08:35 julien_ZTE joined #fuel
08:36 HeOS joined #fuel
08:38 book` joined #fuel
08:47 gongysh joined #fuel
08:48 gongysh_ joined #fuel
08:51 e0ne joined #fuel
08:57 zhangjn joined #fuel
09:04 pradiprwt joined #fuel
09:16 e0ne joined #fuel
09:34 bazilza joined #fuel
09:38 kgalanov joined #fuel
09:45 pal_bth joined #fuel
09:49 strictlyb joined #fuel
09:50 teran joined #fuel
09:53 teran_ joined #fuel
10:01 adanin joined #fuel
10:03 zhangjn joined #fuel
10:11 vstruk_ joined #fuel
10:19 teran joined #fuel
10:19 zhangjn joined #fuel
10:25 omolchanov joined #fuel
10:28 smakar joined #fuel
10:33 adanin RageLtMan: Hi. What ISO do you use? 6.1 or 7.0?
10:33 RageLtMan 7.0
10:33 RageLtMan from two or 3 das ago
10:35 adanin The fastest way is to recreate Nailgun DB with fixed data. All previous environments will be forgotten in this case. Is it ok for you?
10:36 adanin If so, add another deployment mode here https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/fixtures/openstack.yaml#L1475
10:36 RageLtMan I have a fresh fuel install with a new env i can rebuild in a few minutes
10:36 adanin you need to go into nailgun docker container: dockerctl shell nailgun
10:37 RageLtMan i tried setting the mode to multinode via CLI
10:37 RageLtMan thank you much for helping btw, going there now
10:37 RageLtMan in
10:38 adanin then edit the file /usr/lib/python2.6/site-packages/nailgun/fixtures/openstack.yaml
10:39 DaveJ__ joined #fuel
10:39 adanin like I said, add another mode in line 1475
10:40 RageLtMan so "modes: ['ha_compact', 'multinode']"
10:40 adanin yes
10:40 RageLtMan done
10:40 adanin Then run: manage.py dropdb; manage.py syncdb; manage.py loaddefault
10:42 adanin Here is the thing. Although we still keep the “multinode” code in Nailgun and Puppet manifests, we didn’t test it for 7.0. So, it may not work.
10:42 RageLtMan lets see what happens :). thank you very much
10:43 adanin I guess it will work for 6.1 but I’m not sure.
10:45 adanin RageLtMan: btw, it would be nice if you file a bug about your problems with Galera after reboot.
10:46 RageLtMan I've seen a few bugs, but the basic issue is that with a single master, galera tries to find a peer on reboot and fails miserably
10:46 RageLtMan i found references to a script (which i did find, albeit too late) which is supposed to perform a clean shutdown/startup for galera
10:46 RageLtMan concern is what happens in a "not clean" drop?
10:47 RageLtMan also, is dropping the database supposed to be a monumental undertaking, or is the nailgun container just watching me squirm for fun?
10:47 adanin what do you mean by  a "not clean" drop?
10:48 RageLtMan power outage, gremlins, etc
10:50 adanin Sorry, I don’t understand you question. Could you clarify on it?
10:50 RageLtMan If the power goes out and kills the only master node, when it comes up, galera will seek peers which dont exist
10:51 RageLtMan how do we address this today if there's only one master, or all 3 masters get taken out by the same event?
10:52 RageLtMan i've also been looking at '2015-09-14 10:43:16.113 INFO [7f2700c94700] (manage) Dropping database...' for 6m now, is this normal? i see no IO on the underlying storage, and CPUs are in low-usage territory as well
10:52 adanin I thought it works with 1 controller. It just elects itself as a master.
10:52 RageLtMan on fuel7 i rebooted the master in testing and it refused to come up, had this happen a couple times
10:52 RageLtMan nailgun is not very happy with me.
10:53 adanin Press Ctrl-C
10:53 adanin I forgot to tell you shut down Nailgun first.
10:53 RageLtMan seems to be happily ignoring me
10:54 RageLtMan hmm, can i just open another shell and do it from there since the db drop is ignoring my ctrl+c
10:54 RageLtMan killed in background
10:54 adanin Run supervisorctl stop all
10:55 zhangjn joined #fuel
10:55 adanin in nailgun container
10:55 RageLtMan then the drop and such?
10:55 RageLtMan all stopped
10:55 adanin now you may have some stalled stuff in DB.
10:56 RageLtMan there we go
10:56 RageLtMan then supervisorctl start all?
10:57 adanin did you run dropdb again and it finished its work?
10:57 RageLtMan yup
10:57 adanin cool, start all then.
10:57 RageLtMan done
10:57 vsedelnik joined #fuel
10:58 adanin check if it works: run fuel rel
10:58 RageLtMan yup
10:58 RageLtMan creating a new env
11:00 bdudko joined #fuel
11:00 RageLtMan "fuel env --env 1 --set --mode multinode" yields "Following attributes are changed for the environment: mode=multinode"
11:00 RageLtMan thank you very much
11:00 adanin you are welcome.
11:01 bapalm joined #fuel
11:01 adanin btw, why do you “set” a new mode after an env was created? I thought you have to provide it during creation.
11:01 Samos123 joined #fuel
11:01 gongysh joined #fuel
11:03 RageLtMan i created via the ui
11:05 adanin got it.
11:07 adanin RageLtMan: So, galera failed to elect itself all the time or just sometimes?
11:07 RageLtMan Every time in what was probably 3 resets of the env. not sure if it failed EVERY time though, but eventually it did
11:08 adanin did you just power them off? Or graceful restart?
11:12 RageLtMan when possible graceful, issue was that the controller was atop an unstable physical host (controller is virtual, the block devices backing it are ZFS and OpenStack seems to lack native ZFS drivers)
11:13 RageLtMan was actually a thermal problem which appears resolved
11:16 adanin If you accidentally has a diagnostic shapshot of that env it’d be happy to look at it.
11:16 RageLtMan i'll look through the ZFS snaps to see if any might be there, would be i the LVM atop the ZVOL though
11:20 adanin By “a diagnostic snapshot” I meant an archive of logs created by Fuel. It’s hidden under “Support” button on top of UI.
11:21 adanin I guess you didn’t create it.
11:21 RageLtMan i've made a few via the cli
11:22 adanin Cool. So, they should be somewhere in /var/www/nailgun/
11:22 RageLtMan been progressively digging into this deployment/lifecycle management bit. my beef is i chose all the "other" tech in our own stacks, chef, razor, etc, so this is similar stuff but syntactically new
11:22 RageLtMan they'll be in the at folder within the lvm atop a snapshot in zfs, somewhere :)
11:23 RageLtMan kicking off a deployment
11:23 RageLtMan lets see what happens
11:24 RageLtMan hmm fuel doesn't complain at me about there not being 3 masters, promising
11:24 RageLtMan aaaand no
11:24 RageLtMan does this: http://pastie.org/private/2pfmessjpqqjvkiixbmla
11:26 RageLtMan thank you for the assist though, looks like fuel simply can't support it anymore, unless you know of any other knobs to turn for this
11:28 jaypipes joined #fuel
11:28 e0ne joined #fuel
11:38 jaypipes joined #fuel
11:40 adanin RageLtMan: the error means you need to write a new serializer in Nailgun for multinode mode. I dont’ know any workarounds for that.
11:43 RageLtMan hmm. thank you anyway, this has been quite educational
11:44 zhangjn joined #fuel
11:51 jaypipes joined #fuel
11:52 e0ne joined #fuel
12:01 e0ne joined #fuel
12:12 akurenyshev left #fuel
12:13 vsedelnik joined #fuel
12:28 vsedelnik joined #fuel
12:30 neophy joined #fuel
12:38 jaypipes joined #fuel
12:48 championofcyrodi joined #fuel
13:01 vsedelnik joined #fuel
13:08 julien_ZTE joined #fuel
13:20 sc68cal joined #fuel
13:20 sc68cal joined #fuel
13:34 CTWill joined #fuel
13:35 CTWill How much space should /var have free for the fuel master to be happy?
13:35 CTWill I have about 20k
13:52 sergmelikyan joined #fuel
13:53 aliemieshko_ <CTWill>: you need have at least ~ 1gb
13:53 claflico joined #fuel
13:55 aliemieshko_ otherwise your docker containers stop
14:10 CTWill d
14:10 CTWill dop
14:11 CTWill ok /var/log is at 76G
14:11 CTWill im moving it off to a 1.5tb mount on the system
14:25 julien_ZTE joined #fuel
14:26 angdraug joined #fuel
14:30 neophy joined #fuel
14:31 pal_bth_ joined #fuel
14:35 CTWill is there a way to restart all the docker containers at once?
14:35 CTWill with out needing to reboot?
14:37 mwhahaha I think you can do it with dockerctl
14:37 CTWill doing a dockerctl destory all
14:37 mwhahaha no there's another way
14:37 CTWill dockerctl start all
14:37 mwhahaha yea that
14:38 mwhahaha just 'dockerctl start'
14:39 CTWill http://paste.openstack.org/show/461442/
14:42 mwhahaha that's probably a problem
14:42 CTWill :)
14:43 CTWill can I re-pull the docker images?
14:43 mwhahaha The destroy/build process should set them up again
14:43 CTWill is there a seperate build command?
14:44 mwhahaha yea
14:44 mwhahaha dockerctl destroy <all or a specific container>
14:44 CTWill one click fix it
14:44 mwhahaha dockerctl build <all or a specifric container>
14:48 julien_ZTE joined #fuel
14:49 bazilza joined #fuel
14:51 thansen joined #fuel
14:55 CTWill netlink: 20 bytes leftover after parsing attributes.
14:55 CTWill what does that mean?
14:57 mwhahaha annoying centos
14:57 CTWill great loving it
14:57 mwhahaha it's an issue with a newer iproute2 on the older centos kernel
14:58 mwhahaha just fills logs but doesn't actually hurt anything
14:58 CTWill right now it is stuck trying to start nailgun
14:59 CTWill http://paste.openstack.org/show/461457/
14:59 CTWill im getting new iptables denies
15:00 mwhahaha syslog
15:00 mwhahaha did you move your admin interface off of eth0?
15:00 CTWill no I didnt
15:00 CTWill :)
15:01 CTWill this has always been on eth2
15:01 mwhahaha was that from the fuel master?
15:01 CTWill iptables -I INPUT 17 -p tcp -i eth2 -m multiport --dports 4369,5672,15672,61613 -j ACCEPT
15:01 CTWill yes
15:01 CTWill I had to open a ticket for that line
15:02 julien_ZTE joined #fuel
15:02 mwhahaha oh is the rsyslog container up?
15:02 mwhahaha it might be denied if that's down
15:02 CTWill rsyslog      fuel/rsyslog_6.1         Running      fuel-core-6.1-rsyslog
15:03 mwhahaha hmm don't know then
15:03 mwhahaha we didn't actually have a reject all in the iptables i don't think
15:03 CTWill if I deployed a new fuel master
15:04 CTWill how hard would it be to have the two instances of openstack I have deployed seen by the new fuel master?
15:04 CTWill I liked this thing a lot more when it was a VM
15:04 CTWill and not on physical hardware
15:04 mwhahaha if you had a backup
15:04 mwhahaha you might be able to migrate the environments over to the new fuel master
15:05 CTWill well this will be a management call
15:05 CTWill lol
15:05 mwhahaha i know someone was working on a migration script for fuel i think to move it to a libvirt vm
15:05 CTWill could the fact the /var filled up cause this problem
15:05 mwhahaha yes
15:06 jerrygb joined #fuel
15:06 CTWill I moved /var/log to /mnt/log then created a symbolic link so that /var/log -> /mnt/log
15:09 CTWill dhcrelay failed to start on boot
15:11 v1k0d3n joined #fuel
15:18 rmoe joined #fuel
15:20 vsedelnik joined #fuel
15:21 blahRus joined #fuel
15:27 RageLtMan since multinode deployment doesnt work in Fuel 7, how does one deploy a single controller configuration, or more specifically, how are reboots of said single controller handled such that Galera doesnt try to replicate with non-existent masters?
15:30 mwhahaha what do you mean it doesn't work
15:47 angdraug mwhahaha: he means non-HA deployment mode
15:47 mwhahaha no he said multinode doesnt' work
15:48 angdraug multinode is a historic name for the non-HA deployment mode
15:48 mwhahaha worse naming convention ever
15:48 mwhahaha er worst
15:48 angdraug indeed
15:49 mwhahaha anyway i guess we should fix a single node controller reboot?
15:49 angdraug is it broken?
15:49 mwhahaha i don't know thats why I asked
15:49 * mwhahaha pokes RageLtMan
15:56 * RageLtMan looks around
15:57 julien_ZTE joined #fuel
15:57 mwhahaha what's broken?
15:57 RageLtMan galera seems to refuse to start when i have single instance controllers
15:58 mwhahaha so if you deploy a single controller and single compute, you're saying if you reboot the controller it doesn't come back up?
16:02 jacob_ joined #fuel
16:03 Guest92046 Should I go to fuel-dev for questions on Fuel 7.0?
16:07 jeh joined #fuel
16:12 angdraug Guest92046: depends on the kind of questions, not on the release )
16:15 Guest92046 I'm just wondering which plugins are compatible with Fuel 7.0. Any help is appreciated
16:25 RageLtMan mwahahah: sorry for lag, clients and such calling, yes - i've had at least 3 deployments where a crash under the master left it seeking to replicate with peers on reboot, and there arent any
16:29 kgalanov joined #fuel
16:29 teran joined #fuel
16:32 kgalanov joined #fuel
16:39 monester joined #fuel
16:51 julien_ZTE joined #fuel
16:54 mgrohar joined #fuel
17:16 neophy joined #fuel
17:29 RageLtMan Is there a definitive guide somewhere on how to properly bring down a fuel deployed openstack for maintenance?
17:43 mwhahaha RageLtMan: we do have a maintenance mode, https://docs.mirantis.com/fuel/fuel-master/operations.html#maintenance-mode but your issues sounds like it is a bug and needs to get addressed
17:45 julien_ZTE joined #fuel
17:58 angdraug joined #fuel
18:11 RageLtMan whats the syntax for having nonconsecutive VLAN ranges for neutron? so 'network_vlan_ranges = physnet1:1:3' but skipping VLAN 2.
18:24 teran joined #fuel
18:24 e0ne joined #fuel
18:36 teran joined #fuel
18:39 julien_ZTE joined #fuel
18:39 monester joined #fuel
18:47 vsedelnik joined #fuel
18:57 serg_melikyan joined #fuel
18:59 serg_melikyan joined #fuel
19:16 vsedelnik joined #fuel
19:19 jobewan joined #fuel
19:24 sergmelikyan joined #fuel
19:34 julien_ZTE joined #fuel
19:48 vstruk_ joined #fuel
19:50 e0ne joined #fuel
20:03 HeOS joined #fuel
20:20 Guest15777 joined #fuel
20:25 sergmelikyan joined #fuel
20:28 julien_ZTE joined #fuel
20:30 vsedelnik joined #fuel
20:44 az__ joined #fuel
20:45 teran joined #fuel
20:52 hezhiqiang joined #fuel
20:55 adanin joined #fuel
20:59 e0ne joined #fuel
20:59 claflico joined #fuel
21:14 mfisch joined #fuel
21:16 az__ joined #fuel
21:18 vin4gud joined #fuel
21:22 julien_ZTE joined #fuel
21:29 vin4gud Getting the following error message during Network Verification http://paste.openstack.org/raw/461879/, any ideas what could be wrong?
21:31 adanin vin4gud: Which Fuel version do you use and how many vlans you specified for a private range?
21:35 vin4gud Fuel 6.1 VLAN ids 1000 to 1030
21:36 serg_melikyan joined #fuel
21:38 adanin That means nodes 1 and 5 cannot answer via MCollective (which is using RabbitMQ as a transport).
21:38 adanin vin4gud: Do you have any nodes offline right now? If you repeat Network Check you would get the same result?
21:39 vin4gud have 5 slave nodes, and using only 1 controller and 1 compute at a time.... tried with different nodes but always get same error
21:40 adanin vin4gud: Oh, sorry. Whrong guess.
21:41 adanin you may go to the Logs tab, select Astute logs with Debug level and see what exactly commmand failed. Then ssh to one of the nodes (ssh node-1 or ssh node-5) and run the command manually.
21:42 adanin The command should be like “urlaccesscheck check SOME_URL”
21:46 adanin vin4gud: I have to go. Good luck.
21:47 vin4gud thans Andrey,, i will try that
21:50 luz_ joined #fuel
21:53 luz_ hello, so I installed a fuel master node, and I'm having a couple of issues, first of all how do I know that everything is running as expected? 2. At fuel UI there is a warning message about connecting to fuel update repository
21:54 luz_ 3. when rebooting the other servers (expecting it gets pxe boot), it doesn't
21:55 luz_ I'm working with physical hardware... any thoughts?
22:01 jobewan joined #fuel
22:09 e0ne joined #fuel
22:16 julien_ZTE joined #fuel
22:43 jobewan joined #fuel
23:10 julien_ZTE joined #fuel
23:31 WormMan joined #fuel
23:58 julien_ZTE joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary