Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-04-06

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:10 daniel3 joined #fuel
00:23 emagana joined #fuel
00:29 emagana joined #fuel
00:37 rongze joined #fuel
00:38 teran joined #fuel
01:18 emagana joined #fuel
01:38 rongze joined #fuel
01:39 teran joined #fuel
02:12 emagana joined #fuel
02:39 rongze joined #fuel
02:40 teran joined #fuel
02:54 emagana joined #fuel
03:07 rongze joined #fuel
03:23 emagana joined #fuel
03:41 teran joined #fuel
03:55 rongze joined #fuel
04:41 teran joined #fuel
04:52 rongze joined #fuel
05:03 zerda joined #fuel
05:14 Longgeek joined #fuel
05:48 rongze joined #fuel
06:06 emagana joined #fuel
06:16 dklepikov joined #fuel
06:24 Longgeek joined #fuel
06:28 rongze joined #fuel
06:29 aliemieshko_ joined #fuel
06:53 teran joined #fuel
07:16 HeOS joined #fuel
07:35 e0ne joined #fuel
07:47 saibarspeis joined #fuel
07:48 Miouge joined #fuel
07:51 aliemieshko joined #fuel
07:54 teran_ joined #fuel
07:54 emagana joined #fuel
08:17 e0ne joined #fuel
08:49 emagana joined #fuel
08:54 teran joined #fuel
08:57 alecv_ joined #fuel
09:00 e0ne joined #fuel
09:01 aliemieshko joined #fuel
09:12 alecv joined #fuel
09:17 Miouge joined #fuel
09:43 emagana joined #fuel
09:43 Longgeek joined #fuel
10:00 hyperbaba joined #fuel
10:53 azemlyanov joined #fuel
11:31 emagana joined #fuel
11:39 Longgeek joined #fuel
12:23 ddmitriev joined #fuel
12:23 asilenkov_ joined #fuel
12:25 emagana joined #fuel
13:20 emagana joined #fuel
13:22 championofcyrodi joined #fuel
13:22 championofcyrodi mattymo: going to look in to recovering the fuel server today.
13:22 championofcyrodi since monday.
13:22 championofcyrodi things always go well on mondays.
13:25 mattymo championofcyrodi, ok!
13:26 mattymo championofcyrodi, how many containers would you guess are broken?
13:30 championofcyrodi nginx_6.0, ostf_6.0,nailgun_6.0, keystone_6.0 are all exited 5 days ago.
13:30 teran_ joined #fuel
13:31 championofcyrodi one or more of the others seem to depend on nailgun though.
13:31 championofcyrodi so it could just be the nailgun one.
13:31 championofcyrodi astute,rsyslog,rabbit,rsync,mcollective, and cobbler are all up
13:32 championofcyrodi 7 busybox instances all exited 6 months ago as well.
13:32 championofcyrodi which is about the time we installed fuel 5.0
13:32 championofcyrodi actually i guess that was 8 months ago, 6 months ago we updated to 5.1
13:36 teran joined #fuel
13:37 rongze joined #fuel
13:58 ChanServ joined #fuel
13:58 jseutter joined #fuel
13:58 obcecado joined #fuel
13:58 justif joined #fuel
13:58 HeOS_ joined #fuel
13:58 bookwar joined #fuel
13:58 asilenkov joined #fuel
13:58 meow-nofer_ joined #fuel
13:58 Samos123 joined #fuel
13:58 t_dmitry joined #fuel
13:58 alex_didenko joined #fuel
13:58 nurla joined #fuel
13:58 Obi-Wan joined #fuel
13:58 bogdando joined #fuel
13:58 wiza joined #fuel
13:58 martineg_ joined #fuel
13:58 sovsianikov joined #fuel
13:58 zimboboyd joined #fuel
13:58 dpyzhov joined #fuel
13:58 kevinbenton joined #fuel
13:58 mescanef joined #fuel
13:58 maximov joined #fuel
13:58 melafefon__ joined #fuel
13:58 francois1 joined #fuel
13:58 tlbr joined #fuel
13:58 mattymo joined #fuel
13:58 seeg joined #fuel
13:58 SergK joined #fuel
13:58 scheuk_ joined #fuel
13:58 igorbelikov joined #fuel
13:58 kaliya joined #fuel
13:58 mutex joined #fuel
13:58 kybe joined #fuel
13:58 sanek joined #fuel
13:58 jskarpet joined #fuel
13:58 omolchanov joined #fuel
13:58 xek joined #fuel
13:58 okosse joined #fuel
13:58 wayneeseguin joined #fuel
13:58 tobiash joined #fuel
13:58 aarefiev joined #fuel
13:58 youellet joined #fuel
13:58 eren joined #fuel
13:58 mihgen joined #fuel
13:58 holser joined #fuel
13:58 brain461 joined #fuel
13:58 kdavyd joined #fuel
13:58 corepb joined #fuel
13:58 LiJiansheng joined #fuel
13:58 kutija joined #fuel
13:58 dmellado joined #fuel
13:58 daniel3 joined #fuel
13:58 dklepikov joined #fuel
13:58 aliemieshko joined #fuel
13:58 hyperbaba joined #fuel
13:58 azemlyanov joined #fuel
13:58 asilenkov_ joined #fuel
13:58 championofcyrodi joined #fuel
13:58 teran joined #fuel
13:58 evgeniyl___ joined #fuel
13:58 evg joined #fuel
13:58 MorAle joined #fuel
13:58 sbog joined #fuel
13:58 rvyalov joined #fuel
13:58 akislitsky joined #fuel
13:58 bdudko joined #fuel
13:58 meow-nofer joined #fuel
13:58 vkramskikh joined #fuel
13:58 aglarendil joined #fuel
13:58 apalkina joined #fuel
13:58 izinovik joined #fuel
13:58 MiroslavAnashkin joined #fuel
13:58 Bomfunk joined #fuel
13:58 fBigBro joined #fuel
13:58 dmitryme joined #fuel
13:58 aignatov joined #fuel
13:58 bjoernfan joined #fuel
13:58 artem_panchenko joined #fuel
13:58 docaedo joined #fuel
13:58 strictlyb joined #fuel
13:58 Arminder joined #fuel
13:58 jgriffith joined #fuel
13:58 rsFF joined #fuel
13:58 sc-rm joined #fuel
13:58 dkaigarodsev_ joined #fuel
13:58 ddmitriev joined #fuel
13:58 book` joined #fuel
13:58 kat_pimenova joined #fuel
13:58 jkirnosova joined #fuel
13:58 rongze joined #fuel
13:58 HeOS joined #fuel
13:58 aleksandr_null joined #fuel
13:58 BobBall_AWAY joined #fuel
13:58 glavni_ninja joined #fuel
14:01 ddmitriev joined #fuel
14:01 book` joined #fuel
14:01 kat_pimenova joined #fuel
14:01 jkirnosova joined #fuel
14:09 daniel3_ joined #fuel
14:10 tzn joined #fuel
14:18 emagana joined #fuel
14:19 emagana joined #fuel
14:24 mattymo championofcyrodi, sorry I got pulled away a while ago
14:44 mattgriffin joined #fuel
14:52 championofcyrodi mattymo: all good, i did as well.
14:53 mattymo so all 5 of those you mentioned have no persistent data
14:53 mattymo the 3 with persistent data are postgres, cobbler, and astute
14:53 championofcyrodi miroslavanashkin mentioned nailgun having persistent data
14:54 teran joined #fuel
14:54 mattymo data loss there is most harmful.. but it's important to find out what happened to those 3 first
14:54 mattymo it doesn't :)
14:54 championofcyrodi agreed
14:54 mattymo I promise
14:54 championofcyrodi k
14:55 championofcyrodi so just for some background... my disk is NOT full
14:55 mattymo first check postgres.. does `dockerctl check postgres` work?
14:56 championofcyrodi result: http://paste.openstack.org/show/198526/
14:56 jaypipes joined #fuel
14:56 championofcyrodi docker ps -a result: http://paste.openstack.org/show/198527/
14:57 championofcyrodi you can see the first 4 containers are exited, while the let 6 have been up 3 days. (i did a dockerctl backup 3 days ago)
14:57 championofcyrodi when things first went down
14:57 teran_ joined #fuel
14:57 mattymo it looks like it's just postgres
14:58 championofcyrodi whoa. i just noticed that... there i no postgres 6.0 container in my list
14:58 mattymo http://docs.mirantis.com/openstack/fuel/master/operations.html#fuel-master-and-docker-disk-space-troubleshooting
14:58 mattymo so you need to do 1 difficult task first.. which is find the UID of the fuel-core-6.0-postgres container
14:59 mattymo it's somewhere in the log /var/log/docker (or rotated gzipped docker log)
14:59 championofcyrodi gotcha
14:59 mattymo there's steps for how to find that
15:00 mattymo next you have to mount the devicemapper sparse block device manually, copy the data out, then purge the broken container, then start a new postgres container, and finally import the data
15:01 jobewan joined #fuel
15:01 mattymo and find out if there's any corruption. usually it's just indexes which have to be fixed by hand after looking at ostf/keystone/nailgun logs
15:01 championofcyrodi is the "sparse block device" the 10GB of allocated space from the 100GB default pool size devicemapper uses w/ docker on centos?
15:01 mattymo yeah
15:01 mattymo it's all in the link I gave you
15:01 championofcyrodi i've read through that link, and it mentions things like 'sparse block device' but i have trouble verifying if certain terms are what i think they are.  i'll ask you if i run in to anything else i'm confused about.
15:01 championofcyrodi thanks
15:02 mattymo I wrote that doc, but please let me know if you get stuck and I'll be here for the next few hrs to walk you through it
15:06 championofcyrodi mattymo: trying to find the container UID with the docker log in step one.  Step two says, "Once you have the container ID...", but I'm not sure I'm getting the ID in step one: http://paste.openstack.org/show/198538/
15:08 mattymo championofcyrodi, looks weird :( Change -m1 to -m5 -A5 to -C10
15:08 championofcyrodi also, tried to raise m1 to m4 : http://paste.openstack.org/show/198539/
15:09 mattymo you want a line like this: time="2015-04-06T13:33:55Z" level="info" msg="+job log(create, bf207e9cfe00934a02e927006a503324e9e28bb9e2acff1dd39079b7b429e02b, fuel/postgres_6.1:latest)"
15:09 championofcyrodi :-m5 -C10 : http://paste.openstack.org/show/198547/
15:10 mattymo championofcyrodi, try changing grep to zgrep and apply it to any rotated logs in /var/log/docker*
15:10 mattymo if that fails, we can just query sqlite db directly
15:11 championofcyrodi ah that's a good idea. (zgrep...)
15:11 championofcyrodi you know, i'm not seeing any compressed logs in /var/log
15:13 mattymo ok let's go to the db
15:13 mattymo let me get a command for you
15:14 championofcyrodi weird... there is like a 26 byte pgsql executable in /var/log/docker-logs
15:14 championofcyrodi instead of a folder w/ postgres logs
15:14 championofcyrodi postgres container logs*
15:14 mattymo ignore that
15:15 mattymo championofcyrodi, try this: sqlite3 /var/lib/docker/linkgraph.db "select * from edge";
15:15 championofcyrodi 0d8eac1f5b2dfba960a1c0ded5d7b8c54fd25608bf2cf84618584adccba535a0|0|fuel-core-6.0-postgres ?
15:16 mattymo yep that's it
15:16 championofcyrodi i didnt realize docker used sqlite!! awesome
15:16 championofcyrodi i can totally use that for my docker project which requires 3 containers.
15:16 mattymo it's broken somehow though with its link to devicemapper and it won't use the storage
15:16 mattymo the storage exists but docker can't use it
15:16 championofcyrodi hmmm...
15:16 mattymo http://docs.mirantis.com/openstack/fuel/master/operations.html#fuel-master-and-docker-disk-space-troubleshooting back to ehre now
15:16 mattymo here*
15:16 championofcyrodi perhaps a process died before the link information could be persisted to disk?
15:17 championofcyrodi thanks
15:17 mattymo it's just a uid linked to a sparse image id
15:23 championofcyrodi okay, i have the device id, about to run dmsetup and activate the volume and mount it.
15:41 Miouge joined #fuel
15:41 xarses joined #fuel
15:46 daniel3_ joined #fuel
16:00 emagana joined #fuel
16:06 dweaver joined #fuel
16:40 Guest58 joined #fuel
17:09 thumpba joined #fuel
17:10 eriswans joined #fuel
17:11 xarses joined #fuel
17:13 eriswans What’s the recommended way to restart the metadata agents running on a node? “service neutron-metadata-agent status” shows “neutron-metadata-agent stop/waiting” which makes me unsure if I’m supposed to use the init script. (This is Fuel 6.0 in a HA setup.)
17:22 loth joined #fuel
18:03 rongze joined #fuel
18:26 e0ne joined #fuel
18:27 Guest58 joined #fuel
18:32 MiroslavAnashkin eriswans: Please check https://bugs.launchpad.net/fuel/+bug/1397284  You may need to run `chmod 0755 /var/lib/neutron/external/pids/*` first, then apply the patch to OCF scripts.
18:33 MiroslavAnashkin eriswans: On each controller.
18:37 teran joined #fuel
19:00 thumpba_ joined #fuel
19:01 tzn joined #fuel
19:18 championofcyrodi oh man...
19:18 eriswans (The change MiroslavAnashkin mentioned was already in place on all my controllers.)
19:19 championofcyrodi looks like i have about 300 TIME_WAIT connections to my postgres container as the connection test is failing and trying again over and over...
19:19 eriswans I’m in a bit of a pickle though, after rebooting my controllers (one at a time, letting each come back up (or so I thought)) I can’t ping any of my instances.
19:20 championofcyrodi check that out: http://paste.openstack.org/show/198743/
19:21 championofcyrodi it's a wonder the ephemeral ports arent exhausted.
19:23 HeOS joined #fuel
19:27 vt102 joined #fuel
19:29 teran joined #fuel
20:02 xarses joined #fuel
20:09 vidalinux joined #fuel
20:15 rongze joined #fuel
20:24 emagana joined #fuel
20:45 eriswans joined #fuel
21:13 emagana joined #fuel
21:16 rongze joined #fuel
21:54 tzn joined #fuel
21:55 tzn joined #fuel
22:03 teran_ joined #fuel
22:07 tzn joined #fuel
22:08 teran joined #fuel
22:17 tzn joined #fuel
22:17 rongze joined #fuel
22:25 tzn joined #fuel
23:18 rongze joined #fuel
23:27 loth left #fuel
23:54 eriswans joined #fuel
23:55 emagana joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary