Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-04-03

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 rongze joined #fuel
00:06 LiJiansheng joined #fuel
00:10 admin0 joined #fuel
01:04 brain461 joined #fuel
01:05 aleksandr_null joined #fuel
01:07 rongze joined #fuel
01:48 ilbot3 joined #fuel
01:48 Topic for #fuel is now Fuel 5.1.1 (Icehouse) and Fuel 6.0 (Juno) https://software.mirantis.com | Fuel for Openstack: https://wiki.openstack.org/wiki/Fuel | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
02:05 rongze joined #fuel
02:21 xarses joined #fuel
02:52 rongze joined #fuel
02:52 rongze joined #fuel
03:34 Longgeek joined #fuel
03:59 Longgeek joined #fuel
04:47 evgeniyl___ joined #fuel
04:49 kdavyd joined #fuel
05:14 thumpba joined #fuel
05:24 thumpba_ joined #fuel
05:55 corepb joined #fuel
06:06 dklepikov joined #fuel
06:55 admin0 joined #fuel
07:09 e0ne joined #fuel
07:23 admin0 joined #fuel
07:24 e0ne joined #fuel
07:37 stamak joined #fuel
07:55 stamak joined #fuel
08:27 e0ne joined #fuel
08:28 e0ne joined #fuel
08:37 Longgeek joined #fuel
08:55 tatyana joined #fuel
09:02 HeOS joined #fuel
09:02 subscope joined #fuel
09:28 alecv_ joined #fuel
09:59 corepb joined #fuel
10:03 corepb joined #fuel
10:04 corepb joined #fuel
10:07 corepb joined #fuel
10:07 corepb joined #fuel
10:14 stamak joined #fuel
10:15 corepb joined #fuel
10:31 tzn joined #fuel
11:12 admin0 joined #fuel
12:21 e0ne joined #fuel
12:29 artem_panchenko joined #fuel
13:10 homegrown joined #fuel
13:23 admin0 joined #fuel
13:55 mattgriffin joined #fuel
14:09 getup joined #fuel
14:12 teran joined #fuel
14:42 jaypipes joined #fuel
14:55 emagana joined #fuel
14:58 daniel3_ joined #fuel
15:01 rongze joined #fuel
15:03 emagana joined #fuel
15:43 xarses joined #fuel
15:48 blahRus joined #fuel
15:51 championofcyrodi is it possible 'df -h' is reporting free space on my /var mountpoint when it's actually full?
15:59 MiroslavAnashkin No, it requires at least some strange position of the Sun, Moon and stars to happen.
16:04 championofcyrodi hmm.  I guess I'm stuck because the nailgun container is referencing a uuid that just doesnt exist, my disks arent full... and i'm not sure what to do.
16:05 championofcyrodi i was thinking about trying to restore from the dockerctl backup i have from Mar 19th, which would be fine because we did not change any hardware configs since then...
16:05 championofcyrodi but i fear it could get worse.
16:06 championofcyrodi right now openstack juno is running okay...
16:06 championofcyrodi so it just seems to be fuel that is down
16:07 championofcyrodi on a positive note, i was able to get a 10Gbps fibre link up on one of the controllers going to the public lan... so that's a plus.
16:13 championofcyrodi at this point i'm just looking through logs to try and figure out what happened... according to the nailgun log.. http://paste.openstack.org/show/198148/
16:13 championofcyrodi basically out of nowhere the container is just gone...
16:16 championofcyrodi docker log just shows the network proxy being stopped around 09:11 on the same day, because of 'closed network connection'... which i guess just implies the container stopped..
16:17 championofcyrodi and the warning about using centos 2.6.3x kernel w/ docker and being unstable.
16:17 championofcyrodi probably due to devicemapper
16:17 championofcyrodi and old cgroups support i guess
16:18 championofcyrodi hmmm
16:18 championofcyrodi this is in /var/log/messages: Apr  3 11:12:58 fuel kernel: EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts:
16:18 championofcyrodi Apr  3 11:15:45 fuel kernel: EXT4-fs (dm-10): warning: maximal mount count reached, running e2fsck is recommended
16:21 championofcyrodi okay i think i found the problem: http://paste.openstack.org/show/198150/
16:29 stamak joined #fuel
16:32 championofcyrodi well, i have the container id, devicemapper block device ID and docker pool... but running e2fsck -n /dev/mapper/os-root showed some errors.
16:32 championofcyrodi so i guess i need to fix the disk errors before trying to recover the container.
16:34 championofcyrodi sounds like i need to unmount the filesystems first...
16:35 championofcyrodi so i guess I need to boot from live media that supports e2fsck for EXT4
16:35 championofcyrodi does the fuel server have an option in grub for recovery mode?
16:37 mattymo championofcyrodi, I wrote a quick doc on this some time ago
16:37 mattymo we're separating /var/log from /var in 6.1 to avoid this
16:38 mattymo http://docs.mirantis.com/openstack/fuel/master/operations.html#fuel-master-and-docker-disk-space-troubleshooting
16:38 mattymo championofcyrodi, I'm on my way out of the office right now, but leave a message here and I can walk you through more recovery
16:52 championofcyrodi Thanks mattymo
16:52 championofcyrodi I'll read up on it and get back to you.
17:03 teran joined #fuel
17:07 xarses joined #fuel
17:32 loth joined #fuel
17:41 MiroslavAnashkin But it looks rather like Docker hang up. Could you please reboot the master node and check if all containers started successfully?
17:48 championofcyrodi MiroslavAnashkin: i tried that yesterday
17:48 championofcyrodi they came up in the same stae
17:48 championofcyrodi state*
18:10 leakypipes joined #fuel
18:20 rongze joined #fuel
18:25 xarses joined #fuel
18:47 teran_ joined #fuel
18:53 daniel3_ joined #fuel
19:03 daniel3_ joined #fuel
19:03 daniel3_ joined #fuel
19:30 MiroslavAnashkin Then, it looks like devicemapper metadata corruption
19:31 e0ne joined #fuel
19:34 admin0 joined #fuel
19:44 championofcyrodi joined #fuel
19:56 teran joined #fuel
20:22 rongze joined #fuel
20:24 ddmitriev joined #fuel
20:39 admin0 joined #fuel
20:57 daniel3_ joined #fuel
20:58 teran joined #fuel
21:09 MiroslavAnashkin championofcyrodi: Then, it looks like devicemapper metadata corruption. Please try thin_check like following http://stackoverflow.com/questions/24709741/cant-run-docker-container-due-device-mapper-error
21:22 rongze joined #fuel
21:26 admin0 joined #fuel
21:56 admin0 joined #fuel
21:59 teran joined #fuel
22:24 rongze joined #fuel
22:59 teran joined #fuel
23:25 rongze joined #fuel
23:59 admin0 joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary