Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-04-14

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 julien_ZTE joined #fuel
00:45 LiJiansheng joined #fuel
01:02 daniel3_ joined #fuel
01:50 xarses joined #fuel
02:48 glavni_ninja_ joined #fuel
03:01 lee__ joined #fuel
03:13 xarses joined #fuel
03:13 samuelBartel joined #fuel
03:13 asilenkov joined #fuel
04:18 mattgriffin joined #fuel
04:19 thumpba joined #fuel
05:16 thumpba_ joined #fuel
05:39 xarses joined #fuel
05:53 Longgeek joined #fuel
05:56 sambork joined #fuel
06:01 dklepikov joined #fuel
06:08 daniel3_ joined #fuel
06:23 alecv joined #fuel
06:39 kozhukalov joined #fuel
07:12 hyperbaba joined #fuel
07:16 teran joined #fuel
07:24 dmellado joined #fuel
07:45 gongysh joined #fuel
07:58 e0ne joined #fuel
08:01 stamak joined #fuel
08:08 e0ne joined #fuel
08:32 ddmitriev joined #fuel
08:44 tzn joined #fuel
09:23 alecv joined #fuel
09:27 HeOS joined #fuel
09:30 sambork joined #fuel
09:51 geta-tn joined #fuel
09:53 HeOS__ joined #fuel
09:54 BobBall_ joined #fuel
09:57 tobiash_ joined #fuel
10:02 aleksandr_null_ joined #fuel
10:16 alecv joined #fuel
10:20 teran joined #fuel
10:23 sambork joined #fuel
10:24 teran_ joined #fuel
10:42 [HeOS] joined #fuel
11:02 sc-rm joined #fuel
11:05 DaveJ__ joined #fuel
11:05 DaveJ__ Hi
11:06 DaveJ__ Can anyone help me with a fuel postgres issue ?
11:06 DaveJ__ My fuel node ran out of space and now the postgres container wont start
11:06 DaveJ__ it restarst every 7 sec, which is making it hard to get into and discover the issue
11:07 DaveJ__ Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install postgresql' returned 1: rpmdb: /var/lib/rpm/Packages: unexpected file type or format
11:33 tzn joined #fuel
11:38 vt102 joined #fuel
12:59 sambork joined #fuel
13:11 mattgriffin joined #fuel
13:14 DaveJ__ managed to get a bit further
13:14 DaveJ__ it looked like the Packages file was corrupt, so I created a new container from the orginal image, and got a copy of the Packages file
13:14 DaveJ__ now postgres installs when I restart the origina container, but the db doesn't seem to get created
13:15 DaveJ__ https://www.irccloud.com/pastebin/9TNBC63U/postgres_error
13:41 emagana joined #fuel
13:43 MiroslavAnashkin Please check http://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#fuel-master-and-docker-disk-space-troubleshooting
13:57 jskarpet Fuel 6.0 disk full -> postgres container restart every 10 seconds -> how to fix?
14:00 MiroslavAnashkin http://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#fuel-master-and-docker-disk-space-troubleshooting
14:02 jskarpet never gets that far
14:02 jskarpet yum fails
14:02 jskarpet any way to start the container using bash instead of puppet?
14:02 jskarpet so that it lives for more than 10 seconds..
14:03 julien_ZTE joined #fuel
14:06 jaypipes joined #fuel
14:07 aliemieshko first you need to check free space on /var
14:08 jskarpet That parts fixed
14:08 jskarpet logrotate had failed miserablyh
14:09 jskarpet I can't nuke the container either since db dump fails: pg_dumpall: could not connect to database "template1": could not connect to server
14:09 MiroslavAnashkin You have to stop all containers and docker service,  free some disk space in /var/ on the host, then check the host file system, then check device mapper metadata like following http://stackoverflow.com/questions/24709741/cant-run-docker-container-due-device-mapper-error
14:10 aliemieshko please provide us the output : dockerctl list -l
14:10 jskarpet aliemieshko: They're all running
14:10 jskarpet but keystone and postgres keeps dying after 10 s
14:11 jskarpet I can do like: dockerctl start postgres --attach
14:11 jskarpet and see otuput
14:11 jskarpet it runs Puppet, which fails
14:11 jskarpet then container dies
14:11 aliemieshko you have to try steps discribe above
14:17 DaveJ__ jskarpet: I'm experienceing the same issue
14:20 jskarpet following those steps resulted in kernel panic when starting docker again
14:40 MiroslavAnashkin Then, please try to boot master node from some resque CD/USB and check the host file system one more time
14:41 jskarpet recovering from backup atm.
14:41 jskarpet Second time on this Fuel installation!
14:42 jskarpet You'd think that 60GB for logs, with a logrotate set to 100M max filesize should be sufficient, but not
14:42 jskarpet *no
14:43 jskarpet Any ideas on better protection against disk full crashing postgres?
14:43 jskarpet well, container really
14:43 jskarpet corrupting rpm database etc.
14:44 MiroslavAnashkin 60 Gb is minimal configuration. Future Fuel versions create separate partition for the log
14:44 jskarpet 60GB is just the /var partition
14:44 jskarpet well, 66 really
14:45 jskarpet What would be the reccomended max size?
14:47 mattgriffin joined #fuel
14:53 blahRus joined #fuel
14:56 getup joined #fuel
15:25 DaveJ__ This is driving me mad
15:25 DaveJ__ if I do a dockerctl revert postgres
15:25 DaveJ__ even a new container wont start
15:26 DaveJ__ If I completely re-install fuel, if I import the dump from psql, will that restore all config ?
15:26 DaveJ__ i.e. my deployment ?
15:33 jobewan joined #fuel
16:00 Longgeek joined #fuel
16:01 Longgeek joined #fuel
16:10 julien_ZTE joined #fuel
16:15 mattgriffin joined #fuel
16:16 emagana joined #fuel
16:17 mutex joined #fuel
16:25 mattgriffin joined #fuel
16:27 emagana joined #fuel
16:30 championofcyrodi mattymo: been playing with 'dmsetup' very useful knowledge for docker on centos 6.
16:30 championofcyrodi Started setting up the prerequisites for fuel-docs @ stackfoge.
16:31 championofcyrodi still reading and testing though.
16:31 mattymo building all the docs is kind of tough
16:32 mattymo for dmsetup I actually had chats with some devicemapper devels to get some solutions
16:32 championofcyrodi i've done C, java, maven, ant and .NET builds, so I should be able to work through it.
16:33 julien_ZTE joined #fuel
16:33 emagana joined #fuel
16:33 championofcyrodi i noticed i was able to break docker just now on ec2 by having a host process open a working directory on the mount point, and the trying to 'stop' the container.
16:34 championofcyrodi in that the dmsetup status showed it mapped, although i believe the docker sql db differed.
16:35 mattymo yeah mount points are very sensitive
16:36 mattymo in the newer version of docker in fuel 6.1 it's a lot better
16:37 championofcyrodi http://paste.openstack.org/show/203905/
16:40 championofcyrodi essentially, it seems docker does not handle devicemapper resources being busy when you start/stop a container.
16:40 championofcyrodi which could have happened for us during a backup ?
16:41 mattymo dockerctl has a hook in start that tries to fix the mount issue
16:42 mattymo but a generic workaround for what you're seeing is: docker stop $(docker ps -q); umount -l $(grep docker /proc/mounts | awk '{print $2}' | sort -r)
16:45 championofcyrodi right.  that's what i ended up doing
16:46 championofcyrodi well, i just ran mount, and then umount pasting in the containerid's mount point.
16:47 championofcyrodi but also noticed that even after umount, there is still dangling socket in 'dmsetup status'
16:48 championofcyrodi i guess it's a unix socket... 'ls /dev/mapper/'
16:49 championofcyrodi ah.. looks like they are symlinks to /dev/dm-*
16:50 championofcyrodi file type flag of 'b' for those...
16:51 championofcyrodi block device.
16:54 championofcyrodi ah i see, and blockdev --getsize64 /dev/dm-0 is 107374182400 bytes or 100GB, hence the devicemapper pool size limit
17:06 thumpba joined #fuel
17:07 xarses joined #fuel
17:13 e0ne joined #fuel
17:30 mattgriffin joined #fuel
17:35 Arminder joined #fuel
17:57 teran joined #fuel
18:07 Arminder joined #fuel
18:13 newbism joined #fuel
18:14 newbism Hello
18:14 newbism I'm using fuel 6.0. When I discover nodes my Mellanox CX3 card is not discovered
18:32 MiroslavAnashkin Please run `fuel node list`command and get the IP address of one of the bootstrapped nodes with mellanox cards
18:32 MiroslavAnashkin Then, ssh to this node with `ssh <ip address>`
18:34 MiroslavAnashkin And check `lsmod`, `dmesg` and `ip l` - to verify are your Mellanox cards visible, is driver loaded etc.
18:34 MiroslavAnashkin You may also try to start mellanox NIC manually - sometimes NIC simply starts longer than built-in kernel timeout.
18:35 MiroslavAnashkin If NIC is not started or recognized at all - you may have to update the Mellanox drivers in both, bootstrap image and distros.
18:45 championofcyrodi joined #fuel
18:59 newbism MiroslavAnashkin:  if I do echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1 ... the CX3 interface will show up
19:00 newbism # lspci | grep Mel 01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
19:00 newbism mlx4_en 0000:01:00.0: Activating port:1 mlx4_en: eth2: Using 128 TX rings mlx4_en: eth2: Using 8 RX rings mlx4_en: eth2: Initializing port
19:00 e0ne joined #fuel
19:00 newbism thats dmesg after the echo
19:07 championofcyrodi joined #fuel
19:10 teran joined #fuel
19:14 Guest58 joined #fuel
19:19 stamak joined #fuel
19:21 championofcyrodi joined #fuel
19:36 championofcyrodi joined #fuel
19:42 Arminder joined #fuel
19:43 Arminder joined #fuel
19:44 Arminder joined #fuel
19:44 loth joined #fuel
19:45 loth left #fuel
19:45 Arminder joined #fuel
19:46 Arminder joined #fuel
19:47 Arminder joined #fuel
19:48 Arminder joined #fuel
19:49 Arminder joined #fuel
19:50 Arminder joined #fuel
19:51 Arminder joined #fuel
19:52 Arminder joined #fuel
19:52 Arminder joined #fuel
19:53 martineg_ joined #fuel
19:54 Arminder joined #fuel
19:54 Arminder joined #fuel
19:56 [HeOS] joined #fuel
20:03 championofcyrodi joined #fuel
20:36 vt102 joined #fuel
21:02 vt102 joined #fuel
21:04 vt102 joined #fuel
21:06 vt102 joined #fuel
21:07 vt102 joined #fuel
21:10 vt102 joined #fuel
21:12 championofcyrodi joined #fuel
21:12 artem_panchenko joined #fuel
21:24 daniel3_ joined #fuel
21:26 tzn joined #fuel
21:41 teran_ joined #fuel
21:48 teran joined #fuel
23:48 Arminder joined #fuel
23:54 julien_ZTE joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary