Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-05-13

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:35 bogdando joined #fuel
00:38 youellet In HA mode, after a reboot of the active neutron router node, they don't be back... Cannot ping virtual router from node or external network...
00:38 youellet I thinks its an peace maker issu ?
00:52 xarses joined #fuel
00:56 rmoe joined #fuel
01:48 ilbot3 joined #fuel
01:48 Topic for #fuel is now Fuel 5.1.1 (Icehouse) and Fuel 6.0 (Juno) https://software.mirantis.com | Fuel for Openstack: https://wiki.openstack.org/wiki/Fuel | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
02:00 fedexo joined #fuel
03:19 rongze joined #fuel
05:13 brain461_ joined #fuel
05:14 rmoe_ joined #fuel
05:16 jobewan_ joined #fuel
05:16 SergK_ joined #fuel
05:19 glavni_ninja_ joined #fuel
05:19 stamak joined #fuel
05:19 LanceHai- joined #fuel
05:25 eliqiao1 joined #fuel
05:26 rongze joined #fuel
05:58 rongze joined #fuel
06:06 rongze joined #fuel
07:14 MaverickHunter joined #fuel
07:22 vtzan joined #fuel
07:23 samuelBartel Hi all
07:24 samuelBartel i have a question about the 6.1 version
07:24 samuelBartel is the upgrade of the fuel master node and en environement is a feature planned for 6.1 or 7.0?
07:35 pbeskow joined #fuel
07:41 xek joined #fuel
07:42 homegrow_ joined #fuel
07:54 stamak joined #fuel
07:54 dklepikov joined #fuel
07:56 holser_ joined #fuel
08:00 Longgeek joined #fuel
08:06 tzn joined #fuel
08:08 hyperbaba joined #fuel
08:12 hyperbaba_ joined #fuel
08:14 ppal joined #fuel
08:20 HeOS joined #fuel
08:23 hyperbaba__ joined #fuel
08:23 baboune joined #fuel
08:25 e0ne joined #fuel
08:25 baboune hi, fuel 6-0, ran out of disk space in /var, and it might have corrupted the containers.  I am getting the following error: dockerctl stop postgres, Error: No such container: fuel-core-6.0-postgres. Same with start or restart.  cluster i srunning and I can not re-install.
08:26 baboune the image fuel/postgres_6.0                        latest              95203156eb35        4 months ago        211.8 MB is available
08:26 baboune can I recover this?
08:30 e0ne joined #fuel
08:32 martineg_ hm, I'm experiencing the same issue, but w/ sufficient disk space in /var. fs in the postgres container is corrupted.
08:41 kaliya baboune: check https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#fuel-master-and-docker-disk-space-troubleshooting
08:48 baboune kaliya: great!
08:49 kaliya martineg_: how is? Got the container metadata, devicemapper or docker database corrupted?
09:00 baboune so far it looks like there is no error
09:01 baboune at this step right now:  thin_restore -i /tmp/metadata.xml -o /var/lib/docker/devicemapper/devicemapper/metadata
09:01 baboune taking a very long time but it might be normal.
09:01 martineg_ kaliya: devicemapper errors
09:05 martineg_ kaliya: I tried running thin_check, which did not give errors, but I could not start the postgres container afterwards.
09:06 vtzan joined #fuel
09:14 alecv joined #fuel
09:20 rongze joined #fuel
09:22 e0ne joined #fuel
09:23 baboune kaliya: I can not find th eUUID of the container
09:23 baboune kaliya: grep returns nothing grep -m1 -A5 "fuel-core-6-postgres" /var/log/docker
09:24 kaliya baboune: maybe the grep options error, please try to inspect that file with less
09:24 kaliya martineg_: some logs?
09:27 baboune kaliya: those blocks of code, are they bash scripts?
09:28 kaliya baboune: should be command logs
09:34 baboune $(python -c 'import sys; import json; input = json.load(sys.stdin);\ > [sys.stdout.write(str(v["device_id"])) for k, v in input["Devices"].items() if > k == sys.argv[1]]' "579e6092b55f38b7bce17cf8fde0049c3f0cb163e174393bce679187a1977492" < /var/lib/docker/devicemapper/devicemapper/json) -bash: 375: command not found
09:35 baboune it is not straightforward
09:39 neophy joined #fuel
09:40 kaliya baboune: check in /var/log/docker-postgres.log if you find the ID
09:44 baboune kaliya: I found something there, ror: Cannot start container fuel-core-6.0-postgres: Error getting container 579e6092b55f38b7bce17cf8fde0049c3f0cb163e174393bce679187a1977492 from driver devicemapper: Error writing metadata to /var/lib/docker/devicemapper/devicemapper/.json041000306: write /var/lib/docker/devicemapper/devicemapper/.json041000306: no space left on device
09:44 baboune but it is the steps after where it requires finding the device mapper block id
09:44 baboune kaliya: there were no errors i any of the thin commands.  but after service docker start no containers are started
09:45 kaliya no containers at all?
09:46 baboune nope.
09:46 neophy Hi, I am using Fuel 5.1. compute nodes in the cluster don't have SSH access to each other. due to this instance resize fails. It looks like I am affected with this bug: https://bugs.launchpad.net/mos/+bug/1393785
09:46 baboune should they have restarted?
09:46 baboune should I do a "dockerctl start'?
09:47 kaliya baboune: `docker ps` empty?
09:47 neophy adding /bin/bash to nova user is the right solution for this issue?
09:47 kaliya neophy: maybe you can enable the 'nova' user... usermod assigning /bin/bash
09:47 baboune kaliya: yes
09:47 neophy any help on this?
09:48 kaliya baboune: `dockerctl start` fails ?
09:50 baboune kaliya: same pb as before         COMMAND             CREATED             STATUS              PORTS               NAMES [root@kds-cmc-fuel-01 restore]# dockerctl start 2015/05/13 09:49:22 Error: Abort due to constraint violation: constraint failed Error: No such image or container: fuel-core-6.0-postgres Error: No such image or container: fuel-core-6.0-postgres checking container postgres
09:50 baboune kaliya: I did the thin restore but it seems it is the same pb
09:51 baboune kaliya: and after dockertctl start basically no containers are running
09:52 kaliya did you have data in postgres?
09:52 baboune kaliya: do I need to mount it first?
09:52 baboune kaliya: yes.. info about 3 envs, and 37 machines
09:54 kaliya baboune: try to wipe the sqlite3
09:55 kaliya in the guide there is some sqlite3 delete command, you can before mount the recovery dir and get the postgres dump
09:55 kaliya sorry but I must run out now
10:19 getup joined #fuel
10:21 tzn_ joined #fuel
10:24 getup left #fuel
10:37 ilbot3 joined #fuel
10:37 Topic for #fuel is now Fuel 5.1.1 (Icehouse) and Fuel 6.0 (Juno) https://software.mirantis.com | Fuel for Openstack: https://wiki.openstack.org/wiki/Fuel | Paste here http://paste.openstack.org/ | IRC logs http://irclog.perlgeek.de/fuel/
10:37 dklepikov joined #fuel
10:39 homegrow_ joined #fuel
10:55 gabrijela_ joined #fuel
10:59 xdeller joined #fuel
11:14 gabrijela_ Hi, I have a functional deployment 5.1.1 . But now I have a request from the big guys to include active monitoring in the system. I did not activate experimental features on fuel thus did not have zabbix option. I have my own zabbix server and need a templates for openstack monitoring and agents on nodes configured. Is it possible to do so manually on deployed system?
11:22 vtzan gabrijela_, sure but needs some extra work by hand. Here are the templates and configuration file from fuel-library. https://github.com/stackforge/fuel-library/tree/master/deployment/puppet/zabbix
11:28 gabrijela_ vtzan: thank You. Will look into it. So the templates need to go on a zabbix server. The zabbix agent needs to call those helper scripts. What is the way of delivering the data to server ? Snmp or agent is called directly from the server?
11:28 gabrijela_ vtzan: can you give me some conf files from a prod server so i can catch up?
11:28 gabrijela_ vtzan: also the agent part on a node, please
11:36 LanceHaig joined #fuel
11:40 vtzan gabrijela_, all the files are on the url i sent you. these are the files that the puppet deployment uses. I am pretty sure its the agent that the zabbix calls.
11:40 gabrijela_ vtzan: Ok thank you very much.
11:42 holser_ joined #fuel
12:21 ddmitriev joined #fuel
12:53 kaliya joined #fuel
13:02 teran joined #fuel
13:46 hyperbaba vtzan: It's working, but it's hell to set it up manually
13:46 hyperbaba vtzan: Did you tried ceilometer-zabbix-proxy? It looks that it's awesome
13:50 obcecado joined #fuel
13:56 rongze joined #fuel
13:58 holser_ joined #fuel
14:07 holser_ joined #fuel
14:15 jaypipes joined #fuel
14:17 baboune joined #fuel
14:18 baboune hi, my postgresql container is missing.  Fuel 6.  I am following instructions from https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#fuel-master-and-docker-disk-space-troubleshooting.  I can not mount the container.
14:19 baboune dmsetup create "fuel-core-6.0-postgres_recovery" --table "0 20971520 thin /dev/mapper/docker*pool 375" -> device-mapper: reload ioctl on fuel-core-6.0-postgres_recovery failed: No such file or directory Command failed
14:22 baboune how am I supposed to do this data recovery ?
14:28 baboune it is not documented in the doc the "data recovery"
14:29 dmgweb joined #fuel
14:29 baboune plsu the container is gone... It can not be started so I can not dump the pg database either
14:30 dmgweb hello I have a problem with Extraroute in heat, someone help me?
14:42 daniel3_ joined #fuel
14:49 baboune basically the doc requires teh postgres container to be available, otherwise it can not be recovered in the case where the container can not ne found i.e. error: Error: No such image or container: fuel-core-6.0-postgres
14:57 blahRus joined #fuel
14:59 championofcyrodi joined #fuel
14:59 championofcyrodi http://paste.openstack.org/show/221882/ #rabbitmq heartbeat_timeout again...
15:00 championofcyrodi i've launched 2 instances, but now whenever trying to launch a third...  going to look in to the ephemeral ports used and see whats going on.  but i did 'find/replace/ from IPs to hostnames, and it looks like node-46 (my slowest/weakest controller) is an issue
15:06 rmoe joined #fuel
15:22 ub joined #fuel
15:28 baboune joined #fuel
15:33 baboune to sum up, fuel 6, "docker ps" - empty, "dockerctl start " - Error: No such image or container: fuel-core-6.0-postgres.  Can not access the postgres container so can not recover data.  Have container id, and deviceid.  Is there a way to recover the dat ain the "defunct" container? And how?
15:34 baboune the problem origin is that /var filled up
15:35 kaliya baboune: the postgres container, I think very unlikely, mattymo ^^
15:36 baboune kaliya: is not the data still on the host machine FS?
15:37 kaliya did you try to dmsetup the volume?
15:38 kaliya total failure?
15:38 baboune well the dmsetup requires the mount(postgres_recovery
15:38 baboune and I can not do the recovery since there is no container running
15:39 baboune unless I misunderstand the doc?
15:40 kaliya do you have the devicemapper device still around? in /var/lib/docker, you can try to get the /var/lib/postgres dir to some system, and try to dump from that raw data
15:41 baboune kaliya: I did not try to delete the sqllite3 db.  The device id is "375" for device mapper
15:41 xdeller joined #fuel
15:42 kaliya can you enter into /var/lib/docker/devicemapper/mnt/ID/rootfs/ ?
15:44 baboune /var/lib/docker/devicemapper/mnt/ContainerId )where container id is 579e...) is empty
15:44 kaliya rootfs/ ?
15:45 baboune there is no rootfs
15:46 mattymo there's a recovery doc
15:46 mattymo you need the container ID to look up the devicemapper device ID and then manually activate it
15:46 urs2 joined #fuel
15:47 mattymo https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#id97
15:47 mattymo it's tough because filling up the disk corrupted the link between docker and its storage backend... but the storage itself is fine
15:47 mattymo it's a tough and time consuming process to fix
15:47 baboune yes, I used that doc https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#fuel-master-and-docker-disk-space-troubleshooting but the container Id I find in docker-postgres.log has nothing in the /mnt/
15:48 mattymo it won't be moutned
15:48 mattymo mounted*
15:48 baboune the thing is can I not mount it.
15:49 baboune according to doc: dmsetup create "${container}_recovery" --table "0 20971520 thin $pool $device_id"
15:49 baboune this requires thje step to recover data to be done first, and I can not do that
15:49 baboune so unless there is another approach?
15:50 ub left #fuel
15:50 mattymo dmsetup create postgres_recovery --table "0 20971520 thin /dev/mapper/docker-253:2-7752-pool 641"
15:51 mattymo that's what it should look like..
15:51 mattymo look at dmesg to see if you have any file system errors
15:51 baboune There is a total of 6 containers with a rootfs on my host
15:51 baboune mattymo: ok that is a different command. let me try
15:52 mattymo the thin pool device and the device_id are not going to match your host though
15:53 mattymo oh! your thin pool isn't a proper string
15:53 baboune that was going to be my next question
15:53 mattymo pool=/dev/mapper/docker*pool
15:53 mattymo then echo $pool and it should be complete
15:56 baboune it failes.. No such file or directory
15:56 baboune device-mapper: reload ioctl on fuel-core-6.0-postgres_recovery failed: No such file or directory Command failed
15:57 mattymo what command are you running?
15:57 baboune maybe the device id is wrong
15:58 baboune dmsetup create fuel-core-6.0-postgres_recovery --table "0 20971520 thin /dev/mapper/docker*pool 375"
15:58 mattymo still wrong
15:58 mattymo your pool is still wrong
15:58 mattymo baboune, ^
15:59 mattymo fix it. it should look like /dev/mapper/docker-253:2-7752-pool
15:59 baboune mattymo: how do I figure out the pool?
16:00 mattymo ls /dev/mapper
16:00 mattymo it's the only one that contains the word pool
16:00 baboune got it!
16:01 baboune new error
16:01 baboune "no data available"
16:01 mattymo ok maybe you have the metadata block for postgres and not the data one
16:02 mattymo check again in  /var/lib/docker/devicemapper/devicemapper/json
16:02 baboune dmsetup create postgres_recovery --table "0 20971520 thin /dev/mapper/docker-253:2-19447-pool 375"
16:02 mattymo you can use dropbox/google drive to send me your json file and I can try to find it
16:05 baboune https://www.dropbox.com/s/opkgdqhxpcbgzlc/fuel-json?dl=0
16:07 baboune it is there two times I think.. 375 and 374
16:08 ub joined #fuel
16:08 baboune the docker-postgresql.log gave me the following container id: 579e6092b55f38b7bce17cf8fde0049c3f0cb163e174393bce679187a1977492
16:11 baboune mattymo: and here is the docker-postgres.log https://dl.dropboxusercontent.com/u/18136096/docker-postgres.log
16:12 mattymo did you try again with 374?
16:12 mattymo baboune, ^
16:12 baboune yes I did.. same thing
16:14 mattymo it's 375 for sure
16:15 baboune mattymo: do u have teamviewer?
16:15 mattymo I really can't get on that with you, I'm sorry
16:15 championofcyrodi looks like my rabbitmq timeouts could have to do with network link utilization being constantly at 100% due to ceph.
16:16 baboune mattymo: np. any other ideas on how to mount it =
16:16 championofcyrodi and since I am using VLANs on one 1gbps connection for management, storage, pxe, etc...
16:16 championofcyrodi packets get dropped cause of lower prioritization or something
16:16 mattymo baboune, I'm pretty sure the data is destroyed. It's possible that docker tried to make a new device. If that's that true there are 2 more places to check for another container ID. Can you send me /var/log/docker* and /var/lib/docker/linkgraph.db files?
16:16 mattymo destroyed or corrupted
16:16 baboune ok. 1s
16:16 mattymo I'll dig through and try to find another container ID
16:17 championofcyrodi wondering if this is worth 'fixing' or just waiting for 0mq update
16:17 mattymo I don't mean docker-postgres, but the rotated /var/log/docker logs
16:19 baboune but not remote right?
16:20 mattymo correct. I am just trying to see if it's in /var/log/docker*gz
16:20 mattymo there might be others
16:28 baboune one of the nodes under /remote had 151GB of logs
16:32 baboune https://dl.dropboxusercontent.com/u/18136096/linkgraph.db
16:32 baboune mattymo: https://dl.dropboxusercontent.com/u/18136096/dockerlogs.tar.gz and thx
16:39 mattymo baboune, sorry, but you missed the docker log itself (and rotated copies). It contains lines like this: time="2015-05-13T16:33:39Z" level="info" msg="+job images()"
16:41 baboune mattymo: I have this: https://dl.dropboxusercontent.com/u/18136096/docker
16:41 mattymo baboune, any rotated copies of it?
16:42 baboune but no rotated copies.  I cleaned those up this morning when trying to free up space
16:42 baboune mattymo: sorry
16:42 baboune mattymo: there is one more container id at the end of linkgraph.db
16:43 mattymo actually it's fine
16:43 baboune mattymo: it gives me a device id of 385
16:43 mattymo the proper ID when you upgraded is 579e6092b55f38b7bce17cf8fde0049c3f0cb163e174393bce679187a like you said
16:44 mattymo let me research the "no data available" issue
16:44 baboune yes, that gives me device id of 375
16:50 mattymo baboune, this is an issue I haven't seen before. It looks like the next stage is to use thin_dump --repair /dev/mapper/docker-253:2-19447-pool   ... but I have no idea if it will work or if it will destroy the other data
16:51 baboune mattymo: ok.  shall I try?
16:51 mattymo well, I can't recommend it because I'm not sure if it will help or make your issues worse
16:52 baboune I have a backup from 4 weeks ago which could be enough to recover hlaf the cluster.  Would doing the repair prevent restoring a backup?
16:52 baboune plus anyhow at this point w/o postgres there is not much I can do
16:52 mattymo correct
16:53 baboune "bad checksum in superblock"
16:53 baboune that was fast
16:53 mattymo if you decide to restore from backup, then you still have to purge the container from the docker sqlite3 db, documented here: https://docs.mirantis.com/openstack/fuel/fuel-6.0/operations.html#id97
16:53 mattymo it won't be able to restore until it's gone
16:54 baboune mattymo: ok.  The "thin_dump --repair /dev/mapper/docker-253:2-19447-pool" failed with "bac checksum in superblock"
16:54 baboune bac=bad
16:54 mattymo yeah
16:54 baboune so it is over right?
16:54 mattymo did you try the thin_check/dump/restore commands in the link?
16:55 mattymo those point to the metadata device, not the pool
16:55 baboune yes. they all passed w/o problems
16:56 baboune I went to the "service docker restart" step w/o any issues
16:56 mattymo sorry, I'm trying to research your issue more in depth
16:56 mattymo it doesn't look like a common error at all
16:57 mattymo baboune, can you check my private message?
16:58 baboune for the sqlite purge, is this required for all containers? cause the doc only covers one container at a time
16:58 mattymo no
16:58 mattymo just for container 579e6092b55f38b7bce17cf8fde0049c3f0cb163e174393bce679187a
17:04 ub joined #fuel
17:05 baboune ok
17:14 tbharath_ joined #fuel
17:18 tbharath_ Hi, I have openstack  installed on my machine using devstack. How can I install openstack fuel?
17:18 tbharath_ I want to work on openstack fuel UI
17:21 tbharath_ I was confused with the documentation
17:24 docaedo tbharath: Fuel is a tool for installing and managing multiple openstack environments, not something that runs inside an openstack environment
17:25 docaedo Great to hear you're interested in working on the UI though! To get started, you can use the FUel ISO (available at fuel-infra.org) to install Fuel on a VM
17:33 xarses joined #fuel
17:37 ub2 joined #fuel
17:58 rongze joined #fuel
18:13 mattgriffin joined #fuel
18:44 e0ne joined #fuel
18:59 rongze joined #fuel
19:01 Longgeek joined #fuel
19:02 Longgeek joined #fuel
19:03 HeOS joined #fuel
19:04 Longgeek joined #fuel
19:33 e0ne joined #fuel
19:59 e0ne joined #fuel
19:59 homegrow_ joined #fuel
19:59 rongze joined #fuel
20:20 mattgriffin joined #fuel
20:34 mattgriffin joined #fuel
20:37 MaverickHunter joined #fuel
20:41 mattgriffin joined #fuel
20:56 MaverickHunter joined #fuel
21:00 rongze joined #fuel
21:03 xMaverickHunterx joined #fuel
21:06 wiza_ joined #fuel
21:09 mattymo_ joined #fuel
21:09 wayneseguin joined #fuel
21:09 daniel3 joined #fuel
21:10 bogdando joined #fuel
21:23 angdraug joined #fuel
21:37 vtzan joined #fuel
21:41 mattgriffin joined #fuel
22:01 rongze joined #fuel
22:32 tzn joined #fuel
22:43 teran joined #fuel
22:59 holser_ joined #fuel
23:02 rongze joined #fuel
23:12 xarses joined #fuel
23:24 rmoe joined #fuel
23:30 mattgriffin joined #fuel
23:43 mattgriffin joined #fuel
23:46 teran_ joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary