Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-06-25

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 Longgeek joined #fuel
00:31 claflico1 joined #fuel
00:39 claflico joined #fuel
01:24 claflico1 joined #fuel
01:31 dhblaz joined #fuel
01:36 claflico joined #fuel
01:45 ub joined #fuel
01:45 LiJiansheng joined #fuel
02:00 claflico joined #fuel
02:22 claflico joined #fuel
02:45 xarses joined #fuel
03:20 Longgeek joined #fuel
03:21 Longgeek joined #fuel
03:45 saibarspeis joined #fuel
04:09 monester_laptop joined #fuel
04:19 LiJiansheng joined #fuel
04:37 Longgeek joined #fuel
04:47 ub joined #fuel
05:17 Longgeek joined #fuel
05:59 saibarspeis joined #fuel
07:05 dancn joined #fuel
07:32 samuelBartel joined #fuel
07:50 ub joined #fuel
07:54 ub2 joined #fuel
08:09 monester_laptop joined #fuel
08:32 e0ne joined #fuel
09:10 e0ne joined #fuel
09:27 hop joined #fuel
09:33 e0ne joined #fuel
09:37 HeOS joined #fuel
09:49 sergmelikyan joined #fuel
10:34 krogon joined #fuel
10:36 sergmelikyan joined #fuel
10:55 e0ne joined #fuel
10:59 sergmelikyan joined #fuel
11:19 ub2 joined #fuel
11:28 e0ne joined #fuel
11:47 hyperbaba joined #fuel
11:58 stamak joined #fuel
12:06 ub2 joined #fuel
12:33 hammondr joined #fuel
12:41 hammondr I upgraded from fuel 6.0 to fuel 6.1 and seem to have lost my configured environment
12:41 hammondr could anyone here help me with that, please?
12:42 hammondr specifically, I ran the upgrade.sh script. it said it was successful.  I reloaded the fuel webui and didn't see my environment.  I rebooted all the nodes that had been configured in fuel 6.0 and they booted into bootstrap mode rather than booting their pre-existing CentOS
12:50 husanu2 joined #fuel
12:56 husanu1 joined #fuel
13:07 dhblaz joined #fuel
13:13 husanux3 joined #fuel
13:15 husanux4 joined #fuel
13:20 husanux7 joined #fuel
13:25 husanux2 joined #fuel
13:53 hammondr could anyone here help me with a fuel 6.0 to 6.1 upgrade question?
14:36 e0ne joined #fuel
14:45 rmoe joined #fuel
15:04 claflico joined #fuel
15:04 MiroslavAnashkin hammondr: Could you please share the log from upgrade.sh?
15:04 hammondr if I can find it
15:04 evgeniyl___ hammondr: /var/log/fuel_upgrade.log or something like that
15:10 hammondr found it. I'll paste it
15:11 MiroslavAnashkin Anyway, your old containers with the DB and env should be backed up.  Please do not wipe the disks on the available bootstrapped nodes.
15:12 hammondr MiroslavAnashkin: I won't intentionally wipe anything
15:12 hammondr could they have been wiped simply by booting into bootstrap mode?
15:14 hammondr http://paste.openstack.org/show/320995/
15:15 hammondr oops...that paste looks incomplete
15:17 hammondr am I doing something wrong, or can I not paste the entire log to paste.openstack.org ?
15:20 MiroslavAnashkin They may allow to paste first 500 strings only.
15:21 hammondr is there a recommended place to upload files for you to take a look at?
15:22 MiroslavAnashkin You may use google drive, dropbox, box.com or any similar free file share server.
15:23 hammondr http://pastebin.com/MYUCZGMA
15:28 sergmelikyan joined #fuel
15:29 angdraug joined #fuel
15:35 evgeniyl___ hammondr: could you please confirm, that on the master, you have db dump with all of the data which you had had before the upgrade, the path to you db backup is /var/lib/fuel_upgrade/6.1/pg_dump_all.sql
15:39 hammondr evgeniyl___: yes, that file exists
15:42 evgeniyl___ hammondr: could you please clarify, did you run http://paste.openstack.org/show/321031/ commands after the upgrade? Don't run these commands, for now.
15:45 izombix joined #fuel
15:50 hammondr I did run them. they seemed to succeed, though it took several minuts
15:55 ikalnitsky joined #fuel
15:59 kozhukalov_ joined #fuel
16:00 Longgeek joined #fuel
16:00 sergmelikyan joined #fuel
16:00 ikalnitsky o
16:01 evgeniyl___ hammondr: ok, we have theories what could happen, is it production env? Do you need help with restoring of your environment?
16:01 hammondr it is not production, but not exactly throw-away either
16:02 hammondr it would be nicest to restore the env but keep 6.1 (for testing).  2nd nicest is reverting, but then I'm no smarter.
16:02 xarses_ joined #fuel
16:06 akupko joined #fuel
16:06 evgeniyl___ hammondr: first, lets restore cobbler data, run the next command: dockerctl shell 4f48b2bde19e22b60a65b4efd6a55d44e1bb5ac6f663ded9913b57ea81b3f0e7 bash -c 'cp -rn /tmp/upgrade/cobbler_configs/config/* /var/lib/cobbler/config/'
16:07 hammondr Could not get docker ID for container 4f48b2bde19e22b60a65b4efd6a55d44e1bb5ac6f663ded9913b57ea81b3f0e7. Is it running?
16:08 evgeniyl___ Paste output of "docker ps" command
16:10 evgeniyl___ hammondr: ^
16:11 ikalnitsky looks like containers are down
16:11 sergmelikyan joined #fuel
16:11 e0ne joined #fuel
16:11 evgeniyl___ ikalnitsky: I don't think so, ids got changed after patching with yum stuff
16:11 evgeniyl___ ikalnitsky: I think so)
16:12 hammondr [root@fuel etc]# docker ps
16:12 hammondr CONTAINER ID        IMAGE                         COMMAND                CREATED             STATUS              PORTS               NAMES
16:12 hammondr 15f55d6cd948        fuel/mcollective_6.1:latest   "/usr/local/bin/star   19 hours ago        Up 3 hours                              fuel-core-6.1-mcollective
16:12 hammondr 2600ec832e67        fuel/cobbler_6.1:latest       "/bin/sh -c /usr/loc   19 hours ago        Up 3 hours                              fuel-core-6.1-cobbler
16:12 hammondr f200f6fea133        fuel/nginx_6.1:latest         "/usr/local/bin/star   19 hours ago        Up 3 hours                              fuel-core-6.1-nginx
16:12 hammondr 906edc7241cc        fuel/ostf_6.1:latest          "/bin/sh -c /usr/loc   19 hours ago        Up 3 hours                              fuel-core-6.1-ostf
16:12 hammondr ae85c4e2db2a        fuel/nailgun_6.1:latest       "/bin/sh -c /usr/loc   19 hours ago        Up 3 hours                              fuel-core-6.1-nailgun
16:12 hammondr a7290eed5603        fuel/rsyslog_6.1:latest       "/usr/local/bin/star   19 hours ago        Up 3 hours                              fuel-core-6.1-rsyslog
16:12 hammondr 93f754960f1c        fuel/astute_6.1:latest        "/bin/sh -c /usr/loc   19 hours ago        Up 3 hours                              fuel-core-6.1-astute
16:12 hammondr e94be3ec55ba        fuel/rsync_6.1:latest         "/bin/sh -c /usr/loc   19 hours ago        Up 3 hours                              fuel-core-6.1-rsync
16:12 hammondr 7c0abb64d4d1        fuel/keystone_6.1:latest      "/usr/local/bin/star   19 hours ago        Up 3 hours                              fuel-core-6.1-keystone
16:12 hammondr 5e3e8d5a4769        fuel/rabbitmq_6.1:latest      "/usr/local/bin/star   19 hours ago        Up 3 hours                              fuel-core-6.1-rabbitmq
16:12 hammondr 4aa922d5e1e0        fuel/postgres_6.1:latest      "/usr/local/bin/star   19 hours ago        Up 3 hours                              fuel-core-6.1-postgres
16:13 evgeniyl___ hammondr: dockerctl shell 2600ec832e67 bash -c 'cp -rn /tmp/upgrade/cobbler_configs/config/* /var/lib/cobbler/config/'
16:14 hammondr d'oh... cp: cannot stat `/tmp/upgrade/cobbler_configs/config/*': No such file or directory
16:14 hammondr I rebooted the fuel node 3+ hrs ago ... might that have wiped /tmp ?
16:16 evgeniyl___ hammondr: no, I don't think so
16:17 evgeniyl___ ikalnitsky: did we move cobbler stuff to host system?
16:17 hammondr [root@fuel etc]# ll /tmp/
16:17 hammondr total 700
16:17 hammondr -rwx------. 1 root root    480 Apr  6 17:54 ks-script-wN_dVQ
16:17 hammondr -rwxr-xr-x. 1 root root      0 Apr  6 17:54 ks-script-wN_dVQ.log
16:17 hammondr -rwx------. 1 root root   8762 Apr  6 17:54 ks-script-YFMN1O
16:17 hammondr -rw-r--r--  1 root root      0 Jun 25 16:01 logrotate
16:17 hammondr -rw-------. 1 root root      0 Apr  6 17:42 yum.log
16:17 hammondr -rw-------  1 root root    478 Jun 24 19:53 yum_save_tx-2015-06-24-19-53vJozGz.yumtx
16:17 hammondr -rw-------  1 root root  13894 Jun 24 20:28 yum_save_tx-2015-06-24-20-28LI0ogo.yumtx
16:17 hammondr -rw-------  1 root root  13951 Jun 24 20:29 yum_save_tx-2015-06-24-20-29CEdjgO.yumtx
16:17 hammondr -rw-------  1 root root  13951 Jun 24 20:30 yum_save_tx-2015-06-24-20-30pXmwbf.yumtx
16:18 ikalnitsky evgeniyl___, we should move. it as so time ago and i don't remember any changes
16:18 ikalnitsky but..
16:18 ikalnitsky i've met strange bug once
16:18 ikalnitsky by some reason, the /var/lib/fuel_upgrade/ wasn't mounted to /tmp/upgrade inside container
16:19 ikalnitsky i think we need to check this data on the host system first
16:19 ikalnitsky hammondr, could you please check it?
16:19 ikalnitsky ls /var/lib/fuel_upgrade/6.1
16:20 hammondr [root@fuel etc]# ls /var/lib/fuel_upgrade/6.1
16:20 hammondr cobbler_configs  pg_dump_all.sql  pg_dump_all.sql.1
16:20 ikalnitsky Oh! Got it!
16:20 ikalnitsky evgeniyl___, he performed 'dockerctl destroy all' which means we don't mount this folder from host into container
16:21 evgeniyl___ ikalnitsky: yeah, I got it.
16:21 evgeniyl___ ikalnitsky: I'm thinking how to get data from host into the container
16:21 evgeniyl___ ikalnitsky: I'm not sure if "/var/lib/fuel/container_data" directory is mounted
16:22 evgeniyl___ hammondr: could you please run: docker inspect 2600ec832e67
16:36 hammondr http://paste.openstack.org/show/321058/
16:46 claflico joined #fuel
16:52 MiroslavAnashkin evgeniyl___: If container is running - `dockerctl copy`
16:54 hammondr is that a complete command?
16:55 MiroslavAnashkin No, but it has built-in help
16:55 evgeniyl___ MiroslavAnashkin: what is this for?
16:55 MiroslavAnashkin To copy something from host to container
16:56 hammondr [root@fuel etc]# dockerctl copy
16:56 hammondr This command requires two parameters. See usage:
16:56 hammondr /usr/bin/dockerctl copy src dest
16:56 hammondr Examples:
16:56 hammondr /usr/bin/dockerctl copy nailgun:/etc/nailguns/settings.yaml /root/settings.yaml
16:56 hammondr /usr/bin/dockerctl copy /root/newpkg.rpm mcollective:/root/
16:57 hammondr MiroslavAnashkin: I'm not sure what you want me to copy where.  (I'm not a docker user)
16:57 MiroslavAnashkin Yes, both src and dest may be a <container>:<path inside container> or path in the host file system
16:57 hammondr and would it be best to restart the docker container and hope it mounts, or copy the files in?
16:59 MiroslavAnashkin If container starts - you may copy the old data from host to container (EvgenyL will provide what to copy)
17:01 xarses joined #fuel
17:08 evgeniyl___ hammondr: you need to copy data from /var/lib/fuel_upgrade/6.1/cobbler_configs/config to /var/lib/cobbler/config/
17:08 evgeniyl___ hammondr: after that restart your cobbler container, after it's up, try to reboot your node, they should get back to normal state
17:10 evgeniyl___ hammondr: then restore your database copy /var/lib/fuel_upgrade/6.1/pg_dump_all.sql into the container using the command provided by MiroslavAnashkin, and run restore su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"
17:11 evgeniyl___ hammondr: after that restart your nailgun container, it'll apply db migrations
17:11 hammondr ok
17:12 sergmelikyan joined #fuel
17:13 hammondr could you pls help more with the 'docker copy' command?  I tried "dockerctl copy /var/lib/fuel_upgrade/6.1/cobbler_configs/config fuel-core-6.1-cobbler:/var/lib/cobbler/config/" but I get a "docker inspect" usage error
17:17 MiroslavAnashkin `dockerctl copy /var/lib/fuel_upgrade/6.1/cobbler_configs/config cobbler:/var/lib/cobbler/config/`
17:17 MiroslavAnashkin dockerctl uses short container names
17:19 MiroslavAnashkin Or - alternative way: /var/www/nailgun/ is mounted to every container exactly to the same path. Everything in /var/www/nailgun gets automaticalyl shared between containers.
17:19 hammondr thx. that dockerctl copy command worked / did not error
17:20 hammondr "dockerctl restart cobbler" didn't complain either
17:21 MiroslavAnashkin OK. Then, please reboot one of your old nodes to check if it goes to bootstrap
17:23 MiroslavAnashkin Or, check what has been copied. `dockerctl shell cobbler ls -la /var/lib/cobbler/config/`
17:23 hammondr [root@fuel etc]# dockerctl shell cobbler ls -la /var/lib/cobbler/config/
17:23 hammondr total 4
17:23 hammondr drwxr-xr-x 10 root root  135 Jun 24 20:45 .
17:23 hammondr drwxr-xr-x 10 root root 4096 Jun 24 20:46 ..
17:23 hammondr drwxr-xr-x  2 root root   82 Jun 24 20:46 distros.d
17:23 hammondr drwxr-xr-x  2 root root    6 Oct 20  2014 files.d
17:23 hammondr drwxr-xr-x  2 root root    6 Oct 20  2014 images.d
17:23 hammondr drwxr-xr-x  2 root root    6 Oct 20  2014 mgmtclasses.d
17:23 hammondr drwxr-xr-x  2 root root    6 Oct 20  2014 packages.d
17:23 hammondr drwxr-xr-x  2 root root   82 Jun 24 20:46 profiles.d
17:23 hammondr drwxr-xr-x  2 root root    6 Oct 20  2014 repos.d
17:23 hammondr drwxr-xr-x  2 root root   25 Jun 24 20:46 systems.d
17:24 MiroslavAnashkin OK, old data is in place.
17:25 eJunky joined #fuel
17:26 eJunky Hi, can anybody  explain this message to me? 400 Client Error: Bad Request (Deployment tasks not found for 'Juno on Ubuntu 14.04.1' release in the database. Please upload them. If you're operating from Fuel Master node, please check '/etc/puppet' directory.)
17:26 hammondr I've restarted a node.  I'll report back (after lunch).  But it sounds like I still need to restore the postgresql db and restart the nailgun container. yes?
17:27 MiroslavAnashkin Yes.
17:28 hammondr the fuel webui still shows no enviornments ... should that be fixed after the sql/nailgun steps?
17:33 MiroslavAnashkin Yes, envs are in postgres container, inNailgun DB.
17:33 MiroslavAnashkin Please check this chapter first, there are postgres dump and restore procedures: https://docs.mirantis.com/openstack/fuel/fuel-6.1/operations.html#docker-metadata-corruption-loses-containers
17:34 sergmelikyan joined #fuel
17:34 MiroslavAnashkin Then, please backup existing postgres DB with
17:35 MiroslavAnashkin dockerctl shell postgres su - postgres -c 'pg_dumpall --clean' /root/postgres_backup_$(date +"%F-%T").sql
17:38 MiroslavAnashkin Then, please check what is in the 6.0 backup postgres dump file - it may have DB drop commands or may not.
17:39 hammondr I ran ...  dockerctl shell postgres bash -c 'su postgres -c "psql -f /tmp/pg_dump_all.sql postgres"' ... before I saw your note about dumping to a backup file
17:43 MiroslavAnashkin Well, if I remember correct, it is always possible to upgrade old postgres DB one more time.
17:43 claflico joined #fuel
17:45 hammondr thx for your help ... I'll report back after lunch
17:45 DrSlump What's the best way to report error in documentation?
17:48 MiroslavAnashkin DrSlump: Please file new bug with [docs] tag to https://bugs.launchpad.net/fuel/ project, like following: https://bugs.launchpad.net/fuel/+bug/1361282
17:53 xarses joined #fuel
18:02 dontalton joined #fuel
18:20 sergmelikyan joined #fuel
18:48 hammondr MiroslavAnashkin evgeniyl___: the postgres restore completed. I rebooted all nodes, and they are at bootstrap.  the fuel webui does show my old environment as "operational", but it shows 0 nodes allocated to it and 14 unallocated nodes.
18:48 hammondr should I restart the nailgun container or do something else?
18:52 rbrooker joined #fuel
18:53 rbrooker awesome I was hoping to find some info on 6.1
18:53 MiroslavAnashkin Please restart nailgun and nginx containers
18:54 rbrooker Q: am I doing something really wrong, the setup fails at setup on the dhcp check
18:54 rbrooker and my static IPs are not being accepted
18:55 rbrooker Oh yeah, its a 6.1 on vmware vm w
18:55 rbrooker with 2 nics
18:56 MiroslavAnashkin rbrooker: It means some DHCP server answered to Fuel Master inside the Admin/PXE network segment, connected to Fuel eth0.
18:57 hammondr MiroslavAnashkin: I restarted the nailgun and nginx containers. no change in webui
18:57 MiroslavAnashkin hammondr: What does `fuel node list` says?
18:58 hammondr a bunch of nodes in discover status
18:58 hammondr cluster = none; group_id = None, online = True
18:59 MiroslavAnashkin hammondr: Please reboot one of bootstrapped node one more time.
19:00 hammondr ok
19:00 claflico1 joined #fuel
19:00 MiroslavAnashkin hammondr: Bootstrap has special marker so master node always understands if the node bootstrapped or boot to the target installed OS.
19:00 hammondr incidentally, any idea how all of this happened?
19:01 hammondr more info: webui shows the old evn name, but nothing happens when I click on it.
19:01 hammondr it would normally take me to the config options, but not now.
19:01 hammondr node seems to be rebooting into local...
19:02 MiroslavAnashkin hammondr: Ah, we didn't upgraded DB schema after restoring irt from the dump.
19:03 hammondr MiroslavAnashkin: do you know a command I should run?
19:04 hammondr ... and rebooted node did NOT boot local. it is at bootstrap login
19:12 MiroslavAnashkin hammondr: Please run `dockerctl shell cobbler cobbler sync` Doubled "cobbler" is correct here.
19:14 hammondr *** TASK COMPLETE ***
19:15 MiroslavAnashkin Then, please run `dockerctl shell cobbler cobbler system list`
19:16 hammondr [root@fuel etc]# dockerctl shell cobbler cobbler system list
19:16 hammondr default
19:17 MiroslavAnashkin So, no systems in cobbler...
19:17 MiroslavAnashkin Please run this one more time: `dockerctl copy /var/lib/fuel_upgrade/6.1/cobbler_configs/config cobbler:/var/lib/cobbler/config/`
19:18 hammondr ok. I ran it. no output.
19:18 MiroslavAnashkin then `dockerctl restart cobbler` and then `dockerctl shell cobbler cobbler system list`
19:18 hammondr is that supposed to copy in files from a default 6.1? or from my old 6.0 backup?
19:19 MiroslavAnashkin From old backup
19:19 hammondr [root@fuel etc]# dockerctl shell cobbler cobbler system list
19:19 hammondr default
19:21 sergmelikyan joined #fuel
19:22 MiroslavAnashkin Then `dockerctl shell cobbler ls -l /var/lib/cobbler/config/systems.d/`
19:24 hammondr [root@fuel etc]# dockerctl shell cobbler ls -l /var/lib/cobbler/config/systems.d/
19:24 hammondr total 4
19:24 hammondr -rw-r--r-- 1 root root 1236 Jun 25 19:19 default.json
19:25 hammondr but I do see the old files in /var/lib/fuel_upgrade/6.1/cobbler_configs/config, e.g. /var/lib/fuel_upgrade/6.1/cobbler_configs/config/systems.d/node-13.json
19:26 MiroslavAnashkin Hmm... dockerctl somehow failed to copy these files. OK, let us use alternative way.
19:28 MiroslavAnashkin `mkdir /var/www/nailgun/config6.0; cp -r /var/lib/fuel_upgrade/6.1/cobbler_configs/* /var/www/nailgun/config6.0/`
19:29 hammondr ok
19:30 MiroslavAnashkin `dockerctl shell cobbler cp -r /var/www/nailgun/config6.0/config/* /var/lib/cobbler/config/`
19:30 hammondr that worked
19:30 MiroslavAnashkin and `dockerctl shell cobbler ls -l /var/lib/cobbler/config/systems.d/`
19:31 hammondr node files are there
19:31 hammondr [root@fuel etc]# dockerctl shell cobbler cobbler system list
19:31 hammondr default
19:31 hammondr node-13
19:31 hammondr node-15
19:31 hammondr node-16
19:31 hammondr node-17
19:31 hammondr node-18
19:31 hammondr node-19
19:31 hammondr node-20
19:31 hammondr node-21
19:31 hammondr node-22
19:31 hammondr node-23
19:31 hammondr node-24
19:31 hammondr node-26
19:32 MiroslavAnashkin After, please run `dockerctl restart cobbler`
19:32 hammondr done
19:32 MiroslavAnashkin and `dockerctl restart nailgun; dockerctl restart nginx;`
19:32 MiroslavAnashkin and `fuel node list`
19:34 hammondr "fuel node list" does not seem to have different output
19:34 hammondr but a rebooted node seems to be booting normally (not bootstrap)
19:35 MiroslavAnashkin Please run `dockerctl shell cobbler cobbler sync` one more time and then
19:35 monester_laptop joined #fuel
19:35 MiroslavAnashkin `dockerctl shell cobbler ls -l /var/lib/cobbler/config/systems.d/`
19:36 hammondr done. I see node files.
19:36 hammondr "fuel node list" still has output like...
19:37 hammondr 11 | discover | Untitled (30:f7) | None    | 10.12.183.22 | 00:1e:67:6b:30:f7 |       |               | True   | None
19:37 hammondr 12 | discover | Untitled (d2:61) | None    | 10.12.183.7  | c6:5d:29:9e:45:40 |       |               | True   | None
19:37 MiroslavAnashkin If systems remained - cobbler may be considered as restored. Only Astute and postgres left.
19:37 MiroslavAnashkin And yes, these nodes already appeared in the DB twice, one time as discovered.
19:38 hammondr what should "fuel node list" show at this point?
19:38 MiroslavAnashkin Shortest way is to restore postgres DB from dump one more time
19:38 MiroslavAnashkin From 6.0 dump
19:38 hammondr ok; should I stop any containers?
19:39 MiroslavAnashkin No, only restart nailgun, nginx, keystone and ostf after restoring from DB dump
19:39 stamak joined #fuel
19:39 hammondr do I need to drop the db or do anything different this time? should I only run "su postgres -c "psql -f /tmp/pg_dump_all.sql postgres" again?
19:40 claflico joined #fuel
19:41 MiroslavAnashkin `dockerctl shell postgres su - postgres -c "psql nailgun" < <path_to_dump.sql>`
19:42 hammondr was I not supposed to import that file into the _postgres_ db earlier?
19:42 hammondr meaning: should I have imported it into the nailgud db rather than the postgres db?
19:43 MiroslavAnashkin Oh, yes, please copy old dump file to /var/www/nailgun/ and import it with
19:44 MiroslavAnashkin `dockerctl shell postgres su - postgres -c "psql nailgun" < /var/www/nailgun/dump.sql`
19:48 hammondr ok; that is running slowly now
19:50 MiroslavAnashkin If you restarted containers after DB import with `dockerctl restart nailgun nginx keystone ostf` - please run `fuel node list` one more time
19:51 MiroslavAnashkin And check what is in `ls -l /var/lib/fuel_upgrade/6.1/` - I don't remember where old node keys for astute are backed up on upgrade
19:54 hammondr postgres imports (to postgres and nailgun dbs) is not yet finished
19:56 kaliya joined #fuel
20:04 MiroslavAnashkin Hmm, what is the size of the dump?
20:04 hammondr it finished a few min ago.  I'm running "dockerctl restart nailgun nginx keystone ostf" now
20:05 hammondr [root@fuel 6.1]# ll /var/www/nailgun/pg_dump_all.sql
20:05 hammondr -rw-r--r-- 1 root root 1901964 Jun 25 19:45 /var/www/nailgun/pg_dump_all.sql
20:06 hammondr "fuel node list" output seems unchanged.
20:06 hammondr [root@fuel etc]# ls -l /var/lib/fuel_upgrade/6.1/
20:06 hammondr total 3720
20:06 hammondr drwxr-xr-x 3 root root      19 Jun 24 19:57 cobbler_configs
20:06 hammondr -rw-r--r-- 2 root root 1901964 Jun 24 19:57 pg_dump_all.sql
20:06 hammondr -rw-r--r-- 2 root root 1901964 Jun 24 19:57 pg_dump_all.sql.1
20:08 MiroslavAnashkin Hmm. What is in `dockerctl shell astute ls /var/lib/fuel/keys/` ?
20:08 hammondr [root@fuel etc]# dockerctl shell astute ls /var/lib/fuel/keys/
20:08 hammondr 1  2
20:12 MiroslavAnashkin OK, please run `dockerctl shell cobbler cobbler sync` one more time
20:12 MiroslavAnashkin Then, please try to ssh to any existing node
20:12 hammondr ok
20:12 MiroslavAnashkin And check `fuel node list` one more time
20:13 hammondr fuel node list seems unchanged
20:13 hammondr /etc/hosts was modified on fuel master, so nodes names aren't there
20:13 hammondr what do you want me to do after ssh'ing to a node?
20:13 MiroslavAnashkin ssh node-XXX
20:14 MiroslavAnashkin There should be no deployed node names in /etc/hosts on master node
20:14 hammondr [root@fuel etc]# ssh node-13
20:14 hammondr ssh: Could not resolve hostname node-13: Name or service not known
20:15 hammondr never mind!
20:15 hammondr I had tweaked /etc/resolv.conf ... I put it back.  I can ssh by name now.
20:15 hammondr did you want me to test anything on a node?
20:17 MiroslavAnashkin Normally dnsmasq should redirect requests to the nodes properly. it lives inside the cobbler container. Please check the contents of its conf file with
20:18 MiroslavAnashkin `dockerctl shell cobbler cat /etc/dnsmasq.conf | less`
20:19 hammondr I see the hosts in the dnsmasq config
20:21 hammondr "cobbler system list" shows the old nodes (+ default)
20:21 hammondr "fuel node list" shows a bunch in "discover" mode ... I'm not sure what that should show
20:21 hammondr and the webUI still seems pretty broken... I can't see the config for the old env (which still shows 0 nodes) nor can I create a new environment.
20:24 HeOS joined #fuel
20:24 MiroslavAnashkin Probably DB upgrade procedure is hidden somewhere in upgrade script. Looking into source code and see nothing...
20:37 MiroslavAnashkin Well, DB upgrade lives right inside Nailgun,  here: https://github.com/stackforge/fuel-web/blob/stable/6.1/nailgun/nailgun/utils/migration.py
20:37 MiroslavAnashkin So, we need to retrigger DB upgrade
20:38 hammondr is that something I need to do by hand?
20:40 hammondr btw, thanks for all of your help today.
20:41 MiroslavAnashkin Hmm. could you please reboot master node?
20:42 hammondr boom
20:43 MiroslavAnashkin Thats interesting. Once I had to construct my own DB dump manually due to no backup available - and even it worked.
20:50 hammondr I rebooted.  fuel node list does not seem better
20:50 hammondr "cobbler system list" does show nodes
20:51 hammondr can we please pick this up tomorrow?
20:51 hammondr also, do you have any idea why this happened?  it sounded like you saw something in the upgrade log that tipped you off.
20:53 MiroslavAnashkin evgeniyl___ and ikalnitsky saw. They are authors after all. So far only guessed on what might have happened from them both
20:53 hammondr ok; thx
20:53 MiroslavAnashkin s/guessed/guesses/
20:57 MiroslavAnashkin Well. One more option still possible. Please store all contents of var/lib/fuel_upgrade/ somewhere else. For the case.
20:58 MiroslavAnashkin Optionally, you may backup docker containers with `dockerctl backup`
20:59 MiroslavAnashkin Please check if there is enough free space in /var on master node
20:59 hammondr plenty of room
20:59 hammondr [root@fuel ~]# df -h
20:59 hammondr Filesystem           Size  Used Avail Use% Mounted on
20:59 hammondr /dev/mapper/os-root  9.6G  1.1G  8.0G  12% /
20:59 hammondr tmpfs                127G     0  127G   0% /dev/shm
20:59 hammondr /dev/sda3            190M   50M  130M  28% /boot
20:59 hammondr /dev/mapper/os-var   823G   47G  777G   6% /var
21:00 hammondr [root@fuel ~]# du -sh /var/lib/fuel_upgrade/
21:00 hammondr 2.0M    /var/lib/fuel_upgrade/
21:00 hammondr ... I'll be back in 16 hours.  thx again for your help.
21:00 MiroslavAnashkin And try to run the upgrade script one more time. Not sure how well it works, but at least was designed to run multiple times if something fails.
21:00 MiroslavAnashkin OK, see you tomorrow then)
21:02 hammondr do you think it's safe to run even though fuel webui shows no nodes in the old env?
21:05 e0ne joined #fuel
21:09 MiroslavAnashkin If there is backup of old data  - should be quite safe
21:11 MiroslavAnashkin One copy already exists in /var/www/nailgun if you did not removed it
21:13 stamak joined #fuel
21:21 Longgeek joined #fuel
22:32 dontalton joined #fuel
22:37 Longgeek joined #fuel
23:22 rmoe joined #fuel
23:30 sanek joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary