Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-03-18

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 Longgeek joined #fuel
00:26 rmoe joined #fuel
00:50 xarses joined #fuel
00:59 Longgeek joined #fuel
01:46 Rodrigo_US joined #fuel
01:58 LiJiansheng joined #fuel
02:22 IlyaE joined #fuel
02:31 IlyaE joined #fuel
03:31 IlyaE joined #fuel
04:06 melafefon_ joined #fuel
04:16 IlyaE joined #fuel
04:30 zerda joined #fuel
04:30 Longgeek joined #fuel
04:38 Longgeek joined #fuel
05:02 IlyaE joined #fuel
05:15 CheKoLyN joined #fuel
05:41 gongysh joined #fuel
06:40 sambork joined #fuel
06:46 dklepikov joined #fuel
06:52 monester_laptop joined #fuel
06:59 adanin joined #fuel
06:59 sovsianikov joined #fuel
07:05 IlyaE joined #fuel
08:13 alecv joined #fuel
08:32 sambork joined #fuel
08:46 monester_laptop joined #fuel
09:19 e0ne joined #fuel
09:20 teran joined #fuel
09:28 HeOS joined #fuel
09:37 kaliya joined #fuel
09:42 omolchanov joined #fuel
09:45 sambork joined #fuel
09:55 jskarpet joined #fuel
09:56 jskarpet Fuel nodes hanging at: "Cannot get disk parameter", and stopping of deployment seems to take forever.. what can I do?
10:01 tzn joined #fuel
10:11 kaliya hi jskarpet do you see all the disks in the graphical partition manager? (fuel UI)
10:12 sanek joined #fuel
10:12 xek joined #fuel
10:14 omolchanov joined #fuel
10:15 jskarpet yes, but on the first discovery they came up without any disks added
10:17 kaliya maybe the hardware controller is not recognized, do you have info in the master node /var/log/messages?
10:18 jskarpet I have 2 identical machine already in the cluster
10:18 kaliya no sorry, not the right place to look into. You should look into the local messages
10:18 jskarpet I wanted to add 3 more
10:18 kaliya 6.0?
10:18 jskarpet yes
10:22 kaliya jskarpet: can you check on the master node this /var/log/docker-nailgun.log and look at that error, if it reports more details?
10:23 jskarpet no errors
10:23 jskarpet the "Cannot get disk parameter" is from the PXE boot
10:24 jskarpet it never loads any image
10:30 jskarpet How can I override the "stop deployment" step, and remove the newly added nodes?
10:31 kaliya jskarpet: you have to stop the fuel task, in the CLI
10:31 kaliya `fuel task`, find the task that's currently deploying, and kill it `fuel task delete -t <ID>`
10:32 kaliya then you have to reboot the nodes, but if they don't start from PXE with that error (never seen) looks pretty strange
10:32 jskarpet 400 Client Error: Bad Request (You cannot delete running task manually)
10:32 kaliya is the master node healthy? is it working I mean? UI, fuel commands, disk space
10:33 jskarpet yes
10:33 kaliya df -h?
10:33 jskarpet 11%
10:33 kaliya does `fuel task` report the actual deployment attempt?
10:33 kaliya with which status and progress?
10:33 jskarpet progress = 0
10:34 jskarpet running | stop_deployment
10:34 kaliya did you click the button "stop deployment" from the UI?
10:34 jskarpet yes
10:34 jskarpet after the nodes failed to boot from PXE
10:35 kaliya Does `fuel --env <ID> nodes` report those problematic nodes in the env?
10:35 adanin joined #fuel
10:35 jskarpet Yes
10:35 jskarpet they're listed as "provisioning"
10:35 kaliya So deployment has not stopped...
10:36 jskarpet nor started it seems ^^
10:36 kaliya Do you have a deployment task running yet?
10:36 jskarpet yes
10:36 jskarpet also progress = 0
10:36 jskarpet at the previous task id
10:36 kaliya Try to `fuel delete -t <ID>`
10:36 kaliya In case, with -f (force)
10:37 jskarpet deployment or stop deployment?
10:37 kaliya The deployment
10:37 kaliya Probably it didn't get the latest status from the PXE nodes, because they didn't PXEboot
10:38 e0ne joined #fuel
10:39 jskarpet stop_deployment still stuck
10:40 jskarpet is it safe to stop with -f ?
10:40 kaliya It will stop the current deployment task (adding 3 new nodes)
10:42 jskarpet Was thinking about the "stop_deployment" and "deploy, progress=29" ones
10:42 kaliya Yes especially the second
10:42 jskarpet "deploy" task is safe to kill with -f ?
10:42 kaliya jskarpet: check here /var/log/docker-cobbler.log do you see errors?
10:43 kaliya Yes, if there is no other way to kill this deployment
10:44 jskarpet I just want to be sure that I don't mess up the rest of the nodes
10:45 kaliya You'll come back at the situation before adding the last 3 ones
10:45 kaliya But your error is intriguing since I cannot find anything suitable even in the Fuel source code
10:47 jskarpet Gets ip from cobbler during PXE
10:47 jskarpet Trying to load: pxelinux.cfg/mac-stuff
10:47 jskarpet Cannot get disk parameters
10:47 jskarpet boot:
10:47 jskarpet <stop>
10:47 kaliya jskarpet: a good idea would be trying to boot other images yes
10:50 jskarpet after the forceful killing of tasks I seem to be unable to delete the nodes from the environment using GUI as well
10:51 kaliya jskarpet: remove them from CLI
10:52 jskarpet 400 Client Error: Bad Request (Can't unassign nodes with ids <nailgun.db.sqlalchemy.models.node.Node object at 0x3fd3550>,<nailgun.db.sqlalchemy.models.node.Node object at 0x3fd32d0>,<nailgun.db.sqlalchemy.models.node.Node object at 0x3bd3150> if they not assigned.)
10:53 kaliya `dockerctl check` especially postgres
10:54 kaliya evg: jskarpet has a stuck env after adding 3 nodes and stopping the task
10:54 dklepikov joined #fuel
10:55 kaliya jskarpet: are the tasks completely stopped now? do you have others in progress?
10:55 jskarpet no issues as far as I can see
10:55 jskarpet postgres is ready.
10:55 evg kaliya: hi, i'm reading, trying to get what's happening...
11:02 evg jskarpet: and now you're trying to delete the node?
11:02 evg jskarpet: deleting a task can take some time, let's wait a little
11:04 jskarpet I want to add nodes, but that failed
11:04 jskarpet so now I want to try and rediscover them
11:05 jskarpet Before attempting another provisioning step
11:09 evg jskarpet: so you've delete the node sucsessfully?
11:31 jskarpet yes, it just took a while
11:33 evg jskarpet: and what the current problem is? sorry for dullness
11:33 jskarpet Original problem was that PXE boot didn't succeed
11:34 jskarpet They get answer from cobbler, but was stuck with error: "Cannot get disk parameter"
11:34 evg jskarpet: and now?
11:35 jskarpet Still power cycling
11:36 jskarpet I did a reboot, to re-init discovery
11:36 jskarpet pending deployment now
11:36 jskarpet Seems to work now =)
11:37 jskarpet So the issue that remains is: deployment failed to boot node, abort deployment through GUI never completed
11:37 evg jskarpet: nice
11:37 jskarpet The first one I suspect is due to inserting hdd after first discovery
11:38 jskarpet and as such it tried deploying to a node without hdd
11:38 jskarpet not sure how well that is handled
11:38 jskarpet second one came around using -f flag from CLI
11:40 evg jskarpet: I suppose it was because the node stuck in cobbler.
11:40 evg jskarpet: hadn't been deleted from cobbler
11:42 jskarpet I want to replace two controller nodes with two newer nodes. Should I delete, then add new - or add new (complete deployment), then delete old?
11:42 evg jskarpet: you can add hdd to bootstraped node it will be rediscovered after reboot. Not a problem.
11:43 jskarpet so extending hdd on ceph nodes - I would have to delete and discover them again?
11:45 ChrisNBlum joined #fuel
11:45 evg jskarpet: about controllers I'm not sure, Both the way seem similar to me.
11:47 jskarpet one of them leaves a single controller in operation
11:47 jskarpet where the other one gets 5 in operation before removing
11:47 jskarpet but not sure how clusters across 5 controllers impacts compared to 1 standalone
11:47 jskarpet or maybe add/remove one by one?
11:48 evg jskarpet: ceph node - you shouldn't delete an existed ceph node, there is a well discribed procedure for adding disks to ceph.
11:48 bjoernfan Question: Can bridge_mappings in /etc/neutron/plugin.ini have any name or does it have to be something like physnet1:br-foo and not phy-sysm1:br-foo?
11:49 bjoernfan We're trying to make sense of some issues we're having. :)
11:50 jskarpet evg: URI?
11:59 e0ne joined #fuel
12:02 Miouge joined #fuel
12:05 kaliya bjoernfan: can you please ask in #fuel-dev? I'm not sure
12:09 evg jskarpet: http://dachary.org/?p=2428 or http://ceph.com/docs/master/start/quick-ceph-deploy/#expanding-your-cluster for the first glance
12:10 bjoernfan kaliya: Sure, thanks.
12:11 evg jskarpet: There're a lot of ceph experts here may be they'll suggest something more reliable
12:12 jskarpet More information about my previous fail scenario: seems the nodes had misconfigured boot order, and never boot from disk
12:13 jskarpet this is when they get stuck at the "Cannot get disk parameters" during the PXE boot
12:13 jskarpet after the initial PXE install completes
12:13 jskarpet Recovery from this isn't very elegant within Fuel atm. :P
12:14 evg jskarpet: yes, it looks very probable
12:26 dmellado joined #fuel
12:28 azemlyanov joined #fuel
12:47 okosse joined #fuel
12:57 monester_laptop joined #fuel
13:15 sambork joined #fuel
13:18 aliemieshko_ joined #fuel
13:31 jskarpet How can I move Telemetry MongoDB to another host? It's greyed out when I try to add a node with the role, and I'm unable to delete an existing node with it due to the requirement of minimun 3 nodes
13:33 aliemieshko_ <jskarpet>: hi, if you want delete or add mongo, you need to use only cli
13:34 jskarpet I can add mongo role to an existing node using cli?
13:36 aliemieshko_ yes, but only manual
13:37 aliemieshko_ and you need to dig into db
13:38 aliemieshko_ it's not too easy
13:38 jskarpet uh, dig into db?
13:38 jskarpet Is this documented somewhere?
13:39 jskarpet Does it help to delete, and re-add full node?
13:39 aliemieshko_ not yet
13:44 aliemieshko_ first you need delete role from node, then add node in env. After edit ceilometer.conf, keystone.conf, edit ceilometer , edit db nailgun
13:45 aliemieshko_ like I say above it is not too easy
13:47 jskarpet Re-running Puppet wouldn't take care of config changes?
13:51 aliemieshko_ what do you mean ?
13:54 e0ne joined #fuel
14:04 sambork1 joined #fuel
14:09 claflico joined #fuel
15:04 stamak joined #fuel
15:16 Rodrigo_US joined #fuel
15:34 blahRus joined #fuel
16:02 mattgriffin joined #fuel
16:02 rmoe joined #fuel
16:44 xarses joined #fuel
17:06 emagana joined #fuel
17:22 loth joined #fuel
17:59 e0ne joined #fuel
17:59 glavni_ninja_ joined #fuel
18:16 muadgib joined #fuel
18:20 xarses joined #fuel
18:26 HeOS joined #fuel
18:33 thumpba joined #fuel
19:04 monester_laptop joined #fuel
19:09 emagana joined #fuel
19:21 tzn joined #fuel
19:39 bdudko joined #fuel
19:39 dburmistrov joined #fuel
19:40 MorAle joined #fuel
19:40 kdavyd joined #fuel
19:40 aglarendil joined #fuel
20:04 stamak joined #fuel
20:08 teran joined #fuel
20:27 teran_ joined #fuel
20:44 mattgriffin joined #fuel
20:47 sovsianikov left #fuel
21:19 teran__ joined #fuel
21:25 teran joined #fuel
21:44 jobewan joined #fuel
21:45 sovsianikov joined #fuel
21:45 mattgriffin joined #fuel
21:45 sovsianikov left #fuel
22:31 angdraug joined #fuel
23:35 monester_laptop joined #fuel
23:43 e0ne joined #fuel
23:46 BobBall_AWOL joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary