Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-10-29

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 artem_panchenko left #fuel
00:00 artem_panchenko joined #fuel
00:09 kupo24z joined #fuel
00:27 rmoe joined #fuel
00:44 xarses joined #fuel
00:47 Longgeek joined #fuel
00:48 Longgeek joined #fuel
00:51 mattgriffin joined #fuel
01:00 kupo24z xarses: you around?
01:02 kupo24z Getting errors on migrations & resizes with 'no public ip' checked, its trying to communicate over the DHCP PXE IP rather than the management 192.168.0.x network; Unexpected error while running command. Command: ssh 23.80.0.42 mkdir -p /var/lib/nova/instances/9eba2b90-4075-4455-957e-ec4e4e441c87 Exit code: 1
01:03 kupo24z Think it may be a bug
01:13 e0ne joined #fuel
01:21 mattgriffin joined #fuel
02:18 angdraug joined #fuel
02:20 mattgriffin joined #fuel
03:00 mattgrif_ joined #fuel
03:19 teran joined #fuel
04:06 ArminderS joined #fuel
04:36 teran joined #fuel
05:11 anand_ts joined #fuel
05:37 monester_laptop joined #fuel
05:58 xarses kupo24z: can you ssh to the node and run that command using the nova user? (from the source node)
06:06 teran joined #fuel
06:54 teran joined #fuel
06:56 e0ne joined #fuel
07:17 rmoe joined #fuel
07:34 bleachin joined #fuel
07:59 pasquier-s joined #fuel
08:00 bleachin joined #fuel
08:14 bleachin joined #fuel
08:21 bleachin joined #fuel
08:22 monester_laptop joined #fuel
08:29 hyperbaba joined #fuel
08:43 sanek joined #fuel
08:48 e0ne joined #fuel
08:53 dancn joined #fuel
08:54 azemlyanov joined #fuel
09:09 HeOS joined #fuel
09:41 teran joined #fuel
09:47 dklepikov joined #fuel
09:55 adanin joined #fuel
09:59 stamak joined #fuel
10:05 getup- joined #fuel
10:07 getup- joined #fuel
10:16 teran joined #fuel
10:18 e0ne joined #fuel
11:46 vtzan joined #fuel
12:31 Dr_Drache hello all
12:33 Dr_Drache wondering a question on back-up... I have been given a requirement to backup our images once they are in ceph, is there a easy way to make that happen?
12:36 saibarspeis joined #fuel
12:37 merdoc Dr_Drache: backup it befor uploading to ceph (%
12:37 merdoc before*
12:37 Dr_Drache merdoc, right of course... but we have an ingrained process of backup to long term tape.
12:39 Dr_Drache just sorting out how that would work.
12:42 pasquier-s joined #fuel
12:43 azemlyanov joined #fuel
12:45 Dr_Drache merdoc, long story short.
12:45 Dr_Drache there is no "real" way
12:48 Krast joined #fuel
12:59 getup joined #fuel
13:06 gabrijela_ joined #fuel
13:10 teran joined #fuel
13:11 Dr_Drache merdoc, this is going to be bad :(
13:11 Dr_Drache lol
13:12 warpc__ joined #fuel
13:16 Dr_Drache well, I guess I can ignore ceph itself, and just backup images/volumes?
13:35 mattgriffin joined #fuel
13:36 jobewan joined #fuel
13:40 merdoc Dr_Drache: I don't think so. atleast I have not found how to backup images from openstack
13:59 Dr_Drache merdoc, maybe someone else here can help in a bit with questions
14:03 hop joined #fuel
14:07 hop left #fuel
14:10 hop joined #fuel
14:14 teran_ joined #fuel
14:20 mpetason joined #fuel
14:21 championofcyrodi left #fuel
14:21 championofcyrodi joined #fuel
14:29 hop joined #fuel
14:32 Dr_Drache kaliya
14:33 stamak joined #fuel
14:41 bhi Hello again.
14:42 bhi my compute node gets not discovered after sucessful PXE bootstrap
14:42 kaliya hi Dr_Drache
14:42 emagana joined #fuel
14:42 bhi shouldn't it appear automatically if the PXE connection worked?
14:43 Dr_Drache kaliya, I know you may be busy, but I'd like some direction on how to "backup" images/volumes
14:43 Dr_Drache bhi, it should, whats the timeframe?
14:43 bhi uhm maybe 20 minutes ago
14:44 Dr_Drache bhi, it does take a few min, but 20 is too long./
14:44 bhi I had strange messages during boot related to raidcontroller problems but I can see the loginprompt
14:44 kaliya Dr_Drache: you mean exporting snapshots?
14:45 Dr_Drache kaliya, for old school backups in case everything goes to crap.
14:45 kaliya I guess that people in #ceph will know much more than me :)
14:46 kaliya if it's just backuping rbd
14:46 Dr_Drache kaliya, not much action over there.
14:46 Dr_Drache ceph itself doesn't support it.
14:46 Dr_Drache just wondering if you knew of a openstack side
14:48 kaliya no ideas now
14:49 kaliya Dr_Drache: I try to see inside, what do you want to backup?
14:50 Dr_Drache kaliya, just images/volumes
14:50 Dr_Drache already have ways to get all the databases/etc from openstack
14:51 bhi Are there standard credentials to manually login to a bootstrapped system?
14:52 Dr_Drache bhi - > if fuel is default
14:52 Dr_Drache it's root/r00tme
14:52 bhi ok, so maybe this helps for troubleshooting
14:52 kaliya bhi: you can also set SSH keys in the settings pane
14:52 kaliya Dr_Drache: I'm querying for ideas
14:53 bhi yes but first my compute node needs to be discovered right?
14:54 Dr_Drache kaliya, thank you
14:58 bhi hmm I logged into the bootstrapped compute node but all nics are unconfigured
14:59 bhi I tried service network restart but with no effect
15:15 blahRus joined #fuel
15:36 bhi Is there a way to work around this error during bootstrap?
15:36 bhi https://bugzilla.redhat.com/show_bug.cgi?id=541397
16:06 pal_bth joined #fuel
16:07 ArminderS joined #fuel
16:17 jetole joined #fuel
16:23 jetole Hey guys. I am trying to migrate a windows host from esxi to fuel / openstack. I was getting a BSOD on boot (0x0000007b) so I started troubleshooting. I brought the image over to my local workstation and tried to boot it via kvm / virt-manager and got the same BSOD at the same time. The following command did not work however when I added --enable-kvm then it did work and fully booted: sudo qemu-system-x86_64 -m 4096 -alt-grab -localtime -no-kvm -cpu kvm
16:23 jetole 64 -enable-kvm 125115-101999-windows-1.raw
16:24 jetole so it seems to be this option running on kvm/qemu on my workstation would be in direct correlation with why I cannot boot the same VM image on fuel / openstack
16:24 jetole and I'm open to any advice anyone can give
16:24 kaliya jetole: unfortunately, seems a problem isolated to your windows image
16:25 kaliya as I understand, your nova relies on kvm, so how did you convert your image?
16:25 Dr_Drache jetole
16:26 jetole I used qemu-img. The thing is the image will boot and function if I use the -enable-kvm option to qemu-system-x86_64
16:26 jetole @ kaliya
16:26 Dr_Drache you need to apply the reg entry for virtIO before you shut down the Image
16:26 jetole you mean install the virtio drivers?
16:26 Dr_Drache and enable windows to switch disk types on boot
16:27 jetole Dr_Drache: can you elaborate on both of those or point me towards a page with a little more detail ?
16:27 Dr_Drache jetole, will find something, is the windows running virtio before hand?
16:28 jetole it was running on VMWare ESXi. I installed the virtio inf files before powering it off on vmware but I was wondering if I installed the wrong ones or something
16:29 Dr_Drache no.
16:29 Dr_Drache hold on
16:29 jetole k
16:29 jetole thanks
16:30 Dr_Drache this pretty much explains it :
16:30 Dr_Drache http://backdrift.org/converting-windows-guests-from-vmware-esx-to-kvm-with-virtio-drivers
16:31 Dr_Drache the big part is the MERGEIDE file
16:31 Dr_Drache " If you are seeing a bsod (blue screen) with error 0x000007b boot errors, verify that the MergeIDE step was successful"
16:32 jetole yeah that was something we didn't do and I was already looking at this
16:32 jetole but virtio needs to be installed first. Right. doesn't openstack boot off a virtio disk that would BSOD if the virtio drivers aren't there?
16:33 jetole I mean first before importing into openstack
16:33 Dr_Drache no.
16:33 jetole oh
16:33 Dr_Drache the mergeIDE allows windows to change the boot device type.
16:33 jetole I understand that. Like I said I had been looking at that too
16:33 Dr_Drache having the virtio installed first would be BEST, but not required.
16:33 jetole but I thought openstack also used virtio vda disks
16:34 jetole OK
16:34 jetole do you know why the system will boot locally (non openstack) when I used the -enable-kvm option to qemu-system-x86_64 ?
16:34 Dr_Drache my conversions were from KVM (qemu IDE, not virtio) and I require the same process
16:34 rmoe joined #fuel
16:35 Dr_Drache I believe in that case you are overriding the virtio driver
16:36 jetole I looked at the libvirtd xml and it didn't look like it was using virtio
16:36 jetole bus=scsi instead of bus=virtio
16:44 jetole Dr_Drache: I've confirmed that the mergeIDE aspect resolved the problem locally. I'll be re-uploading the updated image to glace but I believe this solves it
16:44 getup joined #fuel
16:44 jetole so are you sure I don't need to install the virtio disk drivers first for Windows? This is a test system and we're documenting as we go but we have a lot that we will be migrating
16:50 championofcyrodi joined #fuel
16:52 kaliya joined #fuel
16:54 Dr_Drache jetole, I am not sure, I didn't have to, might you might.
17:07 jobewan joined #fuel
17:10 jetole Dr_Drache: actually I think you do. Everything I have looked at for creating a image from scratch mentions to have the virtio drivers available to the Windows install CD
17:11 Dr_Drache jetole, like I said, I am not sure. and you very well are right. not debating that. the few windows machines I had to convert, didn't need them. but I can't say for certain if they were scsi or IDE emulated beforehand.
17:12 jetole I don't know 100% either but it just seems logical
17:19 xarses joined #fuel
17:24 emagana joined #fuel
17:32 kupo24z re: xarses: kupo24z: can you ssh to the node and run that command using the nova user? (from the source node) Yes, i can ssh via 'node-X' then running 'mkdir -p /var/lib/nova/instances/9eba2b90-4075-4455-957e-ec4e4e441c87'
17:33 kupo24z Heres the full error from horizon UI http://pastebin.mozilla.org/6987741
17:35 kupo24z xarses: and here is the log from that time http://pastebin.mozilla.org/6987744
17:35 kupo24z (different instance)
17:51 justif joined #fuel
18:01 emagana joined #fuel
18:04 emagana_ joined #fuel
18:11 jobewan joined #fuel
18:13 xarses kupo24z: where you presented with the yes/no save this key prompt?
18:31 vtzan joined #fuel
18:39 vtzan Hi all, how can I make configuration changes on deployed nodes using puppet? I mean after deployment. Thx
18:41 xarses vtzan: we don't run puppet besides the current actions, so running puppet after the fact would be mostly up to you. the fuel master has mcollective running so you could use it to spawn puppet apply accross the nodes, but if the changes aren't reflected in the fuel manifests, then if fuel re-runs puppet the changes will likely be lost
18:42 xarses vtzan: currently the lifecycle is mostly focused on deployment, settings updates and the like will start to get addressed when the granular-deployment blueprint is merged and active around the 7.x release
18:43 vtzan Ok can i add a new manifest with the changes and apply after deployment? Is that possible?
18:44 vtzan That is great to know.I am just looking for a more peoductive method at the moment
18:45 xarses vtzan: lets start with, what are you trying to change?
18:45 vtzan Ok. Lets say for example add ssl support to horizon
18:46 vtzan With custom keys/crt
18:47 xarses vtzan: look over whats being changed as part of https://blueprints.launchpad.net/fuel/+spec/ssl-endpoints
18:48 vtzan A second example would be to change some ceph configuration options on purpose. Or just run custom code for example new osds
18:50 xarses so for both of those they 'conflict' with what fuel already is doing with puppet, so you would want to change the fuel-library manifests to reflect these needs, preferably before deployment
18:50 xarses on the fuel-master node, the fuel-libary modules are in /etc/puppet/modules
18:51 xarses otherwise the code is here https://github.com/stackforge/fuel-library/tree/master/deployment/puppet
18:52 vtzan How do i force to apply changes from puppet manifests after deployment?
18:52 xarses you can use the fuel client on the master node to force the deployment task to re-run
18:52 vtzan Or just add a new manifest and apply it after deployment
18:54 xarses fuel node --node <node id, or comma list of ids> --deploy
18:54 xarses node id's are listed in fuel node
18:54 xarses and the number part of the hostname is also the node id
18:54 vtzan Ok. I might need to create a new task then but its ok i got it.
18:55 angdraug joined #fuel
18:55 xarses I would avoid adding a new manifest to change something that fuel manages because it could become overwritten
18:56 xarses btw, here is the entry point for ceph https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/ceph/manifests/init.pp
18:56 xarses its called from https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/manifests/cluster_ha.pp#L260-270
18:57 xarses there is a similar entry in cluster_simple.pp but that's for non-ha deployments
18:58 vtzan Ok i see. So regarding the ssl we still have road in front of us right?
18:58 xarses if you want to set settings that are not set already you can add entries into ceph_conf like https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/ceph/manifests/conf.pp#L23-37
18:58 xarses for ssl, look over the changes noted in that blueprint
18:59 xarses but ya, its extensive still
19:01 vtzan joined #fuel
19:17 emagana joined #fuel
19:18 championofcyrodi any tips on building a new kernel that supports nfs mounted rootfs options?
19:18 championofcyrodi (like, there already is a built kernel i can use instead of http://wiki.centos.org/HowTos/I_need_the_Kernel_Source)
19:31 emagana joined #fuel
19:33 championofcyrodi nevermind... got the kernel "building" with root nfs support
19:35 kupo24z re: xarses: kupo24z: where you presented with the yes/no save this key prompt? No
19:41 alvinstarr joined #fuel
19:41 alvinstarr i am walking through testing mirantis 5.1 and have what may be some stupid questions.
19:42 teran joined #fuel
19:42 xarses alvinstarr: no such thing
19:42 Dr_Drache alvinstarr, don't we all, don't we all
19:43 alvinstarr Ok you asked for it.... I have gotten a 5 system install up and running to the point where I can see the openstack dashboard and it reports 0 hypervisors.
19:43 alvinstarr I believe I should be seeing 2 systems running kvm/libvirt
19:44 alvinstarr but it looks like the compute nodes don't have libvirtd and associated stuff installed.
19:44 xarses alvinstarr: so you configured 2 of the nods with the compute role?
19:44 alvinstarr yes
19:44 xarses alvinstarr: and the deployment was declared successful?
19:45 vtzan Virtualization is enabled on bios of each compute node?
19:45 vtzan Otherwise kvm module cannot be loaded
19:46 alvinstarr Virtualizaion is enabled and the deployment thinks it is successful.
19:47 alvinstarr I have run up a 4 node test install using RDO
19:47 xarses alvinstarr: from one of the compute nodes please paste the results of: 'source /root/openrc; nova service-list'
19:47 xarses alvinstarr: sorry, from one of the controllers, not compute
19:47 vtzan Kvm module is loaded?
19:47 vtzan Aaxx
19:48 alvinstarr +------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
19:48 alvinstarr | Binary           | Host              | Zone     | Status  | State | Updated_at                 | Disabled Reason |
19:48 alvinstarr +------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
19:48 alvinstarr | nova-consoleauth | node-1.netvel.net | internal | enabled | up    | 2014-10-29T19:48:13.000000 | -               |
19:48 alvinstarr | nova-scheduler   | node-1.netvel.net | internal | enabled | up    | 2014-10-29T19:48:12.000000 | -               |
19:48 alvinstarr | nova-conductor   | node-1.netvel.net | internal | enabled | up    | 2014-10-29T19:48:05.000000 | -               |
19:48 alvinstarr | nova-cert        | node-1.netvel.net | internal | enabled | up    | 2014-10-29T19:48:07.000000 | -               |
19:48 alvinstarr +------------------+-------------------+----------+---------+-------+----------------------------+-----------------+
19:49 xarses and the result of 'pcs status' also please use a paste service like http://paste.openstack.org/  or pastebin and the like
19:52 Dr_Drache xarses, I asked kaliya; but maybe you have an idea, is there a way to "backup" images and volumes from openstack, for an offsite backup?
19:53 xarses Dr_Drache: for volumes you should read about cinder backup
19:53 xarses I think it has some process for images too
19:53 xarses I'm not that familiar with it though
19:55 alvinstarr paste is at http://paste.openstack.org/show/126686/.  pcs?
19:56 alvinstarr looks like the kvm module is not loaded.
19:57 Dr_Drache xarses, i'll look, i'm not really NEEDing automation, just a way to make it work
20:03 xarses Dr_Drache: I think thats point of cinder backups
20:03 xarses http://docs.openstack.org/trunk/config-reference/content/section_backup-drivers.html
20:04 xarses alvinstarr: 'pcs status' or 'crm_mon 1' on one of the controllers should work
20:04 xarses sorry, the second is 'crm_mon -1'
20:06 Dr_Drache xarses, that link is near blank
20:06 xarses Dr_Drache: ya, I know =(
20:06 Dr_Drache "You can restore a volume from a backup as long as the backup's associated database information (or backup metadata) is intact in the Block Storage database"
20:06 Dr_Drache so, it's near useless unless your cluster is still good.
20:07 Dr_Drache or at least your database
20:13 alvinstarr I think I may have found the problem or at least part of it.
20:14 xarses alvinstarr: without the corosync output (crm/pcs) I'
20:14 xarses alvinstarr: without the corosync output (crm/pcs) I'm not sure, but it looks like your nodes cant communicate on the management network
20:15 alvinstarr Is corosync required for a non-HA install?
20:16 xarses alvinstarr: no, its not present if its not a ha install, but you noted earlier you had 5 nodes, and 2 computes. I asked if you had 3 controllers, which if you do is a HA install
20:17 alvinstarr xarses: controller, compute, compute, cinder, telemetry
20:18 xarses ah
20:19 xarses from node-1: 'for each in {2..5}; do ping -c 1 node-${each} ; done'
20:19 alvinstarr I have 4 nics and it looks like on the controller 2 are swapped.
20:19 xarses assuming all your nodes are id's 1..5
20:20 xarses it will ping the mgmt interface
20:22 alvinstarr xarses: Mirantis looks like it makes some decisions about what nic's are associated with the various functional networks and in 3 of the 4 networks thats ok but one is the public interface and is associated with a specific nic.
20:22 alvinstarr It looks like I will need to rebuild the environment.
20:23 xarses alvinstarr: fuel will use a default layout, but you need to ensure that it's correct for your nodes.
20:25 alvinstarr xarses: so the fault is in the default.
20:29 hyperbaba_ Hi,
20:30 hyperbaba_ how can i create a adminURL for identity service in the catalog to be on the public ip. I have problems connecting external apps to openstack. The advertised endpoint is at private ip for identity service and not on public, thus not accessible
20:34 Dr_Drache xarses; it seems this has been in discussion for 3+ years, yet it has never been really fleshed out
20:35 xarses hyperbaba_: you would use the keystone client to set a new adminURL
20:35 xarses Dr_Drache: =/
20:37 Dr_Drache xarses; seems like a oversight that a actual way for DR, is left to "throw more hardware at the cluster"
20:39 hyperbaba_ xarses: changing the existing endpoint or create a new one ?
20:40 xarses hyperbaba_: it looks like you would have to delete the existing endpoint record, and re-create it with the values you want
20:40 hyperbaba_ xarses: hacking the db directly will do the job?
20:41 xarses hyperbaba_: it could, but the keystone client is inteneded to be used for this
20:55 emagana joined #fuel
21:01 emagana joined #fuel
21:05 emagana_ joined #fuel
21:41 angdraug joined #fuel
22:03 emagana joined #fuel
22:06 emagana_ joined #fuel
22:07 kupo24z xarses: what users should i test ' ssh 23.80.0.48 mkdir -p /var/lib/nova/instances/e61e6bd2-69a0-416a-901e-120e74a1c89b' under?
22:07 kupo24z No issues under root
22:08 xarses kupo24z: nova
22:09 kupo24z 'su - nova' work?
22:09 xarses yes
22:10 kupo24z root@node-22:~# su - nova; ssh 23.80.0.48 mkdir -p /var/lib/nova/instances e61e6bd2-69a0-416a-901e-120e74a1c89b
22:10 kupo24z Warning: Permanently added '23.80.0.48' (ECDSA) to the list of known hosts.
22:10 kupo24z root@node-22:~#
22:10 kupo24z Looks fine
22:11 xarses ah
22:11 xarses you should be nova@
22:11 xarses oh
22:11 xarses nova is /bin/balse
22:12 xarses nova is /bin/false
22:12 kupo24z change to bin/bash?
22:12 xarses no need
22:12 xarses sudo -u nova bash
22:12 xarses or sudo -u nova <command>
22:12 kupo24z command need to be quoted?
22:13 kupo24z Hmm, same thing
22:13 xarses should't need it
22:13 kupo24z Warning: Permanently added '23.80.0.48' (ECDSA) to the list of known hosts.
22:13 kupo24z echo $? shows 1
22:13 kupo24z would that be throwing nova off?
22:13 xarses maybe
22:19 xarses hmm
22:19 xarses first lets debug with this
22:19 xarses first lets debug with this
22:20 xarses ssh node-6 -C '/bin/echo $USER $HOSTNAME'
22:20 xarses replace node-6 with your ip
22:21 xarses sudo -u root ssh node-6 -C 'echo $USER $HOSTNAME'
22:21 xarses both should work
22:21 xarses now with sudo -u nova you get nothing
22:21 xarses so it's not staying logged in
22:22 xarses ok, so you have to set nova's default shell back to /bin/bash
22:22 xarses /bin/false wont let it run
22:25 xarses kupo24z: change the shell and report back so we can merge https://review.openstack.org/#/c/130876/
22:30 kupo24z xarses: replace node-6 with destination ip?
22:30 xarses yes
22:30 kupo24z root@node-22:~# sudo -u root ssh 23.80.0.48 -C 'echo $USER $HOSTNAME'
22:30 kupo24z root node-45
22:31 xarses and replace root with nova
22:31 kupo24z root@node-22:~# sudo -u nova ssh 23.80.0.48 -C 'echo $USER $HOSTNAME'
22:31 kupo24z Warning: Permanently added '23.80.0.48' (ECDSA) to the list of known hosts.
22:31 kupo24z no echo with that
22:31 kupo24z thats after changing nova's shell as well on the source, not destination
22:32 xarses dest is the issue
22:32 kupo24z change it on destianation?
22:32 xarses yep
22:32 kupo24z now shows
22:32 kupo24z root@node-22:~# sudo -u nova ssh 23.80.0.48 -C 'echo $USER $HOSTNAME'
22:32 kupo24z Warning: Permanently added '23.80.0.48' (ECDSA) to the list of known hosts.
22:32 kupo24z nova node-45
22:34 kupo24z should i propigate change to all nodes and try to resize again?
22:39 xarses kupo24z: yes please
23:19 xarses kupo24z: better results?

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary