Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-01-29

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
23:03 Bomfunk joined #fuel
23:27 IlyaE joined #fuel
23:31 TVR_ joined #fuel
23:32 TVR_ redeployed a new cluster... once more, a node failed install and am re-deploying it now... HA setup...
23:33 TVR_ the first controller failed...
23:33 TVR_ angdraug .. you on?
23:36 angdraug TVR_: I'm here
23:36 TVR_ I will let you know when it finishes.. and if the ceph backend works
23:36 angdraug if first controller failed, it isn't likely to recover by the end
23:37 TVR_ did you write any of the backend ceph puppet integration?
23:37 angdraug some
23:37 angdraug which bit are you interested in?
23:38 TVR_ ok.. so with the re-deploy.... even if it completes the seccond time.. it is unlikely to work then?
23:38 angdraug depends on why it died in the first place
23:38 angdraug but generally a lot of stuff on other nodes depends on the primary controller
23:39 angdraug if primary-controller role on the first node didn't complete properly, these other bits will fail, too
23:39 TVR_ I was wondering about nova and cinder integration... when I rolled my own for a deployment... I had the compute nodes install BOTH nova and cinder... in here, compute only gets nova....
23:39 angdraug not to mention that if it failed once it's liable to fail again at the same point
23:39 TVR_ is that because nova calls a cinder node to create whatever?
23:40 angdraug cinder node is now called "Storage - Cinder LVM"
23:40 angdraug the reason is that it's only needed when you use LVM backend for cinder
23:40 TVR_ ah.. ok... trying to learn all this stuff...
23:40 TVR_ heh
23:41 angdraug with ceph, you get cinder-volume-manager service, but it doesn't need its own node
23:41 TVR_ so I was adding a not needed part when I installed cinder on my compute nodes then.. ok, got it... good to know
23:41 angdraug and KVM on computes talks directly to OSDs via RBD protocol
23:41 TVR_ yea.. see that.. and auth with the virsh secret for the uuid
23:42 angdraug so ultimate purpose of cinder in this setup is to pass rbd://... URIs to Nova which passes it all the way to KVM
23:42 angdraug yup
23:42 TVR_ that part I had working fine... I just didn't realize the cinder part wasn't needed...
23:42 TVR_ cool.. thanks
01:18 IlyaE joined #fuel
02:27 IlyaE joined #fuel
03:17 vkozhukalov joined #fuel
04:39 ArminderS joined #fuel
06:12 jouston_ joined #fuel
06:41 e0ne joined #fuel
06:46 book` joined #fuel
06:59 jouston_ joined #fuel
07:13 mihgen joined #fuel
07:20 jeremydei joined #fuel
07:48 vk joined #fuel
07:55 vkozhukalov joined #fuel
08:12 amartellone joined #fuel
08:23 alex_didenko joined #fuel
08:25 jouston_ joined #fuel
08:25 jkirnosova joined #fuel
08:30 jouston joined #fuel
08:36 jouston Hi, I've like to add a role which install different set of deb instead of nova. How can I do that?
08:42 miguitas joined #fuel
08:46 evgeniyl` jouston: Hi, if you want to add new role in Fuel, this instruction can be useful for you https://www.mail-archive.com/fuel-dev@lists.launchpad.net/msg00270.html
08:51 vpleshakov joined #fuel
09:13 bookwar joined #fuel
09:30 mrasskazov joined #fuel
09:30 e0ne joined #fuel
09:52 e0ne_ joined #fuel
10:15 e0ne joined #fuel
10:18 e0ne joined #fuel
10:26 e0ne_ joined #fuel
10:32 vkozhukalov Dr_Drache, thanx very much for you help. Now it is clear that the problem is that centos kernel creates device file for smart array as follows /dev/disk/by-path/pci-0000:06:00.0-cciss-disk0 and debian-installer kernel creates instead something like this /dev/disk/by-path/pci-0000:06:00.0
10:44 e0ne joined #fuel
11:22 mihgen joined #fuel
11:39 e0ne joined #fuel
11:41 e0ne joined #fuel
11:42 e0ne joined #fuel
11:43 bookwar joined #fuel
11:57 vk joined #fuel
12:20 TVR_ joined #fuel
12:21 vk joined #fuel
12:30 e0ne joined #fuel
12:41 e0ne joined #fuel
12:44 TVR_ Morning folks
13:05 TVR_ it seems in my environment, the ubuntu os installs faster than the centos variant...
13:21 getup- joined #fuel
13:21 mrasskazov1 joined #fuel
13:31 e0ne joined #fuel
13:39 vkozhukalov joined #fuel
13:55 mattymo TVR_, there's plenty of quirks
13:55 mattymo installer itself is slower in ubuntu, but centos puppet takes longer
13:55 mattymo and swift is unbearably slow in Ubuntu in Python 2.7
13:55 TVR_ interesting....
13:55 TVR_ using ceph backend
13:56 TVR_ installed the centos variant and it kept failing on install of the first node.... so I am trying the ubuntu variant
13:57 TVR_ if this fails.. I will eliminate that node and install again...
13:57 TVR_ who knows.. maybe the nodes disk has issues..
14:04 TVR_ maybe fuel version 5 will have a tab to do an in-depth suite of test on any node selected, if the user wanted to 'certify' the node to be healthy before adding..... seems it would be fairly easy to create as the nodes sit in a ramdisk boot state before deployment anyway...
14:07 mattymo TVR_, feel free to report a bug with your request https://launchpad.net/fuel
14:07 mattymo it's also a good idea to do a CPU burn in and memory test
14:08 TVR_ yes...the nodes I am using are vetted from previous use, but they aren't that old (R610 and R720 Dell's)
14:08 TVR_ cool.. looking at the link
14:09 TVR_ I will deal with support after I see how this install is going... the product looks quite mature though...
14:09 TVR_ kind of fun to play with
14:17 mattymo TVR_, where are you located, if you don't mind me asking?
14:18 TVR_ Massachusetts .. at this second, Watertown
14:19 mattymo oh nice
14:19 mattymo I've been to Boston and Lowell only
14:20 TVR_ cool.. I live outside Lowell in Billerica... and Watertown is just outside Boston
14:44 e0ne_ joined #fuel
14:52 TVR_ I keep getting Deployment has failed. Error occurred while running method 'deploy'. Inspect Orchestrator logs for the details.
14:56 mihgen joined #fuel
15:03 vkozhukalov joined #fuel
15:14 alexz__ joined #fuel
15:18 e0ne joined #fuel
15:18 MiroslavAnashkin TVR_: then, please go to Fuel UI, to the logs tab, select master node, orchestrator and error level and look at the logs
15:21 IlyaE joined #fuel
15:23 vk joined #fuel
15:51 dhblaz joined #fuel
15:53 dhblaz On every instance of 4.0 I have deployed the Platform services (Ceilometer and Heat) functional tests fail.  Do these work for anyone?
15:57 mattymo the Heat tests are actually broken (there's no proxy support in python-heatclient)
15:58 MiroslavAnashkin dhblaz: Heat tests require Heat image uploaded to Openstack. Murano tests require Murano image uploaded to Openstack. So, tests may fail without some pre-condition met.
15:58 mattymo and that^
16:00 dhblaz Where can I get information about what image needs to be present, where to get it and what it needs to be named?
16:00 dhblaz I have a 3 controller HA setup.  Is it correct that the test will never work as mattymo suggests?
16:01 jouston_ joined #fuel
16:12 MiroslavAnashkin Hmm, there is no single word about Heat post-install check in released docs...
16:42 xarses joined #fuel
16:50 vt102 joined #fuel
16:57 vt102 joined #fuel
16:59 vt102 joined #fuel
17:01 e0ne_ joined #fuel
17:13 TVR_ MiroslavAnashkin it would seem I have two errors :
17:13 TVR_ first:   Removing of nodes ["26", "27", "28", "29", "30", "31"] finished with errors:
17:14 TVR_ then after the rebuild:
17:14 TVR_ Upload cirros image failed
17:15 TVR_ so this cluster seems to be having an issue after being deployed with uploading the cirros image... it is a ceph storage and ceph image store...
17:17 Arminder i'm getting issues booting instance using iso as image
17:17 Arminder tried 2-3 different iso's and it says non-bootable
17:18 Arminder i didn't needed till today and found this
17:18 Arminder is it me only?
17:18 Arminder using ceph for images/volumes/object on fuel 4.0
17:22 MiroslavAnashkin TVR_: Cirros image upload sometimes fails by timeout.
17:23 TVR_ should I increase the timeout for it? it has caused my cluster to fail to deploy
17:23 MiroslavAnashkin TVR_: But it may be Ceph configuration issue as well. So. please try upload this image from command line
17:23 TVR_ should I file a ticket with support?
17:23 TVR_ ok.. command...
17:27 TVR_ what command?
17:27 TVR_ where is the image?
17:28 Arminder wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img
17:28 Arminder glance add name='cirros image' is_public=true container_format=bare disk_format=qcow2 < cirros-0.3.0-x86_64-disk.img
17:28 TVR_ ok.. from the main node.. correct?
17:29 Arminder from one of controllers
17:29 Arminder you would need to run this too -> source ~/openrc
17:29 Arminder before running glance command
17:30 MiroslavAnashkin No, from any controller
17:30 angdraug joined #fuel
17:31 TVR_ the source I found yesterday.. yea.. for os env... the wget worked.. but the glance add simply hangs now
17:32 TVR_ Error communicating with http://10.7.212.45:9292 [Errno 32] Broken pipe
17:32 TVR_ let me see what's running
17:33 Arminder its image service on 9292
17:33 Arminder whats for -> ceph -s
17:34 Arminder and does this shows images in the list -> ceph df detail
17:34 TVR_ health HEALTH_WARN 113 pgs degraded; 151 pgs down; 379 pgs peering; 272 pgs stale; 318 pgs stuck inactive; 194 pgs stuck stale; 492 pgs stuck unclean; 3 requests are blocked > 32 sec; 3/11 in osds are down
17:34 TVR_ yes.. images are in the list
17:35 Arminder did you removed any ceph node(s)?
17:35 IlyaE joined #fuel
17:36 TVR_ no.. just did the add nodes, made sure the networks were coddect, and deploy and it just failed and I have not done any changes except to restart service
17:36 Arminder MiroslavAnashkin: any help on boot from iso image?
17:36 TVR_ the 6 nodes are all unchanged, and the disks are all good
17:36 Arminder your ceph cluster is running degraded
17:37 TVR_ I have played with these boxes in other configurations previously... did some of the disks need to be zapped?
17:37 jaypipes_ joined #fuel
17:37 TVR_ so, maybe there was residual from a previous config on some of the disks
17:38 Arminder i nevere required to do anything on nodes and i may have deployed multiple times
17:38 Arminder never*
17:38 TVR_ ok..
17:39 angdraug try ceph osd tree, that should tell you which osds are down
17:39 TVR_ history of this cluster... deployed => failed, so deployed with ubuntu => failed, so redeployed with the centos again as this is what we would use.....
17:40 MiroslavAnashkin Arminder: Please try import ISO to Glance, then create a Bootable Volume from an Image: Then attach the volume to a running instance
17:41 Arminder iso import works fine to glance and shows up fine in glance image-list too
17:41 Arminder ceph images usage too increase to size of iso, so its storing up fine
17:42 MiroslavAnashkin Arminder: This issue looks like yours: https://ask.openstack.org/en/question/1601/how-to-boot-an-iso-instance-on-nova/
17:45 Arminder that is guess is for folsom release
17:45 Arminder the same works fine for my grizzly environment
17:45 Arminder and their is a weird comment to restart all the openstack services
17:47 TVR_ OK.. so what should my next steps be... should I 1) power down the cluster and initialize from bios the drives so as to clear them well, 2.) file a support ticket, or 3) look into resurrecting this, maybe by either restarting ceph or finding out which OSD's are down and remove / reinstall them with ceph-deploy commands?
18:03 MiroslavAnashkin Arminder: Please check libvirt.xml, belonging to the instance you trying to boot from ISO. it may be Havana bug. https://bugs.launchpad.net/nova/+bug/1256906
18:04 dhblaz MiroslavAnashkin: Do you have a suggestion about what to do next or should I dig into the check to see how it works?
18:04 vkozhukalov joined #fuel
18:04 MiroslavAnashkin dhblaz: Simply consider Heat tests as broken.
18:10 TVR_ and my OSD's keep dropping... 12 out of 18 in a down state even though I did a service restart on all ceph nodes and had everything up... great
18:41 angdraug anything useful in /var/log/ceph on osd nodes?
18:57 vk joined #fuel
19:11 e0ne joined #fuel
19:39 IlyaE joined #fuel
20:23 xarses joined #fuel
20:26 designated joined #fuel
20:26 designated anyone active?
20:27 rmoe joined #fuel
20:31 designated is there a way with fuel to have a single 10Gb NIC in a host and have it trunked to the switch?  Everything I've seen so far requires the use of multiple 1Gb NICs, fuel doesn't seem to allow the different networks to exists on a single network interface.
20:34 angdraug it does, but it still requires a separate interface for the admin/pxe network, and, in case of neutron/vlan, for the private network
20:34 angdraug other networks e.g. management/storage can be combined on a single nic
20:37 designated so if i have two 10Gb NICs, I would still require an additional NIC in order to use fuel?
20:38 designated two 10Gb NICs, one to handle neutron/vlan and the other for management and storage, then a third 1Gb NIC for admin/pxe would work?
20:40 vk joined #fuel
20:44 angdraug yeah sounds about right
20:44 angdraug we're working on relaxing that requirements in the future releases, but right now that's what you'd need for neutron/vlan
20:49 designated thanks
20:50 designated btw, I'm sure you know this already but trying to run fuel as a vm in vmware's vmplayer doesn't work very well.  when I tried to deploy to nodes, everything on the box completely freezes.
20:50 designated I'm fairly certain it has something to do with vmplayer but I'm not sure what.
20:51 angdraug centos or ubuntu?
21:01 jouston__ joined #fuel
21:10 designated ubuntu
21:26 TVR_ So I deleted my environment and created a new one. Same issue as previously. my ceph logs show many osd failures, and ceph osd tree
21:26 TVR_ shows half my OSD's are down
21:27 TVR_ I am beginning to suspect the redeployment of the nodes is NOT cleaning up the OSD disks....
21:28 TVR_ since I have the time, I will initialize the disks individually after I take the nodes down and delete this environment and see if that solves my issue.
21:46 e0ne joined #fuel
21:50 dhblaz joined #fuel
21:51 NewChannelsInfo joined #fuel
21:51 NewChannelsInfo left #fuel
21:53 angdraug not sure cleaning the disks really is the problem in your case
21:53 angdraug if fuel provisioning failed to partition them properly they wouldn't be labeled for ceph with sgdisk and wouldn't show up in osd tree in the first place...
21:54 angdraug is ceph complaining about specific drives?
21:56 angdraug btw we do try to clean the drives during provisioning, although we don't nuke the whole drive
21:56 angdraug if there's an area we're missing I'd love to know
22:57 IlyaE joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary