Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2015-05-19

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:22 daniel3 joined #fuel
00:29 mattgriffin joined #fuel
00:36 mattgriffin joined #fuel
00:36 daniel3 joined #fuel
01:04 xarses joined #fuel
02:20 eliqiao joined #fuel
02:21 eliqiao1 joined #fuel
02:21 eliqiao left #fuel
02:21 eliqiao1 left #fuel
02:21 eliqiao1 joined #fuel
02:22 eliqiao1 left #fuel
02:29 fedexo joined #fuel
02:30 pasquier-s joined #fuel
03:39 mattgriffin joined #fuel
04:42 ddmitriev1 joined #fuel
04:59 e0ne joined #fuel
05:04 ddmitriev1 joined #fuel
05:50 gongysh joined #fuel
05:54 e0ne joined #fuel
06:05 mattgriffin joined #fuel
06:08 ub joined #fuel
06:10 Hulda joined #fuel
06:24 pasquier-s joined #fuel
06:25 ub joined #fuel
06:31 dklepikov joined #fuel
06:56 kutija joined #fuel
06:58 kutija_ joined #fuel
07:02 alecv joined #fuel
07:08 pasquier-s_ joined #fuel
07:16 dklepikov joined #fuel
07:26 ub joined #fuel
07:33 sc-rm joined #fuel
07:34 e0ne joined #fuel
07:36 sc-rm Hi, is there anyway to speed ud the spawning of instances. When they reach this point http://paste.openstack.org/show/227898/ they are hanging there for like 10-30 seconds and then continue after there at normal speed. What is making it hang there and is there anything to do about it?
07:38 hyperbaba joined #fuel
07:53 ub joined #fuel
08:17 kutija joined #fuel
08:24 Hulda Hey, how do i add more packages to the Ubuntu repo in fuel?
08:29 pbrooko joined #fuel
08:35 pbrooko joined #fuel
08:40 hyperbaba Hi, is there a fix for "venom" exploit in qemu for 5.1.1 deployments?
08:55 tzn joined #fuel
09:32 kutija_ joined #fuel
09:45 teran joined #fuel
09:48 teran_ joined #fuel
09:50 teran__ joined #fuel
09:52 kutija joined #fuel
09:54 kaliya hyperbaba: we're working on this for releases previous to 6.0 now
09:54 hyperbaba kaliya: Thank you.
10:31 kaliya hyperbaba: devs said that for 5.1.x it's done. You can find packages on http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/ bulletin here https://online.mirantis.com/hubfs/Technical_Bulletins/Mirantis-Technical-Bulletin-12-VENOM-QEMU-Vulnerability-v3-ext.pdf
10:40 kaliya sc-rm: spawning is a complex procedure, involving every program included nova, cinder, neutron, glance, swift, you must find where is the bottleneck, maybe in cloud-init itself?
10:43 sc-rm kaliya: Which process is responsible for the cloud-init?
10:45 kaliya nova
11:06 evg Hulda: just place packages in ubuntu pool then rebuild Packages and Release files. You'll need to install dpkg & dpkg-devel on fuel master
11:07 evg Hulda: I could paste intruction in another place if you need more details
11:17 pbrooko joined #fuel
11:18 dmgweb joined #fuel
11:30 sc-rm kaliya: For instance this is the case http://paste.openstack.org/show/228110/
11:30 sc-rm kaliya: 10 seconds is in my mind slow
11:33 sc-rm kaliya: What is the recomended settings in /etc/cloud/cloud.cfg.d/90_dpkg.cfg for datasource_list?
12:20 Hulda evg: would be very helpful, thank you
12:24 pasquier-s joined #fuel
12:26 pasquier-s_ joined #fuel
12:44 mattgriffin joined #fuel
12:48 evg Hulda: http://paste.openstack.org/show/228163/
12:51 evg Hulda: or you can set up an additional repository alongside on master node in order to not corrupt the main repo
12:54 Hulda evg: thanks alot
13:00 pbrooko_ joined #fuel
13:34 pasquier-s joined #fuel
13:35 pbrooko joined #fuel
13:46 sc-rm kaliya: It’s the ec2 meta-data fetch which is slow, where should I start to look for slow processing with the ec2, because generalt fetch of data inside the instance from external urls is not the speed problem
14:03 kaliya sc-rm: do you inject them through the nova boot option/UI field?
14:04 kaliya seems to be here https://github.com/openstack/nova/blob/stable/juno/nova/api/metadata/base.py#L219
14:04 sc-rm kaliya: Yep, I give it through the horizon
14:05 kaliya mh but I have no idea where the code can get slow there
14:05 kaliya so without these kind of cloud-init, images are instanciated quickly?
14:33 xdeller joined #fuel
14:35 pasquier-s_ joined #fuel
14:44 sc-rm kaliya: I think the line where it does hostname = self._get_hostname()
14:45 sc-rm kaliya: When I disabled the UseDNS for sshd the login speeded up, so maybe there… But I’ll leave for today
15:25 rmoe joined #fuel
15:32 e0ne joined #fuel
15:35 mpetason joined #fuel
15:38 CheKoLyN joined #fuel
15:39 saguilar joined #fuel
16:19 saguilar joined #fuel
16:28 pbrooko joined #fuel
16:34 xarses joined #fuel
16:36 pasquier-s joined #fuel
16:48 blahRus joined #fuel
16:53 e0ne joined #fuel
16:58 e0ne joined #fuel
17:02 daniel3_ joined #fuel
17:12 thumpba joined #fuel
17:14 thumpba_ joined #fuel
17:35 mattgriffin joined #fuel
17:57 pasquier-s joined #fuel
18:10 saguilar joined #fuel
18:15 kutija joined #fuel
18:28 e0ne joined #fuel
18:34 e0ne joined #fuel
18:37 mattgriffin joined #fuel
19:02 pasquier-s joined #fuel
19:09 mattgriffin joined #fuel
19:13 e0ne joined #fuel
19:30 mattgriffin joined #fuel
20:02 kutija just tested the latest 6.1 build #244
20:02 kutija and got this on both storage and compute nodes
20:02 kutija http://paste.openstack.org/show/228542/
20:03 kutija had this issue on #227 build too, but on one of controllers
20:03 kutija it was gone from the build from May 14th but there was that nasty Ceph bug which was presumably fixed and now this
20:04 kutija could someone comment / take a look?
20:09 mwhahaha so that errors if your nodes do not have access to the software repositories
20:10 kutija this happen at 100% deploy status
20:10 kutija just storage left, 3 controllers and one compute were deployed seconds before that
20:10 kutija I've checked connectivity, it's fine
20:10 mwhahaha what does the puppet log say
20:11 mwhahaha alternatively you can manually run the failed task on those nodes: puppet apply --modulepath /etc/puppet/modules /etc/puppet/modules/osnailyfacter/modular/netconfig/connectivity_tests.pp
20:11 mwhahaha and it should tell you what repository it cannot reach
20:12 kutija http://paste.openstack.org/show/228543/
20:12 kutija this is from storage-1
20:13 mwhahaha ERROR: Unable to fetch url 'http://archive.ubuntu.com/ubuntu/', error 'execution expired'. Please verify node connectivity to this URL, or remove it from the settings page if it is invalid. on node node-1.mirantis.com
20:13 mwhahaha that points to network issues after the network configuration task
20:14 mwhahaha or your cluster does not have internet connectivity after the network setup is complete
20:14 kutija it does, checked that
20:15 kutija and there were no network issue, not that I saw any
20:15 mwhahaha so if you wget/curl that url from the node now it works?
20:15 mwhahaha basically we're attempting to fetch that url and it's getting an execution expired which means it's taking longer than 5 or 10 (i can't remember which) seconds to complete
20:15 kutija it works as you expect it to work
20:16 kutija and i\m 100% sure that all nodes had no trouble to connect to the internet at any moment
20:16 kutija I will re-deploy
20:22 kutija this is interesting
20:22 kutija [518] Task '{"priority"=>700, "type"=>"puppet", "uids"=>["1"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/netconfig/connectivity_tests.pp", "timeout"=>600, "cwd"=>"/"}}' failed on node 1
20:22 kutija and if I execute it
20:22 kutija Notice: Compiled catalog for node-1.mirantis.com in environment production in 1.07 seconds
20:22 kutija Notice: Finished catalog run in 0.11 seconds
20:22 kutija no issues at all
20:23 kutija I just tried to re-deploy failed nodes
20:23 mwhahaha did it fail again or work?
20:23 kutija failed
20:24 kutija http://paste.openstack.org/show/228555/
20:25 kutija and If i Execute puppet manually
20:25 kutija everything is fine
20:25 mwhahaha so does it print out the urls when you execute it manually?
20:25 kutija yes
20:25 kutija all of them
20:25 mwhahaha That's weird
20:25 kutija eight
20:25 kutija yes it is
20:26 mwhahaha do you have ipv6 enabled?
20:27 kutija no
20:30 pasquier-s joined #fuel
20:33 mwhahaha can you do a traceroute from one of the nodes that failed to archive.ubuntu.com?
20:34 kutija root@node-6:~# traceroute archive.ubuntu.com
20:34 kutija traceroute to archive.ubuntu.com (91.189.91.23), 30 hops max, 60 byte packets
20:34 kutija 1  ip-1-90-136-213.static.contabo.net (213.136.90.1)  0.303 ms  0.304 ms  0.294 ms
20:34 kutija 2  ffm-b11-link.telia.net (62.115.36.237)  0.460 ms  0.456 ms  0.446 ms
20:34 kutija 3  ffm-b11-link.telia.net (62.115.140.76)  10.073 ms  10.079 ms  10.070 ms
20:34 kutija .....
20:35 kutija I can traceroute on all of them
20:35 kutija both failed and deployed
20:35 mwhahaha i'm downloading #244 to see if i can reproduce may be a while
20:35 kutija allright
20:35 kutija I already tried a build from yesterday
20:35 kutija which had a nasty bug
20:35 kutija with Ubuntu installation
20:36 kutija and one from may 17th which had an issue with docker unpacking
20:36 mwhahaha yea that change was merged in around the 14th i believe
20:36 kutija so I do not need to try those
20:36 mwhahaha for the connectivity_tests
20:36 kutija in the last 19 days I have tried more than 35 deploys
20:36 kutija from 6.0 to 6.1 nightly builds
20:36 kutija all of them failed
20:36 mwhahaha basically it's supposed to pick up on the connectivity issues and fail early before that apt-get task fails or something
20:37 mwhahaha are you doing any special network config for your deployment?
20:38 kutija nope
20:38 kutija nothing fancy
20:38 kutija I have six nodes
20:38 kutija 3 controllers, 2 storages and 1 compute
20:38 kutija HA Deploy
20:38 kutija but I managed to hit every possible bug
20:38 kutija starting from Galera
20:39 kutija to Ceph issue with journal on separate disk
20:39 kutija from May 14th
20:39 mwhahaha neutron with vlan?
20:39 kutija GRE
20:54 teran joined #fuel
21:00 kutija after manual run of puppet command on all failed nodes and re-deploy it managed to go through
21:01 mwhahaha my assumption is that your environment is hitting the low thresholds we have for those tests
21:01 mwhahaha it's currently a 5 second open timeout, a 5 second read timeout and a 15 second overall timeout
21:04 kutija it is possible
21:04 kutija I've managed to finish deploy
21:05 kutija of course nothing is working :D
21:05 kutija can't get a list of flavors, hypervisors...
21:05 kutija sigh
21:05 mwhahaha :(
21:09 kutija is there anything we can do about those timeout values?
21:09 kutija this is second or third time we are hitting this issue
21:13 mwhahaha i'll create a bug
21:13 mwhahaha and increase them
21:13 kutija thanks
21:19 mwhahaha https://bugs.launchpad.net/fuel/+bug/1456805 fyi
21:20 kutija great thanks ;)
21:50 kutija Fix proposed to branch: master
21:50 kutija Review: https://review.openstack.org/184334
21:50 kutija Changed in fuel:
21:50 kutija status:New → In Progress
21:50 kutija sorry wrong window
21:58 teran joined #fuel
22:01 teran joined #fuel
22:10 pasquier-s joined #fuel
22:22 holser_ joined #fuel
22:31 pasquier-s joined #fuel
22:32 holser_ joined #fuel
22:42 samuelBartel joined #fuel
22:55 samuelBartel joined #fuel
23:12 pasquier-s_ joined #fuel
23:13 holser_ joined #fuel
23:20 holser_ joined #fuel
23:29 holser_ joined #fuel
23:31 mescanef joined #fuel
23:37 pasquier-s joined #fuel
23:42 rmoe joined #fuel
23:44 Longgeek joined #fuel
23:54 xarses joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary