Perl 6 - the future is here, just unevenly distributed

IRC log for #openstack-rally, 2015-06-11

| Channels | #openstack-rally index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 meteorfox boris-42: hey
00:00 boris-42 meteorfox: how is going?
00:02 meteorfox boris-42: good, but extremely busy. I've been working on adding support for Rackspace to the Google project, PerfKitBenchmarker. https://github.com/meteorfox/PerfKitBenchmarker/tree/rackspace-provider-dev
00:02 meteorfox boris-42: plus other internal stuff
00:02 meteorfox boris-42: how about you
00:02 meteorfox ?
00:03 boris-42 meteorfox: that one is cloud comparator?
00:03 boris-42 comparator*
00:04 meteorfox boris-42: kind of, it will not compare it for, but you can run the benchmarks against different cloud providers
00:04 meteorfox *it will not compare them for you
00:05 boris-42 meteorfox: heh someday in rally maybe=)
00:05 meteorfox boris-42: the workloads are nice
00:05 boris-42 meteorfox: inside VMs as I assume?
00:05 boris-42 meteorfox: or framework it self?
00:06 meteorfox boris-42: yeah, must of the workloads are for inside the vms, there are some cluster_boot scenarios, but its strength is the data plane
00:07 meteorfox boris-42: it's all through SSH, it provides utilities to add the benchmarks really easily
00:08 meteorfox boris-42: for example, this is the mongodb_ycsb benchmark https://github.com/meteorfox/PerfKitBenchmarker/blob/rackspace-provider-dev/perfkitbenchmarker/benchmarks/mongodb_ycsb_benchmark.py
00:08 boris-42 meteorfox: heh someday we will finish this in rally=(
00:09 meteorfox boris-42: the nice thing about it, is that is very easy to do benchmarks with multiple VMs
00:09 boris-42 meteorfox: it's our goal =)
00:09 meteorfox boris-42: I know :)
00:09 boris-42 meteorfox: to make such framework=)
00:09 klindgren_ joined #openstack-rally
00:10 boris-42 meteorfox: this heavy chain https://review.openstack.org/#/c/177014/ =)
00:10 meteorfox boris-42: at Rackspace, I use Rally for Control Plane (API and stuff), and PerfKitBenchmarker for Data plane
00:11 boris-42 meteorfox: someday I hope you will use for both rally=)
00:12 meteorfox boris-42: yeah, I don't think PerfKit, will go as deep into the control plane, as Rally does
00:13 meteorfox boris-42: I'm part of the PerfKit group, and its mostly academic. MIT and Stanford are part of it too, and they are interested on the workloads, not the API
00:13 boris-42 meteorfox: heh
00:14 boris-42 meteorfox: they have more star on their git repo=)
00:14 meteorfox boris-42: lol yeah
00:15 meteorfox boris-42: but they are Google, and they announced in their blogs
00:16 meteorfox boris-42: it was even on TechCrunch
00:18 boris-42 meteorfox: I should benchmark kubernates=)
00:18 boris-42 meteorfox: to get in their blog=)
00:20 adiantum joined #openstack-rally
00:21 yingjun joined #openstack-rally
00:29 openstackgerrit Merged openstack/rally: Better mocking for self.[admin_]clients in Scenario tests  https://review.openstack.org/188554
00:33 yingjun joined #openstack-rally
00:43 mwagner_afk joined #openstack-rally
00:44 kiran-r joined #openstack-rally
00:48 openstackgerrit OpenStack Proposal Bot proposed openstack/rally: Updated from global requirements  https://review.openstack.org/189463
01:13 BitSmith joined #openstack-rally
01:19 BitSmith joined #openstack-rally
01:45 yingjun joined #openstack-rally
01:52 georgem1 joined #openstack-rally
02:01 georgem1 left #openstack-rally
02:35 yingjun_ joined #openstack-rally
02:36 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer create and list pool  https://review.openstack.org/182685
02:40 BitSmith joined #openstack-rally
02:42 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer create and list pool  https://review.openstack.org/182685
02:47 BitSmith joined #openstack-rally
03:19 openstackgerrit Li Yingjun proposed openstack/rally: Validate tenant related scenarios when using keystone v3  https://review.openstack.org/181321
03:36 arxcruz joined #openstack-rally
03:42 kiran-r joined #openstack-rally
03:52 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer create and list pool  https://review.openstack.org/182685
04:20 mahito joined #openstack-rally
04:23 tfreger joined #openstack-rally
04:28 mahito joined #openstack-rally
04:56 yingjun joined #openstack-rally
04:56 yingjun joined #openstack-rally
04:57 lutostag joined #openstack-rally
05:15 kiran-r joined #openstack-rally
05:20 neeti joined #openstack-rally
05:34 aswadr joined #openstack-rally
05:37 rdas joined #openstack-rally
05:38 nihilifer joined #openstack-rally
05:44 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer create and list pool  https://review.openstack.org/182685
06:14 mahito joined #openstack-rally
06:14 yingjun joined #openstack-rally
06:16 yfried joined #openstack-rally
06:28 nkhare joined #openstack-rally
06:32 yfried yingjun: ping re https://review.openstack.org/182685
06:33 yingjun yfried, lgtm :)
06:34 yfried yingjun: are you familiar with neutron LBaaS?
06:39 yingjun yfried, not a expert of LBaaS, but i am using it
06:40 yfried yingjun: there's a v2 LBaaS. don't know if it's already available or not, but it uses a lot of the same object names
06:40 yfried yingjun: pools, members...
06:42 yfried yingjun: however the objects and the API are completely different. I'm concerned that we will have hard time differentiating between them, so I think we need to have this patch mark which objects/api it's targeting
06:42 yfried yingjun: like s/pool/pool-v1
06:42 yfried yingjun: but I only have a rought idea of the LBaaS and the v2 differences
06:45 yingjun yfried, there is a patch that adds a validator to check component API version https://review.openstack.org/#/c/181321/5/rally/benchmark/validation.py, maybe we can use it in the case?
06:46 yfried yingjun: IDK. I think we need an lbaas guy to take a look at the lbaas stuff before we merge them. I'd hate to have to build more wrappers to solve a problem that we can prevent now
06:52 yingjun yfried, make sense to me:), i’ll remove my +2 for now
06:52 yfried yingjun: do we have someone on the team who might know this?
06:53 yfried boris-42: ^
06:53 yfried kiran-r: ^ we are discussing your patch
06:53 yingjun yfried, not sure, maybe boris-42
06:56 neeti joined #openstack-rally
07:05 kiran-r yfried, yingjun shall I s/pool/pool-v1 and commit again??
07:08 exploreshaifali joined #openstack-rally
07:08 e0ne joined #openstack-rally
07:09 yfried kiran-r: it's not that simple. I don't know lbaas very well, just that there are 2 lbaas version and both use some shared terms but have different api.
07:11 yfried kiran-r: so I suggest that you make clear that you are writing to lbaasV1
07:11 kiran-r yfried, I am thinking in that front.
07:11 yfried kiran-r: so that when you will write the v2, you won't have clashing names.
07:12 yfried kiran-r: I just had an idea. boris-42 introduced something called plugin-namespace. maybe it would be useful here.
07:13 kiran-r yfried: I am clueless..
07:13 kiran-r yfried: What is the next step I should take??
07:13 yfried kiran-r: me too. you should ask him about that
07:14 kiran-r boris-42: ping
07:14 kiran-r boris-42 seems to be inactive in IRC.
07:14 yfried kiran-r: sorry for confusing you. the whole lbaas versions has me confused as well. that's why I want to be extra careful here
07:14 kiran-r yfried: no problem
07:14 anshul joined #openstack-rally
07:14 kiran-r yfried: Please help me close on this. It is taking soooo loong.
07:15 yfried kiran-r: sorry. It would help if we had a lbaas guru on this that could say if/how we should change everything
07:28 e0ne joined #openstack-rally
07:30 kiran-r yfried: ping
07:30 yfried kiran-r: ?
07:30 kiran-r yfried: https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L946-L949 V2 pools https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1065-L1067 v1 pool
07:31 kiran-r yfried: v2 resources starts with lbaas_*
07:31 kiran-r yfried: I think we can differentiate this way
07:31 pbandzi joined #openstack-rally
07:32 yfried kiran-r: ok. but you're gonna have to document this in commint messages and docstrings
07:32 kiran-r yfried: https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L365-L380
07:33 yfried kiran-r: on second thought, that's too confusing
07:33 yfried kiran-r: let's go with v1 and v2
07:33 kiran-r yfried, v1 path is /lb/ while v2 is /lbaas/
07:33 yfried kiran-r: I see
07:33 yfried kiran-r: but rally ppl won't be diving into neutron code for that
07:34 yfried kiran-r: suffix all your terminology with _v1
07:34 kiran-r yfried: True. so it is better to rename s/pool/v1_pool
07:34 yfried kiran-r: or that :)
07:34 kiran-r yfried: I totally understand your point.
07:34 yfried kiran-r: but make it explicit for the LoadBlancer scenario as well
07:35 yfried kiran-r: ie scenarios/neutron/loadblancer_1
07:35 yfried kiran-r: ie scenarios/neutron/loadblancer_v1
07:35 kiran-r yfried: I get it. :)
07:36 yfried kiran-r: and in your docs and commig msgs mention the lb<->lbaas terminology of neutron
07:36 yfried kiran-r: you could also mention how crappy and annoying it is, but that is entirely up to you :)
07:38 kiran-r yfried: :)
07:38 kiran-r yfried: Would you +2 my code if I commit with those changes??
07:38 kiran-r yfried: ;) ^
07:39 yfried kiran-r: I haven't reviewed it in details yet, because this was a big pain point for me, but yeah. in principal I would
07:41 openstackgerrit Oleh Anufriiev proposed openstack/rally: [Fuel] add and remove nodes scenario  https://review.openstack.org/181270
07:41 openstackgerrit Oleh Anufriiev proposed openstack/rally: [Fuel] Environments scenarios  https://review.openstack.org/182873
07:43 kiran-r yfried: What if I name it lb_pool for v1 and lbaas_pool for v2?
07:43 yfried kiran-r: no!
07:43 yfried kiran-r: use v1 and v2
07:43 yfried kiran-r: we can easily change that later
07:44 kiran-r yfried: Reason, neutron differentiates it by lb and lbaas.. :)
07:44 yfried kiran-r: think about the scenario name: lb_loadblancer and lbaas_loadbalancer?
07:44 yfried kiran-r: and look how we are still discussing it because it's unclear
07:44 kiran-r yfried: you are right, lb_loadblancer and lbaas_loadbalancer sounds ugly.
07:45 karimb joined #openstack-rally
07:48 neeti joined #openstack-rally
07:49 arxcruz joined #openstack-rally
07:56 amalinow joined #openstack-rally
08:09 fhubik joined #openstack-rally
08:09 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer create and list pool  https://review.openstack.org/182685
08:19 yingjun joined #openstack-rally
08:35 mahito joined #openstack-rally
08:36 neeti_ joined #openstack-rally
08:40 cdent joined #openstack-rally
08:45 adiantum joined #openstack-rally
08:51 openstackgerrit Mikhail Dubov proposed openstack/rally: Implementation of 'rally task abort' command  https://review.openstack.org/161636
08:57 amaretskiy joined #openstack-rally
09:06 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer (LBaaS v1) create and list pool  https://review.openstack.org/182685
09:08 rook joined #openstack-rally
09:13 tosky joined #openstack-rally
09:42 yingjun joined #openstack-rally
09:46 nkhare joined #openstack-rally
09:47 e0ne joined #openstack-rally
09:53 yingjun joined #openstack-rally
09:58 tosky joined #openstack-rally
10:01 e0ne joined #openstack-rally
10:02 BitSmith joined #openstack-rally
10:03 openstackgerrit Igor Degtiarov proposed openstack/rally: [Ceilometer] Add  scenario for events and traits  https://review.openstack.org/180514
10:25 exploreshaifali joined #openstack-rally
10:27 redixin joined #openstack-rally
10:41 openstackgerrit Mikhail Dubov proposed openstack/rally: Implementation of 'rally task abort' command  https://review.openstack.org/161636
10:43 openstackgerrit Mikhail Dubov proposed openstack/rally: Implementation of 'rally task abort' command  https://review.openstack.org/161636
10:48 pboldin joined #openstack-rally
10:52 nkhare_ joined #openstack-rally
10:59 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add base benchmark for Manila list shares operation  https://review.openstack.org/190580
11:00 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add base benchmark for Manila list shares operation  https://review.openstack.org/184553
11:01 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add base benchmark for Manila list shares operation  https://review.openstack.org/184553
11:13 nkhare_ joined #openstack-rally
11:14 openstackgerrit joined #openstack-rally
11:15 pboldin yfried: hi there
11:15 yfried pboldin: hi
11:16 pboldin yfried: what do you think about the commit message here? https://review.openstack.org/182685/
11:18 pboldin yfried: I really thing that we should forget about the v2 at the moment and only mention that it is implementatioon of v1
11:18 pboldin yfried: like Add Neutron LoadBalancer (LBaaSv1) create and list pool
11:19 yfried pboldin: reviewing it now.
11:20 pboldin yfried: well, changeset was just ok before it started to mention that there is v2 and before renaming
11:20 yfried pboldin: unless you know differently, I am worried that if we don't use "v1" explicitly, we'll run into problems with the v2
11:20 yfried pboldin: it was actually my request
11:20 pboldin yfried: we should not, because as Alex points out, the class is named accordingly
11:20 yfried pboldin: I agree with that
11:21 yfried pboldin: if the class is "v1" then it's obvious the methods are v1 as well
11:21 yfried pboldin: but that's not true for utils
11:21 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer(v1) create and list pool  https://review.openstack.org/182685
11:21 pboldin yfried: yep. so, I think that '_v1_' is only confusing at the moment.
11:21 yfried pboldin: I'm commenting now. let me post it and see what you thing
11:21 yfried think
11:24 kiran-r pboldin, yfried, ameretskiy: I just pushed a new patchset with Alex suggested changes.
11:24 pboldin kiran-r: please, dont rush
11:24 yfried kiran-r: I'm just reviewing the old one.
11:24 kiran-r pboldin, yfried, ameretskiy: I am sorry
11:25 kiran-r pboldin, yfried, ameretskiy: Pardon me.
11:32 yfried kiran-r: posted comments. pboldin please take a look as well.
11:36 e0ne joined #openstack-rally
11:49 openstackgerrit Mikhail Dubov proposed openstack/rally: Implementation of 'rally task abort' command  https://review.openstack.org/161636
12:05 tfreger1 joined #openstack-rally
12:09 aix joined #openstack-rally
12:38 rdas joined #openstack-rally
12:59 tfreger joined #openstack-rally
13:04 mahito joined #openstack-rally
13:15 openstackgerrit Roman Vasilets proposed openstack/rally: Remove deprecated code  https://review.openstack.org/188077
13:25 stpierre joined #openstack-rally
13:37 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer(v1) create and list pool  https://review.openstack.org/182685
13:40 openstackgerrit maplelabs proposed openstack/rally: Add Neutron Loadbalancer(v1) create and list pool  https://review.openstack.org/182685
14:04 exploreshaifali joined #openstack-rally
14:17 psd joined #openstack-rally
14:38 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add scenario for creation and deletion of Manila shares  https://review.openstack.org/190667
14:53 praveens joined #openstack-rally
14:57 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add scenario for creation and deletion of Manila shares  https://review.openstack.org/190667
14:57 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add context for Manila quotas  https://review.openstack.org/188484
15:03 exploreshaifali joined #openstack-rally
15:05 pboldin joined #openstack-rally
15:24 BitSmith joined #openstack-rally
15:24 agarciam joined #openstack-rally
15:25 pboldin https://review.openstack.org/190379/ < review pls
15:26 adiantum joined #openstack-rally
15:29 arxcruz joined #openstack-rally
15:31 praveens1 joined #openstack-rally
15:51 amalinow|2 joined #openstack-rally
15:51 yeungp joined #openstack-rally
15:57 praveens joined #openstack-rally
15:59 PrashantS joined #openstack-rally
16:06 praveens joined #openstack-rally
16:10 kiran-r joined #openstack-rally
16:20 anshul joined #openstack-rally
16:27 e0ne joined #openstack-rally
16:27 amalinow|2 joined #openstack-rally
16:29 openstackgerrit Chris St. Pierre proposed openstack/rally: Overhaul ceilometer utils unit tests  https://review.openstack.org/190712
16:35 jaypipes joined #openstack-rally
16:41 praveens left #openstack-rally
16:50 exploreshaifali joined #openstack-rally
16:56 stpierre sweet, gate looks broken
17:04 openstackgerrit John Wu proposed openstack/rally: [Swift] Add additional Swift scenarios  https://review.openstack.org/185793
17:04 openstackgerrit John Wu proposed openstack/rally: [Swift] Add objects context class  https://review.openstack.org/179578
17:07 philibar joined #openstack-rally
17:08 philibar hey boris-42, the issue we were seeing last time was actually an error somewhere in our config, hence why it would go grab the other config file instead, thanks for the support!
17:08 anshul joined #openstack-rally
17:10 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add scenario for creation and deletion of Manila share networks  https://review.openstack.org/190724
17:12 pboldin joined #openstack-rally
17:36 anshul joined #openstack-rally
17:46 openstackgerrit Valeriy Ponomaryov proposed openstack/rally: Add scenario for creation and listing of Manila share networks  https://review.openstack.org/190737
17:56 tosky joined #openstack-rally
18:06 karimb joined #openstack-rally
18:36 boris-42 philibar: you are welocme
18:39 stpierre boris-42: any idea what might be awry with gate? gate-rally-dsvm-neutron-rally is reliably failing (and taking a long-ass time to do so); the only errors i can find in the logs are, e.g.: http://logs.openstack.org/85/182685/33/check/gate-rally-dsvm-neutron-rally/8925927/logs/devstack-gate-setup-workspace-new.txt
18:40 boris-42 stpierre: I saw it today
18:40 boris-42 stpierre: something got terrible broken
18:40 stpierre cool
18:42 stpierre is there anything i can chase down to try to fix it? or are smarter and more capable people than me already hard at work on it?
18:44 boris-42 stpierre: I am in process of relocation
18:44 boris-42 stpierre: so a bit busy=)
18:44 boris-42 stpierre: so at this momnet of time nobody is working
18:44 boris-42 stpierre: I believe that something bad was merged in devstack or neutron
18:45 stpierre okay, i'll try to dig in
18:45 boris-42 stpierre: so you can try to install devstack with neutron
18:45 boris-42 stpierre: and try to run it
18:45 boris-42 stpierre: as well we need critical bug for this
18:45 stpierre i can try, but devstack and i don't usually get along
18:45 boris-42 stpierre: because something is very bad=(
18:45 stpierre that i can definitely do
18:45 stpierre raise the bug against rally, or...?
18:46 boris-42 stpierre: ya first in rally
18:51 e0ne joined #openstack-rally
19:05 exploreshaifali joined #openstack-rally
19:22 boris-42 stpierre: do not reduce timeout.
19:22 boris-42 stpierre: let's make a patch set that removes all neutron scenarios
19:22 boris-42 stpierre: and try to run it in gates
19:23 stpierre ok
19:25 boris-42 stpierre: I mean you will get result in 20 minutes not in 2hrs=)
19:25 stpierre that's better
19:26 openstackgerrit Chris St. Pierre proposed openstack/rally: DO NOT MERGE: remove neutron jobs  https://review.openstack.org/190771
19:26 boris-42 stpierre: just left single test
19:26 stpierre oh, it needs one test?
19:26 boris-42 stpierre: this won't work
19:26 boris-42 stpierre: yep
19:26 boris-42 stpierre: Authenticate.keystone
19:26 boris-42 stpierre: 1 iteration
19:26 boris-42 stpierre: with 1 user and 1 tenant
19:26 boris-42 stpierre: that will be very fast
19:27 boris-42 stpierre: ya debugging in gates party hard
19:27 openstackgerrit Chris St. Pierre proposed openstack/rally: DO NOT MERGE: remove neutron jobs  https://review.openstack.org/190771
19:28 boris-42 stpierre: btw this may help https://review.openstack.org/#/c/189463/6
19:29 boris-42 stpierre: I just recheked it
19:30 gema joined #openstack-rally
19:33 boris-42 stpierre: btw after fixing gates
19:38 anshul joined #openstack-rally
19:40 BitSmith joined #openstack-rally
19:44 pboldin joined #openstack-rally
19:44 yeungp joined #openstack-rally
19:54 PrashantS joined #openstack-rally
20:03 pboldin joined #openstack-rally
20:10 boris-42 stpierre: gate-rally-dsvm-neutron-rally: SUCCESS
20:10 boris-42 stpierre: LOL
20:10 stpierre ...the hell
20:10 stpierre god dammit
20:10 boris-42 stpierre: we need to run Neutron
20:10 boris-42 stpierre: benchmark
20:10 boris-42 stpierre: that is failing
20:10 boris-42 stpierre: so gates are not totally broken LOL
20:10 stpierre yeah
20:11 stpierre actually, this might be enough, if the devstack logs exist
20:11 stpierre i'll get another build going, but this may give me the info i need
20:11 boris-42 stpierre: just create one more patch
20:11 boris-42 stpierre: with another change-id
20:12 boris-42 stpierre: with neutron benchmark
20:14 openstackgerrit Chris St. Pierre proposed openstack/rally: DO NOT MERGE: only one neutron task  https://review.openstack.org/190796
20:24 itzikb joined #openstack-rally
20:29 itzikb Hi, When running boot-runcommand-delete with times:300 I get errors No more IP addresses available on network 23e5ed3b-1a83-4da2-bb1c-750844156a15
20:30 itzikb Seems like ports are not deleted
20:30 pboldin itzikb: what version do you use?
20:31 itzikb pboldin: 0.0.5
20:31 pboldin itzikb: hm, there were a patch that made sure we remove them. nova-net or neutron?
20:31 itzikb neutron
20:32 pboldin itzikb: well, seems like you will have to remove them by hand. btw, were there any command script values? this could lead to unfreeed ports
20:34 itzikb pboldin: I used the instance_dd_test.sh script
20:34 pboldin itzikb: well, then I want you to reproduce the error while gathering a debug info. this seems like a bug
20:35 openstackgerrit Oleh Anufriiev proposed openstack/rally: [Fuel] add and remove nodes scenario  https://review.openstack.org/181270
20:39 itzikb joined #openstack-rally
20:40 boris-42 itzikb: hey there
20:40 boris-42 itzikb: so this may happen
20:40 itzikb boris-42: Sorry have connecion problems
20:40 boris-42 itzikb: because floating ip addresses are not cleaned up immediately
20:40 boris-42 itzikb: as far as I know..
20:41 boris-42 itzikb: btw are you using latest rally?
20:41 itzikb boris-42: yes
20:41 itzikb Got it fresh from git today
20:42 itzikb boris-42: so I can run just 250 times or so ?
20:43 stpierre i hacked up a context that cleans up floating IPs. i'm guessing that's the wrong solution, though.
20:45 boris-42 stpierre: itzikb https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/vm/vmtasks.py#L103-L105
20:45 boris-42 stpierre: itzikb ^ they are cleaned in scenario
20:45 boris-42 itzikb are there some warrning/errors?
20:46 itzikb boris-42: where in rally or neutron?
20:47 stpierre boris-42: at least in nova-net, that doesn't actually reap the IPs for reuse by another VM.
20:47 boris-42 itzikb: rally
20:47 PrashantS joined #openstack-rally
20:47 boris-42 stpierre: ya because there is some kind of periodic job
20:48 boris-42 stpierre: that cleans them
20:48 boris-42 stpierre: not sure about Neutron, but I think it's the same situtation
20:48 boris-42 itzikb: ^
20:48 boris-42 itzikb: so if there is no errors you will need to do some magic to get more floating IPs
20:49 itzikb boris-42: I don't know why , they have to be freed , I run 40 vms a time so it shouldn't be a problem
20:50 itzikb I have >200 floating ips
20:52 boris-42 itzikb: I believe they are not cleaned up immediately
20:53 boris-42 itzikb: e.g. we send request remove floating ip from VM, but this floating IP is not immediately available for reassign
20:54 openstackgerrit Chris St. Pierre proposed openstack/rally: DO NOT MERGE: only one neutron task  https://review.openstack.org/190796
20:54 boris-42 itzikb: let me ask neutron guys
20:55 itzikb boris-42: I think the problem is with the internal ports and not the floating ips
20:56 itzikb boris-42: While running I have ~10 floating ips used but the number of internal network ports are growing
20:56 boris-42 itzikb: seems quite bad
20:57 boris-42 itzikb: let's try to debug this stuff
20:57 boris-42 itzikb: could you run rally task in debug mode
20:57 boris-42 itzikb: like just vmtask 1 iteration
20:57 boris-42 itzikb: rally -vd task start ...
20:57 boris-42 itzikb:  so we will see all requests to neutron
21:00 boris-42 stpierre: oh you found it https://github.com/openstack/neutron/commit/9d649129ca762f406846190d31997397f0a0c53b ?)
21:00 itzikb joined #openstack-rally
21:00 stpierre "found it" is probably overstating things a bit
21:00 boris-42 stpierre: so we have rally job in neutron
21:00 boris-42 stpierre: you can try to revert this and check
21:00 stpierre but that's the only commit that actually does something that was committed between the last known good build and the first known bad build
21:00 stpierre oh, sweet
21:02 boris-42 stpierre: actually
21:02 openstackgerrit Chris St. Pierre proposed openstack/rally: DO NOT MERGE: only one neutron task  https://review.openstack.org/190796
21:02 boris-42 stpierre: https://review.openstack.org/#/c/176711/
21:02 boris-42 stpierre: seems it works
21:03 boris-42 stpierre: e.g. when they merge this everything worked
21:03 stpierre dang
21:03 boris-42 itzikb: ping
21:04 itzikb boris-42: hey again
21:04 boris-42 itzikb: so did you receive my messages?
21:04 boris-42 itzikb: regarding to run rally in debug mode
21:04 boris-42 itzikb: just one iteration of vmtask
21:04 itzikb boris-42: no but I have the logs
21:05 boris-42 stpierre: take a look here https://review.openstack.org/#/c/189463/
21:06 boris-42 stpierre: seems like there is problem with requriments file
21:06 stpierre i get a 503 from that, so yay gerrit
21:06 stpierre [16:05:54] <fungi> #status notice Gerrit has been restarted to terminate a persistent looping third-party CI bot
21:07 itzikb boris-42: ok I'm running it now
21:07 stpierre i'm not sure what i'm looking at on that review
21:07 openstackstatus NOTICE: Gerrit has been restarted to terminate a persistent looping third-party CI bot
21:08 boris-42 stpierre: now it's ok
21:08 boris-42 stpierre: https://review.openstack.org/#/c/189463/
21:08 boris-42 stpierre: so requirements are changed
21:09 boris-42 stpierre: seems like that is causing bugs
21:09 stpierre why do you think that's causing it?
21:09 boris-42 stpierre: because I just recheked
21:09 boris-42 stpierre: that patch and it works
21:09 boris-42 stpierre: ah no
21:09 boris-42 stpierre: it's old jenkins result
21:10 stpierre yeah, zuul shows that rally-gate-dvsm-neutron-rally is still going on that patchset, which is consistent with it failing
21:13 boris-42 stpierre: ya my fault
21:21 itzikb boris-42: What do you want to see in the debug mode?
21:22 boris-42 itzikb: whole logs
21:22 boris-42 itzikb: paste them please here http://paste.openstack.org/
21:24 itzikb boris-42: Isn't the paste for short ones? It doesn't have attach a file
21:25 boris-42 itzikb: so you can create gist
21:25 boris-42 https://gist.github.com/
21:25 boris-42 itzikb: ^ it doesn't have limitations
21:28 stpierre okay, whatever's happening here is decidedly touchy. both of my attempts to induce a failure with just a single neutron task have passed gate unexpectedly, so it must have something to do with load or concurrency or something
21:29 stpierre i'm taking off in ~10 minutes, but everything i know is in the ticket
21:29 stpierre so hopefully someone else can look at it
21:30 boris-42 stpierre: thanks for working on this
21:36 openstackgerrit Boris Pavlovic proposed openstack/rally: DO NOT MERGE: only one neutron task  https://review.openstack.org/190796
21:37 itzikb boris-42: https://drive.google.com/file/d/0B_sJ3nkK4dLEcHc1dVc2Mm4wbW8/view?usp=sharing
21:40 boris-42 itzikb: please next time send full log
21:40 boris-42 itzikb: including the line "rally task start..."
21:42 itzikb boris-42: ok
21:42 boris-42 itzikb: something is strange
21:42 boris-42 itzikb: I see tons of requests to delete various ports
21:43 boris-42 REQ: curl -i http://10.35.160.29:9696//v2.0/ports/3dbb6aa0-25de-4ae9-b69b-30b7b0fb833e.json -X DELETE -H "User-Agent: python-neutronclient" -H "X-Auth-Token: XXXX"
21:43 boris-42 itzikb: doesn't seem ok ><
21:45 boris-42 шitzikb 2015-06-12 00:29:42.192 991 INFO rally.benchmark.runner [-] Task 6a376e0e-acce-4431-afda-ab9ea5540c02 | ITER: 296 END: Error GetResourceErrorStatus: Resource <Server: rally_volume_XRVqWIVw6Z> has ERROR status.
21:45 boris-42 itzikb: doesn't seem like a single iteration benchamrk log
21:45 boris-42 itzikb: btw seems like vms are not starting
21:46 boris-42 itzikb: could you send logs only of 1 single iteration benchmakr
21:46 boris-42 itzikb: without any other logs
21:46 boris-42 itzikb: that will help
21:48 itzikb boris-42: WHen you say one iteration what do you mean - what values for times and concurrency?
21:48 boris-42 itzikb: times:1 concurrency: 1
21:48 boris-42 itzikb: and single benchmark inside the task
21:49 boris-42 itzikb: and logs only of that run
21:49 boris-42 itzikb: I don't want logs of 300 vms ...
21:49 boris-42 itzikb: btw VM becomes in failure state
21:49 mpopow joined #openstack-rally
21:50 boris-42 itzikb: I am not sure that this is because of not enough floating ips
21:50 boris-42 itzikb: it looks more like not enough fixed ips in network
21:50 boris-42 itzikb: but can't say antyhing more without logs of nova
22:00 mpopow joined #openstack-rally
22:02 itzikb boris-42: Hope it's better now https://drive.google.com/file/d/0B_sJ3nkK4dLEcHc1dVc2Mm4wbW8/view?usp=sharing
22:02 itzikb the command is basically sudo rally -vd --log-file mydebug.log task start boot-runcommand-delete.yaml
22:03 boris-42 itzikb: hm why do you need sudo ouch??
22:04 boris-42 itzikb: wrong link
22:04 boris-42 itzikb: seems like old one debug log
22:04 boris-42 itzikb: 2015-06-12 00:29:11.802 992 INFO rally.benchmark.runner [-] Task 6a376e0e-acce-4431-afda-ab9ea5540c02 | ITER: 276 END: Error IpAddressGenerationFailureClient: No more IP addresses available on network 77d5f606-b463-472f-8774-c394c0842cc2.
22:04 boris-42
22:04 boris-42 itzikb: ^ there can't be ITER 276
22:05 itzikb boris-42: https://drive.google.com/file/d/0B_sJ3nkK4dLEaXJ0QVU3RTBoV0U/view?usp=sharing
22:06 dpaterson joined #openstack-rally
22:07 dpaterson boris-42: Mr. Boris, please review my tempest cli spec at your leasure: https://review.openstack.org/#/c/138785/
22:07 PrashantS joined #openstack-rally
22:08 boris-42 dpaterson: heh that is taking a lot of time....
22:08 boris-42 dpaterson: neverending spec
22:08 dpaterson no shit
22:09 boris-42 dpaterson: =))
22:10 boris-42 itzikb: this one looks better
22:10 dpaterson boris-42: It's key, at least imo, that the interface is in line with rally/tempest integration
22:10 dpaterson Why I am asking you take a looksee
22:10 boris-42 dpaterson: hm but why you are coding it if they don't want?)
22:10 boris-42 dpaterson: maybe it's better to improve rally verify =)
22:11 boris-42 dpaterson: sure looking at it
22:11 dpaterson boris-42: CLI was one of the main objectives discussed at summit
22:11 dpaterson Hoping to have more cycles for rally this release as well.
22:14 boris-42 dpaterson: +1
22:14 boris-42 dpaterson: can't do +2 in tempest sorry=0
22:14 boris-42 itzikb: ya this file is better
22:14 boris-42 itzikb: let me take a look at it
22:14 dpaterson boris-42: thanks and np.
22:25 boris-42 itzikb: the LOG is very very strange
22:25 boris-42 itzikb: take a look at this part http://paste.openstack.org/show/284621/
22:26 boris-42 itzikb: so we are executing SSH command, after that we are removing FloatingIP
22:26 boris-42 itzikb: but not deleting VM..
22:27 boris-42 itzikb: I can't understand how that can happen according to the cod tat we have in master...
22:32 boris-42 itzikb: and I don
22:32 boris-42 itzikb: I don't see logs of VM creation=(
22:34 itzikb boris-42: strange indeed..
22:40 boris-42 itzikb: all logs except nova..
22:40 itzikb boris-42: Did you see the errors regarding security group?
22:41 boris-42 itzikb: so no worries about that
22:42 boris-42 itzikb: it's known issue and i just keep that logs to remember to fix that part
22:42 itzikb too late for me to worry ..
22:42 boris-42 itzikb: so they are cleaned during generic cleanup that affects resources of context (so it tries to delete already delete sec group)
22:43 boris-42 itzikb: I mean that one is not a bug that affects anything except logs
22:43 itzikb makes sense
22:43 boris-42 itzikb: but I am worried about why I don't see logs from nova
22:44 itzikb boris-42: When you run it do you see the logs?
22:45 boris-42 itzikb: run boot-and-delete
22:45 boris-42 itzikb: and yep I see such stuff
22:45 boris-42 REQ: curl -g -i 'http://172.16.5.129:8774/v2/6521e53effa746b6985b784cb361b381/servers/fe170fb7-1426-4d34-8389-02750c464cf7' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: ctx_rally_581fa3a0-09c1-4483-871e-b2b62283a804_tenant_0" -H "X-Auth-Token: {SHA1}8f4c0da92425f7352d1dd107bf40988f804dafd0"
22:47 boris-42 itzikb: same with boot_runcommand_delete
22:47 boris-42 itzikb: I see logs from python nova client
22:47 boris-42 REQ: curl -g -i 'http://172.16.5.129:8774/v2/67701527f4c74d9aa2447666bcb33af5/servers/8573d911-7973-435c-9703-0f91fbd225f4' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: ctx_rally_d3e263eb-ffa0-4184-a80f-227a83dd3219_tenant_0" -H "X-Auth-Token: {SHA1}db89fc1cfca5d001b1fc5f06cf62f109345fa2b1"
22:58 itzikb boris-42: good night
22:58 itzikb boris-42: Hope we'll solve it
22:59 boris-42 itzikb: see you
22:59 boris-42 itzikb: btw what is your time zone?
23:42 adiantum joined #openstack-rally
23:46 openstackgerrit Boris Pavlovic proposed openstack/rally: DO NOT MERGE: only one neutron task  https://review.openstack.org/190796

| Channels | #openstack-rally index | Today | | Search | Google Search | Plain-Text | summary