Perl 6 - the future is here, just unevenly distributed

IRC log for #openstack-rally, 2015-04-03

| Channels | #openstack-rally index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 davideagnello boris-42: no it wouldn't be an issue as long as we can execute specific tests.  We are currently comparing Tempest and Rally as a test platform for our OpenStack service.  I'm still learning about how it works and capabilities.
00:05 boris-42 davideagnello: rally executes only what you are saying and how you are saying
00:05 boris-42 davideagnello: you are passing task file that is run
00:05 boris-42 davideagnello: like this https://github.com/stackforge/rally/blob/master/rally-jobs/rally.yaml
00:05 davideagnello boris-42: ok that's great
00:05 boris-42 davideagnello: so what is specified here will be executed
00:06 boris-42 davideagnello: so if you decide to use rally
00:06 boris-42 davideagnello: for gating
00:06 boris-42 davideagnello: you will have such directory (like in cinder) https://github.com/openstack/cinder/tree/master/rally-jobs
00:06 boris-42 davideagnello: so this is what you will run against patches https://github.com/openstack/cinder/blob/master/rally-jobs/cinder.yaml
00:07 boris-42 davideagnello: and here you can paste any rally plugins https://github.com/openstack/cinder/tree/master/rally-jobs/plugins
00:07 boris-42 davideagnello: (scenarios, context, sla, load generators)
00:07 boris-42 davideagnello: and use them in your gate
00:09 boris-42 davideagnello: so rally is not like unittest framework
00:09 boris-42 davideagnello: where you are running all tests
00:10 davideagnello boris-42: are there projects which have rally plugins defined?
00:11 boris-42 davideagnello: seems like there is https://github.com/stackforge/ec2-api/tree/master/rally-scenarios/plugins
00:11 boris-42 davideagnello: but there is no difference between plugins in rally source code and in that directory
00:11 davideagnello boris-42: been looking at various projects and didn't find any
00:11 boris-42 davideagnello: mechanism is the same
00:12 boris-42 davideagnello: so usually people just merges in rally patches
00:12 boris-42 davideagnello: with scenarios
00:12 davideagnello boris-42: ok, I see
00:13 boris-42 davideagnello: but you can do first in that directory and then some day move to rally source code
00:13 davideagnello boris-42: thanks for your help and clarifications
00:13 boris-42 davideagnello: the reason why we are trying to collect everything in rally
00:13 boris-42 davideagnello: is simplicity for usage by Ops and QA people in real world
00:13 davideagnello boris-42: that makes sense
00:14 boris-42 davideagnello: that just don't want to spend time to understand where to find stuff for benchmarking
00:14 boris-42 davideagnello: and they would like to have at least some guantees
00:14 boris-42 davideagnello: btw I will strongly recommend to read step by step guide https://rally.readthedocs.org/en/latest/tutorial/step_1_setting_up_env_and_running_benchmark_from_samples.html
00:15 boris-42 davideagnello: it will simplify understanding of how rally works and how to use it properly
00:16 davideagnello boris-42: yes, almost there in deploying and running it locally
00:17 boris-42 davideagnello: so basically rally was designed for local and preciese testing
00:17 boris-42 davideagnello: when you would like to test some case exactly
00:17 boris-42 davideagnello: but during the 2 years scope changed a lot
00:18 davideagnello boris-42: what do you mean by precise testing?
00:18 boris-42 davideagnello: like catching some races
00:18 boris-42 davideagnello: measruing perfromance of something and comparing it with different arguments
00:18 boris-42 davideagnello: not like full stack testing
00:18 boris-42 davideagnello: but during the almost 2 years
00:19 boris-42 davideagnello: scope changed a lot
00:19 boris-42 davideagnello: so now we would like to cover case of certification at scale, which means that everything works under expected load
00:19 boris-42 davideagnello: so it's like integration testing++
00:20 davideagnello boris-42: ok cool, I have read something about this on the wiki :)
00:21 boris-42 davideagnello: I will have talk in Vancouver related to this topic
00:21 boris-42 davideagnello: http://openstacksummitmay2015vancouver.sched.org/event/ec4f18c7ff2c0614d2ecff15251e237f#.VR3dBZOUeQw
00:21 boris-42 davideagnello: okay I have to go and sleep a bit =)
00:21 boris-42 davideagnello: it's too late here
00:22 davideagnello boris-42: awesome!  thanks for your help!  have a good evening there
00:24 boris-42 davideagnello: see you
00:33 psd joined #openstack-rally
00:42 openstackgerrit Oleh Anufriiev proposed stackforge/rally: Removed task validation from api.Task.start  https://review.openstack.org/158899
00:51 panbalag joined #openstack-rally
01:00 dpaterson joined #openstack-rally
01:14 openstackgerrit Merged stackforge/rally: Lazy import of keystoneclient in osclients  https://review.openstack.org/169729
01:34 rvcehimanshu joined #openstack-rally
01:42 openstackgerrit Kun Huang proposed stackforge/rally: configure ssl in deployment config  https://review.openstack.org/167471
01:48 ilbot3 joined #openstack-rally
01:48 Topic for #openstack-rally is now ☁ Rally RoadMap: http://goo.gl/JZkmwY ☁ Rally IRC chat logs http://irclog.perlgeek.de/openstack-rally ☁ Key persons to ask:  boris-42, yfried, msdubov, rediskin, andreykurilin, amaretskiy  ☁ Documentation: https://rally.readthedocs.org/en/latest/ ☁ To publish changes to Rally:  https://rally.readthedocs.org/en/latest/contribute.html
02:01 pradeep joined #openstack-rally
02:02 openstackgerrit Kun Huang proposed stackforge/rally: configure ssl in deployment config  https://review.openstack.org/167471
02:07 panbalag joined #openstack-rally
02:08 panbalag left #openstack-rally
02:08 panbalag joined #openstack-rally
02:31 yingjun joined #openstack-rally
02:54 baker joined #openstack-rally
02:59 mwagner_lap joined #openstack-rally
03:12 yingjun_ joined #openstack-rally
03:31 prashantS joined #openstack-rally
03:38 openstackgerrit Kun Huang proposed stackforge/rally: add test for deployment check  https://review.openstack.org/170342
03:39 kun_huang need help on mirantis ci work...  https://review.openstack.org/#/c/167471/
04:09 pradeep joined #openstack-rally
04:15 rvcehimanshu joined #openstack-rally
04:40 pradeep joined #openstack-rally
04:45 prashantS joined #openstack-rally
04:47 zhiyan joined #openstack-rally
04:47 kun_huang joined #openstack-rally
04:49 kun_huang joined #openstack-rally
05:09 yingjun joined #openstack-rally
05:56 yingjun joined #openstack-rally
06:24 pradeep2 joined #openstack-rally
06:32 aswadr joined #openstack-rally
06:41 pradeep joined #openstack-rally
06:54 soumit joined #openstack-rally
06:59 openstackgerrit Vadim Rovachev proposed stackforge/rally: Fix validation for rps parameter  https://review.openstack.org/170160
07:09 e0ne joined #openstack-rally
07:24 e0ne joined #openstack-rally
07:30 arxcruz joined #openstack-rally
07:52 pradeep joined #openstack-rally
08:06 openstackgerrit Sergey Skripnick proposed stackforge/rally: WIP: super new rally_gate.py  https://review.openstack.org/163785
08:08 amaretskiy joined #openstack-rally
08:28 e0ne joined #openstack-rally
08:31 openstackgerrit Vitaly Gusev proposed stackforge/rally: [Ceilometer] Add scenarios for resources and samples  https://review.openstack.org/165092
08:45 openstackgerrit OpenStack Proposal Bot proposed stackforge/rally: Updated from global requirements  https://review.openstack.org/170393
08:50 andreykurilin_ joined #openstack-rally
08:51 openstackgerrit svasheka proposed stackforge/rally: Add unit test for keystone._get_service_by_name  https://review.openstack.org/169034
08:52 rvcehimanshu1 joined #openstack-rally
08:53 openstackgerrit svasheka proposed stackforge/rally: Add keystone create_user_update_password scenario API covered users.update_password  https://review.openstack.org/165422
08:53 openstackgerrit svasheka proposed stackforge/rally: Add keystone benchmark scenarios for roles  https://review.openstack.org/165409
08:53 openstackgerrit svasheka proposed stackforge/rally: Add keystone create_update_and_delete_tenant scenario  https://review.openstack.org/165450
08:56 openstackgerrit Alexander Gubanov proposed stackforge/rally: Adds Nova floating IPs bulk tests  https://review.openstack.org/168054
09:04 pradeep joined #openstack-rally
09:05 psd joined #openstack-rally
09:11 openstackgerrit Pradeep K Surisetty proposed stackforge/rally: Add keystone create_and_list_services scenario  https://review.openstack.org/170401
09:16 svasheka hi guys, dsvm gates still not working?
09:26 amaretskiy svasheka hi, gates worked good yesterday. not sure about what is happening right now
09:26 pradeep svasheka: when was the last time u noticed it ?
09:27 pradeep svasheka: it worked for me this morning
09:30 svasheka pradeep: just half an hour ago
09:30 svasheka rechecked number of commits
09:30 svasheka all faild on dsvm
09:31 pradeep svasheka: i just sent mine. waiting.
09:31 pradeep will let u know
09:33 openstackgerrit Mikhail Dubov proposed stackforge/rally: [Docs] Add tutorial chapter about task templates  https://review.openstack.org/168867
09:38 pradeep1 joined #openstack-rally
09:46 tosky joined #openstack-rally
10:10 kun_huang boris-42: I guess the debate here is whether it helps in the OpenStack Mission -- and my tentative answer to that would be yes.
10:10 kun_huang boris-42: from ttx
10:11 kun_huang boris-42: congs first:)
10:14 pboros joined #openstack-rally
10:36 pradeep joined #openstack-rally
10:52 psd_ joined #openstack-rally
10:53 praveens joined #openstack-rally
10:53 openstackgerrit Pradeep K Surisetty proposed stackforge/rally: Add keystone create_and_list_services scenario  https://review.openstack.org/170401
11:02 openstackgerrit Mikhail Dubov proposed stackforge/rally: Add streaming algorithms and SLA to check for outliers  https://review.openstack.org/148079
11:10 openstackgerrit Mikhail Dubov proposed stackforge/rally: [Docs] Add tutorial chapter about task templates  https://review.openstack.org/168867
11:20 aix joined #openstack-rally
11:23 pradeep1 joined #openstack-rally
11:36 pradeep joined #openstack-rally
11:44 openstackgerrit Roman Vasilets proposed stackforge/rally: Add Http Request Scenarios  https://review.openstack.org/117705
11:45 openstackgerrit Roman Vasilets proposed stackforge/rally: Add Http Request Scenario  https://review.openstack.org/117705
11:51 rook joined #openstack-rally
12:10 svasheka pradeep: hows your commit doing on dsvm gates?
12:17 pradeep svasheka: fails. Some thing wrong with dsvm gates
12:18 pradeep svasheka: lets wait
12:18 pradeep svasheka: all are failing.
12:18 pradeep once dvsm  fixed, they will go through
12:20 svasheka pradeep: ok, ty for the info
12:21 e0ne joined #openstack-rally
12:24 psd__ joined #openstack-rally
12:27 fhubik_meeting joined #openstack-rally
12:27 tosky_ joined #openstack-rally
12:31 fhubik_lunch joined #openstack-rally
12:31 tosky__ joined #openstack-rally
12:41 boris-42 pradeep: svasheka mirantis rally ci doesn't work for now
12:42 boris-42 kun_huang: hehe we need to wait for others
12:46 dpaterson joined #openstack-rally
12:47 openstackgerrit Nikita Konovalov proposed stackforge/rally: [Sahara] Improve Image context  https://review.openstack.org/166859
12:56 pradeep boris-42: ping
12:56 pradeep boris-42:  Do i need resend (https://review.openstack.org/#/c/169598/)  it now with your suggested changed. or it will get merged as it has 2 reviews?
13:20 boris-42 зpradeep I have to go butI will be later and I will take a look
13:36 openstackgerrit joined #openstack-rally
13:45 openstackgerrit Nikita Konovalov proposed stackforge/rally: [Sahara] More debug logging  https://review.openstack.org/170491
13:56 baker joined #openstack-rally
14:03 prashantS joined #openstack-rally
14:09 exploreshaifali joined #openstack-rally
14:24 prashantS joined #openstack-rally
14:39 openstackgerrit joined #openstack-rally
14:42 jaypipes joined #openstack-rally
15:01 e0ne boris-42: ping!
15:21 exploreshaifali joined #openstack-rally
15:28 psd_ joined #openstack-rally
15:29 openstackgerrit Pradeep K Surisetty proposed stackforge/rally: Add heat list_stack_and_events scenario  https://review.openstack.org/170537
15:31 prashantS joined #openstack-rally
15:48 pradeep1 joined #openstack-rally
15:50 prashantS joined #openstack-rally
16:20 baker joined #openstack-rally
17:01 davideagnello boris-42: when comparing Rally with Tempest, why would one choose one over the other?  they solve the same problem, Rally seems more automated in configuring your OpenStack environment and has nice HTML reports that can get generated, benchmarking
17:04 psd_ joined #openstack-rally
17:07 openstackgerrit Roman Vasilets proposed stackforge/rally: Add functional test for task samples  https://review.openstack.org/163870
17:22 exploreshaifali joined #openstack-rally
17:24 openstackgerrit Prabhjyot Singh Sodhi proposed stackforge/rally: Create 'add cloud info' spec file  https://review.openstack.org/169432
17:40 psd__ joined #openstack-rally
17:43 pradeep joined #openstack-rally
18:10 leakypipes joined #openstack-rally
18:29 baker joined #openstack-rally
18:41 aix joined #openstack-rally
19:06 prashantS joined #openstack-rally
19:27 prashantS joined #openstack-rally
19:31 e0ne joined #openstack-rally
19:37 mitz joined #openstack-rally
19:38 prashantS joined #openstack-rally
19:51 rook__ joined #openstack-rally
20:16 psd joined #openstack-rally
20:50 rook__ joined #openstack-rally
20:51 rook___ joined #openstack-rally
21:00 andreykurilin_ joined #openstack-rally
21:09 andreykurilin_ joined #openstack-rally
21:17 prashantS joined #openstack-rally
21:17 rook__ joined #openstack-rally
22:29 rook__ joined #openstack-rally
22:40 harlowja joined #openstack-rally
23:27 openstackgerrit Merged stackforge/rally: Add murano benchmarks  https://review.openstack.org/137661
23:29 boris-42 davideagnello: hi there
23:30 boris-42 davideagnello: so I can provide few samples for key moments if you would like
23:31 Viswanath joined #openstack-rally
23:49 davideagnello boris-42: hello!  yes, that would be great
23:50 psd joined #openstack-rally
23:52 boris-42 davideagnello: so first thing that is very important is checking that things real works and measuring perfromance
23:52 boris-42 davideagnello: so in case of tempest you can test that 1 VM can start, but what will happen if we try to start 10 VM ?
23:53 boris-42 davideagnello: and will cloud stay alive if we try to create-delete vm 10kk times non stop?
23:53 boris-42 davideagnello: so you will need to write N tests to test these N cases
23:54 boris-42 davideagnello: with rally you can write only one scenario in the way that you need
23:54 rook__ joined #openstack-rally
23:54 boris-42 davideagnello: and can you reuse it in different cases
23:55 boris-42 davideagnello: as well splitting success criteria, load generation, context (setup/teardown), scenarios in separated plugins allow you to haevly reuse code
23:55 boris-42 davideagnello: so you don't need to write load generators for each scenario and you don't need to modify scenario to run it with different load
23:55 boris-42 davideagnello: success criteria allows you to not write negative tests
23:56 boris-42 davideagnello: like you would like to test quotas, you are setting CPU=1 and try to run Nova boot scenrario twice
23:56 boris-42 davideagnello: and you need to ensure that 1 time it was created sucessfully
23:56 boris-42 davideagnello: so you don't need to modify scenario and sla just need to combine them
23:56 boris-42 in proper way
23:57 boris-42 davideagnello: the another thing is that all plugins are parametrized
23:57 davideagnello boris-42: awesome, sounds powerful.  I have been playing around with Tempest today and I can see the relations to these limitations
23:57 boris-42 davideagnello: so you don't need to write N scenarios to test N set of arguments that accept nova
23:57 boris-42 davideagnello: and as well splitting of context and scenario is very important
23:58 davideagnello boris-42: that makes sense
23:58 boris-42 davideagnello: if you would like to make clean benchmarks that just starts/deletes VMs and not create users tenatns and sets quotas on each iteration
23:59 boris-42 davideagnello: so benifits of rally frameworks are: 1) flexibility 2) small amount of code 3) high reusability 4) pluguability
23:59 boris-42 5) rich results 6) pretty reports 7) as well rally stores all results for all deployments that you register

| Channels | #openstack-rally index | Today | | Search | Google Search | Plain-Text | summary