Perl 6 - the future is here, just unevenly distributed

IRC log for #puppet-openstack, 2016-06-26

| Channels | #puppet-openstack index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 xingchao joined #puppet-openstack
00:32 markvoelker joined #puppet-openstack
00:45 ayoung PUT https://[::1]:35357/v3/projects/1494113d1d984061​be09bd9dab38beb8/users/ae93abf775e24d06b15738ba4​b094a3f/roles/a8ddbba58b36472dbe2c0b701933c74d  is doing a role assignment for a user on a project, and it looks like a duplicate
00:45 ayoung Not a bug in Keystone,  except maybe if that is how it reports the error
00:46 ducttape_ joined #puppet-openstack
00:47 ayoung INSERT INTO role (id, name, extra) VALUES (%s, %s, %s)']
00:47 ayoung that looks wrong
00:48 ayoung I love that we are goind IPv6 though
00:49 ayoung It looks like it is tripping over the old code we had to set the default Role...hhmm
00:51 ayoung http://git.openstack.org/cgit/openstack/keyston​e/tree/keystone/assignment/controllers.py#n401
00:51 ayoung mwhahaha, EmilienM ^^
00:51 ayoung It looks like something is trying to create a "Member" role
00:52 ayoung and, since it is done as a side effect...it looks wrong
00:54 mwhahaha So I wonder if we have a user role create that's doing it before tempest
00:54 mwhahaha Or should tempest even be doing that
00:55 ayoung mwhahaha, what test is that anyway?
00:55 mwhahaha The setup for nova v2 flavors
00:55 mwhahaha Is what is failing
00:55 mwhahaha Like when it creates the credentials
00:55 mwhahaha Is what I tracked down
00:56 ayoung mwhahaha, so, it looks like an error in Keystone, and I wonder if it is due to the domain specific role code, but we've only now started seeing it?
00:56 mwhahaha So I'm wondering if we are creating a similar role when we run the scenario
00:57 mwhahaha I don't think we should allow a null in that column if it's going to result in a 409
00:57 ayoung OK...lets styart from here
00:57 ayoung http://git.openstack.org/cgit/openstack/keyst​one/tree/keystone/assignment/routers.py#n114
00:58 ayoung its a put so
00:58 ayoung grant_controller, create_grant
00:58 ayoung http://git.openstack.org/cgit/openstack/keyston​e/tree/keystone/assignment/controllers.py#n625
00:59 mwhahaha I'll get to my computer in a bit but like I said I wonder if we're inadvertently creating the same user/role that tempest tries to use
00:59 ayoung mwhahaha, I think it is more subtle
00:59 ayoung http://git.openstack.org/cgit/openstack/keyston​e/tree/keystone/assignment/controllers.py#n631
00:59 ayoung self.assignment_api.create_grant(
01:00 mwhahaha But it seems like a bad thing to allow a null in a constraint like that
01:00 mwhahaha It's the domain that's getting dropped right?
01:00 ayoung http://git.openstack.org/cgit/openstack/key​stone/tree/keystone/assignment/core.py#n339
01:00 ayoung not sure
01:01 ayoung a FULL stack trace would help
01:01 ayoung keystone.common.sql.core   is pretty far down the chain
01:01 ayoung [SQL: u'INSERT INTO role (id, name, extra) VALUES (%s, %s, %s)'] [parameters: ('a312190b7554410ebc815289f7390e8f', u'Member', '{}')]
01:02 ayoung looks like it is trying to create the GLocal Member role
01:02 ayoung and it should not be doing that
01:02 ayoung so first it checks that the role exists self.role_api.get_role(role_id)
01:02 ayoung if domain_id:
01:03 ayoung http://git.openstack.org/cgit/openstack/keys​tone/tree/keystone/assignment/core.py#n1163
01:03 ayoung http://git.openstack.org/cgit/openstack/keystone/​tree/keystone/assignment/role_backends/sql.py#n70
01:03 xingchao joined #puppet-openstack
01:03 ayoung OK that is straight ahead
01:03 ayoung just a fetch, no side effects
01:05 ayoung skip the call to get_domain.  does a d get_project self.resource_api.get_project(project_id)
01:05 ayoung self.driver.create_grant(role_id, user_id, group_id, domain_id,
01:05 ayoung project_id, inherited_to_projects)
01:05 mwhahaha so it looks like perhaps it's old code?
01:05 mwhahaha `domain_id` varchar(64) NOT NULL DEFAULT '<<null>>' is the last column in role
01:05 mwhahaha so if it's not trying to add it, it'll get the <<null>>
01:06 mwhahaha so is it like a v2 create against a v3 table or something
01:06 * mwhahaha is just guessing
01:06 ayoung http://git.openstack.org/cgit/openstack/keyston​e/tree/keystone/assignment/backends/sql.py#n51
01:06 ayoung inserts into the table
01:06 ayoung mwhahaha, red herring
01:06 ayoung domain is a new addition to the role table
01:06 ayoung before that, all roles were global
01:07 ayoung so it should be null 99.99% of the time
01:07 mwhahaha delicious delicious herrings
01:07 ayoung until people start doing domain specific roles...not there yet
01:07 mwhahaha let me go track down the tempest create i had it earlier today
01:07 ayoung the second call does not seem to line up with the first
01:08 ayoung mwhahaha, is 22699  and 22700  the thread/process ID?
01:08 mwhahaha yea i think so
01:08 ayoung 2016-06-25 14:17:14.731 22700 INFO keystone.common.wsgi [req-6654dff5-e5e3-4bc1-ab8e-97066f64ba0f e50189d010664d2bb5e2d1ca9471605e a12c0da88dc542c8abdf55f77998b6f1 - default default] GET https://[::1]:35357/v3/roles
01:09 ayoung that is the API being executed, not the line above then
01:09 ayoung OK.lets take a look at that
01:09 ayoung mwhahaha, feels like a race condition
01:09 mwhahaha probably
01:09 mwhahaha cause it is sporatic
01:09 mwhahaha i noticed it on friday
01:09 mwhahaha and recheck will pass sometimes
01:10 ayoung V3 roles...
01:10 ayoung Standard crud call,
01:10 mwhahaha http://git.openstack.org/cgit/opensta​ck/tempest/tree/tempest/test.py#n351 is the function it's erroring in
01:11 ayoung http://git.openstack.org/cgit/openstack/keyston​e/tree/keystone/assignment/controllers.py#n125
01:11 ayoung http://git.openstack.org/cgit/openstack/keys​tone/tree/keystone/assignment/core.py#n1175
01:12 ayoung http://git.openstack.org/cgit/openstack/keystone/​tree/keystone/assignment/role_backends/sql.py#n38
01:12 ayoung very straight ahead
01:15 ayoung mwhahaha, so we had logic in the v2 api add_user_to_project that created the member role if it did not exist
01:15 ayoung http://git.openstack.org/cgit/openstack/key​stone/tree/keystone/assignment/core.py#n194
01:16 ayoung it feels like that logic, but that should be v2 only and this is a v3 lsit_roles call, should not trigger that
01:17 ayoung mwhahaha, the fact that it is the "Member" role leads me to think something got it out of the Config file, like this code does
01:19 mwhahaha http://logs.openstack.org/05/332805/3/ch​eck/gate-puppet-openstack-integration-3-​scenario001-tempest-centos-7/6718b50/con​sole.html#_2016-06-25_14_28_57_994203
01:19 mwhahaha comes from tempest
01:19 ayoung self.creds_client.create_user_role('Member')  is wrong as a fish in beer
01:19 ayoung well, maybe if you were battering and cooking it
01:19 ayoung but certainly not swiming in my beer
01:20 ayoung mwhahaha, But I think some of the logging has been dropped.
01:20 mwhahaha wonder how to add more
01:21 ayoung mwhahaha, where is tempest/common/dynamic_creds.py _create_creds
01:21 ayoung http://git.openstack.org/cgit/openstack/tempe​st/tree/tempest//common/dynamic_creds.py#n112
01:21 ayoung mwhahaha, http://git.openstack.org/cgit/openstack/tempe​st/tree/tempest//common/dynamic_creds.py#n162  throw a try block around that
01:22 ayoung the role is almost always created
01:22 mwhahaha so bug in tempest?
01:22 ayoung mwhahaha, looks like it
01:22 ayoung mwhahaha, my guess is this
01:23 ayoung two threads running at the same time.  both do get_roles, see the role is missing, one inserts, the other inserts and errors
01:23 ayoung although it does not seem to be doing a "get_roles" call first
01:23 mwhahaha that's old code
01:23 mwhahaha from 2015
01:24 ayoung Maybe the keystoneclient code is less forgiving?
01:24 mwhahaha https://review.openstack.org/#/c/201816/
01:24 mwhahaha yea old code :D
01:24 mwhahaha also year old
01:25 ayoung mwhahaha, so look at Keystone client.  My guess is it used to let a user create a role that already exists, and now it doesn't
01:25 ayoung might even be Keystone  server that used to allow that
01:25 ayoung might have been in error...the constraint might have been put on for domain specific roles and is now catching the duplicate calls, where as before you were getting away with \putting crap in the database
01:26 ayoung I blame you, personally./
01:26 ayoung the error is unfriendly, though
01:27 mwhahaha :D
01:27 mwhahaha just wonder how this started to be a random problem like this week
01:28 mwhahaha i'll just blame mtreinish, i think that's what EmilienM usually does
01:28 * mwhahaha ducks
01:29 EmilienM it's not safe to ping me a saturday night
01:30 EmilienM my sobriety is really unstable
01:30 mwhahaha victory
01:31 mwhahaha :D
01:31 EmilienM https://dashboards.rdoproject.org/rdo-dev
01:31 mwhahaha anyway tempest doesn't use keystone client
01:31 EmilienM this dashboard is super cool
01:31 EmilienM it tells us how far puppet CI is from openstack trunk
01:31 EmilienM (1 day now)
01:31 EmilienM mwhahaha: is it happenning, like all the time?
01:31 EmilienM did something merged in tempest ?
01:32 mwhahaha i noticed it on friday
01:32 mwhahaha ayoung, http://git.openstack.org/cgit/openst​ack/tempest/tree/tempest/services/id​entity/v3/json/roles_client.py#n29
01:32 ayoung must be the server then
01:32 ayoung I think you were lucky befopre
01:32 markvoelker joined #puppet-openstack
01:32 ayoung UI think you were creating slews of Member roles and that is just scary
01:33 EmilienM ayoung: "you"
01:33 mwhahaha https://review.openstack.org/#/c/332805/ we rechecked a bunch of times
01:33 mwhahaha and each time the tests switch on which one is failing
01:33 EmilienM ayoung: we're deploying keystone & tempest from trunk
01:33 ayoung EmilienM, You in the case meaning tempest
01:33 mwhahaha sometimes scenario 1 sometimes 3
01:33 EmilienM so devstack might be affected no?
01:33 ayoung should not
01:33 EmilienM WHY?
01:33 ayoung devstack creates the role exactly once
01:33 EmilienM oops sorry
01:33 EmilienM I told you I have beers in hand
01:33 EmilienM ayoung: don't we ?
01:33 ntpttr- joined #puppet-openstack
01:33 ayoung I don't see an evidence of code that allowed a role create to succeed if the role already existed
01:34 ayoung that leads me to think that there was no uniqueness constraint on role name.  I could go and check, but I am lazy
01:34 ayoung EmilienM, watch out for the fish
01:35 EmilienM I have to google it man
01:35 EmilienM anyway
01:36 mwhahaha anyway we can continue this on monday :)
01:36 mwhahaha sorry to disturb teh drinking
01:36 EmilienM ayoung: if you have time on Monday, we can investigate what our CI does that devstack is not doing
01:36 EmilienM and figure if there is a bug somewhere
01:36 EmilienM now I'll ban you from IRC to make you're enjoying the week end
01:36 EmilienM damn, I don't know how to ban well, please leave
01:37 EmilienM it also reminds me we pin tempest for ubuntu nodes
01:37 ayoung EmilienM, I'm avoiding putting kids to bed
01:37 ayoung this is easier, believe me
01:37 ayoung Putting kids to bed is NP Complete
01:38 openstackgerrit Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge  https://review.openstack.org/333506
01:38 ayoung EmilienM, file a bug against Tempest
01:38 openstackgerrit Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge  https://review.openstack.org/333507
01:38 ayoung its is their mistake
01:38 openstackgerrit Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge  https://review.openstack.org/333508
01:38 openstackgerrit Emilien Macchi proposed openstack/puppet-openstack-integration: CI test - never merge  https://review.openstack.org/333509
01:38 xingchao joined #puppet-openstack
01:38 EmilienM right now I don't even know what to file
01:38 ayoung EmilienM, http://git.openstack.org/cgit/openstack/tempe​st/tree/tempest//common/dynamic_creds.py#n162  throw a try block around that
01:38 ayoung and now to go play dad
01:39 EmilienM we'll do it on Monday with Alex
01:39 EmilienM enjoy weekend folks /me afk
01:41 mwhahaha https://bugs.launchpad.net/tempest/+bug/1596250
01:41 openstack Launchpad bug 1596250 in tempest "setup_credentials failure in FlavorsV2TestJSON" [Undecided,New]
01:42 ayoung EmilienM, https://review.openstack.org/334209
01:46 mwhahaha going to throw a block around that ;0
01:46 ayoung mwhahaha, add to that review, please?
01:46 mwhahaha will do
01:46 ayoung and probablty should be a more specific exception
01:46 mwhahaha k updated
01:47 ilbot3 joined #puppet-openstack
01:47 Topic for #puppet-openstack is now CI status: orange: some async errors with zuul/ansible (WIP by os-infra) - Puppet OpenStack modules http://docs.openstack.org/deve​loper/puppet-openstack-guide/
02:16 xingchao joined #puppet-openstack
02:23 xingchao joined #puppet-openstack
03:49 ducttape_ joined #puppet-openstack
04:53 moravec joined #puppet-openstack
05:21 xingchao joined #puppet-openstack
05:50 xingchao joined #puppet-openstack
07:35 ilbot3 joined #puppet-openstack
07:35 Topic for #puppet-openstack is now CI status: orange: some async errors with zuul/ansible (WIP by os-infra) - Puppet OpenStack modules http://docs.openstack.org/deve​loper/puppet-openstack-guide/
07:40 jmikolajczak joined #puppet-openstack
07:52 ducttape_ joined #puppet-openstack
08:28 jmikolajczak joined #puppet-openstack
08:37 markvoelker joined #puppet-openstack
08:38 xingchao joined #puppet-openstack
08:53 ducttape_ joined #puppet-openstack
09:38 markvoelker joined #puppet-openstack
09:54 ducttape_ joined #puppet-openstack
09:57 xingchao joined #puppet-openstack
10:39 markvoelker joined #puppet-openstack
10:44 crinkle_ joined #puppet-openstack
10:46 alvladeru joined #puppet-openstack
10:55 ducttape_ joined #puppet-openstack
10:59 xingchao joined #puppet-openstack
11:40 markvoelker joined #puppet-openstack
11:56 ducttape_ joined #puppet-openstack
12:40 markvoelker joined #puppet-openstack
12:43 ducttape_ joined #puppet-openstack
13:30 johnmilton joined #puppet-openstack
13:39 xingchao joined #puppet-openstack
13:41 markvoelker joined #puppet-openstack
14:24 ducttape_ joined #puppet-openstack
14:40 xingchao joined #puppet-openstack
14:42 markvoelker joined #puppet-openstack
15:05 ducttape_ joined #puppet-openstack
15:31 ducttape_ joined #puppet-openstack
15:52 _stowa joined #puppet-openstack
16:18 v1k0d3n joined #puppet-openstack
16:39 yolanda joined #puppet-openstack
16:44 markvoelker joined #puppet-openstack
17:18 ducttape_ joined #puppet-openstack
17:42 xingchao joined #puppet-openstack
17:44 markvoelker joined #puppet-openstack
18:00 v1k0d3n joined #puppet-openstack
18:14 v1k0d3n joined #puppet-openstack
18:15 imcsk8 joined #puppet-openstack
18:19 ducttape_ joined #puppet-openstack
18:25 ducttape_ joined #puppet-openstack
18:31 vinsh joined #puppet-openstack
18:43 xingchao joined #puppet-openstack
18:47 markvoelker joined #puppet-openstack
18:49 ducttape_ joined #puppet-openstack
19:34 rcernin joined #puppet-openstack
19:44 xingchao joined #puppet-openstack
20:06 ducttape_ joined #puppet-openstack
20:45 xingchao joined #puppet-openstack
21:46 xingchao joined #puppet-openstack
21:49 markvoelker joined #puppet-openstack
23:45 ducttape_ joined #puppet-openstack
23:47 xingchao joined #puppet-openstack
23:50 markvoelker joined #puppet-openstack

| Channels | #puppet-openstack index | Today | | Search | Google Search | Plain-Text | summary