Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-07-10

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 angdraug Kupo24z1: there's been a custom patch for 4.1.x to rename nodes arbitrarily, I don't think it made it to 5.x
00:13 angdraug https://blueprints.launchpad.net/fuel/+spec/node-naming
00:31 IlyaE joined #fuel
00:38 vogelc joined #fuel
00:38 vogelc how would I build a specific fuel iso, such as #110?
01:00 xarses joined #fuel
01:02 angdraug vogelc: hi, what I meant in that ticket is that you should download it from us
01:02 vogelc How do I do that?
01:03 vogelc angdraug:  did you see my latest update in the ticket?
01:03 angdraug yep
01:03 angdraug I'm trying to confirm that the problem starts with the difference in the environment, not in the iso image
01:04 vogelc angdraug:  I have to step away for a few minutes but I will be back.
01:04 angdraug sure
01:05 angdraug I'm uploading iso #110 to google drive, I'll IM you the link when it's done
01:07 vogelc angdraug: cool - thx
01:09 angdraug btw another theory: could your problem be related to adding osd nodes _after_ initial deployment?
01:10 vogelc we have been seen both today.  We have tried tens of iso's
01:11 vogelc we just spun up a second cluster and that had some failures on initial install
01:13 nick019238981 joined #fuel
01:16 mattgriffin joined #fuel
01:20 angdraug 5.0.1 is still in development, it's a stable series but there can be wrinkles
01:22 angdraug my most recent point of investigation was difference between up and down osd's during activation
01:22 angdraug at some point after activation osd process restarts itself, and when it does it's supposed to report to ceph-mon that it's going down
01:23 vogelc angdraug: So here's the difference, when I do initial deploy I see the osd's go down and come back up. when adding nodes.  they go down and out
01:23 angdraug in ceph-mon logs, you can see that the osd's that are marked down by mon never report themselves going down post-activation
01:23 angdraug what baffles me is node-1 in your first log bundle, the one that had only 2 osd's stay down
01:24 angdraug in all other reports you've posted there was a clear boundary by node
01:24 vogelc right.  the new cluster today had two nodes that way
01:25 angdraug another possibility we can try to eliminate is ceph bug
01:25 angdraug we have Firefly packages for 5.1
01:26 angdraug 5.1 itself is still not ready for you to try, but you can try to upgrade ceph on one of your clusters and see if it changes anything
01:26 vogelc nick019238981 is upgrading the second cluster to 5.1,  so that's probably not going to work either?
01:27 vogelc I have deployed CEPH on it's own but not coupled with fuel
01:27 angdraug 5.1 may fail due to reasons unrelated to ceph
01:29 angdraug the ceph part of it might actually work well
01:29 vogelc what is failing 5.1?
01:29 vogelc I saw lock down was 7/1
01:29 angdraug no, we moved that, we're not past feature freeze yet
01:29 vogelc btw - what is the best way to  pull 5.1?  from master?
01:30 angdraug so far it's the only way to pull it )
01:30 angdraug we won't branch stable/5.1 until soft code freeze, that's weeks away
01:30 vogelc ok that is what we are installing.
01:30 angdraug but after we've got feature freeze it will be just bugfixes, so things will settle down
01:31 vogelc so we have a 9:00 cst call with someone from mirantis.  what code set do you recommend we be at?  5.0/stable?
01:32 angdraug yes
01:32 angdraug btw in 5.1 you'll have this issue: https://bugs.launchpad.net/fuel/+bug/1333814
01:32 angdraug there's a patch attached to the bug but it didn't make its way into package repos for 5.1 yet
01:33 vogelc yuck
01:33 angdraug yeah, we fixed it for ceph-deploy once, but they managed to break it in upstream again
01:33 angdraug when we pulled new version I had to redo the fix
01:34 vogelc bummer......
01:34 vogelc Thanks for all of your help this evening.
01:34 angdraug I suspect ceph 0.80 can be deployed with old ceph-deploy from previous fuel versions, but I haven't tried that
01:34 angdraug no, thanks for coming here with all this!
01:35 angdraug it's a good feedback to have, especially if it ends up with more bugfixes before 5.0.1 is out :)
01:35 vogelc Im sure we'll be talking more after tomorrow.
01:35 vogelc :)
01:35 angdraug did you try your luck with #ceph channel on OFTC yet?
01:36 vogelc I have not, to be honest I thought the ceph version stayed the same from 4.1 to 5.0
01:36 vogelc 4.1 rocked
01:36 angdraug 4.1 to 5.0 yes, 5.0 to 5.1 upgrades from dumpling to firefly
01:36 angdraug oh wait, you can deploy 4.1 on those clusters?
01:37 vogelc 4.1 works just fine
01:37 angdraug oh, now it's even more interesting
01:37 vogelc our cloud peeps wants icehouse
01:38 vogelc I did a nuke and pave btw
01:38 angdraug ok, let me double check package versions
01:38 eshumakher joined #fuel
01:39 angdraug 4.1.1 actually has a more recent ceph version: 0.67.9 vs 0.67.8
01:39 angdraug was it 4.1 or 4.1.1 that you had?
01:39 vogelc 4.1
01:39 angdraug 4.1 had 0.67.5
01:40 vogelc 4.1 was awesome but missing some of the networking features
01:40 angdraug hm. I'm tempted to ask you to try 4.1.1, to see if there was a regression betwen those minor ceph versions
01:40 angdraug not very likely, but easy enough to eliminate
01:41 angdraug we're trying to make 5.0.1 as awesome as 4.1 was
01:41 angdraug fixed tons of stuff
01:41 vogelc I pulled it down last night and tried it.  strange error.  when defining the env it said it failed and I could do it manually.
01:41 angdraug that's why I'm so eager to get to the bottom of your problem )
01:42 vogelc do you know if all drives are wiped during install?
01:45 angdraug yes, and we changed a lot in that area between 4.1 and 4.1.1 and 5.0
01:45 angdraug but so far, the problems with incorrectly wiped drives mostly came up during provisioning
01:45 angdraug meaning that even partitions won't be created
01:45 vogelc ok - nick is trying 5.1 now.  if that blows, we'll do 4.1.1
01:45 vogelc is there an upgrade path from 4.1.1 to 5.0?
01:45 angdraug unfortunately no
01:45 angdraug 5.0 is the first upgradeable realease of fuel
01:45 vogelc damn docker
01:45 vogelc not against it but it was a big change
01:45 angdraug indeed
01:46 angdraug well, a lot of people were asking for upgradeability so there was no way around it
01:47 angdraug did you get that iso #110?
01:47 angdraug that's an interim build of 5.0.1 I made yesterday, the one that worked for us in a similar cluster
01:48 angdraug what time zone are you in? I'm thinking of going home but we definitely should revisit this tomorrow!
01:48 vogelc I did not get the link.  I am in Minnesota or CST
01:49 angdraug sent it again
01:49 angdraug should come up as a private irc message
01:49 vogelc Im kind of slow - got the link
01:49 vogelc have a good onw
01:49 vogelc one
01:49 angdraug thanks, you too!
02:14 fhond joined #fuel
02:32 IlyaE joined #fuel
03:04 IlyaE joined #fuel
03:19 mattgriffin joined #fuel
03:22 IlyaE joined #fuel
04:39 ArminderS joined #fuel
04:45 IlyaE joined #fuel
05:37 AndreyDanin joined #fuel
06:47 IlyaE joined #fuel
06:50 ddmitriev joined #fuel
06:58 pasquier-s joined #fuel
07:22 Longgeek joined #fuel
07:23 al_ex10 joined #fuel
07:25 Longgeek joined #fuel
07:33 hyperbaba joined #fuel
07:47 hyperbaba joined #fuel
07:53 hyperbaba hi there , where can i find fuel5.0.1 upgrade tarball?
08:15 Arminder joined #fuel
08:15 pasquier-s joined #fuel
08:18 guillaume__1 joined #fuel
08:37 e0ne joined #fuel
08:52 brain461 joined #fuel
09:05 mrasskazov joined #fuel
09:05 aglarendil joined #fuel
09:07 nurla joined #fuel
09:15 accela-dev joined #fuel
09:25 e0ne joined #fuel
09:29 dotty joined #fuel
09:31 geekinut1h joined #fuel
09:58 bookwar joined #fuel
09:59 hyperbaba_ joined #fuel
10:03 hyperbaba__ joined #fuel
10:04 pasquier-s_ joined #fuel
10:13 neith joined #fuel
10:16 scroiset joined #fuel
10:21 brain461 joined #fuel
10:24 Longgeek joined #fuel
10:45 AndreyDanin joined #fuel
10:50 ddmitriev joined #fuel
11:09 warpc_ joined #fuel
11:20 e0ne joined #fuel
11:57 radosevic joined #fuel
11:57 radosevic hello
11:58 radosevic is there any way i can upgrade running fuel 5.0 to 5.0.1
11:58 guillaume__1 joined #fuel
11:58 pasquier-s joined #fuel
12:05 radosevic anyone?
12:09 evg_ radosevic: hi, yes. there will be a tarball soon.
12:09 radosevic so there isn't one yet?
12:11 evg_ radosevic: there is unofficial one
12:12 radosevic where i can get it, please?
12:12 evg_ radosevic: there hasn't been official v5.0.1 yet
12:12 evg_ radosevic: ok. moment
12:13 radosevic evg_: thanks
12:13 radosevic evg_: waiting... no problem
12:50 al_ex11 joined #fuel
13:11 tatyana joined #fuel
13:12 e0ne joined #fuel
13:43 pasquier-s joined #fuel
13:47 guillaume__1 joined #fuel
13:51 Arminder- joined #fuel
13:53 guillaume__1 joined #fuel
13:59 mattgriffin joined #fuel
14:02 jobewan joined #fuel
14:17 wrale I'm having an issue where the second and third controller plus all compute node installs fail.. The first controller is fine.. On 5.0.1, GRE, HA, Ubuntu, Ceph for all backing.  Would choosing CentOS be any better?
14:24 radosevic wrale: what's in error log for that nodes?
14:28 wrale radosevic: they were puppet errors, but i couldn't find the initial failure that caused them.. the fuel db is busted now for some reason (tried to reset the env and stopped midstream)..
14:29 radosevic wrale: yes... puppet, but you have to be more specific...
14:29 al_ex11 joined #fuel
14:32 wrale radosevic: agreed.. but i'm not troubleshooting that problem at the moment.. i'm going to try with centos
14:32 wrale i thought it may be a well known issue as described by the node affliction
14:35 Longgeek joined #fuel
14:40 wrale How do I set the MTU for my NIC's RE: https://bugs.launchpad.net/fuel/+bug/1274789 ?
14:43 meow-nofer_ joined #fuel
14:49 mattymo joined #fuel
14:51 angdraug joined #fuel
14:55 IlyaE joined #fuel
14:59 georgem2 joined #fuel
15:20 evg_ wrale: I may be wrong but I think only via "fuel --downlod/ediiting yaml file/--upload"
15:20 evg_ wrale: see here http://docs.mirantis.com/openstack/fuel/fuel-5.0/reference-architecture.html#id49
15:21 evg_ wrale: but, sorry, I myself don't see the ready recipe
15:23 wrale thanks evg_ .. i'll try to figure it out
15:23 ArminderS joined #fuel
15:27 xarses joined #fuel
15:52 tatyana joined #fuel
15:52 vogelc_ joined #fuel
15:53 vogelc_ anyone running into deploying ceph storage nodes where radom osd's report as down?
15:57 AndreyDanin joined #fuel
16:28 IlyaE joined #fuel
16:30 wrale evg_: it looks like i can set the mtu in the interfaces block, but it seems quite undocumented
16:30 brain461 joined #fuel
16:34 evg_ wrale: indeed undocumented. port_properties/interface_properties in The "Transformations" Section?
16:43 evg_ wrale: aaaa, sorry, misread.
16:48 e0ne joined #fuel
16:59 wayneeseguin joined #fuel
17:02 mutex joined #fuel
17:04 whytewol1 joined #fuel
17:06 whytewol1 I'm having an issue with a couple of discovery servers they pxe boot off of eth0, but once the discovery system is booted eth0 has become another ethernet port and and one of the 10GB cards is now eth0. this is on the latest 5.0
17:11 evg_ wrale: I suppose it was "network_scheme: interfaces: eth0: L2: mtu: 1500". Am I right?
17:12 wrale evg_: i would have thought the same, but the source code makes it out to be under L3 instead of L2, which makes little sense to me
17:15 tatyana joined #fuel
17:17 crandquist joined #fuel
17:18 whytewol1 okay, never mind this looks like it might be a network or switch issue with turning off the port while we reboot the server
17:31 tatyana joined #fuel
17:36 ArminderS joined #fuel
17:39 georgem2 whytewol1:enable portfast on the switch ports
18:11 IlyaE joined #fuel
18:25 vogelc_ joined #fuel
18:33 WhyteWolf georgem2: it is enabled on the switch ports. that was one of the first things we checked :/ we are looking into some kind of powersaving feature now.
18:35 WhyteWolf trouble shooting this issue sucks cause these dells take forever to reboot
18:40 AndreyDanin joined #fuel
18:44 Kupo24z1 Is there any docs on replacing a controller with a new server?
19:03 AndreyDanin joined #fuel
19:21 e0ne joined #fuel
19:26 sanek joined #fuel
19:28 e0ne joined #fuel
19:46 angdraug joined #fuel
19:47 Kupo24z1 xarses: MiroslavAnashkin angdraug I'm getting a 'Invalid staet of instance files on shared storage' when evacuating, http://pastebin.mozilla.org/5541877 is there an open bug for this?
19:47 Kupo24z1 state*
19:49 Kupo24z1 Since its using ceph ephemeral is the --on-shared-storage needed?
19:51 angdraug probably not
19:51 Kupo24z1 we get {"badRequest": {"message": "host and onSharedStorage must be specified.", "code": 400}} if we try the request without it
19:53 angdraug see https://bugs.launchpad.net/nova/+bug/1250751 and https://review.openstack.org/91722 for context
19:54 angdraug "shared storage" in nova assumes NFS in many places
19:54 angdraug the commit above removes that assumption for the case of live migration
19:54 angdraug but there sure are many more places inside nova that still assume the same thing
19:57 Kupo24z1 angdraug: it appears the evacuate works via CLI, just the API gives us the troubles
19:57 Kupo24z1 ill keep digging
19:58 Kupo24z1 well maybe not, it might have just booted from the base image heh
19:59 angdraug what's the exact command you're using?
19:59 Kupo24z1 just 'nova evacuate c1c9378c-beb2-4d4b-9f81-aff6ab016e44 node-24'
20:00 Kupo24z1 judging how the inode table was expanded and SELinux initialized the drive it looks like it booted from the base image snapshot
20:11 angdraug why not "nova live-migration c1c9378c-beb2-4d4b-9f81-aff6ab016e44 node-24"?
20:13 Term1nal joined #fuel
20:13 Term1nal Just installed Mirantis Fuel master node with the ISO. Used fuelmenu to change the IP address, though I'm unable to reach the fuel webui.
20:21 MiroslavAnashkin Term1nal: Did you finished installation? It may take ~1hour to install master node after you exit fuelmenu.
20:22 Term1nal well.. what happened, was I plopped in the disc, hit fuel install
20:22 Term1nal and it just did everything on its own without any notifications or interaction.
20:22 Term1nal formatted my drives without warning (not that I cared I was going to anyway, just saynig) and eventually landed on a prompt
20:23 Term1nal I'm running it on a beefy Xeon quad core with SSDs so it didn't take NEARLY that long..
20:23 Term1nal I ran fuelmenu once I got the centos login prompt modified with the fuel URL
20:23 MiroslavAnashkin So, you did ran fuelmenu after the installation finished?
20:23 Term1nal I never had any chance to run it -before-
20:24 MiroslavAnashkin You had a chance to run it -in scope- ;)
20:25 sanek joined #fuel
20:25 Term1nal I can rerun the installation I suppose, but is changing the IP before even ever touching the install/webui really that detrimental?
20:26 MiroslavAnashkin Yes, it is mandatory to make all necessary changes to network setup in scope of master node installation. http://docs.mirantis.com/fuel/fuel-5.0/user-guide.html#download-and-install-fuel
20:29 Term1nal ahh, tab
20:29 Term1nal blegh
20:38 vogelc_ Does any have a 5.0 cluster with more than 80 ceph OSDs?
20:39 angdraug vogelc_: hi, I thought your problems are starting much sooner than 80 OSDs?
20:40 angdraug as in, around 20-30 OSDs?
20:41 vogelc_ angdraug:  that was happening when we were only deploying.  today we tried deploying all 8 nodes at once.  one one cluster we got 80 osds up,  the other we got 81 up out of a total of 88.
20:41 vogelc_ angdraug:  that was happening when we were only deploying 4 nodes at a time
20:42 angdraug this indicates that it's a function of time, not number of OSDs
20:42 angdraug at some point in time, OSDs stop talking to ceph-mon, and after that point, no new OSDs ever come up
20:43 vogelc_ angdraug: I talked to some people at inktank and showed them our logs.  they said the osds look fine they are just not joining the cluster.  your point about time kind of makes sense.
20:43 MiroslavAnashkin What if we try to re-deploy one of failed OSDs manually?
20:44 Kupo24z1 angdraug: i can confirm it issues a rebuild if you dont include shared storage on evacuate
20:44 vogelc_ I tried to redeploy one osd twice, and no luck
20:44 Kupo24z1 and your other question, we are simulating a downed compute node so live migrate wont be an option; ERROR: Compute service of node-25 is unavailable at this time. (HTTP 400)
20:48 angdraug ah, in that case you might need another patch for nova
20:48 Kupo24z1 should we file a bug in nova?
20:48 angdraug yes please
20:48 angdraug actually..
20:48 angdraug https://bugs.launchpad.net/nova/+bug/1249319
20:49 angdraug is that it?
20:50 angdraug https://bugs.launchpad.net/nova/+bug/1332536 also relevant: you don't see it yet but you will
20:50 angdraug hm no 1249319 is a separate problem
20:50 angdraug so yeah, please raise a new bug
21:04 Kupo24z1 angdraug: if we manualy copy the instance directory over to the destination server before we take the node offline, then attempt a evacuation using shared storage would it work theoretically?
21:05 AndreyDanin joined #fuel
21:05 Kupo24z1 (assuming in a production setting the instance directories are backed up then restored to destination nodes on request)
21:06 angdraug yes, that might work
21:06 angdraug I'll need to spend some time with evacuate code path to be more confident in saying that
21:06 angdraug but what you're saying is very logical
21:59 Kupo24z1 angdraug: so i tested it out, looks like it just rebuilt the image from source again however i found another workaround
21:59 Kupo24z1 editing nova.instances node references in the table then issuing a reset-status active, then hard reboot
22:00 Kupo24z1 that works fine
22:03 Kupo24z1 heres the posted bug https://bugs.launchpad.net/nova/+bug/1340411
22:05 angdraug thanks!
22:07 Kupo24z1 do you know the difference between nova.instances:host, nova.instances:launched_on, and nova.instances:node ?
22:07 Kupo24z1 Just making sure we dont modify a column we dont need to
22:11 sanek joined #fuel
22:15 angdraug vogelc_: good news, we have reproduced your problem locally
22:18 angdraug Kupo24z1: no
22:18 angdraug don't know
22:33 jseutter joined #fuel
22:33 bas_ joined #fuel
22:33 dmitryme joined #fuel
22:33 AndreyDanin joined #fuel
22:33 Arminder joined #fuel
22:36 bookwar joined #fuel
22:36 23LAA0QIE joined #fuel
22:36 wrale joined #fuel
22:36 mihgen joined #fuel
22:36 MorAle joined #fuel
22:36 dburmistrov joined #fuel
22:36 evg_ joined #fuel
22:36 book` joined #fuel
22:36 christop1eraedo joined #fuel
22:36 monester_ joined #fuel
22:36 akislitsky joined #fuel
22:36 MiroslavAnashkin joined #fuel
22:36 meow-nofer joined #fuel
22:36 sbog_ joined #fuel
22:36 izinovik joined #fuel
22:36 holser joined #fuel
22:36 alex_didenko joined #fuel
22:36 vkramskikh joined #fuel
22:36 apalkina joined #fuel
23:25 intLabs joined #fuel
23:31 intLabs Hi, I’ve been using fuel as part of a larger project that I am now looking to release as an ISO for easy installation by users, modified to install a custom dashboard and additonal services. Does anyone have any advice on the appropriate way to credit the developers; my understanding is that all code within the fuel project is licenced under Apache 2.0 (as is my own code), but I would appreciate input that anyone has?
23:31 mattgriffin joined #fuel
23:45 eshumakher joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary