Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-12-30

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 taea00 Thanks semiosis I just realized that I can install the glusterfs client on the client machine instead of NFS.  Pretty awesome.
00:00 semiosis there you go
00:06 hchiramm joined #gluster
00:11 diegows joined #gluster
00:13 RicardoSSP joined #gluster
00:30 hchiramm joined #gluster
00:36 Pupeno joined #gluster
01:11 Pupeno joined #gluster
01:27 Pupeno joined #gluster
01:30 hagarth joined #gluster
01:54 bala joined #gluster
01:56 msmith_ joined #gluster
02:16 harish joined #gluster
02:29 side_control joined #gluster
02:59 anoopcs joined #gluster
03:00 hchiramm joined #gluster
03:02 anoopcs joined #gluster
03:16 suman_d_ joined #gluster
03:24 msmith_ joined #gluster
03:25 weykent joined #gluster
03:37 lalatenduM joined #gluster
03:51 itisravi joined #gluster
03:53 side_control joined #gluster
03:59 kanagaraj joined #gluster
04:03 RameshN joined #gluster
04:10 lalatenduM joined #gluster
04:11 nbalacha joined #gluster
04:11 nbalacha joined #gluster
04:13 aravindavk joined #gluster
04:20 shubhendu joined #gluster
04:28 kumar joined #gluster
04:41 side_con1rol joined #gluster
04:44 saurabh joined #gluster
04:48 ndarshan joined #gluster
04:51 side_control joined #gluster
04:51 bala1 joined #gluster
04:59 rafi1 joined #gluster
05:00 atinmu joined #gluster
05:03 soumya_ joined #gluster
05:05 spandit joined #gluster
05:09 prasanth_ joined #gluster
05:14 aravindavk joined #gluster
05:18 hchiramm joined #gluster
05:22 saurabh joined #gluster
05:27 kdhananjay joined #gluster
05:45 badone joined #gluster
05:46 ndarshan joined #gluster
05:47 hchiramm joined #gluster
06:04 raghu` joined #gluster
06:07 nishanth joined #gluster
06:12 lalatenduM joined #gluster
06:14 dusmant joined #gluster
06:14 smohan joined #gluster
06:19 fandi joined #gluster
06:25 ppai joined #gluster
06:26 soumya_ joined #gluster
06:32 ndarshan joined #gluster
06:37 rjoseph joined #gluster
06:38 fandi joined #gluster
06:50 fandi joined #gluster
07:10 fandi joined #gluster
07:20 dusmant joined #gluster
07:27 hagarth joined #gluster
07:30 atalur joined #gluster
07:33 suman_d_ joined #gluster
07:43 kovshenin joined #gluster
07:51 aravindavk joined #gluster
08:01 harish joined #gluster
08:11 fsimonce joined #gluster
08:25 anil joined #gluster
08:49 dusmant joined #gluster
08:50 nishanth joined #gluster
08:57 Hoggins joined #gluster
09:02 Hoggins Hello everyone, I believe I'm facing a split-brain scenario on a two-nodes (replicated) system, possibly after a network outage or a reboot of one of the two bricks... not sure... anyway, brick1 is using 100% CPU, while the other one is idle. First one reports things such as "0-thedude-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:7170854e-8c58-46a4-b77f-4451f8682286> (7170854e-8c58-46a4-b77f-4451f
09:02 Hoggins 8682286)" and "0-thedude-client-0: bailing out frame type(GlusterFS 3.3) op(OPEN(11)) xid = 0x447 sent = 2014-12-30 07:54:09.184529. timeout = 1800 for 192.168.148.3:49152" while the second one keeps reporting only "0-thedude-replicate-0: Another crawl is in progress for thedude-client-1"... where should I start ?
09:03 Hoggins (I can't mount on any client, and I have stopped all the client machines... only the Gluster servers are left alive)
09:07 elico joined #gluster
09:12 nangthang joined #gluster
09:13 JoeJulian Hoggins: use fpaste.org and copy/paste the relevant log info so I can have a little more context.
09:14 Hoggins will do, thanks !
09:14 JoeJulian What version?
09:16 Hoggins 3.5.3
09:17 Hoggins http://ur1.ca/j9nwo
09:18 Hoggins (not sure if I'm pasting the needed info)
09:22 JoeJulian Meh, that's just a self-heal daemon log.
09:23 JoeJulian You said you couldn't mount the volume from a client?
09:23 Hoggins yep
09:23 Hoggins do you need other logs ?
09:24 Hoggins oh, maybe that volume's specific log
09:24 pcaruana joined #gluster
09:24 Pupeno joined #gluster
09:25 harish joined #gluster
09:30 JoeJulian Truncate a client log, try to mount a client, paste that log.
09:39 Hoggins ok, I'm doing it
09:45 shubhendu joined #gluster
09:46 ndarshan joined #gluster
09:52 nishanth joined #gluster
10:06 fandi joined #gluster
10:11 glusterbot News from newglusterbugs: [Bug 1016494] Volume status operation after remove-brick is started on a volume fails, until remove-brick commit or remove-brick stop is done. <https://bugzilla.redhat.com/show_bug.cgi?id=1016494>
10:12 hagarth joined #gluster
10:12 fandi joined #gluster
10:19 pp__ joined #gluster
10:32 atalur joined #gluster
10:33 Hoggins JoeJulian: here you go, logs from the client trying to mount http://ur1.ca/j9obv
10:34 partner_ hmm not sure if this is wise to run 4.5.6 and 3.6.1 together, i haven't got even volume up and already my glusterd fails to start after reboot
10:34 partner_ also: State: Peer Rejected (Connected)
10:41 glusterbot News from newglusterbugs: [Bug 1176011] Client sees duplicated files <https://bugzilla.redhat.com/show_bug.cgi?id=1176011>
10:43 Pupeno joined #gluster
10:43 Pupeno joined #gluster
10:48 partner_ [2014-12-30 10:33:04.133431] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
10:48 partner_ nice, nothing running on the box, just peered with the existing ones and day after rebooted..
10:50 lalatenduM joined #gluster
10:51 partner_ hmm they all more or less look like rejected, only difference here is the version.. no time to play with this right now
10:57 Pupeno joined #gluster
11:13 calum_ joined #gluster
11:17 ghenry joined #gluster
11:38 partner_ hmph, downgrading made no difference, new hosts stay rejected
11:38 ppai joined #gluster
11:41 glusterbot News from newglusterbugs: [Bug 1177767] make uninstall leaves two symlink files <https://bugzilla.redhat.com/show_bug.cgi?id=1177767>
11:42 ndarshan joined #gluster
11:42 shubhendu joined #gluster
11:44 dusmant joined #gluster
11:59 Pupeno joined #gluster
11:59 Pupeno joined #gluster
12:06 nbalacha joined #gluster
12:09 hchiramm joined #gluster
12:10 partner_ new attempt..
12:11 glusterbot News from newglusterbugs: [Bug 1177773] [RFE][HC] – Brick scaling and balancing when adding storage on hyper converged nodes. <https://bugzilla.redhat.com/show_bug.cgi?id=1177773>
12:11 glusterbot News from newglusterbugs: [Bug 1177775] [RFE][HC] – Brick-rebalance and brick-replace when losing host on hyper converged nodes. <https://bugzilla.redhat.com/show_bug.cgi?id=1177775>
12:12 partner_ nope, still turns into rejected (connected)
12:13 partner_ i don't understand..
12:14 partner_ peered host says: State: Accepted peer request (Connected)
12:14 kanagaraj joined #gluster
12:16 fandi joined #gluster
12:24 partner_ now they both say rejected
12:28 partner_ http://fpaste.org/164279/42457141/
12:29 partner_ however i can see all the existing volumes and their info on the new box even thought they both report rejected. is this some known issue as i think since we upgraded to 3.4.5 we haven't added a single peer ? if so, any workarounds? (yes, going to google for more hints)
12:31 partner_ maybe this one: https://bugzilla.redhat.com/show_bug.cgi?id=1051992 thought my peer file says state=6
12:31 glusterbot Bug 1051992: unspecified, unspecified, ---, bugs, NEW , Peer stuck on "accepted peer request"
12:41 glusterbot News from newglusterbugs: [Bug 1177791] [RFE][HC] - add parameters and default policies for data centers running hyper converged nodes to utilize images locality. <https://bugzilla.redhat.com/show_bug.cgi?id=1177791>
12:42 calum_ joined #gluster
12:42 RameshN joined #gluster
12:47 pcaruana joined #gluster
12:47 nangthang joined #gluster
12:49 TvL2386 joined #gluster
12:54 hagarth joined #gluster
13:01 partner_ i seem to be able to peer between the new boxes without issues.. even checked the maximum amount of peers but it seems to be 64, i only have 12 and now trying to add 4 more
13:03 Hoggins JoeJulian: thanks for your help this morning... Anyway, I couldn't find any useful information in the logs, and I solved the problem entirely... simply by stopping and starting the volume again, with all clients disconnected
13:11 glusterbot News from newglusterbugs: [Bug 1051992] Peer stuck on "accepted peer request" <https://bugzilla.redhat.com/show_bug.cgi?id=1051992>
13:13 partner_ ^ added some story of mine there, not sure if its exactly related but similar
13:13 Fen1 joined #gluster
13:23 LebedevRI joined #gluster
13:24 Pupeno_ joined #gluster
13:39 bala joined #gluster
13:40 lalatenduM joined #gluster
13:48 msmith_ joined #gluster
13:57 partner_ seems i can make it into state=4 aswell ie. State: Accepted peer request (Connected) but no bonus, as useless state
13:58 lalatenduM joined #gluster
14:02 partner_ oh, can do even state=5 ie. sent and received peer request.. come on, pair you bloody boxes, can't be that hard
14:03 soumya_ joined #gluster
14:25 virusuy joined #gluster
14:34 partner_ probably figured it finally out :/
14:36 partner_ or not but at least i got 1 peered with a single existing one..
14:39 nbalacha joined #gluster
14:41 rotbeard joined #gluster
14:56 hagarth joined #gluster
15:00 nishanth joined #gluster
15:01 diegows joined #gluster
15:03 partner_ ok, finally they all connected, case closed, thanks and sorry for the noise
15:06 elico joined #gluster
15:12 nbalacha joined #gluster
15:14 hagarth joined #gluster
15:19 wushudoin joined #gluster
15:24 RameshN joined #gluster
15:29 coredump joined #gluster
15:38 diegows joined #gluster
15:39 kovshenin joined #gluster
15:42 calisto joined #gluster
15:43 jamesc joined #gluster
15:56 roost joined #gluster
16:10 diegows joined #gluster
16:18 msmith_ joined #gluster
16:19 coredump joined #gluster
16:33 B21956 joined #gluster
16:35 LebedevRI joined #gluster
16:39 msp3k joined #gluster
16:41 shubhendu joined #gluster
16:42 msp3k Stupid question: Can I configure gluster to talk to clients on one ethernet device, and cross-talk between nodes on a second ethernet device?  And if so, is there an example configuration somewhere that I can look at?
16:52 Pupeno joined #gluster
16:52 vimal joined #gluster
16:55 msp3k left #gluster
16:56 calisto1 joined #gluster
17:04 sage_ joined #gluster
17:05 Pupeno joined #gluster
17:06 Pupeno joined #gluster
17:06 Pupeno joined #gluster
17:10 jobewan joined #gluster
17:13 hchiramm_ joined #gluster
17:26 nishanth joined #gluster
18:12 Gilbs joined #gluster
18:15 Gilbs Anyone seen issues with Geo-replication not replicating current files, but replicating anything new?
18:20 hagarth Gilbs, sending out an email on gluster-users with as many details might help in getting back a response to this query. Please include as many details about your deployment when you do.
18:21 msmith_ joined #gluster
18:23 Gilbs I would but I'm moving jobs tomorrow, just trying to do a litlte extra effort before I head out.   I'll most likely have to have someone do it after I go.
18:24 JoeJulian I asked a question of you yesterday... I can't remember what it was now...
18:25 Gilbs DoD got me back in their cluches, off to Germany I go.  Bye gluster guys :(
18:26 JoeJulian Is that a good thing?
18:27 Gilbs Yeah, I had to pull up roots here, but living off uncle sam's dime isn't a bad life.
18:28 Pupeno joined #gluster
18:30 JoeJulian Well, best of luck to you then. Maybe you'll find a way to use gluster for the dod.
18:33 Gilbs I'll try, that and haproxy.
18:34 hagarth Gilbs: +1, good luck with the move
18:35 Gilbs They are packing us up as I speak, they move fast.  :I
18:36 msmith_ joined #gluster
18:38 codex joined #gluster
19:25 jobewan joined #gluster
19:35 Pupeno_ joined #gluster
19:49 partner ah, lovely europe, +0 degC, raining water, what a new year coming up :)
19:58 diegows joined #gluster
20:00 msmith_ joined #gluster
20:02 SOLDIERz joined #gluster
20:15 partner back to the topic, i've got a 5 TB replica 2 volume that is almost full and no room to expand on the host
20:15 partner i'd like to migrate it to another datacenter but i don't know if i can complete the task before i have to physically move the server to that same location
20:16 partner can anybody even remotely give guestimate on how long could such task take? or if i start it and run out of time i guess i can just stop the process (ie. never reach the "commit" phase)?
20:17 partner 20M inodes in use so probably lots of small files
20:20 partner i have no other parameters other than the source host is on gigabit and target is 10 gig (and so is the link to remote), ping roundtrips will increase from 0.1 ms to roughly 4.5 ms due to distance
20:20 msmith_ joined #gluster
20:26 roost joined #gluster
20:28 partner no worries, i would have not blamed anybody for giving "wrong" estimates, just trying to figure out if its worth the effort even try as plain fix-layout on other volume takes 1,5 months according to my estimates :)
20:38 jobewan joined #gluster
20:38 JoeJulian I think you could theoretically base your estimate on bytes of data over transit bandwidth. Maybe add 20% for latency/
20:39 JoeJulian I wonder if geo-rep's rsync uses compression...
20:43 partner no idea how gluster does the replace-brick, how conservative it is on bandwidth/io/whatnot in order not to affect the quality of service towards the clients
20:43 JoeJulian Oh, replace-brick... I wouldn't.
20:44 msmith_ joined #gluster
20:44 partner no? i heard some rumours since 3.3 about that but at least the commandline still offers it
20:45 JoeJulian They took out the migration, so it's basically replace-brick and self-heal.
20:46 Gilbs looks like geo-rep does have compression, here is how someone needed to turn it off due to cpu issues:  http://www.gluster.org/pipermail/gluster-users/2014-November/019631.html
20:47 partner as its replica 2 i would have wanted to get neither one out of the box before its physically moved to new location. sure, the stuff will stay up while move is in progress but i have to migrate the data off anyways
20:48 partner kind of like the idea when i don't have to worry about the storage layer details but in this case i kind of want to care
20:48 Gilbs Found this on my previous question, but the next question is where is the geo-rep index located so I can delete it?
20:48 Gilbs "You can enforce a full sync of the data by erasing the index and restarting GlusterFS Geo-replication"
20:53 partner crap, i guess i'm out of options then, i _loved_ the replace-brick functionality as it gave me the same capabilities as i have with for example vmware storage vmotion
20:54 JoeJulian partner: down one server, bring it up in the new DC. You'll lose redundancy during the transfer and have to do some self-heal, but that seems the least of all evils to me.
20:54 ildefonso joined #gluster
20:55 JoeJulian Gilbs: Should be under .glusterfs somewhere.
20:55 partner JoeJulian: that was the original plan, take the half of the replica down and move it physically and let self-heal take care of rest
20:56 Gilbs Thanks.
20:57 partner i would have rather wanted still to move half of it already now as i don't have too many days in between moving the other replica partner, couple of days i think, probably won't self-heal the full volume by that time
21:00 partner in a safe way, that is.. could kill the glusterfs process to make it fail.. just sounds really ugly approach..
21:01 partner JoeJulian: alright, thanks, i need a beer to get over this, owe one more
21:10 partner would it be bad option to add couple of bricks to that replica 2 and then command remove brick for the old ones, data gets migrated to new ones, i guess this is the "new" current approach? and will it fail badly when the one box is taken down for half a day?
21:11 partner maybe some rebalance in between to get stuff migrated partially nicely to the other side?
21:12 partner had to postpone the move already by one week due to issues with gluster earlier, running out of time here :/
21:16 partner i'm puzzled, i can't replace a server with a new one anymore for the bricks it holds? that's like, backwards in functionality?
21:16 partner emphasis on "controlled and safe manner"
21:23 ildefonso joined #gluster
21:29 partner i apogolize for the tone, just spent whole day figuring out why i couldn't peer the new hosts for this operation and once i figured it out it turns out i can't or shouldn't do it. i'll background myself, thanks, bye
21:32 JoeJulian No worries. I'm not ignoring you, just having a heated discussion at work.
21:40 partner not expecting you Joe to resolve every and all of my problems neither, was not targetted at you by any means, was just getting a bit frustrated here. been stressed out with the whole move, couple of dozen of racks to move, not too much time given either
21:41 JoeJulian Been there.. I sympathize.
21:42 partner thanks, i'll go and see how House would solve this, board and couple of pensils is a good start :o
21:43 partner and tomorrow is fun anyways, i'll have my 5yr daughter helping me all day, first task is to go via datacenter to replace a broken disk, maybe she figures this all out :)
21:44 JoeJulian +1
21:55 jobewan joined #gluster
21:56 dv8goat joined #gluster
22:01 dv8goat hi all… I have a use case question..I am considering using glusterfs to create a personal cluster for video storage across four 1U servers each with 8 tbs of storage.. Since this is a personal cluster, not all nodes will be powered up all the time.. There might be just one or two of the four powered up to access specific resources. Is Gluster a good fit for this use case?
22:07 ama I'm no expert, but I'd say probably not.  I think you'd be better having separate filesystems on each server and your videos maybe clasified on them by subject or something?
22:08 JoeJulian A "cold storage" plex backend would be kind-of nice.
22:09 JoeJulian I wish plex wasn't closed-source. Adding hooks in to swift would be really nice.
22:10 dv8goat dang =)
22:10 ama How abut Kodi?
22:14 kovshenin joined #gluster
22:19 dv8goat I'll have to check Kodi out, i've only used plex
22:19 JoeJulian Kodi's xbmc, a front-end. Plex is the back-end server.
22:21 jobewan joined #gluster
22:24 daMaestro joined #gluster
22:24 Pupeno joined #gluster
22:29 ildefonso joined #gluster
22:37 Gilbs left #gluster
23:03 cristian left #gluster
23:04 nueces joined #gluster
23:05 ildefonso joined #gluster
23:08 iPancreas joined #gluster
23:09 iPancreas has anyone tried creating a gluster "cluster" in SoftLayer using virtual server instances?
23:10 iPancreas Love to hear from anyone with actual experience on the subject of creating it in a virtualized environment
23:13 JoeJulian I frequently test to find bugs that are reported here in kvm. Many people use gluster in AWS (so Xen). I've heard of people using docker. Not sure about SoftLayer.
23:14 iPancreas The thing is, this time is for a "production" environment
23:14 iPancreas Rackspace maybe?
23:15 JoeJulian rackspace would be xen.
23:16 iPancreas right...
23:16 iPancreas guess I'll have to try
23:17 JoeJulian There's nothing functionally different between aws and rackspace. Just overcommit and network latency fluctuations.
23:23 iPancreas how about disk io?
23:23 iPancreas I was looking at SoftLayer's disk options for virtual instances... and there are quite a few options...
23:23 JoeJulian I count that as overcommit.
23:24 iPancreas that's probably what will either make it work or kill it
23:24 JoeJulian With public clouds, that's kind-of what you're going to get.
23:25 JoeJulian Otherwise, you're talking 24k+/mo for private clouds
23:26 ildefonso or just get dedicated servers
23:26 roost joined #gluster
23:26 JoeJulian Or that, though you'll probably be paying more for hardware, rack and network over 5 years.
23:27 JoeJulian Depends on your business plan, of course.
23:27 ildefonso it depends on the amount of traffic you manage.
23:27 iPancreas yes... obviously getting dedicated equipment is the better option...
23:27 ildefonso softlayer includes 20TB with any dedicated server.
23:27 iPancreas but the storage need are quite small..
23:27 iPancreas not a lot of traffic
23:28 ildefonso iPancreas, in that case, dedicated would end up being more expensive...
23:28 iPancreas of course.. dedicated would be much much more expensive
23:28 iPancreas specially since it would require at least two nodes
23:28 ildefonso iPancreas, there is a breaking point, where dedicated starts to make sense, but it depends on what you want to do.
23:29 iPancreas im guessing we can always start with a virtual environment
23:30 iPancreas and if it ends up running short of expectations then move everything to a dedicated environment
23:30 ildefonso or running off your DSL connection from a home PC :P
23:32 ildefonso there is people who believes that moving into "the cloud" will give them better performance, or magically make their poorly designed app "highly available"... it is amazing how much confusion the "cloud" term created among managers.
23:33 JoeJulian Ask 3 and get 3 different definitions.
23:33 calum_ joined #gluster
23:35 _Bryan_ joined #gluster
23:36 Pupeno_ joined #gluster
23:45 semiosis iPancreas: two awesome features for gluster that AWS has are availability zones & system snapshots
23:45 semiosis idk about softlayer, but rackspace definitely doesnt have those
23:46 semiosis i'm sure you can get gluster working in softlayer, but how well will it fail?
23:47 semiosis and when it fails, how easily & rapidly can you restore it?
23:47 ildefonso semiosis, for softlayer you would select a different datacenter for each of your instances (availability zone)
23:47 JoeJulian rackspace (all openstack) has zones and snapshots.
23:47 semiosis ildefonso: whats the latency between those DCs?  AWS AZs are <1ms which is essential for gluster
23:47 semiosis JoeJulian: rs snapshots don't capture attached block devices, only the system disk
23:48 JoeJulian Correct. cinder volumes are snapshotted separately. But you can also boot from cinder volumes under openstack so that's kind-of a moot difference.
23:49 ildefonso semiosis, how far away are those AZ? as for softlayer, these are located on different geographical areas.
23:50 semiosis ildefonso: AWS has regions and each region has availability zones.  within a region the AZs are independent datacenters with their own internet & power connected to each other by low latency ethernet links
23:51 ildefonso semiosis, latency between two (random) DC in softlayer is relatively high, ~30ms.
23:51 semiosis i'm a big fan of AWS :)
23:52 ildefonso I just feel their storage to be a bit... sluggish, but you will find the same problem with pretty much any SAN-based storage, so, it is not really their fault.
23:52 PeterA joined #gluster
23:52 ildefonso main issue is disk latency for synchronous writes (a really important number for PostgreSQL performance)
23:52 semiosis yep, ebs latency can be annoying
23:53 semiosis db workloads are hard
23:53 ildefonso yeah, more often than not we end up using dedicated servers for highly loaded DBs
23:53 ildefonso however, I had a good experience with ZFS and AWS a few months ago, performance was surprisingly good.
23:54 semiosis cool!  would love to hear more about it but getting ready to leave shortly
23:55 semiosis later
23:55 ildefonso c-ya!
23:57 iPancreas joined #gluster
23:58 iPancreas sorry.. got disconnected and lost a few of your comments
23:59 iPancreas yes... I think of using different DCs in SoftLayer

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary