Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-02-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 haomaiwa_ joined #gluster
00:18 Logos01 JoeJulian: Is there anything fundamentally broken about creating a gluster volume on top of two servers if they have the exact same data already present in the designated brick directories?
00:18 Logos01 I mean, aside from needing to ls every last file individually from a single client in order to ensure that their gfid xattrs are properly set
00:39 jwang_ joined #gluster
00:40 JoeJulian Logos01: the self-heal daemons start by default on all the servers.
00:41 JoeJulian So yeah, if two clients lookup the same file, they'll create conflicting metadata.
00:41 JoeJulian Plus, they'll still have to walk the whole file as part of the self-heal so you're not gaining anything.
00:44 Logos01 JoeJulian: So if I only have one client initially?
00:44 JoeJulian Like, only one server?
00:44 Logos01 No, two brick hosts but one mounting client.
00:44 Logos01 I didn't see any heal activity occurring.
00:45 Logos01 But running "find" certainly is spawning a lot of traffic.
00:45 JoeJulian Try it. I expect problems.
00:45 Logos01 I'm seeing a lot of warning traffic like the following two log lines:
00:46 Logos01 [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-libreta_pdf-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
00:46 JoeJulian yep
00:46 Logos01 [2016-02-09 00:45:39.349499] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-libreta_pdf-client-1: remote operation failed. Path: /PATH/TO/FILENAME_REDACTED.pdf (00000000-0000-0000-0000-000000000000) [No data available]
00:47 Logos01 But when I go into the actual brick-servers and getfattr the actual file (rather than the gluster-fuse mount), everything looks fine and synchronized on both sides.
00:47 JoeJulian What you're doing is not supported. It will, by definition, produce undefined results.
00:48 JoeJulian If you're sure you know what it's supposed to be doing and can verify that it's doing it, great. I've helped people out of the situtation they've caused by doing this before.
00:48 Logos01 Right. Thing is, I let these volumes sit idle for a good twenty or thirty minutes before running that "find" command, and it never showed any self-heal activity ongoing that I could see in logs, nor were gfid's getting populated.
00:48 JoeJulian but things change, so who knows.
00:48 Logos01 So what sort of problems might I encounter should I let this complete?
00:49 JoeJulian mismatched gfids, mostly.
00:49 jwang joined #gluster
00:49 Logos01 The thing is, I couldn't see any other way to create them.
00:49 Logos01 (the gfid's)
00:50 JoeJulian Oh, that was someone else I answered today! I was thinking I was continuing a conversation from earlier. (shows how well I pay attention to names)
00:51 JoeJulian If it was me, I'd start with only one populated brick.
00:52 Logos01 Heh. That would take a week or two to replicate.
00:53 JoeJulian How did it get there in the first place?
00:53 jwang_ joined #gluster
00:53 JoeJulian It should take the same amount of time.
00:57 Logos01 JoeJulian: I used ZFS send/recv over netcat.
00:57 Logos01 No file reconciliation.
00:57 Logos01 Interestingly ...
00:57 Logos01 http://fpaste.org/320103/79443145/
00:57 glusterbot Title: #320103 Fedora Project Pastebin (at fpaste.org)
00:58 Logos01 Can't find any discrepancy in gfid's so far, and I've checked a few tens-of-thousands of files.
00:59 Logos01 JoeJulian: Also, it *did* take about three/four days when I did the naive block stream replication via ZFS.
01:01 haomaiwa_ joined #gluster
01:02 JoeJulian Right? So why waste the effort?
01:03 Logos01 Hrm?
01:04 Logos01 What level of risk is the mismatch of gfid's?  I've now checked 464945 files, can't find any mismatched gfid's.
01:05 JoeJulian From my perspective, pretty high. But then again, I only talk to the people that have problems.
01:05 Logos01 True enough.
01:05 Logos01 But I meant more what's the rate of incidence of mismatched gfid's?
01:05 Logos01 I guess that's unanswerable.
01:06 JoeJulian Right?
01:06 Logos01 lol
01:06 Logos01 I guess I'll find out.
01:06 JoeJulian My usual solution is to wipe one of the bricks and heal...full.
01:06 JoeJulian So I don't bother coming up with statistics.
01:06 Logos01 gfids are tracked in .glusterfs as well aren't they.
01:07 JoeJulian I guess a more minimal solution would be to do an xattr comparison and then pick one to remove (including the gfid hardlink) and then heal...full.
01:07 JoeJulian Right, hardlinks.
01:07 Logos01 What's really interesting is those "remote operation failed" warning messages seem to coincide perfectly to the files that have had their gfid values propagated.
01:07 JoeJulian @lucky what is this new .glusterfs
01:07 glusterbot JoeJulian: https://joejulian.name/blog/what-is-this-new-glusterfs-directory-in-33/
01:07 JoeJulian Right
01:08 JoeJulian First, it tries to read the gfid file, when it doesn't exist it creates one.
01:08 Logos01 I mean, so far everything that it's hit -- which is upwards of a million or so files -- has had the gfid properly replicated for both servers.
01:08 Logos01 But even when I tried to kick off a heal I didn't see any gfids get created.
01:08 JoeJulian Cool
01:09 JoeJulian Are there any files that it hasn't hit that have gfids?
01:09 JoeJulian That's the scary part.
01:09 Logos01 No.
01:09 Logos01 I did a few "find /gluster -exec getfattr -d -m . {} +"
01:10 Logos01 and didn't see anything but SELinux context info for a good fifteen+ minutes after I set up the volumes.
01:10 JoeJulian Heh
01:10 JoeJulian Maybe afr2 handles this better.
01:12 JoeJulian I guess where I'm really going with this is that I would not recommend creating a deployment strategy that depends on something that is outside of the design scope.
01:12 JoeJulian It could change and break your design at any release.
01:13 Logos01 Fair.
01:14 Logos01 I mean I suppose there's nothing especially stopping me from forcing one of the bricks out, reducing to replica 1, and then running a full heal, then adding the other brick back with the directories purged and run a rebalance.
01:15 Logos01 Aside from that this would take... a while.
01:15 Logos01 I'll keep an eye on things and if I start seeing mismatched gfid's then I'll definitely do that.
01:16 Logos01 Actually I should probably give up on this for now and wait until certain other things finish.
01:29 jmarley joined #gluster
01:43 djgerm joined #gluster
01:53 nishanth joined #gluster
02:01 haomaiwa_ joined #gluster
02:20 theron joined #gluster
02:31 calavera joined #gluster
02:47 necrogami joined #gluster
03:01 haomaiwa_ joined #gluster
03:02 EinstCrazy joined #gluster
03:04 jeek joined #gluster
03:17 shyam joined #gluster
03:22 calavera_ joined #gluster
03:24 calavera joined #gluster
03:29 Nebraskka joined #gluster
03:30 gem joined #gluster
03:45 calavera_ joined #gluster
03:47 gildub joined #gluster
03:49 atinm joined #gluster
03:51 calavera joined #gluster
03:56 calavera_ joined #gluster
03:56 shubhendu joined #gluster
04:01 haomaiwang joined #gluster
04:04 nickage joined #gluster
04:04 nickage what is the right way to rplace brick w/o data loss?
04:06 ovaistariq joined #gluster
04:07 ramteid joined #gluster
04:08 ovaistar_ joined #gluster
04:15 calavera joined #gluster
04:22 the-me joined #gluster
04:24 nbalacha joined #gluster
04:34 JesperA joined #gluster
04:34 poornimag joined #gluster
04:36 bharata-rao joined #gluster
04:43 vimal joined #gluster
04:43 ovaistariq joined #gluster
04:58 nehar joined #gluster
05:01 gowtham joined #gluster
05:01 hgowtham joined #gluster
05:01 haomaiwa_ joined #gluster
05:06 ndarshan joined #gluster
05:07 pppp joined #gluster
05:08 jiffin joined #gluster
05:10 calavera joined #gluster
05:12 dusmantkp_ joined #gluster
05:15 skoduri joined #gluster
05:17 Manikandan joined #gluster
05:18 aravindavk joined #gluster
05:29 kanagaraj joined #gluster
05:30 karnan joined #gluster
05:32 Apeksha joined #gluster
05:32 jeek joined #gluster
05:35 ashiq joined #gluster
05:36 djgerm what's the max number of peer's you can have?
05:37 calavera_ joined #gluster
05:45 karthikfff joined #gluster
05:50 calavera joined #gluster
05:52 ramky joined #gluster
05:54 sakshi joined #gluster
05:56 Saravanakmr joined #gluster
05:58 nickage joined #gluster
06:01 haomaiwa_ joined #gluster
06:07 unlaudable joined #gluster
06:08 anil joined #gluster
06:12 karnan joined #gluster
06:19 nishanth joined #gluster
06:22 atinm joined #gluster
06:25 rafi joined #gluster
06:27 kotreshhr joined #gluster
06:30 itisravi joined #gluster
06:36 David_Varghese joined #gluster
06:43 JoeJulian nickage: replicated or not?
06:44 JoeJulian djgerm: technically there is no limit but tcp ports, but I don't know how far that's been tested functionally.
06:44 djgerm awesome
06:44 djgerm I think I only need 6 total
06:45 JoeJulian I've seen many installations with way more than that.
06:45 djgerm but… i have another question: when is geo distribution necessary?
06:46 nickage replicared but itstoo late I borked whole server
06:46 JoeJulian when you want a replica someplace else. Usually someplace high latency.
06:46 JoeJulian djgerm: you're not doing replica 6, are you?
06:47 nickage tried to replace brick and even glusterd is not starting anymore
06:47 JoeJulian nickage: Uh-oh. Need to fix it?
06:47 djgerm uh oh. you phrase it like that, I get worried :)
06:48 djgerm i was thinking replica six. but each pair of 2  are geographically isolated… with ~30-90 ms latency between them
06:48 JoeJulian djgerm: yeah, it's kind-of pointless to do more than replica 3. This isn't a mirroring service, it's a clustered filesystem. Unless you're exceeding the read capacity of three servers, you're not getting any benefit from the additional replicas.
06:49 djgerm the benefits I'm looking for are fault tolerance per site, and globally.
06:49 nickage JoeJulian: yeah, I am looking, not sure if it is still fixible
06:50 djgerm so I want to be able to lose a pair (site) or lose ones and twos without interruption to filesystem.
06:50 JoeJulian And you also don't want replicas with that kind of latency. A lookup could take 180ms.
06:50 JoeJulian per file
06:51 djgerm so that would be time for geo distribution then?
06:51 JoeJulian It sounds like you want a replica 2 master that distributes to regional replica 2 volumes using geo-replicate
06:51 djgerm it does sound like that!
06:51 JoeJulian nickage: Tell me what steps got you to where you are.
06:53 djgerm thanks JoeJulian for helping me figure out the solution I want! Now for the docs to do that.
06:53 nickage JoeJulian: this manual https://gluster.readthedocs.org/en/latest/Administrator%20Guide/Managing%20Volumes/#replace-brick step where I had to do "Execute replace-brick command"
06:53 glusterbot Title: Managing Volumes - Gluster Docs (at gluster.readthedocs.org)
06:55 nickage didn't get successfull reply, nothing at all from gluster
06:55 JoeJulian Sorry, I already know what the manual says. I probably had a hand in coming up with some of the text.
06:57 SOLDIERz joined #gluster
06:58 djgerm hrmm…. geo-replicated slaves probably can't be written to huh…
06:58 JoeJulian Sorry, no.
06:58 djgerm heh. I want replica 2 master/master geo :)
06:59 dusmantkp_ joined #gluster
06:59 JoeJulian ... and a pony. ;)
06:59 djgerm hehehe.
07:00 JoeJulian There's not really anything that does that today.
07:00 djgerm I tried XtreemFS
07:00 djgerm It sucks, XTREEMly
07:00 djgerm but is pretty promising
07:00 [Enrico] joined #gluster
07:01 djgerm what if… i had replica 6 with my pairs of geo dispersed nodes, but had local VIPs for clients mounting?
07:01 djgerm does the latency come in due to the nature of synchronous writes?
07:01 JoeJulian @mount server
07:01 glusterbot JoeJulian: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns, or (#2) One caveat is that the clients never learn of any other management peers. If the client cannot communicate with the mount server, that client will not learn of any volume changes.
07:02 JoeJulian The replication happens at the client.
07:02 haomaiwa_ joined #gluster
07:02 djgerm I see.
07:03 JoeJulian Sorry to dash your hopes and run, but I'm off to bed. 11pm here in Seattle.
07:04 djgerm 11pm here in Santa Cruz :)
07:04 djgerm it's the story of my life. Thanks for your help!
07:04 djgerm aside from lag, any other issues with my idea?
07:04 JoeJulian Oooh, I like Santa Cruz. Probably about the only place in California I could say that about.
07:05 djgerm Yeah…. it's not what it used to be… but it's hanging in there.
07:05 JoeJulian Oh, where's my rant in gluster-users...
07:06 JoeJulian There's this old article that's still pretty valid: https://joejulian.name/blog/glusterfs-replication-dos-and-donts/
07:06 glusterbot Title: GlusterFS replication do's and don'ts (at joejulian.name)
07:06 Humble joined #gluster
07:06 djgerm hehe… I know I know, it sounds crazy… but what if… I don't really care if it's a really slow filesystem?
07:06 nickage omfg I fixed it
07:06 nickage 0_0
07:06 JoeJulian If it fits your use case, then it's a good idea
07:06 JoeJulian Good job nickage!
07:07 djgerm Good work nickage!
07:07 JoeJulian djgerm: I'm consistently ranting that nobody engineers to suit their criteria. If you're doing that, then more power to you.
07:08 nickage gluster updated brick text file but the rest configs remained the same old it coldn't start because of wrong(updated) vol file
07:08 nickage so i just manually revert changes back
07:08 djgerm I maximize the work I don't do… so I only engineer to my criteria :)
07:10 JoeJulian Ah, there's my other rant: https://www.mail-archive.com/gluster-users@gluster.org/msg23295.html
07:10 glusterbot Title: Re: [Gluster-users] Unexpected behaviour adding a third server (at www.mail-archive.com)
07:11 JoeJulian Doesn't exactly fit what you're now asking, but fwiw.
07:13 JoeJulian Yes. If you don't care about that latency it should work great. I would look at quorum capabilities and decide how to best use it for your situation. With the increased possibility of network partition, it might be important to you.
07:17 jtux joined #gluster
07:21 natarej_ joined #gluster
07:21 atalur joined #gluster
07:23 Bhaskarakiran joined #gluster
07:25 Wizek joined #gluster
07:28 mhulsman joined #gluster
07:29 atinm joined #gluster
07:32 hgowtham joined #gluster
07:33 ovaistariq joined #gluster
07:35 ovaistar_ joined #gluster
07:40 [Enrico] joined #gluster
07:44 atalur joined #gluster
07:46 vmallika joined #gluster
07:49 [diablo] joined #gluster
07:49 [diablo] joined #gluster
07:50 kovshenin joined #gluster
07:54 [diablo] joined #gluster
08:01 haomaiwang joined #gluster
08:17 anrao joined #gluster
08:17 ivan_rossi joined #gluster
08:27 Vaelatern joined #gluster
08:27 deepakcs joined #gluster
08:28 gowtham joined #gluster
08:33 deepakcs left #gluster
08:34 Manikandan joined #gluster
08:35 ovaistar_ joined #gluster
08:49 fsimonce joined #gluster
08:53 ctria joined #gluster
08:58 dusmantkp_ joined #gluster
09:01 haomaiwa_ joined #gluster
09:14 ashiq joined #gluster
09:19 glafouille joined #gluster
09:23 [diablo] G'morning #gluster
09:24 [diablo] guys is anyone running a CTDB setup with GlusterFS? I'm looking at the lockfile setup, and I see in the docs they recommend creating a dedicated volume for the lockfile... I'm toying with the idea of creating a file, formating it to XFS, mounting it and using it as the fs for the lockfile
09:27 dizzystreak joined #gluster
09:27 [diablo] or better to create a dedicate partition, and create bricks _
09:27 [diablo] ?
09:35 kdhananjay joined #gluster
09:36 ovaistariq joined #gluster
09:39 shubhendu joined #gluster
09:52 toshywoshy joined #gluster
09:54 nehar joined #gluster
10:00 shubhendu joined #gluster
10:01 haomaiwang joined #gluster
10:08 itisravi joined #gluster
10:12 malevolent joined #gluster
10:12 xavih joined #gluster
10:12 muneerse2 joined #gluster
10:18 ashka joined #gluster
10:19 Bhaskarakiran joined #gluster
10:20 dizzystr_ joined #gluster
10:22 [Enrico] joined #gluster
10:23 dizzyst__ joined #gluster
10:24 dizzys___ joined #gluster
10:27 [Enrico] joined #gluster
10:28 dizzystreak joined #gluster
10:30 oliva joined #gluster
10:30 dusmantkp_ joined #gluster
10:41 xavih joined #gluster
10:41 malevolent joined #gluster
10:46 Akee joined #gluster
10:56 Manikandan joined #gluster
10:59 Manikandan REMINDER: Gluster Community Bug Triage meeting at 12:00 UTC (~in 60 minutes) in #gluster-meeting
11:01 haomaiwa_ joined #gluster
11:04 oliva hi there I'm trying to setup an hpc cluster with ~40 nodes. All nodes are connected to a san with fc, and to a infiniband network. I would like that only some dedicated storage nodes access the san and export the filesystem to the rest of the cluster via infiniband. For now I'm not sure about how many storage nodes will be needed since the range of appliaction vary a lot. The space assigned by the storage g
11:04 oliva uys is fixed, but they can give me as many lun as I need. Does it make sense to use glusterfs for such a setup? Is it possible to combine clvmd and glusterfs such that if tomorrow I need to add a new storage node I only create re-arrange lvm volumes?
11:04 nehar joined #gluster
11:07 nehar joined #gluster
11:20 Bhaskarakiran joined #gluster
11:24 ovaistariq joined #gluster
11:37 ashiq joined #gluster
11:39 sakshi joined #gluster
11:41 B21956 joined #gluster
11:57 rastar joined #gluster
12:00 glafouille joined #gluster
12:01 haomaiwa_ joined #gluster
12:04 Manikandan REMINDER: Gluster Community Bug Triage meeting *started* in #gluster-meeting
12:11 nehar joined #gluster
12:14 gem joined #gluster
12:26 ira joined #gluster
12:33 post-factum oliva: shouldn't you use ocfs2/gfs2 instead?
12:35 rastar joined #gluster
12:43 shubhendu joined #gluster
12:45 nishanth joined #gluster
12:56 dusmantkp_ joined #gluster
12:57 paraenggu joined #gluster
12:59 paraenggu left #gluster
13:01 haomaiwa_ joined #gluster
13:03 johnmilton joined #gluster
13:03 oliva post-factum: gfs2 is recommended up to 16 nodes, I never used ocfs2, does it have a good support community?
13:04 kdhananjay joined #gluster
13:04 johnmilton ive inherited a shaky glusterfs cluster w/two nodes. yesterday i added two bricks (one on each node) to a volume and extended, then rebalanced it. the problem is it was only rebalanced on one node.
13:05 johnmilton i think im gonna have to delete the volume and resync it from the other node
13:06 johnmilton that will work, right?
13:12 [diablo] Afternoon guys....
13:12 ovaistariq joined #gluster
13:12 post-factum oliva: well, Oracle suggests it
13:13 [diablo] guys if a volume is started, and replicated across 2 x nodes...and one machine is shutdown down, the volume is still started.... if the 2nd node is restarted, it connects to the running volume... all good
13:13 post-factum oliva: dunno if it is a measure if good support community. i'd say no :)
13:13 [diablo] however, if both of the nodes are shutdown, and the machines started again... the volume is running once booted...
13:13 post-factum s/if good/of good/
13:13 glusterbot What post-factum meant to say was: oliva: dunno if it is a measure of good support community. i'd say no :)
13:14 [diablo] as there are hooks, (for CTDB), my question is this: does the volume get 'started' again (thus run the start hooks), or does it continue in the state it was at shutdown, and not run the hooks
13:17 [diablo] RH ship hook scripts that help (more hinder imo) the configuration and starting of ctdb, smb, editing fstab, blah blah.... but they're really screwing up the system... they don't play nicely with systemd
13:17 oliva post-factum: looks like I should change distro to use it
13:18 plarsen joined #gluster
13:19 unclemarc joined #gluster
13:20 oliva there is no support for ocfs2 in centos
13:21 Bhaskarakiran joined #gluster
13:24 deepakcs joined #gluster
13:24 djgerm left #gluster
13:25 deepakcs left #gluster
13:32 post-factum oliva: I've prepared ocfs2 module once for centos kernel via dkms, but haven't tested it yet
13:32 arcolife joined #gluster
13:35 oliva I think I would switch to debian in case ... I'm trying to understand what's the status of the software in debian.
13:41 dusmantkp_ joined #gluster
13:41 poornimag joined #gluster
13:45 robb_nl joined #gluster
13:46 arcolife joined #gluster
13:47 haomaiwang joined #gluster
13:48 nishanth joined #gluster
13:53 shubhendu joined #gluster
13:53 Wizek joined #gluster
13:54 post-factum oliva: you could try Oracle UEK kernel instead, it fits OK to CentOS
14:01 haomaiwang joined #gluster
14:25 nehar joined #gluster
14:27 wnlx joined #gluster
14:28 dusmantkp_ joined #gluster
14:29 theron joined #gluster
14:30 ndarshan joined #gluster
14:33 portante joined #gluster
14:51 coredump joined #gluster
14:51 bennyturns joined #gluster
14:54 aravindavk joined #gluster
14:59 hamiller joined #gluster
15:01 ovaistariq joined #gluster
15:17 skylar joined #gluster
15:22 coredump joined #gluster
15:25 shaunm joined #gluster
15:34 jmarley joined #gluster
15:35 wushudoin joined #gluster
15:37 telmich good day
15:37 wushudoin joined #gluster
15:42 farhoriz_ joined #gluster
15:43 hagarth joined #gluster
15:45 telmich I'm trying to backup 9.1T of data to another host and the backup-procses takes a couple of days already
15:45 julim joined #gluster
15:45 telmich I was wondering if there is a way to make the backup (essentially ccollect with rsync under the hood) faster?
15:49 kovsheni_ joined #gluster
15:49 dizzystreak you could  try tar over nc
15:50 dizzystreak sth like: tar cf - SOURCE_DIR | nc DST_SRVR DST_PORT
15:50 skoduri joined #gluster
15:51 farhoriz_ joined #gluster
15:52 [diablo] guys, anyone have any idea why when I try to connect via CIFS I get this http://pastie.org/pastes/10715068/text?key=cdc1prbtkwelymtijheq
15:52 glusterbot Title: Private Paste - Pastie (at pastie.org)
15:52 farhoriz_ joined #gluster
15:52 [diablo] it won't mount ...
15:52 mobaer joined #gluster
15:52 farhoriz_ joined #gluster
15:52 [diablo] the samba conf is using the vfs object as glusterfs
15:55 neofob joined #gluster
15:55 ira What versions of what are in use?
15:55 ira Looks like a version issue of some form..
15:55 ira (Just reading the messages, as a naive person.)
15:55 [diablo] afternoon ira ... well, we're using the RHGS
15:55 [diablo] let me check the exact version they're using
15:56 [diablo] glusterfs-server-3.7.1-16.el7rhgs.x86_64
15:56 [diablo] so we're using the ctdb also shipped by RH
15:56 ira [diablo]: Have you contacted Red Hat?
15:56 [diablo] not yet
15:57 [diablo] actually we've had such a head f* with it... we're at the point now of having no other option
15:57 [diablo] we're just in our last attempts before doing so
15:57 [Enrico] joined #gluster
15:57 ira rpm -qa | grep samba
15:58 [diablo] once sec, pastie is down
15:59 [diablo] http://paste.ubuntu.com/15002526/
15:59 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
16:00 [diablo] ctdb-4.2.4-9.1.el7rhgs.x86_64
16:00 [diablo] and that also
16:00 ira That looks right...
16:00 [diablo] can I only guess so
16:00 [diablo] :-/
16:01 mobaer joined #gluster
16:01 ira What is the exact issue.
16:02 [diablo] ira, every time I connect via SMB it fails, and I see that in the logs
16:02 ira Sure... What fails?
16:02 ira And what is connecting.
16:02 [diablo] Ubuntu desktop
16:02 [diablo] just via file manager
16:03 [diablo] it sees the share announced , but can't connect
16:04 [diablo] http://paste.ubuntu.com/15002574/
16:04 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
16:05 [diablo] thats the share conf
16:06 ira That's generic alas.  Do you set any options in the conf?
16:06 [diablo] ira, only those which RHGS docs advised
16:07 [diablo] it auto configures itself somewhat
16:07 [diablo] https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-SMB.html#sect-Sharing_Volumes_over_SMB
16:07 glusterbot Title: 7.3. SMB (at access.redhat.com)
16:07 [diablo] thats the docs we've been following
16:07 ira Yeah... Though it doesn't always get the permissions on the share right.  :)
16:08 [diablo] hmm
16:08 [diablo] interesting
16:08 ira That's one of the causes of such things, there's about 2-3 of these causes.
16:08 [diablo] OK
16:08 ira Support should be able to step you through it.
16:09 [diablo] well, I see also the old way was to mount with fuse, and share the mounted dir
16:10 [diablo] ira, we'll raise a ticket tomorrow ...
16:11 [diablo] ira,  thanks for the info and taking a look tho
16:11 ira Honestly, I suspect support will be able to help you :).  Or at least they'll get all the debugging info etc.
16:11 kenansulayman joined #gluster
16:11 ira Don't fear support. :)
16:11 mobaer1 joined #gluster
16:11 [diablo] haha
16:11 [diablo] only the 1st liners
16:11 [diablo] ;-)
16:13 [diablo] ira, tbh we've had a right head f* with the RHGS ...
16:17 Ethical2ak joined #gluster
16:17 Ethical2ak Hi
16:17 glusterbot Ethical2ak: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:18 Ethical2ak I just want to know if there's any ETA for GlusterFS 3.8 ?
16:19 JoeJulian Typically 6 months between feature releases.
16:19 ndevos Ethical2ak: *estimated* time of arrival is in the direction of april/may, earlier is very unlikely
16:20 ndevos Ethical2ak: is there a particular feature that you are interested in, or why do you ask?
16:20 hagarth joined #gluster
16:20 Ethical2ak Yea i think JoeJulian know my case. We really need that new feature to control the iops of the healing.
16:22 ndevos ah, ok, that is not ready yet, and I do not know if it is possible to backport it to 3.7
16:22 Ethical2ak We can live without it for the moment
16:22 Ethical2ak it's just a bit complicated when we have splitbrain
16:22 JoeJulian I still wish I knew why it only affects a minority of users.
16:23 Ethical2ak We have several production that pushing hard on our GlusterFS system , lots of file , and the healing is to aggressive
16:24 Ethical2ak We need to disable it when the production is working
16:24 Ethical2ak if not , there's some upload that don't pass.
16:25 Ethical2ak We enable the healing in the weekend  , to fix splitbrains and rebalance everything.
16:26 Ethical2ak Another thing that is annoying
16:27 Ethical2ak Is when we enable/disable the cluster.entry-self-heal
16:27 Ethical2ak gluster volume set myvolume cluster.entry-self-heal off
16:27 Ethical2ak we got the error ''Connection failed. Please check if gluster daemon is operational.''
16:27 Ethical2ak This behavior was not in 3.4
16:28 Ethical2ak When this happen , it's a micro downtime for the production
16:38 Ethical2ak Because we need to restart the service , umount/remount our volume and then rerun the command again and it pass properly.
16:43 CyrilPeponnet joined #gluster
16:47 CyrilPeponnet Hey guys, I have a weird issue... when starting geo-rep on a volume, both of the brick of the master are crashing... https://gist.github.com/CyrilPeponnet/b67b360f186f31d34d8f
16:47 glusterbot Title: gist:b67b360f186f31d34d8f · GitHub (at gist.github.com)
16:47 Melamo joined #gluster
16:47 CyrilPeponnet centos7 gluster 3.6.5
16:47 CyrilPeponnet When I try with a fake volume it works fine.
16:48 shubhendu joined #gluster
16:50 cpetersen joined #gluster
16:51 hagarth CyrilPeponnet: looks like a crash in changelog. we are debugging a similar crash in mainline.
16:52 CyrilPeponnet ha !
16:52 CyrilPeponnet is there any "workaround" ?
16:52 hagarth CyrilPeponnet: can I share your gist on the email thread?
16:52 CyrilPeponnet I also had a  Metadata corruption detected at xfs_attr3_leaf_write_verify...
16:52 CyrilPeponnet sure go ahead
16:53 CyrilPeponnet I can reproduce it everytime
16:53 hagarth CyrilPeponnet: not that I am aware of, will check with the changelog developers.
16:53 CyrilPeponnet I manage to bring back the brick by restarting glusterd on each node
16:53 CyrilPeponnet and the geo-rep finished it's init
16:54 CyrilPeponnet but, every few hours, the active brick is going down...
16:54 CyrilPeponnet and lead to strange situation like:
16:54 hagarth CyrilPeponnet: https://bugzilla.redhat.com/show_bug.cgi?id=1221629
16:54 glusterbot Bug 1221629: high, unspecified, ---, vshankar, ASSIGNED , Bitd crashed
16:54 CyrilPeponnet https://gist.github.com/CyrilPeponnet/427bb76f72ab793d8cd0
16:54 glusterbot Title: gist:427bb76f72ab793d8cd0 · GitHub (at gist.github.com)
16:55 hagarth CyrilPeponnet: will add you to CC on that email thread.
16:55 CyrilPeponnet sure
16:57 hagarth done
16:57 ekuric joined #gluster
17:00 CyrilPeponnet thanks!
17:00 CyrilPeponnet I really I could give geo-rep a second chance, but looks not well engaged...
17:03 hagarth CyrilPeponnet: happy to help with that effort
17:03 hagarth CyrilPeponnet: 3.7 has had a lot of stability fixes for geo-replicaiton. If you can start testing there, I can help with the engagement from our developers.
17:04 CyrilPeponnet well migration to 3.7 in production is not planned for now.
17:04 CyrilPeponnet too afraid to discover blocking issues ;p
17:05 hagarth CyrilPeponnet: hmm okay, any possibility of a small test cluster for testing?
17:05 David_Varghese joined #gluster
17:06 CyrilPeponnet Yes but this will not be representative of the production. I already tested 3.7 (and 3.6.5) on staging setup, and everytime everything went fine. It's always in production with 11TB data and 1k clients that we encounter issues.
17:06 calavera joined #gluster
17:07 farhoriz_ joined #gluster
17:09 CyrilPeponnet and as we are extenting our labs I have a brand new gluster cluster I tried to geo-rep with the first one...
17:10 hagarth CyrilPeponnet: ok, am very much interested in making this work in production for you. do you have a current list of issues with geo-replication in 3.6.5 that you can email me?
17:10 CyrilPeponnet last time I tried geo-rep it was transatlantique with a poor wan link, it worked, but then took too much time to convert so we disabled it and use custom rsync scripts then. Now that I have the perfect condition to use geo-rep, well it crashed my two brick by night, make all QA testing failed ;/
17:11 CyrilPeponnet @hagarth Appart the fact it's not working, no. I had to tweak syncd for a symlink update *after* target as been sent but that's all.
17:12 CyrilPeponnet I'm no pretty familiar how geo-rep is working under the hood.
17:12 CyrilPeponnet *no/now
17:13 hagarth CyrilPeponnet: ok, I will start an email thread (again) with our developers and you. Let us see how that progresses.
17:13 Manikandan joined #gluster
17:17 hagarth1 joined #gluster
17:18 cpetersen I have a question.
17:19 cpetersen Heal info says I have two nodes with one GFID that needs healing.
17:19 cpetersen Heal statistics says I have none.
17:19 cpetersen When I drill down to the directory of the GFID, there is nothing there.
17:20 cpetersen Does that mean this is a directory?
17:20 cpetersen Do I need to clear the attributes on the GFID path?
17:20 cpetersen Errr, folder?
17:22 cpetersen The GFID is 837160e7-5c11-421e-8bf6-0762d9af2812, but if I go down in to .glusterfs/83/71/ there is nothing there.
17:24 uebera|| joined #gluster
17:25 JoeJulian Sounds like a leftover in .glusterfs/indices/xattrop
17:26 CyrilPeponnet @hagarth1 thanks :)
17:26 cpetersen Why would that happen?
17:27 cpetersen I manually started a heal.  Now statistics shows it as 1 to heal.
17:27 cpetersen But it didn't do it automatically.
17:27 JoeJulian Not sure. Maybe a file was deleted while it was pending heal?
17:27 rcampbel3 joined #gluster
17:27 JoeJulian I know there's code that's supposed to cover that scenario, but <shrug>
17:27 cpetersen :)
17:28 JoeJulian If you can figure out how to make it happen, I'm sure the devs would be glad to fix it.
17:28 cpetersen Is there any way to find out what that GFID references now or do I just have to go and delete that entry from the xattrop directory to clear it up?
17:28 cpetersen OK.  :)
17:29 JoeJulian For that matter, go ahead and file a bug. I'm not sure if there's no matching gfid file to heal that there's any reason not to just delete the xattrop file.
17:29 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
17:30 theron joined #gluster
17:30 sonicrose joined #gluster
17:31 CyrilPeponnet well @hagarth1 geo-rep is currently in bad shaped, not syncing anything else: [2016-02-09 09:30:30.898741] W [syncdutils(slave):480:errno_wrap] <top>: reached maximum retries (['.gfid/8f691bb5-9867-4deb-86c5-a0a0101a0c3d', 'glusterfs.gfid.newfile',
17:32 sonicrose hi gluster folks...  I am looking to setup a new gluster cluster but i'm having some trouble with performance and i'm kinda stuck on what to do next
17:33 CyrilPeponnet I hope the slave volume is not screwed...
17:33 pppp joined #gluster
17:34 cpetersen Weird.  I don't have any GFID references in xattrop for that GFID.
17:35 sonicrose is it possible to hard mount gluster without using NFS?
17:35 theron joined #gluster
17:37 djgerm joined #gluster
17:38 rafi1 joined #gluster
17:38 JoeJulian sonicrose: Do you mean like https://gluster.readthedocs.org/en/latest/Administrator%20Guide/Setting%20Up%20Clients/#mounting-volumes
17:38 glusterbot Title: Setting Up Clients - Gluster Docs (at gluster.readthedocs.org)
17:40 cpetersen Oddly, after manual heal, node 3 used to say it had a heal-failed entry, now it doesn't and node 2 does now.
17:40 cpetersen Perhaps it is now healing?!
17:40 cpetersen This makes me worried.  Time to check some logs.
17:44 CyrilPeponnet @hagarth1 is there a way to gather more logs, I have some time right now. There is no DEBUG for diagnostics.brick-sys-log-level.
17:45 CyrilPeponnet oh ok not the good option ;p
17:48 farhoriz_ joined #gluster
17:49 B21956 joined #gluster
17:50 anrao joined #gluster
17:50 cpetersen @JoeJulian, how did you find the file path from the GFID again?
17:50 cpetersen The commands in the Gluster article aren't working for some reason.
17:50 JoeJulian @gfid
17:50 glusterbot JoeJulian: The gfid is a uuid that's assigned to represent a unique inode that can be identical across replicas. It's stored in extended attributes and used in the .glusterfs tree. See http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/ and http://joejulian.name/blog/what-is-this-new-glusterfs-directory-in-33/
17:51 JoeJulian hrm, no, that's not what I was looking for
17:51 cpetersen ty
17:51 JoeJulian @gfid lookup
17:51 cpetersen :o
17:51 JoeJulian @lookup
17:52 JoeJulian @gfid resolver
17:52 glusterbot JoeJulian: https://gist.github.com/4392640
17:52 JoeJulian cpetersen: ^
17:52 cpetersen ahar
17:53 jiffin joined #gluster
17:54 hagarth1 CyrilPeponnet: a recent blog from one of our geo-replication developers can possibly help in understanding internals - http://hrkscribbles.blogspot.com/2016/02/understanding-internals-of-gluster-geo.html
17:54 glusterbot Title: scribbles...: Understanding the Internals of Gluster Geo-replication Setup! (at hrkscribbles.blogspot.com)
17:55 CyrilPeponnet I have brick debug log when crashing if it can help
17:55 CyrilPeponnet not sure there is more info on it...
17:55 sonicrose JoeJulian: thanks for that, no i don't see a "hard" mount option there.   On NFS, you can have soft or hard mounts as defined in your mount -o  ... i'm storing VHD files and I prefer hard mount because it will not return IO errors to the guest OS, instead the VM will wait or hang until storage is available again
17:56 CyrilPeponnet (and the link you send me is exactly how I've set up the geo-rep except for the medata volume)
17:56 CyrilPeponnet becaise of Applicable for gluster version >= 3.7.0
17:57 sonicrose JoeJulian: if you sent IO errors to the guest OS, then they start thinking that they need an fsck and will mark their root are read-only...   using a gluster fuse mount, I had a small network disrupt and all the VMs had to be restarted and many had to run fsck /chkdsk when they rebooted it was yucky.  I never had that problem using the NFS hard mount before and I dont see a FUSE/gluster alternative :(
17:58 CyrilPeponnet @hagarth1 I guess my issue could be related to  c) It enables 'changelog' (changelog xlator) and 'geo-replication.indexing' (marker xlator)
17:59 sonicrose perhaps from a different perspective entirely: is there any preferred volume options that should be used when storing large VHDs that are being actively used by many VMs?
18:00 jwang joined #gluster
18:00 hagarth1 CyrilPeponnet: yes, both are necessary for geo-replicaiton to function
18:00 cpetersen lol why am I getting command not found when running your gfid-resolver
18:00 cpetersen LS
18:00 cpetersen LS
18:00 cpetersen :s
18:00 hagarth1 it should not result in a crash :)
18:00 JoeJulian It's not mine
18:01 cpetersen Sorry.
18:01 CyrilPeponnet @hagarth1 with 3.6.5 I need the shared storage as well ?
18:01 JoeJulian And I have no idea why your install doesn't have `ls`.
18:02 CyrilPeponnet and btw redhat storage 3.1 is correponding to which gluster release (and 3.0 ?)
18:02 hagarth1 CyrilPeponnet: not needed. shared storage provides better resilience when a node goes down.
18:02 CyrilPeponnet ok
18:02 JoeJulian If that had been mine, I would have used sys functions instead of a shell command. :D
18:02 hagarth1 CyrilPeponnet: 3.1 corresponds to 3.7.x and 3.0 corresponds to 3.6.x
18:02 CyrilPeponnet thz
18:02 bennyturns joined #gluster
18:03 JoeJulian because mine would have been in python
18:04 bennyturns joined #gluster
18:06 CyrilPeponnet @hagarth1 So management gave me till tomorrow evening to use geo-rep, afterward I will have to fallback to rsync scripts.... :/
18:07 hagarth1 CyrilPeponnet: let me shoot an email right away
18:07 sonicrose JoeJulian: why is XenServer so weird about using gluster over NFS?  From the hypervisor i can read/write to the nfs mount at 100+ MB/sec, but within a VM with it's VHD on gluster over NFS, i seem to max out at 7MB/sec  even on 10GbE
18:09 ovaistariq joined #gluster
18:10 JoeJulian sonicrose: no idea. I use kvm.
18:11 JoeJulian With kvm, it's super fast.
18:11 sonicrose i know i've tried it :( unfortunately i'm locked into xenserver at the moment
18:11 hagarth1 CyrilPeponnet: done
18:11 cpetersen lol gfid-resolver didn't come up with anything
18:12 cpetersen It spit out a space.
18:13 CyrilPeponnet @hagarth1 thanks I'm providing more info
18:17 JoeJulian cpetersen: when you ls -l the gfid file, is the number after the permissions, "1"?
18:18 cpetersen Hmmm that's probably because the file doesn't exist anymore.
18:18 cpetersen Correct it is.
18:18 JoeJulian Then yeah, there is no file.
18:20 cpetersen well shit
18:20 cpetersen :)
18:22 cpetersen I think I forgot to delete the indices after I resolved the split-brain last week
18:22 * cpetersen blushes
18:39 lanning sonicrose: Xen connects the image to the VM via the loopback block device.  That breaks up the read/write size to very small size. Bad for Gluster performance.
18:40 lanning KVM QEMU talks to GlusterFS directly via libgfapi. Massive speed up.
18:43 theron joined #gluster
18:47 ovaistariq joined #gluster
18:57 sonicrose lanning: thank you very much for the explanation.  something like that is what i was dreading...
19:04 lanning technically that IO breakup is bad for any high latency IO. (ie. network based IO)
19:04 mhulsman joined #gluster
19:06 sonicrose lanning: it is worst going nfs to gluster for sure.  i replaced the NFS mount with a symbolic link to the same files on a fuse mount and the VM works much faster but still not superfast
19:06 CyrilPeponnet nfs is better for small files, gluterfs for big files, and libgfap even faster for vms.
19:07 rafi joined #gluster
19:09 uebera|| joined #gluster
19:09 lanning FUSE does have some code for read/write coalescing, but libgfapi removes a lot of the latency issues with ping/poinging data in and out of the kernel.
19:09 ueberall joined #gluster
19:09 ueberall joined #gluster
19:10 JoeJulian nfs is *not* better for small files. RAM is better for small files.
19:11 lanning :)
19:11 theron joined #gluster
19:11 lanning rsync to tmpfs
19:11 JoeJulian And that's what you're getting with an nfs mount. You're reading from ram a lot because the kernel caches stuff.
19:11 CyrilPeponnet well hosting homedir over nfs if way faster than hosting them over gfs, this is my feedback
19:11 post-factum failover with nfsis pita
19:11 JoeJulian So you could be getting stale stuff.
19:11 post-factum s/nfsis/nfs is/
19:12 glusterbot What post-factum meant to say was: failover with nfs is pita
19:12 rwheeler joined #gluster
19:13 post-factum i'd prefer using ceph rbd for vm images and glusterfs for file storage
19:13 JoeJulian I still like having faster vm images over using ceph.
19:14 post-factum well, they are not on par with local ssd, but 10gbe makes thing work faster
19:14 lanning I'm using glusterfs via kvm qemu libgfapi. works great. :)
19:14 post-factum lanning: what about healing after 1 node fail?
19:15 JoeJulian same hardware, same network, same hypervisor (kvm): gluster measured out over 3x faster than ceph.
19:15 CyrilPeponnet healing is a pain with qcow2 imgs...
19:15 CyrilPeponnet I stopped using replicated setup for that usage.
19:15 JoeJulian ^ yeah, we use raw images.
19:15 lanning I use raw, also
19:16 post-factum raw is not suitable for us
19:16 JoeJulian Why?
19:16 JoeJulian Just curious
19:16 post-factum we exploit snapshots with overprovisioning and fast image cloning
19:16 CyrilPeponnet why use raw over qcow2 @JoeJulian ?
19:16 lanning I backup the content of the images, not the images. :)
19:16 JoeJulian I will admit, I do love ceph's ability to spin up 100 vm's from the same image instantly.
19:17 CyrilPeponnet same here qcow2 used for snapshoting / backing file.
19:17 JoeJulian I've not found snapshotting to be of any value.
19:17 post-factum we constantly upgrade base image, take new snapshot of it and clone for new vms
19:18 CyrilPeponnet ok for us. spawning 50 vms based on the same base image is a huge plus.
19:18 JoeJulian CyrilPeponnet: I use raw because it requires less overhead.
19:18 post-factum JoeJulian: with ceph we also use raw, but stored on rbd, that's why trim works, and they are overprovisioned :)
19:19 post-factum much cheaper in terms of space, especially ssd
19:19 hagarth joined #gluster
19:19 JoeJulian In a perfect world, gluster on btrfs will give us the last feature ceph has over gluster.
19:19 post-factum JoeJulian: "in a perfect world" is a key phrase, unfortunately
19:19 JoeJulian It's coming.
19:19 post-factum in a perfect world cephfs should work for us as well, but it doesn't
19:20 JoeJulian You can get that with the gluster BD translator, but that requires it backed with lvm - which isn't a bad thing just doesn't fit with our design.
19:20 post-factum same for us, no lvm
19:21 post-factum btw, looking forward to reiser4 developer response, i've sent him questions for interview, and ceph/glusterfs were among them
19:22 toshywoshy joined #gluster
19:24 JoeJulian Now that I've said that it doesn't fit... I may change my mind. I'm going to have to take another look at that in the next product iteration.
19:26 JoeJulian I wonder why my brain does that. As soon as I say, "No" or "That won't work" I immediately try to invalidate my statement. :)
19:28 lanning I do the exact same thing...
19:32 lanning My theory is that I need to be correct. So if I correct myself, I am still correct. If someone else corrects me, then I am wrong. :)
19:33 sonicrose im really stuck trying to decide what to do with my platform :/  as of right now i have no clue what i should do...  so many options...  im curious how others would choose to set this up...  i'm working with 8 servers here, 4 as virtualization nodes and 4 as storage nodes...  xenserver is required on the 4 virt nodes, and I have gluster running successfully in dom0...  each of the virt nodes has dual 10 GbE and 2 x 4TB SATA disks.
19:34 sonicrose each of the storage nodes has dual 1GbE and 5 x 6TB SAS drives
19:35 sonicrose i have a ~15TB volume just using the local storage of the XenServers... without even involving any storage nodes
19:35 sonicrose and it runs quite fast even over just 1 of the 10GbEs 300MB/sec write and 600MB/sec reads
19:35 JoeJulian I start with defining my needs. What service is this system that you're engineering supposed to provide? What are the operational requirements? Then engineer a system that provides those.
19:37 sonicrose thanks JoeJulian , the need is to run about 100 VMs, and not have them be laggy.   the 4 xenservers have enough capacity to run those VMs, now I just need to select a shared storage method so that the VMs are Agile and can live migrate between hosts
19:38 sonicrose HA is a requirement
19:39 sonicrose thats about it.  I think i have the right hardware for the task... and I get really good performance as well until i get to the virtualization layer in Xen there where as lanning said, it's breaking up the data into very small writes
19:39 sonicrose it works better omitting nfs completely and going to a FUSE mount instead
19:40 sonicrose however i'm worried about stability in that case
19:40 JoeJulian the fuse mount is way more stable than nfs.
19:40 JoeJulian And it's HA by design
19:40 sonicrose JoeJulian: if theres a network interruption, i think the FUSE mount will break
19:40 sonicrose whereas NFS just seems to 'resume'
19:41 JoeJulian The gluster mount will wait for ping-timeout. Adjust it longer if you see the need (less than 30 minutes though).
19:41 JoeJulian Generally your VM's filesystem is going to time out first and go read-only anyway.
19:42 sonicrose i'm trying to prevent the case of a VM filesystem going read-only yes.
19:42 sonicrose i can do that right now over NFS by using the 'hard' mount option...
19:42 sonicrose JoeJulian: how is frame-timeout different
19:43 sonicrose i also want it to failover quickly if a storage server goes offline
19:43 JoeJulian How? Time still progresses and when an nfs hard connection is lost.
19:43 JoeJulian And with the fuse mount, there is no "failover" it's an always-on connection to all the servers.
19:44 JoeJulian frame-timeout is how long any one framed transaction can linger
19:44 sonicrose JoeJulian: i used to suffer many many cases of read-only root VMs  until i found xenserver defaults is for "soft" mount.  after I changed that to "hard" mount, i've never had a read-only root since then because the VM will simply freeze, forever, until storage is restored, or until it is force shutdown
19:45 sonicrose basically any IO errors forwarded into the guest OS will make it go read-only
19:45 sonicrose using hard mount prevents sending any IO errors, instead it waits and continuously retries
19:45 sonicrose i much prefer this over soft mounts
19:47 sonicrose i basically want to avoid IO errors in the event there is a say 30 second outage of the network
19:48 sonicrose nfs hard mounts seem to tolerate 'brief' disruptions very gracefully.  i'm worried about the FUSE mount being so graceful
19:48 sonicrose JoeJulian: if i understand you correctly, setting a ping timeout of say, several minutes, would prevent IO errors being returned for atleast several minutes?
19:48 theron joined #gluster
19:53 JoeJulian 42 seconds is already the default.
19:53 JoeJulian ext4 times out in 30 seconds.
19:53 cpetersen joined #gluster
19:54 JoeJulian But you can both lengthen the ping-timeout safely - and you can change ext4's behavior to not go read-only: https://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/
19:54 glusterbot Title: Keeping your VMs from going read-only when encountering a ping-timeout in GlusterFS (at joejulian.name)
19:57 sonicrose JoeJulian: thanks you :D  so i think i am going to just write a replacement mount command so that when xenserver tries to mount nfs we symlink to a gluster mount instead...  thank you!
19:59 JoeJulian You're welcome. :)
20:02 armyriad joined #gluster
20:05 hagarth joined #gluster
20:07 nickage_ joined #gluster
20:14 edong23 joined #gluster
20:18 crashmag joined #gluster
20:31 timotheus1_ joined #gluster
20:37 post-factum JoeJulian: talking about failover and ping timeout... is that reasonable to keep 42s for local cloud interconnected with 10gbe?
20:38 post-factum in fact, for us if hypervisor failed to respond in 5 secs, it is a disaster
20:40 post-factum so i guess, 10 secs should be sufficient for vm storage as well as file storage
20:44 theron joined #gluster
20:45 JoeJulian @ping-timeout
20:45 glusterbot JoeJulian: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
20:46 CyrilPeponnet @hagarth btw issuing gluster vol set usr_global diagnostics.brick-log-level CRITICAL crashed all the bricks of my vol https://gist.github.com/CyrilPeponnet/11954cbca725d4b8da7a
20:48 JoeJulian Ah, changelog_rollover again.
20:48 JoeJulian bug 1221629
20:48 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1221629 high, unspecified, ---, vshankar, ASSIGNED , Bitd crashed
20:48 CyrilPeponnet same issue than the geo-replication issue I have ?
20:49 JoeJulian I suspect so
20:49 mhulsman joined #gluster
20:54 CyrilPeponnet god... each time I touch this gluster it just falls appart...
20:55 CyrilPeponnet how come setting a log level option can make crash the changelog thinhgy...
20:57 calavera joined #gluster
20:58 JoeJulian Ah... it's *because* of the geo-rep configuration.
20:58 CyrilPeponnet look that from the logs that it tried to change the changelog stuff
20:58 CyrilPeponnet not sure why...
20:58 CyrilPeponnet but this could cause the crash
20:58 clyons joined #gluster
21:00 CyrilPeponnet The funniest is that I have geo-rep running with to other volumes... and they are not going bad
21:01 JoeJulian Oh, well that's sure changed completely in master...
21:01 farhoriz_ joined #gluster
21:01 m0ellemeister joined #gluster
21:01 JoeJulian https://github.com/gluster/glusterfs/blob/release-3.6/xlators/features/changelog/src/changelog.c#L1641-L1651
21:01 glusterbot Title: glusterfs/changelog.c at release-3.6 · gluster/glusterfs · GitHub (at github.com)
21:02 JoeJulian So I take that to mean that it happens when you reconfigure a volume.
21:04 rwheeler joined #gluster
21:05 CyrilPeponnet make sense
21:05 CyrilPeponnet starting geo-rep will enable the changelog options,
21:05 CyrilPeponnet changing log-level on brick, not sure to get why it needs to touch the changelog options but from the logs sounds related.
21:08 CyrilPeponnet Anyway I send a bunch of logs and trace to the thread launched by @hagarth, let's see...
21:08 CyrilPeponnet btw setting the geo-rep to config change-detector xsync, doesnt crash the brick
21:08 B21956 joined #gluster
21:09 CyrilPeponnet (but I had to shut it down because it was overloading the disks as our gluster are pretty loaded on week days)
21:09 CyrilPeponnet (hopping this will be usefull for you guys)
21:10 JoeJulian It will be for me. We're planning on doing geo-rep by the end of the year.
21:14 CyrilPeponnet good luck :p 2 years I tried, 2 years it fails ;p
21:17 JoeJulian I'm rather persistant and I can usually pinpoint bugs when I'm given the time to dig in to them.
21:18 CyrilPeponnet Same here, except I don't have all the gluster knowledge you have.
21:18 JoeJulian It's just time.
21:18 JoeJulian Plus, hanging out here helping everybody else.
21:19 JoeJulian When you figure out enough other people's problems, you end up having very few yourself. :D
21:23 hagarth CyrilPeponnet: 3.7 is where I am most confident about geo-replication as there have been nice enhancements to improve resilience.
21:24 CyrilPeponnet Yeah I've seen that.
21:25 CyrilPeponnet but you know with 5 cluster setup in prod and 2k clients, updating is a long journey :/ even with puppet help
21:31 theron_ joined #gluster
21:32 JoeJulian The hardest part is the self-heals.
21:33 JoeJulian I considered doing a "pkill -9 -f gluster" on the servers and seeing if I could get the upgrade done inside of ping-timeout so heals wouldn't be necessary.
21:33 arcolife joined #gluster
21:34 Wizek joined #gluster
21:35 JoeJulian I suppose NSR will cure that. Then it'll just have to replay the journal.
21:37 dlambrig_ joined #gluster
22:05 theron joined #gluster
22:09 moss JoeJulian: I could use your help :)
22:09 rwheeler joined #gluster
22:10 moss I've got a 3 node Gluster replicate cluster - I've created a new server which is to act as a client and mount the data to a directory.  It does do this, however, when I try to write to the directory it claims that it is read only.  I do not have read only specified in my fstab.. what gives?
22:12 JoeJulian If you have configured quorum then it means that the client isn't connecting to all the bricks.
22:12 JoeJulian (or more than half)
22:12 moss should i need to configure quorum with 3 nodes?
22:13 JoeJulian "need to": no. Should?: yes. Did you?: No idea.
22:13 moss I did not :)
22:13 moss Can you point me to any docs on configuring quorum with 3 replicate nodes?
22:14 moss ah i figured it out
22:14 moss I had to add firewall rules on all 3 nodes for the client
22:20 JoeJulian That would do it.
22:20 moss JoeJulian: I also did: gluster volume set all cluster.server-quorum-ratio 51%
22:21 hagarth joined #gluster
22:21 JoeJulian You'll also need to enable it for your volumes: gluster volume list | xargs -I{} gluster volume set {} cluster.server-quorum-type server
22:22 moss I only have 1 volume
22:22 moss I think
22:23 JoeJulian Hehe
22:23 cpetersen joined #gluster
22:24 moss fantastic
23:01 neofob left #gluster
23:07 calavera joined #gluster
23:09 jhc76 I'm running into weird behavior with samba-vfs-glusterfs module. is there a channel for this?
23:13 JoeJulian What're you seeing?
23:16 jhc76 Hi JoeJulian. samba with samba-vfs-glusterfs mounting the distribution mode volume. It shows directories and files. when I create a folder and files, it does create it. but when I delete it through cifs share, it disappears but it really didn't. as if it only deleted the metadata. the folder and file is still intact.
23:17 jhc76 but if I were to mount the volume using gluster-client, then everything works like it supposed to.
23:17 jhc76 I'm on 3.7 for both glusterfs and the samba-vfs module
23:19 JoeJulian Anything in any logs? I'm not sure where the samba puts the libgfapi log. Maybe it's under /var/log/glusterfs?
23:20 JoeJulian ira was here earlier. he knows the vfs really well, not sure if he's still around though.
23:21 hagarth jhc76: would you be able to describe this problem with versions of samba and gluster etc. on gluster-users?
23:21 hagarth jhc76: a developer would be able to help then
23:22 jhc76 I'm looking at the log I set to dump to /var/log/glusterfs/ with verbosity to 10. but it's cryptic to my eyes.
23:22 * JoeJulian goes back to his $dayjob then...
23:22 jhc76 yes. I'll do that. thanks hagarth
23:24 hagarth once obnox is online, he might be able to help with this
23:24 jhc76 ok. try joining gluster-users channel. I'm the only one there.
23:25 CyrilPeponnet mailing list :)
23:25 JoeJulian @mailing lis
23:25 glusterbot JoeJulian: I do not know about 'mailing lis', but I do know about these similar topics: 'mailing list', 'mailing lists'
23:25 JoeJulian @mailing list
23:25 glusterbot JoeJulian: the gluster general discussion mailing list is gluster-users, here: http://www.gluster.org/mailman/listinfo/gluster-users
23:25 jhc76 ah got it
23:25 jhc76 thanks sirs
23:30 cpetersen joined #gluster
23:38 theron joined #gluster
23:38 cpetersen Whelp, I was able to successfully achieve vCenter HA functionality on my setup.  :)
23:39 cpetersen Still have the NFS-Grace resource error.
23:39 cpetersen And I briefly had a split-brain but it resolved itself somehow.
23:40 JoeJulian :( That sounds sad. I'm sorry you have to continue to dump money into vmware.
23:41 cpetersen You raise a valid point.
23:41 cpetersen I've never used any Linux based virtualization technologies.
23:41 cpetersen I've used VirtualBox alot but that's Java.
23:41 cpetersen And not really an enterprise solution.
23:41 cpetersen KVM is the option of choice right?
23:42 JoeJulian It is for me. I tried out several of them once and they all performed more or less the same, so I went with kvm since it required the least amount of work to be useful.
23:43 cpetersen Are there any good stacks that you would recommend, or just go with RHEL installing manually?
23:44 csim ovirt ?
23:44 cpetersen My company is very much where a lot of IT companies are, VMware is the trusted standard.
23:44 JoeJulian I'm doing rack-at-a-time stuff, so we're using openstack. A lot of people love ovirt.
23:45 cpetersen Unfortunately VSAN is a bit too expensive.
23:46 JoeJulian We have customers that are in that same place. Luckily we're a datacenter so we can let them expand into our hardware using their own vcenter, and simultaneously offer them openstack so they can learn and migrate their deployments without that fear of loss.
23:47 toshywoshy joined #gluster
23:48 cpetersen Our VP of engineering mentioned that he saw a VSAN-esque implementation "tool" that was put in to an open beta by EMC.  But the tool they were using could pool redundant storage.  I'm not sure if that was DRBD or Gluster or Ceph, but that's what took me down this road.  Finding a solution for shared VM storage without spending $40k on VMware VSAN.
23:48 JoeJulian It's their own proprietary tool.
23:49 cpetersen SO you've heard of this?
23:49 JoeJulian Storage is what I do. I've heard of all of them.
23:49 JoeJulian Netapp has something similar.
23:50 cpetersen So in your opinion that you have coached me with, Gluster would be your chosen solution?
23:50 cpetersen :)
23:50 JoeJulian It *is* my chosen solution.
23:51 plarsen joined #gluster
23:52 JoeJulian We/I am waffling on whether I still want ceph, too. I suspect I'll keep it and offer both fast (gluster) and rapid deployment (ceph) until gluster can use btrfs snapshotting.
23:53 cpetersen I didn't go with Ceph because I have limited resources (compute) and it has a lot of components to get it setup.
23:54 cpetersen But would you say it is quicker to deploy from the ground up than what I did with Gluster?
23:54 cpetersen And Ganesha?
23:54 JoeJulian Ceph's an 800 lb gorilla, to be sure.
23:55 JoeJulian And no. Getting ceph installed right takes significantly longer than gluster. And required a lot more admin time to maintain.
23:55 cpetersen IC.  OK.
23:55 cpetersen Now I am super interested in trying out KVM.
23:56 cpetersen To see how it functions doing HA and VM migrations.
23:56 cpetersen Completely new to me, but if it's free and can do what VMware can do, my company can benefit from that.
23:57 JoeJulian But... it is nice if you're a user that needs dynamic load management and you might suddenly need to spin up 100 vm's instantly. Ceph does that with a metadata operation instantly.
23:57 JoeJulian Save them 100k in licensing and demand a raise. ;)

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary