Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-07-02

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 JoeJulian Yes, it can be rather mind boggling when you're first trying to compare apples to orchards.
00:00 TaiSHi I'm still trying to grab the concept of 'replica 3'
00:01 JoeJulian @brick order
00:01 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
00:01 JoeJulian So that's replica 2, of course. Replica 3 just means there are three redundant bricks that hold your data.
00:02 TaiSHi Ah, my initial thought, exact same data on all 3 servers
00:02 JoeJulian So in a 3 nine SLA, you have a 0.1% probability of a server failing. Since the same data still resides on the other two servers, your clients (even if those clients are also the servers) will still have access to the data during that outage.
00:03 JoeJulian The odds of two servers failing simultaneously are .001%. All three .0000001%
00:03 JoeJulian I find reality often kicks probability in the ass, but that's still pretty good.
00:04 TaiSHi All 3 would need to fail for the data to be completely unreachable
00:04 JoeJulian Right
00:04 JoeJulian You could have 10000 web servers serving off those three servers and as long as the iowait time meets your needs, you have enough servers.
00:04 JoeJulian You could have 10000 web servers serving off those three servers and as long as the iowait time meets your needs, you have enough *storage* servers.
00:05 TaiSHi I'll have to monitor it closely to see if it borks somewhere with the iowait, given the case I'll have to rethink all over
00:06 JoeJulian I would imagine that by the time those three servers *don't* meet your needs, you'll have a much stronger understanding of why they don't.
00:06 TaiSHi This is the smaller of all sites, last (and biggest) one will be ~10 webservers
00:07 JoeJulian Then you can add replicas or distribute your files to satisfy the load as is appropriate at the time.
00:07 TaiSHi Still I could propose a 4 (small/5 dollar) server cluster with replica 2 eventually
00:07 JoeJulian Of course, that change can be done live without any downtime.
00:07 TaiSHi Live? Seriously?
00:08 JoeJulian yes
00:08 TaiSHi I was looking at your post ( http://joejulian.name/blog/how-to-expand-gl​usterfs-replicated-clusters-by-one-server/ ). Wouldn't a set up with 2 brick per server with distributed-replica setup be more efficient and give more space ?
00:08 glusterbot Title: How to expand GlusterFS replicated clusters by one server (at joejulian.name)
00:10 JoeJulian That was just to explain the logic of distributing your risk as you advance from two servers to three, instead of the typical 2 to 4.
00:10 TaiSHi Yeah but it served me to understand that one server can have multiple bricks
00:11 TaiSHi My mind seems to be adjusting to gluster still :P
00:12 TaiSHi I'll go with replica 3 for now, and hopefully I'll understand more by the time the big (10 webservers) project kicks in
00:13 TaiSHi I appreciate all your help JoeJulian
00:13 JoeJulian apples and orchards. One person can eat an apple very quickly. It's much more efficient to feed one person an apple than it is to give them an orchard. But if you want to feed 1000 people, that one apple is no longer efficient. It's much more efficient to send them in to the orchard to eat. Sure, it takes them a little longer to get their apple, but you can feed that 1000 in only a slightly greater time than it took to feed the one.
00:15 Pupeno joined #gluster
00:17 TaiSHi It's a good analogy and I think I understand it applied to this case. The idea of "migrating" a volume to other server set without downtime is great
00:17 TaiSHi Oh, one more, one more, if my volume is replica 3 and with 3 nodes
00:18 TaiSHi How could I convert it to replica 2 with 4 nodes ?
00:29 JoeJulian TaiSHi: That's a lot more complicated. The best way would be to use remove-brick to reduce it to a replica 2 volume with 2 bricks, then add-brick 2 bricks.
00:30 TaiSHi I recall trying that and receiving an error yesterday, well, firing up 3 VMs to try it again lol
00:31 JoeJulian You would if you don't format the former 3rd brick before trying to re-add it.
00:32 TaiSHi The error was when trying to remove it
00:32 JoeJulian remove-brick $vol replica 2 $third_brick
00:33 TaiSHi exactly what I did, let me see if I can reproduce the error
00:34 harish joined #gluster
00:41 TaiSHi Do I have to stop the volume for that ?
00:45 TaiSHi It just worked on debian, bad thing that debian's default version is 3.2.7... ¬¬
00:51 srjb joined #gluster
00:52 coredump joined #gluster
00:55 TaiSHi What would happen if a peer would die permanently and I would like to set up a peer with the same name?
01:04 gildub joined #gluster
01:05 hagarth joined #gluster
01:06 B21956 joined #gluster
01:27 hagarth joined #gluster
01:45 TaiSHi Whenever I cancel an rsync to a glusterfs mount I have to remount it due to Transport endpoint is not connected (107)
01:45 TaiSHi Any reason why that might be ?
01:47 gildub joined #gluster
01:48 glusterbot New news from resolvedglusterbugs: [Bug 764655] NetBSD port <https://bugzilla.redhat.com/show_bug.cgi?id=764655>
01:57 StarBeas_ joined #gluster
01:58 jvandewege_ joined #gluster
01:59 coredump joined #gluster
02:11 Ark joined #gluster
02:13 RameshN joined #gluster
02:23 harish joined #gluster
02:34 codex joined #gluster
02:35 srjb joined #gluster
02:45 srjb joined #gluster
02:48 srjb joined #gluster
02:57 theron joined #gluster
03:17 plarsen joined #gluster
03:32 bharata-rao joined #gluster
03:45 shubhendu joined #gluster
03:51 itisravi joined #gluster
03:57 TaiSHi Hmm seems gluster isn't having fun with me running almost 100k files
03:58 srjb joined #gluster
04:03 ndarshan joined #gluster
04:14 eryc joined #gluster
04:14 theron joined #gluster
04:15 Pupeno joined #gluster
04:16 rastar joined #gluster
04:26 kumar joined #gluster
04:30 RameshN joined #gluster
04:40 dusmantkp__ joined #gluster
04:40 spandit joined #gluster
04:44 nbalachandran joined #gluster
04:48 ndarshan joined #gluster
04:52 ppai joined #gluster
04:53 psharma joined #gluster
04:54 kshlm joined #gluster
04:58 prasanthp joined #gluster
05:01 davinder16 joined #gluster
05:12 nishanth joined #gluster
05:13 rjoseph joined #gluster
05:15 RameshN joined #gluster
05:22 vpshastry joined #gluster
05:23 lalatenduM joined #gluster
05:28 bala joined #gluster
05:30 RioS2 joined #gluster
05:32 kdhananjay joined #gluster
05:38 hagarth joined #gluster
05:41 ndarshan joined #gluster
05:50 rjoseph joined #gluster
05:50 srjb joined #gluster
05:52 bala joined #gluster
06:01 ktosiek joined #gluster
06:07 aravindavk joined #gluster
06:09 meghanam joined #gluster
06:09 meghanam_ joined #gluster
06:13 mbukatov joined #gluster
06:20 raghu joined #gluster
06:24 sahina joined #gluster
06:28 saurabh joined #gluster
06:31 vkoppad joined #gluster
06:34 aravindavk joined #gluster
06:55 vimal joined #gluster
06:59 fsimonce joined #gluster
07:02 Philambdo joined #gluster
07:09 keytab joined #gluster
07:14 karnan joined #gluster
07:15 deepakcs joined #gluster
07:16 bene2 joined #gluster
07:22 Ark joined #gluster
07:41 Philambdo joined #gluster
07:45 Pupeno joined #gluster
07:56 ctria joined #gluster
08:10 kumar joined #gluster
08:15 liquidat joined #gluster
08:19 ricky-ti1 joined #gluster
08:30 calum_ joined #gluster
08:40 [o__o] joined #gluster
08:44 andreask joined #gluster
08:55 ProT-0-TypE joined #gluster
08:57 Pupeno_ joined #gluster
09:09 Norky joined #gluster
09:39 rwheeler joined #gluster
09:39 calum_ joined #gluster
09:46 andreask joined #gluster
09:48 Pupeno joined #gluster
10:04 rjoseph joined #gluster
10:09 hagarth joined #gluster
10:11 Norman_M joined #gluster
10:20 Norman_M Hey guys! Did anyone of you tried to put user homes on a gluster cluster and mount it on login? We are trying it using nfs over gluster and got some performance issues and one in two days kernel panic on the nodes. Now we are looking for professional help solving these issues. Before we had a single nfs server without gluster and this worked out fine but was not high availible. There are 120 clients reading their home directories from the cluster. So
10:20 Norman_M my question is: Is there anybody who can help us here or personal in the area around Berlin?
10:20 ppai joined #gluster
10:23 JackPeter joined #gluster
10:37 Slashman joined #gluster
10:42 JackPeter Hi folks, I need some help with replacing a node in replicated volume. I added a new node the and started the replace-command, but nothing happens
11:00 hagarth joined #gluster
11:03 harish joined #gluster
11:07 edward1 joined #gluster
11:13 Pupeno joined #gluster
11:14 suliba joined #gluster
11:29 ramteid joined #gluster
11:34 andreask joined #gluster
11:34 meghanam joined #gluster
11:36 LebedevRI joined #gluster
11:39 ppai joined #gluster
11:39 andreask joined #gluster
11:40 Norky Norman_M, we've done a similar thing for a customer here inthe UK
11:40 Norky yes, we've found metadata performance to be a bit poor
11:40 Norky so parsing directories which contain many files takes a long time
11:41 Norky make sure you're using a recent version so that it includes readdirplus() support, which goes some way to helping that
11:42 Norky also, be aware that accessing a gluster volume via NFS wont be highly-available
11:43 Norman_M Do you mean a recent Version of gluster? We have got "glusterfs 3.4.4 built on Jun 12 2014 01:48:40"
11:43 Norman_M on the nodes
11:44 Norky you can make it slightly more available by using automount on the clients, so that when it comes to mount a volume, if one server is down it will use another
11:44 Norky yes, 3.4.4 does include it I think
11:44 Norky by "nodes" you mean the gluster servers?
11:45 Norman_M We are aware of that and want to change to the gluster client but the clients mounting the nfs are 32 bit and we have ot upgrade them first
11:45 glusterbot New news from resolvedglusterbugs: [Bug 1112520] Use History API when available instead of switching back to xsync <https://bugzilla.redhat.co​m/show_bug.cgi?id=1112520>
11:45 julim joined #gluster
11:46 Norman_M by nodes I mean servers, right
11:46 Norky what are the performance issues you're seeing? Slow directory browsing?
11:46 Norky the metadata performance can affect a lot of different things
11:47 Norman_M the system load on the servers is pretty high and the cpu load too. The clients are not that fast opening applications than on pure nfs, because it takes some time for the nfs to load the user config files for the applications
11:48 Norman_M our config is this
11:48 Norman_M Volume Name: gluster_dateisystem
11:48 Norman_M Type: Replicate
11:48 Norman_M Volume ID: 2766695c-b8aa-46fd-b84d-4793b7ce847a
11:48 Norman_M Status: Started
11:48 Norman_M Number of Bricks: 1 x 2 = 2
11:48 Norman_M Transport-type: tcp
11:48 Norman_M Bricks:
11:48 Norman_M Brick1: filecluster1:/mnt/raid
11:48 Norman_M Brick2: filecluster2:/mnt/raid
11:48 Norman_M Options Reconfigured:
11:48 Norman_M nfs.enable-ino32: on
11:48 Norman_M performance.cache-size: 2GB
11:48 lalatenduM joined #gluster
11:48 Norman_M diagnostics.brick-log-level: WARNING
11:48 Norman_M diagnostics.client-log-level: WARNING
11:48 Norman_M nfs.addr-namelookup: off
11:48 Norman_M performance.cache-refresh-timeout: 60
11:48 Norman_M performance.cache-max-file-size: 100MB
11:48 Norman_M performance.write-behind-window-size: 10MB
11:48 Norman_M performance.io-thread-count: 22
11:48 Norman_M performance.stat-prefetch: off
11:49 Norky are the application executalbes/libraries held on the gluster servers
11:49 Norky ?
11:49 Norky er, no pasting please
11:49 Norman_M this is the best working config at the moment
11:49 Norky at least, not more than 3 or 4 lines
11:49 Norman_M sorry... didn't know it
11:49 Norman_M won't happen again
11:49 Norky there's a recommended pastebin, cannot recall what it is atm
11:50 Norman_M nope the executables are locally, only the user homes are on the gluster servers
11:50 Norky okay, so it's just .rc type files?
11:50 hagarth joined #gluster
11:52 Norman_M what do you mean with .rc? there are only files located in a normal linux user home directory, mostliy small ones, configs and settings and the mail files from thunderbird
11:52 Norman_M and personal files, documents
11:53 Norky ls ~/.*rc   will show you small config files
11:53 Norky larger applications might have entire configuration directories
11:54 Norman_M in the home itself is only one rc file, otherwise there are directories
11:54 Norman_M like .thunderbird, .libreoffice, -firefox and so on
11:54 Norky have a look at all home directory and subdirectories, see if any of those contain large numbers of files
11:55 Norman_M they contain a lot of small files
11:55 Norky yeah, those will contain not just configuration but "state" and cached data
11:55 Norman_M yep... the firefox cache f. e. is overmounted by an temp fs
11:56 Norman_M but the rest is used via nfs
11:57 rastar joined #gluster
11:58 Norman_M do you have any recommendations for us?
12:01 Norky if you can, improve the latency of the underlying network (might require changing hardware); work out and address the cause of the high load on the servers; more memory never hurts (ours have 32GiB each); try the GlusterFS native (FUSE-based) protocol - build a test client if you have to
12:02 Norman_M and you think if we use the glusterfs client instead of nfs it will solve those issues?
12:02 Norky as for performance tunables, I don't *think* we used any more than you did
12:03 glusterbot New news from newglusterbugs: [Bug 1113066] DHT: Log new layout of directory generated during directory self healing <https://bugzilla.redhat.co​m/show_bug.cgi?id=1113066>
12:03 Norman_M The Network is new and fast... The servers only got 8 GB ram each, that may be a little low...
12:03 Norky I'm honestly not sure - I think it did for the customer I'm thinking of, I'm trying to find my notes
12:03 Norman_M Thanks a lot
12:04 Norky I'm not an expert - there should be people on later who might have better advice
12:04 Norman_M ok, I am patient :) thanks so far for your help
12:05 Norky certainly with regard to glusterfs itself, rather than infrastructure stuff
12:06 Norman_M I hope there will be someone who can help us, our admin is running out of hair XD
12:07 Norky for reference, we're getting ~0.25 ms ping time from client to server
12:07 Norky latency is more important than bandwidth
12:08 Norky certainly for small files
12:08 Norman_M the ping time in our setting is ~0.15 ms
12:08 Norky that's pretty reasonable, assuming Ethernet
12:08 Norky or are you using IB?
12:09 Norman_M nope it's 1GB Ethernet but we have got a very low hopcount and a fibre connection to the server rooms
12:10 Norman_M client - ethernet switch - ethernet/fibre switch - fibre/ethernet switch - server
12:15 deepakcs joined #gluster
12:18 rjoseph joined #gluster
12:18 ppai joined #gluster
12:22 nshaikh joined #gluster
12:27 B21956 joined #gluster
12:28 diegows joined #gluster
12:33 JackPeter does somebody know when the moving/copy command start when replacing a brick?
12:40 chirino joined #gluster
12:54 julim joined #gluster
12:58 theron joined #gluster
13:02 bennyturns joined #gluster
13:05 japuzzo joined #gluster
13:06 obelix_ joined #gluster
13:18 theron_ joined #gluster
13:19 theron joined #gluster
13:20 maasm joined #gluster
13:22 maasm Instead of going through this convoluted way of solving split-brain file by file: https://github.com/gluster/gluster​fs/blob/master/doc/split-brain.md
13:22 maasm Is it not possible to simply do something like "gluster volume testvol1 peer1 overwrites all on peer2"  ??
13:22 glusterbot Title: glusterfs/doc/split-brain.md at master · gluster/glusterfs · GitHub (at github.com)
13:25 premera joined #gluster
13:26 jobewan joined #gluster
13:26 Pupeno_ joined #gluster
13:27 maasm or perhaps something like: "gluster vol heal testvol01 peer1 wins force"  ?
13:31 hagarth maasm: policy based split-brain resolution enhancement intends doing that - http://www.gluster.org/community/doc​umentation/index.php/Features/pbspbr
13:31 glusterbot Title: Features/pbspbr - GlusterDocumentation (at www.gluster.org)
13:33 maasm hagarth: So fairly new then. But good, I'll need to keep an eye on that one!
13:33 glusterbot New news from newglusterbugs: [Bug 1101111] [RFE] Add regression tests for the component geo-replication <https://bugzilla.redhat.co​m/show_bug.cgi?id=1101111>
13:34 mjsmith2 joined #gluster
13:34 hagarth maasm: yes, any help you can provide us in shaping the enhancement right would be great!
13:35 julim joined #gluster
13:37 maasm hagarth: well I like this one as it goes about it in the "Policy" direction. Meaning automated recovery. Even better. Would like lot's off logging with that though ;-)
13:37 hagarth maasm: right :)
13:48 weirdan joined #gluster
13:50 weirdan hi. I'm seeing memory leaks (leading, eventually, to OOM killer being invoked). Any idea on how to solve this?
13:53 gmcwhistler joined #gluster
13:55 theron joined #gluster
13:58 kshlm joined #gluster
13:58 marbu joined #gluster
14:01 marbu joined #gluster
14:01 Ark joined #gluster
14:02 gmcwhistler joined #gluster
14:04 julim joined #gluster
14:14 nbalachandran joined #gluster
14:19 wushudoin joined #gluster
14:20 theron joined #gluster
14:23 srjb joined #gluster
14:26 davinder16 joined #gluster
14:29 ndk joined #gluster
14:31 mortuar joined #gluster
14:35 andreask joined #gluster
14:37 plarsen joined #gluster
14:41 rjoseph1 joined #gluster
14:42 theron joined #gluster
14:43 mortuar_ joined #gluster
14:43 bala joined #gluster
14:43 mbukatov joined #gluster
14:53 bennyturns joined #gluster
14:53 kkeithley GlusterFS Community Meeting in six minutes in #gluster-meeting on FreeNode
14:57 theron joined #gluster
15:04 ndk` joined #gluster
15:05 jdarcy joined #gluster
15:07 ProT-0-TypE joined #gluster
15:09 vpshastry joined #gluster
15:13 kshlm joined #gluster
15:18 xrsa joined #gluster
15:18 bala joined #gluster
15:20 ramteid joined #gluster
15:24 andreask joined #gluster
15:33 jbrooks left #gluster
15:34 glusterbot New news from newglusterbugs: [Bug 1115132] glusterfs two node replication test appeared to result in data corruption <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115132>
15:35 theron joined #gluster
15:41 jbrooks joined #gluster
15:42 Phil___ joined #gluster
15:43 Phil___ Is there a bug with the replace-command wihtin a ditstributed-replicated volume??
15:45 tg2 joined #gluster
15:49 vpshastry joined #gluster
16:00 theron joined #gluster
16:01 B21956 joined #gluster
16:17 sputnik13 joined #gluster
16:18 sputnik13 joined #gluster
16:28 mjsmith2 joined #gluster
16:32 rikki joined #gluster
16:38 Mo_ joined #gluster
16:50 Matthaeus joined #gluster
16:56 mjsmith2 joined #gluster
16:57 MacWinner joined #gluster
17:00 Matthaeus joined #gluster
17:02 rikki left #gluster
17:03 AaronGr left #gluster
17:04 vpshastry joined #gluster
17:10 plarsen joined #gluster
17:22 ninkotech joined #gluster
17:23 JoeJulian weirdan: what version?
17:23 weirdan 3.4.4
17:24 weirdan the servers use 3.4.3 actually, but the leak is on the client side
17:26 JoeJulian hmm, I've not noticed any leaks with 3.4.4. oom killer doesn't kill the offender, though. Are you sure it's the client?
17:26 weirdan oomkiller is triggered on the client side
17:27 weirdan glusterfs client process was using ~5G of memory when that happened, if I'm reading oom killer message correctly
17:28 weirdan this might be related: https://bugzilla.redhat.co​m/show_bug.cgi?id=1085511
17:28 glusterbot Bug 1085511: urgent, urgent, ---, pkarampu, VERIFIED , Perf: OOM when running performance regression tests(iozone sequential writes)
17:29 ninkotech joined #gluster
17:29 weirdan does anyone know if it's going to be fixed in 3.4 branch?
17:30 kkeithley do you think that's the same as https://bugzilla.redhat.co​m/show_bug.cgi?id=1112844
17:30 glusterbot Bug 1112844: urgent, high, 3.4.5, csaba, ASSIGNED , OOM: observed for fuse client process (glusterfs) when one brick from replica pairs were offlined and high IO was in progress from client
17:30 JoeJulian nice
17:32 kkeithley if so, it's included (i.e. fixed) in 3.4.5beta1
17:32 weirdan looks pretty close, however I believe both replicas were online in my case
17:33 kkeithley you can rpms from http://download.gluster.org/pub/glust​er/glusterfs/qa-releases/3.4.5beta1/
17:33 glusterbot Title: Index of /pub/gluster/glusterfs/qa-releases/3.4.5beta1 (at download.gluster.org)
17:34 glusterbot New news from newglusterbugs: [Bug 1073217] /var/lock/subsys/glusterfsd missing <https://bugzilla.redhat.co​m/show_bug.cgi?id=1073217>
17:34 weirdan well, I'm on a deb based system, so I'd have to wait for https://launchpad.net/~semiosis​/+archive/ubuntu-glusterfs-3.4 to have updated
17:34 glusterbot Title: ubuntu-glusterfs-3.4 : semiosis (at launchpad.net)
17:34 kkeithley yep
17:35 weirdan thanks, that looks promising indeed
17:37 weirdan also I'm getting a lot of error messages like these:
17:37 weirdan [2014-07-02 07:20:01.710587] E [iobuf.c:733:iobuf_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/​xlator/debug/io-stats.so(io_stats_readv+0x16e) [0x7faaa133a79e] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/x​lator/performance/md-cache.so(mdc_readv+0x15f) [0x7faaa1550cbf] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/x​lator/performance/quick-read.so(qr_readv+0x4a) [0x7faaa175e0fa]))) 0-iobuf: invalid argument: iobuf [2014-07-02 07:20:01.710627] E [
17:38 JoeJulian weirdan: semiosis ususally builds qa and beta releases by request. If you can't wait for release, I'm sure he would be willing to build the current beta of 3.4.5.
17:38 weirdan that's good to know, thanks
17:38 JoeJulian I've been seeing a lot of those too.
17:39 cogsu_ joined #gluster
17:45 semiosis bug 961615
17:45 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=961615 urgent, high, ---, pkarampu, CLOSED CURRENTRELEASE, Hypervisor mount crashed after rebalance
17:46 Peter1 joined #gluster
17:53 semiosis weirdan: uploading 3.4.5beta1 to the ubuntu-glusterfs-3.4qa ppa now
17:54 weirdan wow, thanks
17:54 semiosis yw
17:56 semiosis it's in the build queue now.  could be a couple hours or more before it's published
17:56 semiosis https://launchpad.net/~semiosis/+arch​ive/ubuntu-glusterfs-3.4qa/+packages
17:56 glusterbot Title: Packages in “ubuntu-glusterfs-3.4qa” : ubuntu-glusterfs-3.4qa : semiosis (at launchpad.net)
17:57 zerick joined #gluster
17:59 Peter1 anyone experienced quota keep going up even files has been cleaned up?
17:59 Peter1 http://fpaste.org/115108/43239481/
17:59 glusterbot Title: #115108 Fedora Project Pastebin (at fpaste.org)
18:06 zerick joined #gluster
18:06 sonicrose joined #gluster
18:07 Dragon joined #gluster
18:09 sonicrose avati or JoeJulian, would you happen to know if it's possible to recompile fuse from source so that I can get direct_IO working with gluster client on a 32-bit EL5 box?  Joe mentioned the kernel fuse doesn't support direct_IO, could I compile a newer version of the fuse.ko from source or am I wasting my time?
18:11 sonicrose im working with Citrix XenServer and the dom0 is 2.6.32.43-0.4.1.xs1.8.0.853.170791xen  i686/i386, and I can't change it.  gluster client works OK, just no direct IO which is cooked into the hooks.  I tried liboindirect but it caused instability, as I dont want gigs of VHDs getting read thru the VFS cache
18:12 sonicrose my workaround for now is to run a gluster server VM to get NFS exports, but i'd like to cut out NFS entirely
18:14 JoeJulian sonicrose: As long as fuse is a module and not compiled in, then it should be possible.
18:16 Peter1 JoeJulian: do u happen to know if quota not able to release space after file got clean up?
18:16 JoeJulian no clue
18:20 sonicrose ty JoeJulian and thx for your reply earlier
18:31 Peter1 0-cli: Failed to get default soft limit
18:31 Peter1 what does that means?
18:33 Peter1 0-management: Received lock RJT from uuid: what is lock RJT?
18:34 Slashman joined #gluster
18:42 qdk joined #gluster
18:42 _Bryan_ joined #gluster
18:44 plarsen joined #gluster
18:46 obelix_ joined #gluster
18:50 ghenry joined #gluster
18:50 ghenry joined #gluster
18:52 JoeJulian RJT would be a reject
18:52 JoeJulian so uuid whatever rejected the lock request
18:53 JoeJulian Since that's 0-management that's something to do with glusterd, like a volume change or something like that.
19:03 Peter1 ic
19:03 Peter1 thanks!
19:04 Peter1 and then i got these
19:04 Peter1 http://fpaste.org/115134/04327875/
19:04 glusterbot Title: #115134 Fedora Project Pastebin (at fpaste.org)
19:04 Peter1 so many locking errors
19:05 Peter1 are these bugs on locking? files seems intact and users able to access volumes
19:05 JoeJulian If I were diagnosing it, and I'd stopped all my glusterd and started them all again, I would do some wireshark captures to try to find out what it's trying to do.
19:06 JoeJulian No, they have nothing to do with the client/brick connection, they're all management.
19:06 Peter1 what should i look at from wireshark?
19:07 JoeJulian I honestly don't know. It's all about finding patterns.
19:08 Peter1 what kind of network traffic pattern could cause locking?
19:09 theron joined #gluster
19:09 JoeJulian Something needs changed. One glusterd will send out a lock request to all the rest. If it gets the lock from all of them, then it sends the change. Once the change is complete, it sends the unlock request.
19:11 Peter1 ok
19:14 DV joined #gluster
19:16 sonicrose i wonder if that's why my rewrite speeds are horribad...   Writes and Reads individually are like 600MB/sec but when i do a re-write op its like 5MB/sec
19:21 coredump joined #gluster
19:33 theron joined #gluster
19:35 sonicrose i think i might be seeing another issue on 32-bit EL5, not sure... not related to the previous statement...   NFS server keeps crashing... i enabled debug logging and i got a crash dump but not sure how to tell what went wrong, would anyone donate a few mins to have a look?  http://sonicrose.com/crashlog.tar.gz  is the nfs.log and the core dump
19:35 sonicrose oops should http://sonicrose.com/crashed.tar.gz
19:36 JoeJulian sonicrose: Can you please file a bug report on this? Include that file.
19:36 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
19:37 sonicrose Im not a dev :p
19:37 JoeJulian Also, if you could install the debug packages. open that core file with gdb and "thread apply all bt" and include that as well
19:37 JoeJulian If you were a dev I'd ask you to fix it. :P
19:39 sonicrose before doing so i guess i could try disabling some performance xlators first
19:40 sonicrose i had bigger problems til i disables stat-prefetch... i'll turn off all performance things and retry
19:40 JoeJulian Not really. That bug information is very valuable to a lot of people. You would be doing the community a great service by reporting that bug.
19:43 sonicrose well, i do want gluster to do well...  after working on enterprise storage stuff like netapp and equallogic, gluster is a breath of fresh air... it only took me 6 months of fiddling with it to find a fast and stable configuration... i'm just trying to make it faster, i think cutting out the NFS protocol would do that, so either getting Direct_IO to work on fuse mount would be best then i dont have to worry about NFS crashing... i
19:43 sonicrose best thing but i've seen that there can be problems doing nfs mounts to 127.0.0.1
19:43 sonicrose i might be going down a bad rabbit hole
19:44 pureflex joined #gluster
19:45 sonicrose alrighty i made a bugzilla account :)... you're all in trouble now
19:46 Rydekull joined #gluster
19:50 theron joined #gluster
19:53 theron_ joined #gluster
19:54 TaiSHi left #gluster
19:54 theron_ joined #gluster
19:59 sonicrose so turns out its not NFS related... even doing a fuse mount is crashing with read/writes
19:59 sonicrose i guess should re-word my bugreport
20:01 sonicrose hey, quick question, on that O_DIRECT thingy... would it be possible that this could be fixed by a fuse libs patch in the gluster fuse stuff?  It seems that's how avati got O_DIRECT to work in the first place, and works on EL6
20:05 glusterbot New news from newglusterbugs: [Bug 1115648] Server Crashes on EL5/32-bit <https://bugzilla.redhat.co​m/show_bug.cgi?id=1115648>
20:05 pureflex joined #gluster
20:09 sonicrose where do i get gdb?
20:09 sonicrose i installed debuginfo-rpm but didnt get that command
20:15 sonicrose when using NFS... the NFS server is effectively using the FUSE client to make the gluster mount right?  I think this is not a gluster server issue... its a gluster client issue... the client crashes and that's why NFS stops working
20:16 semiosis sonicrose: fuse is not involved with the nfs server.  everything else is the same as the fuse client, without the actual fuse
20:17 theron joined #gluster
20:29 mjsmith2 joined #gluster
20:38 theron joined #gluster
20:39 andreask joined #gluster
20:40 theron joined #gluster
21:00 [ilin] left #gluster
21:01 jag3773 joined #gluster
21:14 Matthaeus joined #gluster
21:14 obelix_ joined #gluster
21:29 andreask joined #gluster
21:39 theron joined #gluster
22:05 pureflex joined #gluster
22:28 Matthaeus joined #gluster
23:19 theron joined #gluster
23:47 Ark joined #gluster
23:49 coredump joined #gluster
23:56 gildub joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary