Camelia, the Perl 6 bug

IRC log for #gluster, 2013-08-15

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 failshell joined #gluster
00:11 crashmag joined #gluster
00:12 arusso joined #gluster
00:36 robo joined #gluster
00:47 plarsen joined #gluster
00:48 Guest23519 joined #gluster
00:56 Durzo does 3.4 support network encryption ?
00:56 Durzo i remember talk about it
00:56 Durzo cant see anything about it.. unless its on by default
01:14 semiosis i dont think so
01:20 failshell joined #gluster
01:22 kevein joined #gluster
01:25 chirino joined #gluster
01:36 harish joined #gluster
01:38 zwu joined #gluster
01:38 failshell joined #gluster
01:55 MugginsM joined #gluster
01:55 rcoup joined #gluster
02:12 robo joined #gluster
02:34 awheeler joined #gluster
02:36 chirino joined #gluster
02:41 sprachgenerator joined #gluster
02:44 chirino joined #gluster
02:56 bala joined #gluster
02:57 chirino joined #gluster
03:32 JoeJulian Durzo: Actually, I'm pretty sure that it does...
03:36 JoeJulian Durzo: This ci test should be fairly easy to use as documentation (for now): https://github.com/gluster/glusterfs​/blob/master/tests/bugs/bug-873367.t
03:36 glusterbot <http://goo.gl/CWVsO> (at github.com)
03:37 Durzo doesnt look like it
03:37 JoeJulian Oh?
03:37 Durzo atleast not in the semiosis packages.. there are no ssl keys included
03:37 JoeJulian That's because there are no ssl keys included.
03:38 Durzo so its not on by default
03:39 semiosis o_O
03:39 Durzo i see.. i need to set server.ssl and client.ssl on the volume
03:39 JoeJulian Did you even read the test that I referred you to? That's how it's tested, so that's also how you would set it up.
03:39 JoeJulian Ah, now you did. :D
03:39 Durzo yes im reading it now
03:39 JoeJulian Yes.
03:40 Durzo can you specify the path to the certs, or is it just going to use /etc/ssl/glusterfs.* ?
03:43 JoeJulian I don't see any way of specifying other than at compile time.
03:43 Durzo is there any way to see/list all the available options on a volume? even defaults
03:44 JoeJulian No
03:44 Durzo not even looking at source?
03:44 JoeJulian Oh, well that...
03:44 semiosis 'gluster volume set help' and the ,,(undocumented options) page
03:44 glusterbot Undocumented options for 3.4: http://goo.gl/Lkekw
03:44 Durzo ahh wow thanks semiosis
03:44 semiosis yw
03:45 JoeJulian Oh, right... I forgot about that one...
03:45 JoeJulian https://github.com/gluster/gluster​fs/blob/master/xlators/mgmt/gluste​rd/src/glusterd-volume-set.c#L348
03:45 glusterbot <http://goo.gl/MOcr1g> (at github.com)
03:46 JoeJulian I use "git grep" a lot...
03:47 hagarth joined #gluster
03:51 Durzo i guess enabling ssl would cause quite an increase in cpu / bandwidth
03:52 semiosis cpu mostly
04:01 rcoup joined #gluster
04:09 lurbs joined #gluster
04:10 lurbs Hey, I'm having issues setting up Geo Replication using a non-privileged account, and mountbroker.  The slave is failing with:
04:11 lurbs http://paste.nothing.net.nz/30f01e
04:11 glusterbot Title: [untitled] (at paste.nothing.net.nz)
04:11 rcoup joined #gluster
04:11 lurbs The same setup seems to work if I swap to using root, instead of the non-privileged account.
04:11 MugginsM it worries me how many familiar names are in this channel
04:11 lurbs MugginsM: Ha!
04:13 lurbs 12.04 LTS and GlusterFS 3.4.0 from the Semiosis PPA, if that makes a difference.
04:16 JoeJulian lurbs: That makes sense. It's hard to set the trusted. attributes or the ownership if you're a non-privileged account.
04:17 lurbs Isn't that the point of mountbroker, though?
04:17 lurbs Or am I missing something entirely.
04:17 JoeJulian but then again, I haven't really used geo-rep so I could easily be wrong...
04:18 JoeJulian 110 = ETIMEDOUT
04:22 JoeJulian Is there anything more in log-file=/var/log/glusterfs/geo-replication-sla​ves/32ba9ec7-487a-42c7-a4ca-28d41ec07bf7:gluste​r%3A%2F%2F127.0.0.1%3Atest-volume.gluster.log ?
04:23 lurbs http://paste.nothing.net.nz/e2452e
04:23 glusterbot Title: [untitled] (at paste.nothing.net.nz)
04:26 lurbs The other concerning part is the "failed to fetch volume file (key:test-volume)"
04:27 JoeJulian Is this the master or the slave?
04:27 lurbs On the slave.
04:29 JoeJulian So you're geo-replicating to a gluster volume then, right?
04:30 JoeJulian If I'm reading this right...
04:31 lurbs Yep: http://paste.nothing.net.nz/52c5fb
04:31 glusterbot Title: [untitled] (at paste.nothing.net.nz)
04:34 lurbs And I've added what I believe to be the correct options for the mountbroker: http://paste.nothing.net.nz/7d994d
04:34 glusterbot Title: [untitled] (at paste.nothing.net.nz)
04:36 fidevo joined #gluster
04:44 JoeJulian Ok, how about the glusterd.vol.log from the slave?
04:47 lurbs etc-glusterfs-glusterd.vol.log?
04:48 JoeJulian yes
04:49 lurbs http://paste.nothing.net.nz/a2fb9c
04:49 glusterbot Title: [untitled] (at paste.nothing.net.nz)
04:52 JoeJulian Well that's odd...
04:53 JoeJulian That would make sense that it times out though... According to the timestamps, glusterd wasn't running at the moments that it tried to look up the volume.
04:54 JoeJulian I would chalk that up to unsynchronized clocks, but these logs are both on the same machine...
04:55 lurbs I can double check the clocks, no problem.
04:55 JoeJulian but this is all the slave...
05:01 lurbs I'll debug more into why that mount is failing on the slave.  Seems to be where the problem is.
05:02 JoeJulian I figured it was probably reading from socket failed. Error (No data available), peer (127.0.0.1:24007)
05:03 lurbs Yeah.  I saw people having similar problems without turning server.allow-insecure to on, because without that it'll drop connections from high ports.
05:04 lurbs But it didn't seem to make a difference.
05:05 JoeJulian you did create /var/mountbroker-root
05:06 JoeJulian Right, I thought about that, but since you have rpc-auth-allow-insecure on in the vol file, that should allow that for 24007.
05:06 lurbs drwx--x--x 4 root root 4096 Aug 15 13:37 /var/mountbroker-root/
05:07 lurbs If that doesn't exist, or has different ownership/permissions then the gluster daemon won't even start.
05:07 JoeJulian since glusterd is brokering the mount for the unprivileged user, the server.allow-insecure shouldn't be necessary.
05:19 JoeJulian Ah, ok... all the useful information from glusterd is at the debug level.
05:23 JoeJulian lurbs: if it makes you feel any better, I've duplicated your issue.
05:25 lurbs Heh, that does. :)
05:26 lurbs Thanks for looking into it, BTW.
05:29 kevein joined #gluster
05:41 rcoup joined #gluster
05:44 rgustafs joined #gluster
05:58 mohankumar joined #gluster
06:02 lalatenduM joined #gluster
06:09 jtux joined #gluster
06:47 piotrektt joined #gluster
06:47 piotrektt joined #gluster
06:52 ekuric joined #gluster
06:53 manik joined #gluster
06:56 ricky-ticky joined #gluster
07:01 vimal joined #gluster
07:03 JoeJulian @ports
07:03 glusterbot JoeJulian: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
07:17 kevein joined #gluster
07:19 JoeJulian lurbs: this is really wierd. The client sends the SYN to the server on 127.0.0.1:24007. That packet is never ACKed. Falling alseep now, though. Will have to come back at this later.
07:26 crashmag joined #gluster
07:41 andreask joined #gluster
07:47 mohankumar joined #gluster
07:51 kevein joined #gluster
07:56 asias joined #gluster
07:57 mgebbe_ joined #gluster
07:58 manik joined #gluster
07:59 mgebbe_ joined #gluster
08:22 Norky joined #gluster
08:50 spider_fingers joined #gluster
08:56 giany joined #gluster
08:57 giany hello, any idea why if I run df on 2 separate servers I get different sizes?
09:03 mbukatov joined #gluster
09:20 ababu joined #gluster
09:25 JusHal joined #gluster
09:27 asias joined #gluster
09:28 JusHal How can I set ownership on a glusterfs mount? I thought ' /usr/sbin/glusterfs  --fuse-mountopts=user_id=501,group_id=501' would do the trick, but it does not seem to work. Even in /proc/mounts is stays 'user_id=0,group_id=0'.
09:49 vimal joined #gluster
09:53 manik joined #gluster
10:00 JusHal joined #gluster
10:26 tobias- doing a new install of glusterfs in debian wheezy; it seems that it doesn't mount after network is enabled although _netdev is set in options. I guess this is a common problem but google is tricking me for solutions, any takers?
10:28 manik joined #gluster
10:33 duerF joined #gluster
10:40 kkeithley1 joined #gluster
10:40 SteveCooling Hi guys. I'm having trouble sorting out a failure after upgrading to 3.4. I have a volume that mounts fine, but breaks on "ls" with the message "Transport endpoint not connected". The client log says readv failed, and ends in a backtrace, but I cannot make use of it alone. Log: https://dl.dropboxuserconten​t.com/u/683331/test.log.txt
10:40 glusterbot <http://goo.gl/nAhnLo> (at dl.dropboxusercontent.com)
10:41 SteveCooling Servers running CentOS 6 64-bit and Glusterfs 3.4.0-8 RPMS. Client is 32.bit RHEL 5 and GlusterFS 3.4.0-8 RPMs
10:59 spider_fingers left #gluster
11:00 spider_fingers joined #gluster
11:03 vimal joined #gluster
11:12 JusHal joined #gluster
11:18 manik joined #gluster
11:18 manik joined #gluster
11:26 jporterfield joined #gluster
11:27 B21956 joined #gluster
11:47 JusHal joined #gluster
11:50 robo joined #gluster
11:55 ujjain joined #gluster
12:07 fcami joined #gluster
12:14 jclift_ joined #gluster
12:31 sprachgenerator joined #gluster
12:33 NeatBasis joined #gluster
12:34 mbukatov joined #gluster
12:43 awheeler joined #gluster
12:47 awheeler joined #gluster
12:48 zetheroo joined #gluster
12:49 zetheroo say you have two servers running a replica 2 gluster ... server 1 and server 2 ... the gluster is made on server 1 ...
12:51 zetheroo now on server 1 you would obviously do "mount server1:/gluster /mount/path/" ... but what happens if you do the following on server 2 "mount server2:/gluster /mount/path" ?
12:53 rcheleguini joined #gluster
12:55 dewey joined #gluster
12:58 hagarth joined #gluster
13:00 aliguori joined #gluster
13:16 bennyturns joined #gluster
13:36 chirino joined #gluster
13:42 semiosis :O
13:42 awheeler joined #gluster
13:42 awheele__ joined #gluster
13:45 chirino joined #gluster
13:47 failshell joined #gluster
13:48 harish joined #gluster
13:53 kaptk2 joined #gluster
14:00 kanagaraj_ joined #gluster
14:01 jmalm joined #gluster
14:01 lpabon joined #gluster
14:02 chirino joined #gluster
14:06 zetheroo ?
14:12 aliguori joined #gluster
14:15 chirino joined #gluster
14:17 tqrst- :O
14:17 sjoeboo joined #gluster
14:19 hagarth :O
14:20 awheeler joined #gluster
14:23 sjoeboo joined #gluster
14:27 zetheroo did anyone see my question above/
14:27 zetheroo ?
14:33 zaitcev joined #gluster
14:34 tqrst- zetheroo: in both cases, your volume will be mounted in /mount/path
14:37 tqrst- (the same volume, since there's only one)
14:37 awheele__ joined #gluster
14:39 zetheroo but one command is mounting the gluster from server1, and the other command is mounting a gluster from server2 ... and there was never a gluster created on server2 ..
14:42 _pol joined #gluster
14:43 bugs_ joined #gluster
14:44 chirino joined #gluster
14:46 JoeJulian ~mount server | zetheroo
14:46 glusterbot zetheroo: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds
14:48 zetheroo so do I even need to mount server1:/gluster on the client? -- or is the gluster live before being even mounted ... ?
14:48 zetheroo the issue we are having has to do with running VM's from gluster
14:48 spider_fingers left #gluster
14:49 JoeJulian SteveCooling: Nice crash you've got there. :(
14:50 bradfirj zetheroo: You must mount the volume before attempting to read or write from it
14:50 JoeJulian zetheroo: Are you using the native qemu links?
14:50 bradfirj r/w directly to the brick doesn't work and may break everything
14:50 zetheroo when one of two replicated bricks goes down the filesystems of the VM's on the brick which is still up go into read-only and don't come back out until the brick that went down is restored and the VM's rebooted
14:51 JoeJulian Which distro?
14:54 JoeJulian zetheroo: ^
14:54 zetheroo Ubuntu on the host
14:54 zetheroo the VM's are Linux and Windows
14:54 zetheroo Ubuntu 12.04 Server ... libvirt, KVM basic setup
14:55 JoeJulian I'm not sure how/if upstart handles the shutdown of glusterfsd during a shutdown. semiosis?
14:57 JoeJulian What may be happening is that your client loses connection to a replica server before it gets the FIN from glusterfsd closing. This will cause the client to hang for ping-timeout waiting for a theoretically temporary network interruption to recover.
14:57 JoeJulian @ping-timeout
14:57 glusterbot JoeJulian: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
14:57 LoudNoises joined #gluster
15:00 zetheroo I already tried bringing that timeout to 3s to see if the VM's would stay alive long enough for the gluster to come out of it's timeout period but that did not work either
15:00 robo joined #gluster
15:06 JoeJulian Just to make sure I'm not coming across as ambiguous, I'm not suggesting that adjusting ping-timeout is in any way the correct solution. Check "gluster volume status" and make sure everything's up. Check your client log and make sure you're connecting to all your bricks. Make sure glusterfsd stops (SIGTERM) before the network does to ensure the TCP connection is closed correctly.
15:14 semiosis JoeJulian: no idea.
15:16 zetheroo we have another new pair of exactly identical servers setup in identical fashion to the pair we have in production mode ... we are using this new pair for testing purposes ... so far I have been unable to reproduce the read-only issue there :P
15:17 lalatenduM joined #gluster
15:20 sprachgenerator joined #gluster
15:23 redragon_ joined #gluster
15:24 redragon_ so my question today is, if I have a raid failure under gluster (on brick) will gluster detect this and stop feeding data from that brick, and not attempt to heal it, it should i/o errors
15:24 semiosis redragon_: maybe
15:25 semiosis there are many kinds of failures, which may cause different behavior
15:26 zetheroo we had an idea here ... but not entirely sure if it would work ... basically making glustered bricks from a RAID ... the idea being that if a disk in the RAID dies, the gluster would not notice and the brick would remain running as normal ...
15:27 semiosis that's a very vague idea.  what kind of raid?  how did the disk die?
15:28 zetheroo raid1 ? the disk had a mechanical failure ... for instance ...
15:28 semiosis sounds hard to test
15:29 zetheroo yep ...
15:30 zetheroo tomorrow I will move one of the VM's onto the test environment and see if I can recreate the read-only issue
15:31 zetheroo left #gluster
15:31 manik joined #gluster
15:38 redragon_ i've got 4 brick replica with raid 0 under it for speed and trying to find a way to test so that gluster will drop using a brick if that brick has an issue but haven't done so well
15:39 redragon_ I may need to setup an snmp listener for raid failure and have it shut down glusterd
15:40 JoeJulian shutting down glusterd won't do anything. You'll need to kill the server ,,(process) associated with that brick.
15:40 glusterbot I do not know about 'process', but I do know about these similar topics: 'processes'
15:40 JoeJulian @processes
15:40 glusterbot JoeJulian: The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/F6jqx for more information.
15:40 JoeJulian @alias process processes
15:40 glusterbot JoeJulian: Error: No factoid matches that key.
15:40 JoeJulian @alias processes process
15:40 glusterbot JoeJulian: The operation succeeded.
15:41 redragon_ so it will need to shut down glusterfsd
15:41 semiosis redragon_: see bug 832609
15:41 glusterbot Bug http://goo.gl/WLzCC urgent, high, ---, rabhat, ASSIGNED , Glusterfsd hangs if brick filesystem becomes unresponsive, causing all clients to lock up
15:41 semiosis redragon_: and also bug 971774
15:41 glusterbot Bug http://goo.gl/h2W1JU medium, medium, ---, ndevos, CLOSED CURRENTRELEASE, [RFE] Improve handling of failure of a disk, raid array or raid controller
15:41 jcsp joined #gluster
15:42 redragon_ thanks JoeJulian
15:45 gkleiman joined #gluster
15:46 JoeJulian Hey, cool. I found a way to make glusterd unresponsive in 3.4.0.
15:46 redragon_ well says its in 3.4.0 but running 3.4.0 and we used omconfig to drop a raid (the one used for gluster source) and the brick remained up
15:46 redragon_ congrats
15:47 JoeJulian file a bug
15:47 glusterbot http://goo.gl/UUuCq
15:47 redragon_ will do more testing to make sure and build a replication plan for the bug report
15:47 JoeJulian That was for me...
15:48 JoeJulian I use glusterbot for bookmarks. :)
15:55 redragon_ lol
16:06 robo joined #gluster
16:12 lpabon joined #gluster
16:15 JoeJulian lurbs: bug 997576
16:15 glusterbot Bug http://goo.gl/D5yfwL high, unspecified, ---, kparthas, NEW , glusterd becomes unresponsive when acting as mountbroker
16:20 zerick joined #gluster
16:49 Mo__ joined #gluster
16:50 glusterbot New news from resolvedglusterbugs: [Bug 953887] [RHEV-RHS]: VM moved to paused status due to unknown storage error while self heal and rebalance was in progress <http://goo.gl/tw8oW>
17:13 lpabon_ joined #gluster
17:13 lpabon_ joined #gluster
17:15 lpabon joined #gluster
17:18 bstr joined #gluster
17:32 robo joined #gluster
18:01 jdarcy joined #gluster
18:01 Technicool joined #gluster
18:03 SteveCooling JoeJulian: about my crash. i'm back, and i can consistently reproduce. what can i do?
18:23 johnmark SteveCooling: file a bug report, for one
18:23 glusterbot http://goo.gl/UUuCq
18:32 rwheeler joined #gluster
18:47 robo joined #gluster
18:52 chirino joined #gluster
18:55 rwheeler joined #gluster
18:59 lpabon joined #gluster
19:11 bennyturns joined #gluster
19:15 chirino joined #gluster
19:20 lpabon joined #gluster
19:24 chirino joined #gluster
19:34 chirino joined #gluster
19:58 _pol joined #gluster
20:11 chirino joined #gluster
20:24 lpabon joined #gluster
20:24 lurbs joined #gluster
20:27 chirino joined #gluster
20:30 GomoX joined #gluster
20:30 GomoX Hello
20:30 glusterbot GomoX: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
20:31 GomoX I wonder if Gluster is a good solution for the following scenario, and if so, what would be the recommended tuning options for it
20:32 GomoX I have a bunch of application servers that serve customer web applications in a sharded distribution (i.e a single customer is typically served by one or just a few of the servers within the app server pool)
20:32 GomoX I would like to store customer files in a shared volume for operational simplicity, but I would like to keep good performance
20:32 GomoX (i.e compared to local storage)
20:33 JoeJulian "application servers" typically seem to be ,,(php) and thus the following is apropos:
20:33 glusterbot php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details.
20:34 GomoX I can definitely live with some replication lag , but I would like to keep the scaling overhead to a minimum
20:34 GomoX JoeJulian: this is PHP indeed but my concern is mostly file storage, not PHP scripts
20:35 GomoX Will read the link though
20:35 GomoX Ideally, what I want is the ability to fail over easily from a specific app server
20:35 GomoX And not lose too much performance while gaining that ability
20:36 GomoX Maybe Gluster is not the right tool for this, but if so, any ideas?
20:36 JoeJulian "compared to local storage" is comparing ,,(apples to orchards)
20:36 glusterbot Don't get stuck micro-engineering for scale http://goo.gl/rxZi1
20:36 JoeJulian Fault tolerance is something that gluster excells at.
20:37 JoeJulian and spelling is obviously something I don't... <sigh>
20:37 PatNarciso dittoe
20:38 GomoX I'm hardly microengineering :) i'm outgrowing a simpler solution
20:38 JoeJulian :)
20:39 JoeJulian Some people don't get scale and expect local performance from a CAP limited system.
20:39 PatNarciso @CAP
20:39 JoeJulian @lucky CAP
20:39 glusterbot JoeJulian: http://www.cap.org/
20:39 JoeJulian hmm, nope
20:39 GomoX Nah, my system is sharded already so I can grow easily in size but the operational complexity is what I'd like to avoid
20:40 GomoX So if I can have a fault tolerant file system that gives a specific client good behaviour on a small subset of the data that is good enough for me
20:40 JoeJulian I think you would like gluster then. Try it out, you can have it up and running within minutes.
20:41 GomoX I have used Gluster for VM image storage already, and performance kind of sucks (different workload altogether)
20:41 GomoX Hence the 2nd part of the question
20:41 GomoX Is there a way to tell Gluster to relax its sync expectations?
20:41 GomoX So that I get better performance on a (hopefully, somehow cached) subset of the data at the expense of transactional guarantees
20:41 JoeJulian Yeah, btw... qemu and GlusterFS 3.4 now have a native interface that performs 6X (or better) faster than what you probably tested.
20:42 JoeJulian What language is your app?
20:42 GomoX JoeJulian: definitely looking forward to that when I upgrade my VM cluster :) (I still use Gluster though, but only for SMB servers or so, no apps running in those VMs)
20:43 GomoX JoeJulian: this is a PHP application
20:43 GomoX JoeJulian: I will keep in mind the APC/stat things mentioned before, although I have that in place already IIRC
20:43 JoeJulian bah, I missed that earlier...
20:44 JoeJulian As long as you're not doing a lot of directory listings, everything else should be pretty performant.
20:45 a2_ and if you do a lot of directory listing, the readdirplus feature should help significantly
20:45 GomoX JoeJulian: is there a client side setting that might be interesting WRT to this local "buffering" i'm mentioning?
20:45 GomoX So I can google something more specific
20:45 a2_ JoeJulian, have you gotten around to upgrading to centos 6.4, to test out readdirplus?
20:45 JoeJulian My applications keep track of what files exist for my application using database tables. I look up the filename in the table so there's no ...
20:46 a2_ ah ok
20:46 JoeJulian a2_: Taht's awesome. I haven't.
20:46 JoeJulian So is that just a fuse thing? Does it require 3.4 or does that exist in 3.3?
20:46 PatNarciso @lucky readdirplus
20:46 a2_ 3.4..
20:46 glusterbot PatNarciso: http://goo.gl/ZKKN6i
20:47 JoeJulian I'm hoping to do that over laborday weekend.
20:47 JoeJulian Still on 3.3 but I want to hit 3.4 before Black Friday.
20:47 a2_ but the change in fuse is minimal to backport to 3.3 i guess.. (http://review.gluster.org/3905 is the major one)
20:47 glusterbot Title: Gerrit Code Review (at review.gluster.org)
20:48 JoeJulian Speaking of...
20:48 JoeJulian a2_: The wiki states, "3.3.2 is the latest version of GlusterFS recommended for production environments." Do you want that changed to 3.4?
20:49 a2_ i'd wait for 3.4.1 for production environments
20:49 JoeJulian ok
20:49 JoeJulian I found something that doesn't like the new threaded glusterd last night...
20:49 JoeJulian bug 997576
20:49 glusterbot Bug http://goo.gl/D5yfwL high, unspecified, ---, kparthas, NEW , glusterd becomes unresponsive when acting as mountbroker
20:51 JoeJulian I'm guessing you guys will look at it and go, "Oh! I know why!" but I couldn't figure it out.
20:51 GomoX From an operational point of view, I have noticed that gluster versions increaste quite fast, and there is some work involved in upgrading, any comments on that?
20:51 JoeJulian GomoX: Yes, that work has gone away. :D
20:52 GomoX Or should I expect to copy stuff over to new volumes for upgrading without or with minimal downtime?
20:52 a2_ JoeJulian, unack'ed syns are typically a firewall problem. the acks should be sent by the kernel independent of what the application is doing.
20:52 JoeJulian a2_: Through lo?
20:52 GomoX JoeJulian: what do you mean by that?
20:53 a2_ through wherever.. ack'ing a syn is not the responsibilty of the server process
20:53 a2_ it's a tcp/ip stack responsibility
20:53 JoeJulian a2_: lo... there's no firewall on the loopback connector...
20:53 a2_ hmm
20:53 JoeJulian yeah, that's what I said...
20:54 a2_ iptables -L was empty?
20:54 JoeJulian I tried every trace I could think of.
20:54 JoeJulian yes
20:54 a2_ what was glusterd doing? is there a gdb backtrace of what it was busy doing / not doing?
20:55 a2_ getting a 'thread apply all bt full' is of extreme help in such "hangs"
20:56 JoeJulian Added the backtrace
20:57 JoeJulian I think I have a statedump too if that's of any use
20:59 RicardoSSP joined #gluster
20:59 RicardoSSP joined #gluster
20:59 JoeJulian full, huh... never used that yet...
21:00 a2_ yeah, thread apply all bt full is exceptionally useful when _not_ compiled with -O2, but very useful nonetheless even with -O2
21:00 JoeJulian heh
21:04 a2_ JoeJulian, i see the problem..
21:05 JoeJulian Oh, good.
21:05 GomoX Last Q, any tips assuming my Gluster filesystem will be built on top of a cloud environment? (i.e virtualized bricks or something like that)
21:05 GomoX Looking at you, @glusterbot
21:05 GomoX :)
21:05 JoeJulian I love and hate that... I've looked at that for over 2 hours. You look at it for a few seconds. :)
21:07 andreask joined #gluster
21:08 JoeJulian GomoX: https://wiki.ubuntu.com/UbuntuCloudDay​s/23032011/ScalingSharedStorageWebApps
21:08 glusterbot <http://goo.gl/mvahcC> (at wiki.ubuntu.com)
21:10 JoeJulian lurbs: Tried to send you an email about your bug: 451 4.3.5 Server configuration problem
21:10 GomoX Looks interesting, thanks
21:11 a2_ JoeJulian, can you test this patch:
21:11 JoeJulian sure
21:11 a2_ http://review.gluster.org/5636
21:11 glusterbot Title: Gerrit Code Review (at review.gluster.org)
21:11 chirino joined #gluster
21:17 a2_ JoeJulian, please vote (tested +1) on the patch if it works for you
21:17 JoeJulian will do
21:17 semiosis @semiosis tutorial
21:17 glusterbot semiosis: http://goo.gl/6lcEX
21:18 semiosis JoeJulian: ^
21:18 JoeJulian semiosis: Yeah, that's the one I was looking for... ;)
21:19 chirino joined #gluster
21:22 MugginsM joined #gluster
21:26 _pol joined #gluster
21:31 B21956 left #gluster
21:41 Technicool joined #gluster
21:42 johnmark @channelstats
21:42 glusterbot johnmark: On #gluster there have been 169532 messages, containing 7186043 characters, 1200274 words, 4805 smileys, and 638 frowns; 1057 of those messages were ACTIONs. There have been 64889 joins, 2026 parts, 62882 quits, 21 kicks, 164 mode changes, and 7 topic changes. There are currently 190 users and the channel has peaked at 226 users.
21:47 B21956 joined #gluster
21:49 chirino joined #gluster
21:59 chirino joined #gluster
22:00 jag3773 joined #gluster
22:06 lurbs JoeJulian: Hey, thanks again for looking into that bug.  I saw the bug report (and patch!) this morning.
22:06 JoeJulian You're welcome.
22:06 jbrooks joined #gluster
22:07 JoeJulian It caught my interest...
22:07 lurbs I was one of the lucky ones, apparently. :)
22:07 JoeJulian Heh
22:07 JoeJulian You're using ubuntu iirc, right?
22:08 lurbs Yep.  3.4.0 from Semiosis' PPA on 12.04 LTS.
22:08 JoeJulian @meh
22:08 glusterbot JoeJulian: I'm not happy about it either
22:08 semiosis whats the problem?
22:08 lurbs semiosis: https://bugzilla.redhat.com/show_bug.cgi?id=997576
22:08 glusterbot <http://goo.gl/D5yfwL> (at bugzilla.redhat.com)
22:08 glusterbot Bug 997576: high, unspecified, ---, kparthas, NEW , glusterd becomes unresponsive when acting as mountbroker
22:08 JoeJulian Then you'll have to build your own test. I would have pointed you to packages on koji if you were using rpms. :D
22:09 chirino joined #gluster
22:10 semiosis can you give me an executive summary?  are you saying localhost mount doesn't work?
22:10 JoeJulian It's a geo-replication thing
22:11 a2_ it's a deadlock in geo-replication's mount broker functionality
22:11 JoeJulian Using non-root users for geo-replication
22:11 semiosis ohh
22:14 JoeJulian man... it's been a long time since I've looked at the spec. It's gotten very complex
22:25 chirino joined #gluster
22:27 JoeJulian lurbs: That patch works for me.
22:27 GomoX Is replace-brick the good approach for resizing a gluster fs without increasing the number of nodes?
22:27 lurbs JoeJulian: Excellent, I'll give it a go when I manage to find the time.
22:28 semiosis GomoX: if you use replication, just take a brick out of service (by killing its glusterfsd process) and expand the underlying storage
22:28 semiosis GomoX: that's what I do
22:31 GomoX I am planning a cloud deployment where resizing volumes might not be possible
22:34 semiosis s/expand/replace/
22:34 glusterbot What semiosis meant to say was: GomoX: if you use replication, just take a brick out of service (by killing its glusterfsd process) and replace the underlying storage
22:34 chirino joined #gluster
22:35 GomoX Yeah, but I have to deal with the copying of the data myself because I can't really resize the volume in the cloud
22:35 GomoX I was wondering if Gluster could do that for me
22:36 semiosis sure, replace-brick will migrate data.  or you can let replication heal the data
22:36 semiosis you should try both approaches, see which works best for you
22:38 GomoX Actually your very talk on Ubuntu IRC says all bricks should have the same size
22:38 semiosis as a best practice, to keep things manageable, bricks should be the same size
22:38 semiosis however it's not a requirement
22:39 GomoX Oh OK
22:39 GomoX So a workable resize procedure would be to replace the bricks one by one with larger ones
22:39 GomoX Right?
22:39 semiosis yes that has worked well for me over the years
22:40 semiosis i even used that same strategy to replace ext4 with xfs on all my bricks
22:40 semiosis zero downtime :)
22:40 GomoX Sounds great
22:43 semiosis bbl
22:44 Technicool joined #gluster
22:50 chirino joined #gluster
22:59 chirino joined #gluster
23:00 LaurentPointCa joined #gluster
23:01 matthewh joined #gluster
23:02 matthewh Hi, I'm new to gluster and am having real problems setting up gluster 3.4 on ubuntu.  Either the documentation is plain wrong or it's completely broken.
23:03 LaurentPointCa Good day everyone! Following some problems, one brick was unavailable for some time, which lead to a lot of files on the other replica node that needs to be healed. Now, when I try to run any "volume heal gv0 info" command, the command itself times out after 2 minutes, and the cluster lock times out after 10. Is there a way to heal this volume even if the heal info command is impossible to view because it takes too long to pr
23:04 matthewh I have 4 nodes.  I can peer probe.  Peer status shows 3 peers.  From the other nodes, peer probe also shows 3 nodes (excluding itself).  The problem is that volume create always fails.  Even if I leave out gluster1 from the volume create, the log file is showing that gluster1 is not in the peer in cluster state.
23:05 LaurentPointCa matthewh: What is the error message you see when you try to create the volume?
23:05 matthewh Is that a bug in 3.4?
23:06 matthewh ok, so I have 4 nodes and one brick on each node.  The eror I am seeing in etc-glusterfs-glusterd.vol.log is The error I'm getting is that the host "gluster1" is
23:07 matthewh [2013-08-15 22:52:49.464416] E [glusterd-volume-ops.c:780:glu​sterd_op_stage_create_volume] 0-management: Host gluster1 is not in 'Peer in Cluster' state
23:07 matthewh [2013-08-15 22:52:49.464431] E [glusterd-syncop.c:767:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host gluster1 is not in 'Peer in Cluster' state
23:08 matthewh Let me do that again... to make sure the options match up with the error
23:08 matthewh Command line is:  gluster volume create gv0 transport tcp gluster2:/export/brick1 gluster4:/export/brick1
23:08 matthewh volume create: gv0: failed
23:09 chirino joined #gluster
23:09 LaurentPointCa matthewh: what if you try to add the bricks one by one? Are all 4 producing that error?
23:10 matthewh I'll try that
23:10 LaurentPointCa matthewh: Also, in peer status, do you see "Peer in cluster" or "Accepted peer request" ?
23:11 matthewh I'm actually wanting a 2 replica distributed setup using all the bricks I have available (which is 4).
23:11 LaurentPointCa then I think you should have "replica 2" as part of your volume create command
23:11 matthewh I'm seeing "Peer in cluster" for every other peer (excluding the one I'm running the command from).  But if you do go to another host and run the command you see every other peer
23:12 matthewh This is actually what I want to do.  gluster volume create gv0 replica 4 transport tcp gluster1:/export/brick1 gluster2:/export/brick1 gluster3:/export/brick1 gluster4:/export/brick1
23:13 matthewh oops, replica 2 it should be
23:14 matthewh The error is "volume create: gv0: failed: Host gluster1 is not in 'Peer in Cluster' state" when run from gluster1
23:15 matthewh But peer status from another node shows:  Hostname: gluster1
23:15 matthewh Port: 24007
23:15 matthewh Uuid: 09a3bacb-558d-4257-8a85-ca8b56e219f2
23:15 matthewh State: Peer in Cluster (Connected)
23:19 chirino joined #gluster
23:20 awheeler joined #gluster
23:21 LaurentPointCa matthewh: What if you try to create only the first replica set with nodes 1 and 2? Or if that fails, then create one with 3-4? Does that work?
23:21 matthewh Just tried that Laurent.  The error I get is a very helpful: volume create: gv0: failed
23:23 matthewh By the way, the documentation is a bit confusing.  What I've done is create a freshly mounted partition at /export/brick1 and that's what I'm using in the volume create line.  But in other places I see examples of directories created below brick1 and that used as the volume create line.  Which is the correct way? does it matter?
23:24 LaurentPointCa The way you did it is correct, partitions mounted to /export/brickN and then using that mount point as the gluster volume create command
23:29 matthewh Oddly enough, even this fails.  gluster volume create gv0 transport tcp gluster2:/export/brick1
23:29 matthewh from gluster1.
23:30 matthewh I should point out that these gluster volumes are running on EC2.
23:30 matthewh So each one is behind a NAT of some sort
23:30 matthewh I've opened up TCP/UDP ports 111 + TCP 2407-24012
23:31 chirino joined #gluster
23:38 fidevo joined #gluster
23:41 chirino joined #gluster
23:41 awheeler joined #gluster
23:53 tzero joined #gluster
23:54 chirino joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary