Camelia, the Perl 6 bug

IRC log for #gluster, 2013-05-08

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 jurrien joined #gluster
00:06 vpshastry joined #gluster
00:18 fidevo joined #gluster
00:20 stoile_ joined #gluster
00:20 johnmark_ joined #gluster
00:21 frakt joined #gluster
00:21 mynameisbruce joined #gluster
00:21 ndevos|recovery joined #gluster
01:02 yinyin joined #gluster
01:09 inevity joined #gluster
01:09 baul joined #gluster
01:17 fidevo joined #gluster
01:19 kevein joined #gluster
01:24 premera joined #gluster
01:25 Shdwdrgn ah I see I finally got reconnected
01:25 Shdwdrgn seem to be having issues with ipv6 tonight
01:36 Shdwdrgn damn, no wonder nobody's talking... wrong channel!
01:41 yinyin joined #gluster
01:58 bharata joined #gluster
02:35 bulde joined #gluster
02:40 yinyin joined #gluster
02:43 vshankar joined #gluster
02:53 lalatenduM joined #gluster
03:30 fidevo joined #gluster
03:32 mohankumar__ joined #gluster
03:41 saurabh joined #gluster
04:20 shanks joined #gluster
04:21 montyz1 joined #gluster
04:24 yinyin joined #gluster
04:25 pithagorians joined #gluster
04:25 shylesh joined #gluster
04:43 bala1 joined #gluster
04:47 vpshastry joined #gluster
04:47 vpshastry left #gluster
04:54 montyz1 joined #gluster
04:56 bulde joined #gluster
04:58 glusterbot New news from newglusterbugs: [Bug 960818] Installing glusterfs rpms on a pristine f19 system throws "error reading information on service glusterfsd". <http://goo.gl/b7ZLa>
05:03 aravindavk joined #gluster
05:18 vpshastry joined #gluster
05:25 Shdwdrgn joined #gluster
05:25 badone joined #gluster
05:26 lalatenduM joined #gluster
05:27 bala1 joined #gluster
05:27 bulde joined #gluster
05:27 lala_ joined #gluster
05:32 samppah @latest
05:32 glusterbot samppah: The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
05:43 deepakcs joined #gluster
05:43 bala1 joined #gluster
05:50 rastar joined #gluster
06:05 bulde1 joined #gluster
06:20 jtux joined #gluster
06:48 ekuric joined #gluster
06:58 ngoswami joined #gluster
07:01 ctria joined #gluster
07:03 ndevos joined #gluster
07:05 lkthomas joined #gluster
07:05 lkthomas hey guy
07:05 lkthomas can I form a local disk to disk distributed replication volume ?
07:06 lkthomas I know gluster could deal with network volume but I want a simple solution on local disk to disk replication
07:16 pithagorians joined #gluster
07:20 samppah lkthomas: you mean you want to replicate between local disks.. ie. use glusterfs to handle replication instead of raid?
07:20 rotbeard joined #gluster
07:24 ricky-ticky joined #gluster
07:24 dobber_ joined #gluster
07:31 hchiramm__ joined #gluster
07:37 SteveCooling Is anyone in here good with the GlusterFS Geo-replication? I'm having my replication stop and restart with "argument list too long" errors, and I'm unable to find any information on whether GlusterFS takes care of this in any way.
07:42 lkthomas samppah: yes
07:42 lkthomas I have another question, how to preserve existing data on disk if I want to add gluster ?
07:43 lkthomas should I just add a directory as brick 1 then add brick 2 as replicate volume ?
07:43 NuxRo lkthomas: afaik the directories need to be emtpy
07:43 lkthomas errr
07:43 NuxRo so you need to start with emtpy dirs, create the volume, mount it then copy over the data
07:44 lkthomas anyone could confirm this as well ?
07:51 pithagorians hi. where can i find more  details about how glusterfs failover is done?
07:51 pithagorians i had a situation - 2 nodes in replica mode, the second one had a bad hdd and the entire cluster didn't work
07:53 lkthomas define didn't work
07:54 NuxRo pithagorians: i believe unless you remove that HDD, gluster will keep trying to use it hence causing problems, at least until some timeouts are reached. for how long did the cluster not work?
07:55 pithagorians NuxRo: let's say 2 h
07:55 pithagorians first the support said it's raid controller
07:55 pithagorians they changed it
07:55 pithagorians after that they understood that it's a disk
07:55 pithagorians part of the raid massive
07:56 pithagorians on which the partition used by bricks works
07:56 NuxRo that's a long time
07:56 NuxRo have you tried to send this to the mailing list?
07:56 lkthomas NuxRo: how does gluster define a brick is dead ?
07:56 pithagorians NuxRo: why time matters ?
07:56 NuxRo lkthomas: no idea
07:57 pithagorians is it related to timeout ?
07:57 NuxRo pithagorians: well, i would expect gluster to honour some timeouts
07:57 NuxRo but 2h sounds like a long time
07:57 pithagorians ah
07:57 pithagorians that's right
07:57 lkthomas pithagorians: I bet it's possible for you to remove a host from gluster
07:57 NuxRo lkthomas: i'm not that intimate with gluster, sadly
07:57 lkthomas heh
07:58 lkthomas that's something you have to test before put into production yeah ?
07:58 pithagorians lkthomas: it's not failover already
07:58 NuxRo so your setup is not replicated?
07:58 pithagorians when i have to take out a node member from a cluster
07:58 pithagorians yes, it's replica 2
07:58 lkthomas pithagorians: can you do manual failover ?
07:58 pithagorians i guess so
07:58 jikz joined #gluster
07:58 lkthomas shutdown the failed server
07:59 pithagorians yes
07:59 lkthomas native client should pick up the change
07:59 pithagorians only like this
07:59 lkthomas what's other option ?!
07:59 pithagorians or change the firewall
07:59 lkthomas if it's replica, then nothing need to worry
07:59 pithagorians to block all communication with first node
07:59 NuxRo yup, somehow cut it out of the setup, repair the problem, add it back
07:59 lkthomas I am not sure how is your setup works, but if it's linked by heartbeat, it might fence itself
07:59 pithagorians the problem is entire communication with glusterfs was broken
08:00 NuxRo pithagorians: in this time, have you tried taking down the problematic server and see if communication is restored with gluster?
08:00 pithagorians and glusterfs failover didn't count the second node as dead
08:00 pithagorians it was restored
08:01 lkthomas brb
08:01 pithagorians when took it down
08:01 pithagorians but again
08:01 NuxRo so gluster started to work again after you took out the bad server
08:01 pithagorians it's not failover
08:01 NuxRo true, it's not what I'd expect either
08:01 pithagorians or better say - not the failover i expected
08:01 NuxRo I would take this on the mailing list and/or the redhat bugzilla
08:02 pithagorians let me look into
08:02 pithagorians do i have to register?
08:02 pithagorians i never reported into mail lists
08:03 NuxRo well, you need to register here http://www.gluster.org/interact/mailinglists/
08:03 glusterbot Title: Mailing Lists | Gluster Community Website (at www.gluster.org)
08:03 NuxRo or you can submit a bug in bugzilla here https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS (also requires registration)
08:04 glusterbot <http://goo.gl/UUuCq> (at bugzilla.redhat.com)
08:04 pithagorians thx
08:05 bulde joined #gluster
08:07 hchiramm__ joined #gluster
08:08 tshm joined #gluster
08:10 ollivera joined #gluster
08:10 tshm Hi all! I have a weird thing occurring on a Gluster test setup here, with three 2-way replicates (2+2+2 bricks). getfattr -n trusted.glusterfs.pathinfo returns different hashvalues for the replicates, depending on which subfolder you're in. Shouldn't the same hash ranges apply to all subdirectories, or did I misunderstand something?
08:11 tshm for one directory:
08:11 tshm trusted.glusterfs.pathinfo="(dht-layout (replicate-0 1431655765 2863311529) (replicate-1 2863311530 4294967295) (replicate-2 0 1431655764))"
08:11 tshm for another:
08:11 tshm trusted.glusterfs.pathinfo="(dht-layout (replicate-0 0 1431655764) (replicate-1 1431655765 2863311529) (replicate-2 2863311530 4294967295))"
08:16 lkthomas interesting
08:16 lkthomas when I create replica volume with data in brick 1 dir, it does not show
08:22 yinyin_ joined #gluster
08:24 spider_fingers joined #gluster
08:25 NuxRo lkthomas: so you just confirmed I was right? :)
08:28 glusterbot New news from newglusterbugs: [Bug 960867] failover doesn't work when a hdd part of hardware raid massive becomes broken <http://goo.gl/6usIi>
08:32 tshm lkthomas: What is it that doesn't show?
08:33 tshm Does the directory not show at all?
08:33 andreask joined #gluster
08:33 lkthomas old data before join as brick didn't appear
08:33 lkthomas NuxRo: brick creation works
08:33 lkthomas but old data wouldn't show
08:33 tshm oh
08:33 lkthomas rebalance doesn't work
08:34 tshm Did you force rebalance?
08:34 lkthomas force? I didn't know there is a force option ?
08:35 tshm Rebalancing is not done automatically, when just adding a brick.
08:35 tshm http://gluster.org/community/documentation/​index.php/Gluster_3.2:_Rebalancing_Volumes
08:35 glusterbot <http://goo.gl/tWGZe> (at gluster.org)
08:35 lkthomas but I was talking about data before gluster being created
08:36 lkthomas there is no force option
08:36 tshm But you did run the rebalance command?
08:37 tshm That's what I meant by "force", maybe I was a bit unclear
08:37 tshm I'm very new to Gluster myself, so I'm afraid I can't help you more than that...
08:38 lkthomas ...
08:38 lkthomas I did rebalance but it say nothing neeed to balance
08:38 lkthomas nevermind
08:38 lkthomas I use rsync to resync all data
08:40 tshm okay
08:48 SteveCooling Guys, how do I find out if a particular fix is in my running version of GlusterFS? (3.3.1-1 RPM packages). This is the patch: http://review.gluster.org/#/c/3917/
08:48 glusterbot Title: Gerrit Code Review (at review.gluster.org)
08:48 NuxRo SteveCooling: rpm -q --changelog ?
08:50 manik joined #gluster
08:52 NuxRo or just check /usr/libexec/glusterfs/pyt​hon/syncdaemon/resource.py
08:58 SteveCooling NuxRo: the fix was committed after 3.3.1 was released, and cannot find any mention in changelog
08:59 SteveCooling guess i'll be waiting like a child for 3.3.2
09:03 vimal joined #gluster
09:05 duerF joined #gluster
09:05 rastar1 joined #gluster
09:06 Norky joined #gluster
09:06 chlunde_ joined #gluster
09:07 lhawthor_ joined #gluster
09:08 NuxRo SteveCooling: you could try one of the qa builds of 3.3.2, http://bits.gluster.org/pub/glu​ster/glusterfs/3.3.2qa2/x86_64/
09:08 glusterbot <http://goo.gl/GhzCd> (at bits.gluster.org)
09:09 hagarth__ joined #gluster
09:10 SteveCooling NuxRo: 3.3.2 doesn't have the fix either. extracted the files from the glusterfs-geo-replication-3.3.2qa1-1 package.
09:11 NuxRo right
09:11 chirino joined #gluster
09:11 primusinterpare1 joined #gluster
09:11 ehg_ joined #gluster
09:12 SteveCooling no --files-from in that file
09:12 SteveCooling like it should be accarding to http://review.gluster.org/#/c/3917/1/xlator​s/features/marker/utils/syncdaemon/resource.py
09:12 glusterbot <http://goo.gl/vnLvi> (at review.gluster.org)
09:12 nueces_ joined #gluster
09:12 SteveCooling s/accarding/according/
09:13 glusterbot SteveCooling: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
09:14 NuxRo lol
09:14 SteveCooling i guess i might be able to work around this by doing a manual sync first?
09:15 georgeh|workstat joined #gluster
09:15 SteveCooling worth a shot maybe
09:16 ollivera Is there any way to get the hashing range assigned to a specific brick?
09:18 zykure joined #gluster
09:24 ekuric joined #gluster
09:26 VSpike joined #gluster
09:27 glusterbot New news from resolvedglusterbugs: [Bug 826512] [FEAT] geo-replication checkpoint support <http://goo.gl/O6N3f> || [Bug 847839] [FEAT] Distributed geo-replication <http://goo.gl/l4Gw2>
09:27 vpshastry joined #gluster
09:28 bala1 joined #gluster
09:29 glusterbot New news from newglusterbugs: [Bug 960889] G4S: PUT/GET/HEAD/DELETE request where file and containers are named in UTF-8 format fails <http://goo.gl/DIoKl>
09:32 lkthomas ok, I have a question on volume size, if brick 1 = 2TB, brick 2 = 3TB, what's the end replica volume size ?
09:36 NuxRo lkthomas: 2 TB ?
09:36 NuxRo i believe the size is that of the smallest brick
09:36 lkthomas really ?
09:36 NuxRo much like in "raid"
09:36 lkthomas anyone could confirm that ?
09:36 NuxRo in "replica" mode, of course
09:36 NuxRo lkthomas: you can confirm it :)
09:37 lkthomas haha, well
09:39 lkthomas I think setting quota is more reliable
09:39 lkthomas instead of using hit or miss approch
09:40 NuxRo if you are going to use quota, better mount your xfs filesystem with allocsize=4096
09:40 NuxRo it was recommended to me by one of the devs
09:41 lkthomas why
09:41 lkthomas why not set at glister level
09:47 NuxRo this is for the underlying filesystem, i understand it helps if you want to use gluster quota
09:47 NuxRo is this what you asked?
09:47 lkthomas yep
09:47 lkthomas I think it's ok to set it
09:47 lkthomas as long as I am not losing data on it
09:53 lh joined #gluster
09:53 lh joined #gluster
09:59 glusterbot New news from newglusterbugs: [Bug 959069] A single brick down of a dist-rep volume results in geo-rep session "faulty" <http://goo.gl/eaoet> || [Bug 960913] do not do root squashing for the clients mounted in the storage pool <http://goo.gl/iUogO>
10:15 bala1 joined #gluster
10:15 rastar joined #gluster
10:21 vpshastry joined #gluster
10:22 sgowda joined #gluster
10:25 manik joined #gluster
10:30 thekev joined #gluster
10:30 manik joined #gluster
10:40 mohankumar joined #gluster
10:50 vpshastry joined #gluster
10:52 VSpike I just did ifdown eth0 && ifup eth0 on one of my gluster servers, and two clients have lost their mountpoints
10:52 VSpike is that expected?
10:52 krishna_ joined #gluster
10:55 VSpike Oh, looks like it is http://www.gluster.org/community/documentation​/index.php/Transport_endpoint_is_not_connected
10:55 glusterbot <http://goo.gl/eY1k0> (at www.gluster.org)
10:56 jtux joined #gluster
10:57 glusterbot New news from resolvedglusterbugs: [Bug 960913] do not do root squashing for the clients mounted in the storage pool <http://goo.gl/iUogO>
10:59 glusterbot New news from newglusterbugs: [Bug 960944] G4S: DELETE on container that does not exist returns 503 instead of 404 <http://goo.gl/9jf93>
10:59 VSpike So would you actually get better HA with NFS and ucarp?
11:00 VSpike Seems odd that a clustered replicating filesystem client can't survive a momentary outage on one server in the pool
11:01 lkthomas this is stupid, gluster does not have ubuntu raring package
11:01 lkthomas for 3.3
11:01 lkthomas I am compiling my own binary now
11:06 NuxRo lkthomas: you mean to say, ubuntu doesn't have glusterfs package :)
11:06 NuxRo semiosis maintains a ppa for ubuntu afaik, but might be only for lts releases (makes sense)
11:07 lkthomas yeah
11:08 lkthomas I understand the logic behind but I don't like the fact
11:15 vpshastry joined #gluster
11:16 lkthomas how could I enable extend attr on glusterfs ?
11:21 lkthomas nevermind
11:21 lkthomas I think I fix it by putting mount option acl
11:24 krishna_ joined #gluster
11:33 jclift joined #gluster
11:34 zykure joined #gluster
11:43 tshm I've got another question: I'm trying to trigger a self-heal on a replicate volume (v3.1.3), running the method described at http://gluster.org/community/documentation/index.p​hp/Gluster_3.1:_Triggering_Self-Heal_on_Replicate
11:43 glusterbot <http://goo.gl/pMJrJ> (at gluster.org)
11:43 tshm but it takes AGES for anything to show up!
11:43 tshm Also tried only the find part, i.e. 'find /storage -print0'  and it never finishes, although it's a really tiny volume (and nothing should be in need of healing as of now). Also no printouts whatsoever. How come?
11:51 nickw joined #gluster
11:59 glusterbot New news from newglusterbugs: [Bug 960961] G4S: Nested directories and files are not listed with GET calls which results in failure of some swift upstream functional test cases <http://goo.gl/h8R2f>
12:04 vpshastry2 joined #gluster
12:05 spider_fingers joined #gluster
12:15 satheesh joined #gluster
12:17 H__ What can cause this -> E [posix.c:1755:posix_create] 0-vol01-posix: setting xattrs on /bricks/f/somefile failed (Operation not supported)
12:29 krishna__ joined #gluster
12:30 smellis joined #gluster
12:35 klaxa|web joined #gluster
12:36 klaxa|web hi, glusterfs 3.2, split-brain, i just delete the file i don't want from one of the bricks and open the file in the glusterfs mount? that should initialize self-heal, right?
12:36 klaxa|web or rather, i just delete the file i don't want anymore, right?
12:38 H__ in 3.2 that should be enough afaik yes (note that 3.3+ has to be handled different)
12:40 klaxa|web alright, thakns
12:40 klaxa|web *thanks
12:47 bet_ joined #gluster
12:54 kkeithley In 3.2.x you need to stat the file from a client to initiate self-heal. In 3.3.x the glustershd (gluster self heal daemon) will do it automatically, but you can stat it to make it heal sooner
13:02 dmojoryder Isn't there a way when mounting using the native client to specify multiple vol servers? I know I have seen it but I don't recall the specific syntax
13:03 kkeithley you only need to specify one vol server. The client will get all the vol servers from the one you specify
13:03 kkeithley and mount only groks one server anyway AFAIK
13:04 dmojoryder kkeithley: I am aware of that, but if the single server is down at time of mount...
13:04 kkeithley If it can't reach the one, you'll know about it pdq
13:04 kkeithley one successful mount is all it takes. The other servers are persisted by glusterfs after that
13:05 kkeithley I think....
13:05 kkeithley hmmm
13:05 dmojoryder Redhat gluster docs have this: backupvolfile-server=server name - name of  the backup volfile server to mount the client. If this option is added  while mounting fuse client, when the first volfile server fails, then  the server specified in backupvolfile-server option is used as volfile server to mount the client.
13:06 hagarth joined #gluster
13:06 dmojoryder I will try
13:06 brunoleon_ joined #gluster
13:17 spider_fingers joined #gluster
13:19 aliguori joined #gluster
13:29 glusterbot New news from newglusterbugs: [Bug 960985] G4S: HEAD Request for a file does not reflect correct content length of file (a constant 108 is shown) <http://goo.gl/sfaly>
13:36 bugs_ joined #gluster
13:36 dialt0ne joined #gluster
13:37 H__ What can cause this -> E [posix.c:1755:posix_create] 0-vol01-posix: setting xattrs on /bricks/f/somefile failed (Operation not supported)
13:38 H__ getfattr -d -e hex -m . /bricks/f/somefile show xattr's just fine
13:38 montyz joined #gluster
13:43 plarsen joined #gluster
13:52 manik1 joined #gluster
13:55 sjoeboo joined #gluster
13:56 spider_fingers joined #gluster
14:00 glusterbot New news from newglusterbugs: [Bug 960999] G4S: Ranged GET requests failes to respond properly to some byte range requests <http://goo.gl/vFZwj>
14:00 spider_fingers1 joined #gluster
14:03 vpshastry joined #gluster
14:10 semiosis lkthomas: meh
14:11 semiosis helpful feedback, even constructive criticism, is welcome.  calling things stupid is neither.
14:12 semiosis i uploaded 3.3.1 for raring to the PPA, it should be avalable in an hour or so
14:15 dustint joined #gluster
14:15 semiosis VSpike: fuse mounts connect directly to all bricks after retrieving volume information from the ,,(mount server)
14:15 glusterbot VSpike: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
14:22 vpshastry1 joined #gluster
14:24 fabien joined #gluster
14:33 daMaestro joined #gluster
14:45 chlunde joined #gluster
14:53 chirino joined #gluster
14:59 failshell joined #gluster
14:59 failshell im concerned about disaster recovery with our gluster clusters
15:00 failshell how do you backup the data in it?
15:01 NuxRo failshell: geo-replication might be one way, or just copy the files to some other place
15:02 failshell our VMs are backed up by Veem. im gonna have to test a restore
15:03 rastar joined #gluster
15:03 jthorne joined #gluster
15:05 aravindavk joined #gluster
15:07 morse joined #gluster
15:08 montyz1 joined #gluster
15:21 spider_fingers1 left #gluster
15:23 andrewjs1edge joined #gluster
15:29 vpshastry joined #gluster
15:49 jag3773 joined #gluster
15:53 VSpike semiosis: so why would down/up on the interface on one server cause the mount to become "transport endpoint not connected" on the two clients which happen to mention it in their /etc/fstab?
15:54 semiosis VSpike: possibly because those clients were never connected to the other bricks in the first place?
15:55 semiosis truncate your client log, remount the client, check the log for connectivity messages
16:02 plarsen joined #gluster
16:05 lkthomas semiosis: LOL
16:05 lkthomas semiosis: you know I am talking about you!
16:05 VSpike semiosis: hmm, good point.. thanks
16:05 lkthomas semiosis: thanks anyway even I compile my own binary
16:08 tryggvil_ joined #gluster
16:15 lkthomas so, if I use replica 2, does read performance will be double ?
16:16 semiosis something like that
16:16 semiosis clients will read from whichever replica answers first on lookup
16:16 semiosis per file
16:17 semiosis so with many clients, and all else being equal, reads should be balanced between replicas
16:19 lkthomas so session is define by client
16:19 semiosis idk what you mean by "session"
16:19 semiosis thats not a glusterfs term afaik
16:20 lkthomas ok, as long as a ocnnection from same IP session without disconnect which is keep alive, that's one session
16:20 lkthomas so on replica 2, only one replica disk will resposne to my request
16:20 semiosis talking about fuse clients, btw...
16:20 semiosis the client is connected to all bricks (all replicas)
16:21 semiosis before a file is opened, the client polls the replicas & whichever answers first serves reads for that file to the client
16:21 lkthomas ok
16:22 lkthomas I think I need to do some performance tuning
16:23 lkthomas I am running replica 2 disk, writing file into replica volume cause CPU just up 100%
16:23 lkthomas maybe the CPU just slow :P
16:23 lkthomas http://ark.intel.com/products/33079
16:23 glusterbot Title: ARK | Intel Xeon Processor E5405 (12M Cache, 2.00 GHz, 1333 MHz FSB) (at ark.intel.com)
16:23 lkthomas what do you expect from 6 years old CPU could do ?
16:32 cyberbootje1 hi, what is the latest stable glusterfs version?
16:32 cyberbootje1 not the one on the website but personal opinions :-0
16:32 cyberbootje1 :-)
16:34 thomasle_ joined #gluster
16:42 bulde joined #gluster
16:45 Mo__ joined #gluster
16:47 Supermathie semiosis: PONG
17:00 cfeller_ joined #gluster
17:07 bennyturns joined #gluster
17:10 zaitcev joined #gluster
17:11 \_pol joined #gluster
17:12 failshell has anyone ever experienced data loss with gluster?
17:14 thomaslee joined #gluster
17:16 portante joined #gluster
17:25 shylesh joined #gluster
17:28 soukihei joined #gluster
17:34 lpabon joined #gluster
17:36 \_pol_ joined #gluster
17:44 \_pol joined #gluster
17:54 failshell let's say i'm storing a file of 10GB on a volume with 2 replicas. does that mean the entire file will be stored on 2 blocks? or is it going to split it around?
17:55 rwheeler joined #gluster
17:56 Supermathie failshell: distributed volume -> one block per file. (with replica 2, 2 blocks per file)
17:56 Supermathie s/block/brick
17:56 failshell so each replica is store entirely on each brick
17:56 failshell ok
18:03 failshell so its normal that the data is not roughly the same on all the bricks?
18:09 Supermathie yep
18:10 bulde joined #gluster
18:11 failshell can i have bricks with different storage size?
18:12 H__ yes, with latest qa2 release on 3.3-release branch where the disk.minfree works again
18:23 failshell so in theory, if i had a disaster, all i need to restore the data, is to get the latest snapshots for the VM, and I could rebuild the data set, albeit manually
18:24 failshell cause each file is at least on 2 bricks
18:31 glusterbot New news from newglusterbugs: [Bug 958781] KVM guest I/O errors with xfs backed gluster volumes <http://goo.gl/4Goa9>
18:33 Supermathie failshell: From what I understand, the volume self-healing daemon should take care of rebuilding the data set. But I don't know if reverting to a snapshot is appropriate for recovery.
18:33 Supermathie Good question though
18:35 failshell i can easily mount the volumes and rsync the data to another VM
18:35 failshell but at some point, considering i have double the data in gluster, + a backup
18:35 failshell storage wise, its getting expensive
18:36 Supermathie Correct, redundancy is expensive :)
18:37 failshell backing up a distributed solution, i wonder if that's too much
18:37 andrei_ joined #gluster
18:37 andrei_ hello guys
18:37 Supermathie failshell: C'mon man, RAID (or in this case mirrors) is not backup. It's an availability solution.
18:38 failshell i know
18:38 andrei_ I was wondering if someone could help me with structuring two glusterfs servers which have different hardware and the number of disks
18:38 andrei_ i want to get as much flexibility from the storage so that I can expand it in the future
18:39 vpshastry joined #gluster
18:39 Supermathie andrei_: It's a trap!
18:40 andrei_ that doesn't sound reassuring )))
18:40 andrei_ why do you say so?
18:40 Supermathie andrei_: Is this a case of trying to plan too far ahead and overcomplicating things, or do you have an expansion plan already in mind :)
18:41 andrei_ ah, okay
18:41 andrei_ i do not currently have the exact figures in mind
18:41 andrei_ it's just an estimation
18:41 andrei_ however, I would like to be able to grow without redoing things
18:42 duerF joined #gluster
18:42 andrei_ at the moment i've got one pretty new server and one server which is 3 years old
18:42 andrei_ the new one has 8 disks + 1 spare
18:42 andrei_ they are 3TB each done in raid10
18:43 andrei_ the old server has just 8 disks with raid6
18:43 andrei_ i would like to create a redundant glusterfs cluster
18:43 manik joined #gluster
18:43 andrei_ but planning to have a 3rd server in about 6 months to add to the cluster
18:44 andrei_ could I achieve this with glusterfs?
18:45 piotrektt_ joined #gluster
18:49 dustint_ joined #gluster
18:49 Supermathie andrei_: with glusterfs you have to manually specify which bricks are replicated. (think RAID1)
18:50 andrei_ so, i guess if I create a two server setup in a replicated manner I would need to add not one, but two servers with my next storage upgrade
18:50 andrei_ is that correct?
18:50 andrei_ there is no raid5 equivalent setup with glusterfs at the moment, is there?
18:51 Supermathie andrei_: Correct & correct
18:51 Supermathie andrei_: (note that you mirror bricks, not servers. So you could do a RAID1E type config but that means rebuilding your volume)
18:53 andrei_ not really sure what you mean
18:53 Supermathie mirror s1b1 to s2b1, s2b2 to s3b1, s3b2 to s1b2, and so on
18:54 dustint_ joined #gluster
19:13 tech_ joined #gluster
19:14 andrei_ thanks
19:15 tech_ Hello, I am looking to set up a replicated gluster volume using NFS as the client. Most of the stuff I've read says that you need the gluster client to start self healing. Is this correct?
19:17 Supermathie tech_: You need the gluster *CLI* to tell gluster to start self-healing outside of its schedule.
19:21 tech_ How do I do that?
19:21 tech_ What I'm seeing uses the client
19:25 Supermathie tech_: 'gluster volume heal vol0 full'? That's the CLI.
19:29 tech_ I can run that on either server?
19:29 Supermathie tech_: the *client* is the FUSE daemon...
19:30 Supermathie tech_: Yeah, you can issue commands from any server.
19:30 tech_ thank you very much
19:30 Supermathie welcome
19:30 tech_ I was looking at this "find <gluster-mount> -noleaf -print0 | xargs --null stat >/dev/null 2>/var/log/gluster/<gluster-mount>-selfheal.log"
19:31 tech_ which looked like it had to be ran from one of the clients
19:31 Supermathie That'll trigger the healing as a side-effect of stat()ing the files, yeah. Which you may have needed to do pre... 3.2?
19:32 tech_ Ah ok, it was in the 3.2 documentation for triggering a self heal
19:34 _pol joined #gluster
19:34 _pol joined #gluster
19:37 elyograg joined #gluster
19:39 elyograg We are finally getting around to trying to really use glusterfs.  We have nfs mounted from linux clients with no trouble, but now we're trying it from Solaris.  Solaris can mount the volume, but not subdirectories.
19:39 elyograg I went through the things listed on this post with my test volume but it doesn't seem to be working: http://www.gluster.org/pipermail/g​luster-users/2012-July/033867.html
19:39 glusterbot <http://goo.gl/8VJGk> (at www.gluster.org)
19:40 elyograg or maybe it's working now.  cow-orker says it works. :)
19:40 Supermathie Oh wow those directions are wacky.
19:41 Supermathie I'd be curious to see a pcap from Linux & Solaris and compare them
19:45 elyograg well, it works now, so I'm not going to look into it further unless it breaks.
19:46 Supermathie ... at 3AM :)
19:47 elyograg apparently now -o proto=tcp is apparently required to mount from linux.
19:47 semiosis ~nfs | elyograg
19:47 glusterbot elyograg: To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
19:56 a2_ our UDP rpc implementation uses glibc, and is not sophisticated
19:56 a2_ tcp based mounting supports NFS subdir mounts
19:56 Supermathie elyograg: more specifically, you need vers=3 if everything else is configured right. For some reason (I don't know for sure) looks like client systems supporting nfsv4 go to port 2049 WITHOUT contacting portmapper. Derp?
19:56 a2_ solaris insists on UDP based mount
19:57 a2_ Supermathie, yes, nfs clients with v4 support go straight to 2049 without portmapper. with 054c1d7eb3782c35fc0f0ea3a5fd25337d080294 you don't need to specify vers=3
19:59 Supermathie a2_: I'm just wondering why the client does that - there's nothing (that I can find) in the spec to say to do so. And nfsv4 registered with portmapper
20:00 failshell im wondering, in /etc/fstab, i specify ro, but when i run mount, it displays rw. the FS is still read-only. but that's just weird and confusing
20:00 a2_ i think it was because nfsv4 got an iana assigned port number? (while v3 did not? maybe i'm wrong)
20:01 a2_ i think the 2049 port number made it to the spec, making portmapper kinda redundant.. again, not sure
20:02 a2_ here you go, rfc 3530: Historically, NFS version 2 and version 3 servers have resided on
20:02 a2_ port 2049.  The registered port 2049 [RFC3232] for the NFS protocol
20:02 a2_ should be the default configuration.  Using the registered port for
20:02 a2_ NFS services means the NFS client will not need to use the RPC
20:02 a2_ binding protocols as described in [RFC1833]; this will allow NFS to
20:02 a2_ transit firewalls.
20:02 semiosis a2_: careful glusterbot doesnt kick you for flooding ;)
20:02 Supermathie Yeah, that's what I was reading. SHOULD != MUST, and a client that gets a RST on port 2049 really ought to ask portmapper if he hasn't already.
20:03 a2_ this is a lower case "should", not "SHOULD"
20:04 Supermathie fair enough... still...
20:04 a2_ it clearly says the "NFS client will not need to use RPC binding protocols"
20:05 Supermathie ahhhhhhhhhh yes.
20:05 aliguori joined #gluster
20:05 Supermathie There's the rub.
20:06 Supermathie Still, Linux is making a bad assumption that the server is version 4. It should fall back if it doesn't get a response on 2049.
20:06 jbrooks joined #gluster
20:07 dialt0ne left #gluster
20:10 a2_ "doesn't get a response" == "retransmit". that's the RPC way
20:10 Supermathie Well, it did get a response, it got an RST.
20:10 a2_ RST is not an RPC response
20:10 a2_ an RPC response is only a well-crafted RPC reply header
20:11 a2_ i'm not arguing that it makes sense.. i'm only saying how things work
20:12 Supermathie From the RPC side, yeah. The client mount program should interpret "Connection refused" as "Hmmm, NFSv4 isn't running, I should contact portmapper for v1-3"
20:12 a2_ that means the client mount program understands the nuances of TCP
20:12 a2_ which is blasphemy
20:14 a2_ client mount program never even realizes that "connection refused" happens.. the connection refused is silently consumed by the rpc layer with a retransmit/reconnection attempt after a delay
20:14 Supermathie Nope:
20:14 Supermathie strace says: mount("fearless1:/gv0", "/mnt/tmp", "nfs", 0, "vers=4,addr=192.168.0.96,clienta"...) = -1 ECONNREFUSED (Connection refused)
20:15 a2_ that's.. odd, probably "special handling" of initial connection failure
20:15 a2_ you won't see that behavior for an already established session
20:16 Supermathie but if you want to debate semantics of RPC, you can start with answering why gluster can't respond to multiple RPC requests in a single TCP packet (ooooohhhhhhhhhh crossing the layers yeah)
20:16 Supermathie :D
20:17 a2_ no, gluster shouldn't care if multiple rpc requests came in the same packet or not
20:17 Supermathie shouldn't, I agree.
20:18 a2_ it does not even realize if multiple rpc requests arrived in the same IP packet or same rpc request was split across multiple packet or combinations of these.. we only treat it like a tcp stream and read header by header till boundaries
20:19 a2_ will need a state dump and core dump to see what's happening when those requests arrive
20:20 a2_ i can see how it could potentially mess up if those multiple rpc requests were in the same _udp_ packet
20:20 Supermathie a2_: I have an applicable 15GB corefile here...
20:20 a2_ but we don't do rpc over udp (well, we do in a very limited way for MOUNT, but that's not involved in  your scenario)
20:21 a2_ are there lot of pending call stacks?
20:21 a2_ ctx->this->pool->{all_stacks}
20:21 a2_ *this->ctx->pool->{all_stacks}
20:22 Supermathie will be able to examine it in a bit... and I may no longer have the right symbols. I can replicate it though.
20:22 a2_ make sure you compile with CFLAGS="-g -O0"
20:23 Supermathie a2_: yes, lots of pending RPC calls. This is my 'Reply submission failed' problem: https://bugzilla.redhat.com/show_bug.cgi?id=960141
20:23 glusterbot <http://goo.gl/RpzTG> (at bugzilla.redhat.com)
20:23 glusterbot Bug 960141: urgent, unspecified, ---, vraman, NEW , NFS no longer responds, get  "Reply submission failed" errors
20:33 Supermathie Oh god, I wonder if this is a TCP windowing problem...
20:34 a2_ do you know who sent the first RST? client or server?
20:34 Supermathie client sends RST after no response from the server after 10s
20:35 Supermathie Oh, and the RPC frames aren't being responded to for a while *before* errors start showing up in gluster logs
20:35 a2_ does the client send the 0-byte payload rpc header to "ping" the server for responsiveness?
20:37 Supermathie Ahhhhh wait NO, it's the server RSTing.
20:37 a2_ oh?
20:39 Supermathie I have a complete TCP connection... here... https://www.dropbox.com/s/41ejnrs​r9qnd8id/gluster-rpc-failure.pcap
20:39 glusterbot <http://goo.gl/6V0UU> (at www.dropbox.com)
20:41 _pol joined #gluster
20:42 a2_ there's a whole lot of retransmissions happening
20:42 _pol joined #gluster
20:44 Supermathie OK, this particular connection is a bit of a mess, wow... hmmm...
20:45 Supermathie have to get home, may take another crack at this later.
20:45 a2_ what's the time difference between packet number 4 and 6? i don't see timing info in the capture
20:45 a2_ and time difference between 4 and 19
20:45 Supermathie a2_: It's in there. 0.2s
20:45 Supermathie 4->19: 25.5s
20:47 a2_ are you using a floating IP?
20:49 failshell hmm how come df doesnt report the actual amount of data in a volume?
20:49 a2_ Supermathie, do you have a similar trace against knfsd?
20:52 genewitch joined #gluster
21:14 montyz1 joined #gluster
21:18 andreask joined #gluster
21:20 andrei_ does anyone have experience with gluster running on zfs?
21:21 theron joined #gluster
21:35 JoeJulian andrei_: Yes and no. The logically accurate answer to your question as stated is yes. There are people with experience running gluster on zfs. Unfortunately the no is that I don't think any of them hang out here.
21:38 semiosis andrei_: http://www.gluster.org/community/do​cumentation/index.php/GlusterOnZFS
21:38 glusterbot <http://goo.gl/BG4Bv> (at www.gluster.org)
21:50 andrei_ thanks!
22:07 fidevo joined #gluster
22:28 war|child joined #gluster
23:13 montyz joined #gluster
23:39 toddstansell we run gluster on zfs. seems to be working fine.  using zfsonlinux.org stuff, not fuse zfs.
23:43 _pol joined #gluster
23:44 _pol joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary