Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 y4m4 semiosis: well that is what CROS doesn't want
00:00 y4m4 :P
00:00 semiosis oh hm
00:00 CROS Hmm, well, the localhost I would imagine would "mostly" respond first, no?
00:00 semiosis CROS: you'd think so, unless maybe its disk is busy doing other things?
00:02 semiosis CROS: varnish cache is great for caching http responses, i have glusterfs -> apache -> varnish
00:02 semiosis aw
00:02 mattr01 semiosis: I was thinking of just using mod_cache, varnish much better?
00:02 CROS_ joined #gluster
00:02 CROS_ sorry, got booted
00:02 y4m4 ah funny thing is there is read-subvolume option still
00:02 semiosis CROS: varnish cache is great for caching http responses, i have glusterfs -> apache -> varnish
00:02 CROS_ varnish, eh?
00:03 semiosis y4m4: no cli support though :(
00:03 y4m4 well it can be fixed through client side --xlator-option mounting
00:03 semiosis ?!
00:03 semiosis is this a new feature?
00:03 mattr01 once I get the hadoop working out in jBoss it will connect using the API but to mostly server small images that will get uploaded and such .. I want to cache the images near httpd..
00:03 mattr01 so I guess varnish is an option
00:04 y4m4 --xlator-option=XLATOR-NAME.OPTION=VALUE
00:04 y4m4 Add/override an option for a translator in volume
00:04 y4m4 file with specified value
00:04 y4m4 this ^^
00:04 semiosis wow cool
00:04 y4m4 on both server1 and server2
00:04 CROS_ what's that for?
00:04 y4m4 may be it might work
00:04 * semiosis will test
00:05 y4m4 mount.glusterfs doesn't provide that as a proper option but individually 'glusterfs' command can be used
00:07 semiosis right
00:07 semiosis so not easily available from fstab
00:07 semiosis still curious to see if it works
00:07 y4m4 it would
00:13 Troy joined #gluster
00:21 CROS_ joined #gluster
00:24 y4m4 semiosis: it works
00:24 y4m4 CROS_: seems like there is solution for you to avoid local traffic
00:24 y4m4 :P
00:25 y4m4 CROS_: there might be issues, but its upto you if you want it
00:25 semiosis y4m4: --xlator-option=foo-replicate-​0.read-subvolume=foo-client-1
00:25 y4m4 semiosis: yeah
00:25 semiosis this doesnt seem to have any effect.  what am i doing wrong?
00:25 y4m4 it has
00:25 y4m4 its using foo-client-1 now
00:26 y4m4 read_child is set to '1'
00:26 y4m4 do another mount with 'foo-client-0'
00:26 semiosis ok
00:26 y4m4 you can see the read_child is set to '0'
00:26 y4m4 which is basically what should be
00:27 CROS_ hmmm
00:27 semiosis so i am mounting on one of the two servers, and whichever i set that to i get xfer over the network, never localhost
00:27 y4m4 hmm semiosis may be that option is deprecated
00:28 semiosis using glusterfs 3.3.1 btw
00:28 CROS_ I have 3.3.1 as well
00:28 y4m4 yeah correct me too
00:28 y4m4 i just saw read_child change back again
00:29 y4m4 weird
00:29 CROS_ =[
00:29 y4m4 seems like the txn_type check for lookup is overriding
00:29 y4m4 the read-subvolume option
00:29 y4m4 :-)
00:29 y4m4 that is where it calculates latency
00:29 y4m4 of first lookup
00:30 hagarth joined #gluster
00:31 y4m4 semiosis: can you check the gluster dump for both the 'glusterfs' process through 'kill -USR1' ?
00:31 y4m4 semiosis: i mean client
00:31 semiosis ok
00:31 y4m4 grep for 'read_child'
00:32 y4m4 lets see if its same for both the clients
00:32 y4m4 for me its different
00:32 y4m4 and that is how it should be
00:32 y4m4 with read-subvolume option
00:36 semiosis glusterdump.client-1.dump:read_child=1
00:36 semiosis glusterdump.client-0.dump:read_child=0
00:36 semiosis hmm
00:46 semiosis its different, but doesnt seem to have any effect
00:47 y4m4 yeah - it must be that that part of the code is ignored
00:47 y4m4 now
00:47 y4m4 i mean it does what is supposed to do but doesn't really work
00:47 y4m4 :-)
00:47 y4m4 semiosis: bummer
00:52 semiosis more for CROS_than for me, but yeah i guess a bummer all the same
00:55 CROS_ =\
00:55 CROS_ well, tahnks for the help guys
00:57 semiosis yw
00:58 CROS_ left #gluster
01:00 mooperd joined #gluster
01:19 yinyin joined #gluster
01:45 hflai joined #gluster
01:49 JoeJulian @later tell Troy If it was a user-configurable file, it would be in /etc. The fact that it's in /var/lib implies that it's a application state file and therefore controlled by the application.
01:49 glusterbot JoeJulian: The operation succeeded.
01:56 _pol joined #gluster
01:56 alex____ joined #gluster
01:57 alex____ I want to ask a question about geo-replication
01:58 alex____ anybody here?
01:58 _pol joined #gluster
01:59 alex____ I using "gluster volume geo-replication testvolume glusterslave:/ftp_home/testvolume start" to setup geo,it worked
02:00 alex____ but when i change hostname glusterslave's ipaddress, it not work
02:00 alex____ gluster output "Faulty"
02:00 alex____ i' sure i update the host file and restart glusterd
02:03 alex____ joined #gluster
02:03 alex____ ?
02:05 alex____ joined #gluster
02:07 alex____ left #gluster
02:08 alex____ joined #gluster
02:09 alexbot joined #gluster
02:09 alexbot hello
02:09 glusterbot alexbot: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
02:10 alex____ left #gluster
02:12 alexbot I have a question about geo-replication
02:13 alexbot I setup my geo-replication using cmd "gluster volume geo-replication testvolume glusterslave:/ftp_home/testvolume start", it worked
02:14 alexbot but when i change hostname "glusterslave's ipaddress", i not worked, gluster output "Faulty", what happned? i just change ipaddress but not hostname,
02:16 yinyin joined #gluster
02:40 jchen joined #gluster
02:43 JoeJulian alexbot: You'd have to check the logs and see.
02:44 raven-np joined #gluster
02:50 _pol joined #gluster
02:56 alexbot which log should i check?
02:57 JoeJulian I don't remember off the top of my head... Look under /var/log/glusterfs when you try to start geo-replication and see which one(s) changed most recently.
03:00 alexbot the log output as following:
03:00 alexbot [2013-02-28 18:39:28.507453] I [gsyncd:286:main_i] <top>: syncing: gluster://localhost:testvolume -> ssh://glusterslave:/ftp_home/testvolume/
03:00 alexbot [2013-02-28 18:39:28.928333] E [syncdutils:131:exception] <top>: FAIL:
03:00 alexbot Traceback (most recent call last):
03:00 alexbot File "/opt/glusterfs/3.2.2/local/libexec/glus​terfs/python/syncdaemon/syncdutils.py", line 152, in twrap
03:00 alexbot tf(*aa)
03:01 alexbot File "/opt/glusterfs/3.2.2/local/libexec/g​lusterfs/python/syncdaemon/repce.py", line 118, in listen
03:01 alexbot rid, exc, res = recv(self.inf)
03:01 alexbot File "/opt/glusterfs/3.2.2/local/libexec/g​lusterfs/python/syncdaemon/repce.py", line 42, in recv
03:01 alexbot return pickle.load(inf)
03:01 alexbot EOFError
03:01 alexbot but if change hostname to ipaddress, it worked
03:01 JoeJulian Use a pastebin like fpaste.org next time, please.
03:01 alexbot ok, tks
03:01 JoeJulian First of all, you're using a version with known critical bugs.
03:02 alexbot glusterfs 3.2.2 built on Jul 14 2011 13:34:25
03:03 JoeJulian I'm sorry if you read that as a question. It was a statement. You *are* using a version with critical flaws.
03:04 alexbot er~, it was solved if i switch to new version?
03:04 JoeJulian I don't know, but you're far less likely to lose your volume data.
03:05 yinyin joined #gluster
03:05 JoeJulian Just to confirm, you have said both: "when i change hostname glusterslave's ipaddress, it not work" and "if change hostname to ipaddress, it worked"
03:05 pipopopo_ joined #gluster
03:06 alexbot "when i change hostname glusterslave's ipaddress, it not work"
03:12 JoeJulian The rpc handling has change a lot since that version. I'm trying to look through the code but there's too many changes to track. I would strongly recommend upgrading to at least 3.2.7 if not 3.3.1
03:12 yinyin_ joined #gluster
03:12 JoeJulian @repo
03:12 glusterbot JoeJulian: I do not know about 'repo', but I do know about these similar topics: 'repository', 'yum repo', 'yum repository', 'git repo', 'ppa repo', 'yum33 repo', 'yum3.3 repo', 'repos'
03:12 alexbot ok, tks, i will switch to new version and re do
03:12 JoeJulian Do you use yum or apt?
03:12 alexbot ok, tks, i will switch to new version and retry
03:13 alexbot no ,i use tar.gz source
03:13 JoeJulian Which distro do you use?
03:13 alexbot CentOS release 5.5 (Final)
03:14 JoeJulian Then you should be using the ,,(yum repo)
03:14 glusterbot kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
03:14 alexbot 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
03:14 alexbot yes, i got it
03:15 sgowda joined #gluster
03:15 zhashuyu joined #gluster
03:17 yinyin joined #gluster
03:23 alexbot another question about geo-replication
03:24 alexbot I have some init data in my volume "testvolume" on localsite1 , i clone these data to machine localsite2
03:24 alexbot then ,i setup geo-replication between localsite1 and localsite2,the geo-replication running ok, but the data was deleted on localsite2
03:39 hagarth joined #gluster
03:46 vshankar joined #gluster
03:54 hflai joined #gluster
04:06 raven-np joined #gluster
04:13 y4m4 joined #gluster
04:20 bstansell___ joined #gluster
04:29 deepakcs joined #gluster
04:32 vshankar joined #gluster
04:32 zhashuyu joined #gluster
04:33 lala joined #gluster
04:35 bulde joined #gluster
04:43 hagarth joined #gluster
04:46 shireesh joined #gluster
04:49 bstansell_ joined #gluster
04:52 yinyin joined #gluster
04:56 yinyin_ joined #gluster
04:56 koudaddy joined #gluster
04:58 koudaddy Hey guys, I searched all over the internet for solution for this error I'm getting while using the gluster-client 3.3.1 that says "remote operation failed: Stale NFS file handle"
04:58 koudaddy Anyone around that can help?
05:12 yinyin joined #gluster
05:13 sahina joined #gluster
05:16 alexbot gluster's version and os info
05:17 mohankumar joined #gluster
05:22 glusterbot New news from resolvedglusterbugs: [Bug 764838] Rebalance start failure message is not appropriate <http://goo.gl/SdWJu> || [Bug 763637] Mounting volume is not working when any one server is down <http://goo.gl/UsFcm> || [Bug 763638] After setting the allowed hosts GNFS in not working as per specified allowed hosts <http://goo.gl/SXJIK> || [Bug 763534] Gluster volume creation failed <http://goo.gl/Y5rL9>
05:24 vpshastry joined #gluster
05:26 bala joined #gluster
05:30 pipopopo joined #gluster
05:37 raghu joined #gluster
05:38 krishnan_p joined #gluster
05:40 satheesh joined #gluster
05:42 krishnan_p JoeJulian, Hi
05:49 bstansell_ joined #gluster
05:49 mohankumar joined #gluster
06:04 timothy joined #gluster
06:07 vshankar joined #gluster
06:09 rastar joined #gluster
06:14 shylesh joined #gluster
06:20 krishnan_p joined #gluster
06:21 hagarth joined #gluster
06:23 vimal joined #gluster
06:26 _pol joined #gluster
06:29 vpshastry joined #gluster
06:36 Humble joined #gluster
06:38 vijaykumar joined #gluster
06:40 ramkrsna joined #gluster
06:47 sgowda joined #gluster
06:50 guigui1 joined #gluster
07:01 raven-np1 joined #gluster
07:08 mohankumar joined #gluster
07:10 mooperd joined #gluster
07:12 ThatGraemeGuy joined #gluster
07:14 raghu joined #gluster
07:18 raven-np joined #gluster
07:25 jtux joined #gluster
07:29 raghu joined #gluster
07:37 Nevan joined #gluster
07:44 ctria joined #gluster
07:47 vpshastry joined #gluster
07:55 ngoswami joined #gluster
07:57 hybrid5121 joined #gluster
08:01 jtux joined #gluster
08:08 romero left #gluster
08:11 krishnan_p joined #gluster
08:12 samppah @yum repo
08:12 glusterbot samppah: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
08:13 hagarth joined #gluster
08:16 tjikkun_work joined #gluster
08:18 hybrid5122 joined #gluster
08:18 alexbot left #gluster
08:19 hybrid5121 joined #gluster
08:28 Oneiroi joined #gluster
08:29 mohankumar__ joined #gluster
08:36 Oneiroi joined #gluster
08:37 andreask joined #gluster
08:55 sgowda joined #gluster
08:56 dobber_ joined #gluster
08:57 Staples84 joined #gluster
09:01 ramkrsna joined #gluster
09:03 rastar joined #gluster
09:06 hagarth joined #gluster
09:09 NeatBasis joined #gluster
09:23 duerF joined #gluster
09:26 rgustafs joined #gluster
09:28 mgebbe joined #gluster
09:29 mgebbe joined #gluster
09:49 hagarth joined #gluster
09:54 alex____ joined #gluster
09:55 alex____ left #gluster
10:06 vpshastry joined #gluster
10:07 glusterbot New news from newglusterbugs: [Bug 916934] Rebalance failures/Very slow <http://goo.gl/Mf5aC>
10:31 puebele joined #gluster
10:37 glusterbot New news from newglusterbugs: [Bug 916406] NLM failure against Solaris NFS client <http://goo.gl/uGTJA>
10:38 ctria joined #gluster
10:56 deepakcs joined #gluster
10:56 vijaykumar joined #gluster
11:02 raven-np1 joined #gluster
11:05 rastar joined #gluster
11:19 raven-np joined #gluster
11:22 joeto joined #gluster
11:25 vpshastry joined #gluster
11:27 edward1 joined #gluster
11:34 ctria joined #gluster
11:41 jclift_ joined #gluster
11:54 mohankumar__ joined #gluster
11:58 rgustafs joined #gluster
12:21 sahina joined #gluster
12:24 kkeithley joined #gluster
12:28 ctria joined #gluster
12:34 mohankumar__ joined #gluster
12:41 lh joined #gluster
13:04 vimal joined #gluster
13:16 gbrand_ joined #gluster
13:16 yinyin joined #gluster
13:19 bala joined #gluster
13:29 mooperd joined #gluster
13:38 JasonL_Sage joined #gluster
13:38 gbrand__ joined #gluster
13:57 rwheeler joined #gluster
13:58 dustint joined #gluster
14:03 aliguori joined #gluster
14:16 alex88 joined #gluster
14:16 alex88 hi guys, i've messed a bit with a server config and I've both glusterd and glusterfsd, both 3.3.1.. which should be the correct name?
14:22 jskinner_ joined #gluster
14:22 __Bryan__ joined #gluster
14:26 stopbit joined #gluster
14:26 hagarth joined #gluster
14:39 theron Good Morning johnmark. :)  I'm always forgetting to _away
14:39 cyberbootje joined #gluster
14:39 elyograg alex88: both are required, they do different things.  ,,(processes)
14:39 glusterbot information.
14:39 glusterbot alex88: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more
14:39 theron and thanks for the ^5
14:46 raven-np joined #gluster
14:53 bala joined #gluster
14:58 johnmark theron: I always forget that as well
14:58 dustint joined #gluster
14:58 jbrooks joined #gluster
14:59 Norky joined #gluster
14:59 lpabon joined #gluster
15:00 gbrand_ joined #gluster
15:01 jskinner joined #gluster
15:07 rwheeler joined #gluster
15:08 nueces joined #gluster
15:12 jskinne__ joined #gluster
15:19 theron joined #gluster
15:19 daMaestro joined #gluster
15:25 yinyin joined #gluster
15:29 martoss joined #gluster
15:43 akshay joined #gluster
15:43 hateya joined #gluster
15:44 m0zes gluster.org is down? http://www.downforeveryoneorjustme.com/gluster.org
15:44 glusterbot <http://goo.gl/pw9Db> (at www.downforeveryoneorjustme.com)
15:45 martoss joined #gluster
15:45 lpabon joined #gluster
15:45 dustint joined #gluster
15:45 17WAA70PK joined #gluster
15:45 nixpanic joined #gluster
15:45 jds2001 joined #gluster
15:45 rwheeler joined #gluster
15:45 ehg joined #gluster
15:45 twx joined #gluster
15:45 Ramereth joined #gluster
15:45 errstr joined #gluster
15:45 stigchristian joined #gluster
15:48 ndevos m0zes: works for me
15:48 m0zes ndevos: it was gone for a bit. oh well :) thanks
15:50 martoss hey folks, what should I do if I get State: Peer Rejected (Connected) from gluster peer status? UUID match the ones in peer/* respectively.
15:52 puebele joined #gluster
15:56 xiu joined #gluster
15:56 xiu hi, how do i trigger a self heal on a particular file with gluster 3.3 ?
15:56 ndevos martoss: all storage server shouls have a different UUID in /var/lib/glusterd/glusterd.info
15:58 ndevos ~targeted self heal | xiu
15:58 glusterbot xiu: http://goo.gl/E3b2r
15:58 ndevos ~cloned servers | martoss
15:58 glusterbot martoss: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again.
15:59 xiu ndevos: yep that's what i did but it stopped at half the file
15:59 xiu 1GB out of 1.8
15:59 ndevos xiu: no idea, I'd check for errors in the logs...
16:00 hagarth joined #gluster
16:04 alex88 elyograg: sorry for the delay and thanks for the tips
16:04 jason_sage Hi guys, I was wondering if you'd be willing to help with a serious problem that was either caused or exposed by gluster.
16:04 alex88 btw guys, I've two servers, going to be more in some months.. I've to decide if to use HW raid1 and gluster distributed volume or HW raid0 and gluster replicated volume
16:05 alex88 I was thinking of the second for when I'll need to put in maintenance mode a node, and I can shutdown and reboot it without any problem..
16:05 jason_sage The short version is that we have two servers running Ubuntu Server 12.04 and, once gluster was installed and the machines rebooted, they're unable to comminucate with the network.
16:05 alex88 any advice on what to select?
16:05 alex88 jason_sage: seems not a gluster related problem? have you messed with iptables?
16:06 alex88 have you debugged why they aren't able to connect?
16:07 jason_sage alex88: We've been working on it. There is no obvious problem in ifconfig, in the hardware (since it can ping on a local lan) or in fstab. Our network admin is in Mexico for two weeks, and we don't know what kind of setup he did to get it working. It looks like he tried to debug one machine and never found a solution.
16:07 jason_sage We've gone to google, gluster.org and history logs, but we can't see anything.
16:08 akshay joined #gluster
16:09 jason_sage Our theory is that gluster is doing some sort of interference when the system reboots, but we have no idea how to chase that.
16:11 johnmark m0zes: it was down for a bit for software upgrades. For about a minute.
16:13 jdarcy joined #gluster
16:16 hybrid512 joined #gluster
16:16 semiosis :O
16:17 jason_sage alex88: Of course, most of that is guess work right now. The only commonality between the two systems is that gluster got installed, they appeared able to connect to the network, ping was working. Then, once the machines were rebooted, they were no longer able to see out.
16:19 kr4d10 joined #gluster
16:19 xiu ndevos: yep i don't have any but thanks anyway
16:20 ndevos xiu: hmm, and you have enough space on the brick I assume? (thats all I can think of atm)
16:21 jclift_ jason_sage: I wonder if there's some kind of dependency conflict.  It'd be super unusual, but that's the only thing easily coming to mind. :(
16:21 semiosis targeted self heal was my workaround for lack of a self heal daemon before 3.3, it's no longer necessary.  you can always force a self heal by running 'stat <file>' through a client mount, and since 3.3 there is a 'gluster volume heal' command which can force a volume wide check
16:21 Norky jason_sage, you say that "it can ping on a local lan"
16:22 Norky do you mean that the servers can both communicate on the 'local' subnet?
16:22 jclift_ jason_sage: Unfortunately, I don't use Ubuntu at all so I have no idea what commands you'd need to run to check. :/
16:22 xiu ndevos: yep :/
16:23 martoss ndevos, yes, the uuids in glusterd.info are different and match the ones in peer/ filenames on the respective other sides.
16:24 ndevos martoss: okay, and all servers have the same glusterfs version?
16:25 neofob So does anyone run gluster on 32bit OS in production?
16:26 semiosis neofob: it's been done
16:27 semiosis jason_sage: can you please clarify "it can ping on a local lan" and "no longer able to see out"
16:28 semiosis jason_sage: what commands work, what dont?
16:28 neofob semiosis: just wonder because i only run gluster on my ARM boxes (CuBox)
16:28 semiosis wow gluster on arm
16:29 semiosis what version?
16:30 ndevos neofob: same here, 2x efikamx with fedora 17/18 and kkeithley arm rpms
16:30 martoss ndevos, yep - 3.3.1
16:31 martoss two ubuntu machines with packages from ppa and one powerpc NAS on which I've build debian packages myself.
16:31 jason_sage semiosis: sorry, morning meeting.
16:31 ndevos martoss: hmm, I'm not sure what could cause that rejected state then :-/
16:31 neofob my semiosis: 3.3 on Debian Wheezy hardfloat on ARM
16:32 martoss ndevos, ok thx I'll check further if I find sth.
16:32 jason_sage semiosis: The primary server mounts a gluster file system from a local loopback from its two exposed bricks. (I think this is right, like I said, our SysAdmin is gone).
16:32 neofob CuBox (http://www.solid-run.com/) has esata port with port multiplier and 1G of RAM
16:32 neofob the new version has 2G
16:32 glusterbot Title: SolidRun - Home of the CuBox (at www.solid-run.com)
16:33 ndevos martoss: probably there is something in the glusterd logs, thats where I would start searching
16:33 jason_sage semiosis: The machine can ping to that local loopback and gets its file system and can get pings returned from that. When we attempt to ping into the primatry server or ping out to anything that's not local loopback, we get host unreachable.
16:34 semiosis ah ok
16:34 semiosis cant imagine how glusterfs would affect that
16:35 jason_sage semiosis: The secondary server, which was meant to be the mirror machine for the primary, hit this same problem (we suspect) during setup, and it's got the same symptoms. The mirroring never completed, since the server was rebooted and then lost its network access.
16:35 semiosis jason_sage: could you prepare a pastie/gist with the output of 'ifconfig', 'route -en', iptables -nvL', 'iptables -t nat -nvL' and 'cat /etc/network/interfaces' please?
16:35 jason_sage semiosis: That's where we are, too, but it's the only thing that changed on those machines between having connectivity and not having connectivity.
16:36 jason_sage semiosis: Yeah, give me some time. With these problems, I have to run those right on the server
16:36 semiosis sure
16:37 Norky is ipmi/remote console not an option?
16:38 jason_sage Norky: No, it's not, since the machine is walled off.
16:40 jason_sage brb, I'm going to switch to  my laptop in the server room
16:41 hagarth joined #gluster
16:46 Jason_Sage joined #gluster
16:50 rwheeler joined #gluster
16:52 alex88 please help, I've removed a volume
16:52 alex88 and trying to recreate using same path
16:52 alex88 but I get path/ or a prefix of it is already part of a volume
16:52 glusterbot alex88: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
16:53 alex88 glusterbot: thanks :) so fast :P
16:53 semiosis glusterbot: awesome
16:53 glusterbot semiosis: ohhh yeeaah
16:53 NuxRo joined #gluster
16:56 Humble joined #gluster
16:56 plarsen joined #gluster
16:58 raven-np joined #gluster
17:01 jason_sage semiosis: took some doing, but here they are:
17:01 jason_sage http://pastebin.com/BZzZeMEm
17:01 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
17:01 jason_sage ok
17:02 jason_sage http://fpaste.org/OX4W/
17:02 glusterbot Title: Viewing Paste #281398 (at fpaste.org)
17:02 jskinner_ joined #gluster
17:02 alex88 on the admin pdf, Creating Distributed Replicated Volumes section
17:02 jason_sage http://fpaste.org/EsIr/
17:02 glusterbot Title: Viewing Paste #281399 (at fpaste.org)
17:03 alex88 it shows that with multiple bricks per server, a file goes in the brick of another server, that's by default? and, is it really that way?
17:03 alex88 no file will be stored on different bricks of same server?
17:03 jason_sage http://fpaste.org/YXBw/
17:03 glusterbot Title: Viewing Paste #281400 (at fpaste.org)
17:03 jason_sage http://fpaste.org/XGtk/
17:03 glusterbot Title: Viewing Paste #281401 (at fpaste.org)
17:04 jason_sage http://fpaste.org/Qbtt/
17:04 glusterbot Title: Viewing Paste #281402 (at fpaste.org)
17:04 jason_sage those are the requested commands in order
17:05 Norky the broadcast addres for eth0:1 makes no sense with that address/netmask
17:05 JoeJulian ~brick order | alex88
17:05 glusterbot alex88: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
17:05 semiosis jason_sage: you have no default route
17:05 xian1 joined #gluster
17:06 jason_sage semiosis: I'm sorry, I'm generally a java dev. I understand what you're saying. Do you have a pointer to a fix?
17:07 Norky the RX and TX packet counters show these machine have sent and received some traffic
17:07 jason_sage Norky: That is true, but it's on local loopback.
17:07 semiosis Norky: agreed, looks like the netmask should be 255.255.255.0 instead of current 255.255.0.0.  would that explain why the default route (which would be on that interface) is missing from the routing table?
17:08 Norky under eth0, 11.4MB RX
17:08 jason_sage is that on the 172.20.0.0 address?
17:08 Norky 172.20.201.15  Bcast:172.20.201.255  Mask:255.255.0.0    is contradictory
17:09 Norky either mask should be 255.255.255.0 as semiosis says, or broadcast should be 172.20.255.255
17:10 semiosis Norky: look at the network address in the config file, it would need to change as well.  network and bcast agree with each other & a /24 netmask
17:10 jason_sage that's in the ip tables, yet?
17:10 Norky semiosis, true
17:10 alex88 JoeJulian: thanks for the tips
17:11 Norky 172.X. is available as a /16
17:12 alex88 also, our use will be with large files written just one time and readed tons of times.. I think that distributed replication is the correct way to have better read performance and maintenance without downtime
17:12 semiosis doesnt /16 on a host seem unusual?  i'd usually expect to see such a large subnet on a router, not a host
17:12 akshay left #gluster
17:12 Norky *shrug* we use it in our 'build' network
17:13 jason_sage semiosis, Norky: What I'm hearing is that I should alter the eth0:1 entry in ifoncifg to read Mask: 255.255.255.0
17:13 semiosis jason_sage: it's worth a shot
17:13 mooperd joined #gluster
17:13 JoeJulian 172.16 is a /12 private address block.
17:13 jason_sage semiosis: Okie dokie, I'll give it a shot
17:14 semiosis jason_sage: you should also have a default route appearing in 'route -en' looking something like "0.0.0.0         10.168.1.1      0.0.0.0         UG        0 0          0 eth0" <-- notice "destination" and "genmask" columns both all-zero
17:14 Norky 172.16.0.0 through to 172.31.255.255, in fact
17:14 martoss joined #gluster
17:14 Norky err, n/m, that;'s what you just said, JoeJulian
17:14 JoeJulian Hehe
17:15 semiosis jason_sage: you dont have any default route, which means you should be able to reach other hosts on lans directly connected to your nics, but not things beyond the lan, like 8.8.4.4
17:15 JoeJulian Norky and I know how to use whois. ;)
17:15 jason_sage semiosis: Is that something *I* put in, or should it appear after I make this change?
17:15 semiosis jason_sage: you have one 'gateway' line in yrou interfaces file, which seems to have been ignored, i hope because of this contradictory config Norky identified
17:15 Norky private addresses is the only rfc number I can remember
17:16 semiosis jason_sage: hopefully fixing the netmask will allow the default route to be added
17:16 jason_sage ok. So it's the one change and test?
17:16 semiosis yep, make that netmask 255.255.255.0 for the eth0:1 interface in the config and reboot
17:16 jason_sage ok
17:17 Norky jason_sage, the networking config in your case is not simple, I'd even go so far as to say "odd"
17:18 jason_sage Norky, semiosis: I'm getting pushback from partners I'm working with saying that the 255.255.0.0 mask is correct and is our default network, which is a /16
17:19 semiosis well if that's correct then the "network" and "broadcast" lines in that interface are both wrong
17:19 JoeJulian if it were me, I'd remove the broadcast and network lines from that interfaces file.
17:19 semiosis if the netmask is /16 then "network should end with "0.0" and broadcast should end with "255.255"
17:19 mohankumar joined #gluster
17:19 semiosis s/\"network/network/
17:19 glusterbot What semiosis meant to say was: if the netmask is /16 then network should end with "0.0" and broadcast should end with "255.255"
17:21 JoeJulian If you have any other machines on that network that do communicate with the router, I'd check their netmask. That network could be a /12 making the netmask 255.240.0.0.
17:21 semiosis in any case, none of this is gluster related :)
17:22 semiosis how could the presence/absence of glusterfs on the host have any effect on connectivity?
17:22 JoeJulian This is network admin shouldn't have gone on vacation without making sure his configuration would withstand a reboot related. (is there an irc channel for that?)
17:22 jason_sage Could not agree more.
17:23 Humble joined #gluster
17:24 * JoeJulian wonders if anyone he knows uses any private addresses other than 10. and 192.168....
17:25 mattr01 JoeJulian: we use 158 for some reason
17:25 mattr01 I cant stand it actually and im new here
17:25 larsks I often use 172.16.x for VPN addresses, since it seldom conflicts with the local wireless network :).
17:25 semiosis larsks: +1
17:25 semiosis i use it on openvpn
17:25 mattr01 I remember the old 2wires used t ouse 172
17:25 larsks Yup.
17:25 Norky we use 172.16.0.0/16 on our 'build' network
17:30 kkeithley 158? As in 158.x.x.x?
17:32 jason_sage semiosis: No improvement. I've been told that the primary problem is that eth0 isn't doing dhcp even though all other machines on the network are and they're all running the same basic configuration.
17:33 JoeJulian kkeithley: Who wants to connect to the Department of Defense anyway? ;)
17:33 jason_sage and when we do service networking start, it hangs, when we do service networking stop, we just get a waiting response
17:33 kkeithley I could tell you, but then I'd have to, er, never mind.
17:35 redsolar joined #gluster
17:38 CROS_ joined #gluster
17:39 JoeJulian jason_sage: We're all pretty helpful folks by nature, we do also try to take into account the 175 lurkers that try to learn more about GlusterFS by scanning what we talk about in here. So with regret I'd like to redirect you to #networking
17:39 CROS_ joined #gluster
17:40 jason_sage JoeJulian: I appreciate the help. My partner left, so I'm implementing those suggestions now. If it doesn't work, I'll switch rooms.
17:40 jason_sage question first: is rcpbind a gluster op?
17:41 kkeithley rpcbind a gluster op? I don't know what that means.
17:42 JoeJulian rpcbind is it's own thing, but gluster utilizes that for nfs
17:42 jason_sage Ok, because on startup, I get 'cannot open /run/rpcbind/rpcbins.xdr no such file'
17:42 jason_sage same with portmap.xdr
17:43 _pol joined #gluster
17:45 kkeithley Do you have the rpcbind package installed? (Did you build gluster from source? Because if you installed rpms, rpcbind should have been installed too, if you used yum; or you should have been prompted to install it, if you used rpm.)
17:47 kkeithley Hmm, which reminds me, someone told me recently that RHEL5 uses something other than rpcbind, by default (or otherwise). Ring any bells anyone?
17:47 jason_sage kkeithley: We did not build from source, we got the deb package following the installation instructions on gluster.org. And we have the rpcbind package installed and running.
17:47 JoeJulian kkeithley: portmapper, iirc.
17:48 kkeithley Oh, deb package.
17:50 kkeithley So, deb package from gluster.org for Debian, or deb package from semiosis' ppa for Ubuntu?
17:50 jason_sage kkeithley, let me look at the history and verify that.
17:50 JoeJulian kkeithley: checking now... I have a EL5 install that I haven't upgraded yet...
17:51 kkeithley Every time I think I'm done for the time being with packaging, one more thing turns up
17:51 JoeJulian portmap
17:54 __jason_sage__ left #gluster
17:55 kkeithley thanks. But I have to wonder how the RHEL5 builds in EPEL work. There's no rpcbind in EPEL, so where does it get that from?
17:56 rubbs joined #gluster
17:59 hagarth joined #gluster
17:59 rubbs question: when I created a new volume and then did a `echo "test" > /usr/web/testing.txt` I got a "invalid Argument error. here is my client log. http://dpaste.com/1009115/
17:59 glusterbot Title: dpaste: #1009115: invalid argument, by rubbs (at dpaste.com)
18:03 Mo__ joined #gluster
18:03 JoeJulian rubbs: The problem is further up in the log where you get "No route to host" errors on two servers.
18:03 JoeJulian Probably iptables I would guess.
18:04 JoeJulian Since both servers in one replica set are down, that causes "0-usrweb-dht: found anomalies in /. holes=1 overlaps=0"
18:05 JoeJulian With a 1:4 probability, "hello.txt" hashes out to 3868025292 which is supposed to be assigned to that missing distribute subvolume.
18:06 rubbs JoeJulian: yup... fat fingered a command on the last two boxes
18:06 rubbs thanks for helping me catch that.
18:06 rubbs now I know what to look for ;)
18:06 JoeJulian three... connection to 10.0.8.241:24007 failed
18:07 JoeJulian That's interesting. I haven't encountered that before.
18:08 JoeJulian "0-usrweb-client-1: changing port to 24013 (from 0)" So it picked up the port from one of the servers but then later it failed on 24007. It's trying to get the brick port from that server.
18:09 JoeJulian I wonder why it doesn't just use the info it already had?
18:09 rubbs yeah I just did a telnet to that port on that IP and it worked
18:09 mohankumar joined #gluster
18:11 rubbs I'm gonna try again here... one sec
18:12 rubbs firewalls are set correctly, but no go
18:12 rubbs still getting the same error
18:12 rubbs I did change the hostnames to these machines earlier today.
18:12 jason_sage kkeithley: sorry for the delay, the package was installed through aptitude via ppa:semiosis
18:13 JoeJulian Check tcpdumps at both ends. "No route to host" has to be coming from somewhere.
18:13 rubbs but I also deleted all volumes. and started completely new ones (didn't reuse names or dirs)
18:13 JoeJulian @ppa
18:14 glusterbot JoeJulian: The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY
18:14 kkeithley okay, so maybe semiosis might have some thoughts about why you're getting those xdr file errors
18:14 rubbs JoeJulian: kk. I'll do that
18:17 JoeJulian @yum
18:17 glusterbot JoeJulian: I do not know about 'yum', but I do know about these similar topics: 'yum repo', 'yum repository', 'yum33 repo', 'yum3.3 repo'
18:17 JoeJulian @yum repo
18:17 glusterbot JoeJulian: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
18:20 CROS_ left #gluster
18:31 zaitcev joined #gluster
18:35 _pol joined #gluster
18:36 disarone joined #gluster
18:36 _pol joined #gluster
18:37 jason_sage Hey guys, I just wanted to say thanks for the help. We're gonna give up for now and go an entirely different way. No more 11 hour days.
18:37 jason_sage left #gluster
18:38 larsks Hey, there's an interesting question on community.gluster.org (http://goo.gl/3MJwe) regarding whether it makes more sense to have one big brick on top of a RAID or lots of little bricks on top of a JBOD.  I'm curious if there's a general consensus here?
18:38 glusterbot Title: Question: Multiple bricks from JBOD better than one brick from RAID? (at goo.gl)
18:40 elyograg larsks: I was faced with that recently.  My original plan was to go with JBOD, but ended up going with two RAID5 volumes per server.  same loss of space as raid6, but maintains some independence of spindles.  what finally sold me was how much effort I'd have to expend if a drive failed, and I know that drives WILL fail.
18:41 elyograg using Dell PE R520 servers.  twelve 3.5" bays, OS is on a 2.5" SAS raid1 that's separate from the front bays.
18:42 larsks elyograg: Hmmm, interesting point.  Makes me wonder how exactly Linux responds to a failed disk on a mounted filesystem.
18:42 larsks elyograg: I guess instead of simply swapping in a new disk, you would have remove the brick from gluster, unmount the filesystem, swap disks, reformat, remount, and add back to gluster, yes?
18:43 elyograg larsks: with good hardware, it should fail in a way that's detectable and can be handled properly, but that was another consideration ... would the OS notice and take it offline rather than just never responding to the I/O request?
18:43 elyograg larsks: that's it in a nutshell.  very messy.  with RAID, you just swap the disk and the controller melds it all back together.
18:44 larsks elyograg: Thanks.  That definately helps me.  You should just copy and paste that onto community.gluster.org :).
18:44 elyograg if you can't afford the raid5/6 write penalty, then the other way would obviously work better except when there's a failure.
18:48 tjikkun joined #gluster
18:48 tjikkun joined #gluster
18:51 tryggvil joined #gluster
18:57 elyograg I just created an account on the community site and put up an answer to that question.
19:00 larsks Hopefully that helps the poster, too.
19:24 Troy joined #gluster
19:25 layer7switch joined #gluster
19:58 lpabon_ joined #gluster
20:04 lpabon joined #gluster
20:04 medhu joined #gluster
20:05 medhu Hi. I just installed glusterfs 3.3.1 on amazon ec2 debian instances
20:06 medhu Created a replica based volume of 2 nodes
20:06 medhu volume info from both nodes working fine
20:06 medhu but when i mount i am not getting a bigger volume
20:07 medhu and neither the files i touch(creatre) dont appear in the other server
20:07 medhu i tried creating it in replicated + distributed way following the document
20:08 medhu the logs say that "Stopping crawl as < 2 children are up"
20:09 medhu and client-1: connection to my.ip.add.ress:24007 failed (Connection timed out)
20:09 medhu but a telnet on those port works fine
20:13 elyograg medhu: if it's replicated at replica 2, your volume will be half the size of the total space.  if bricks on the two servers are equal size, then the volume will be as big as one server's brick(s).  If you are not seeing files created on both servers, then it sounds like you are creating the files on the brick, not a client-mounted volume.
20:14 elyograg having equal size bricks is very much preferred for proper operation.
20:14 rubbs JoeJulian: Not sure why this fixed it, but I deleted the volume, deleted all the brick directories, restarted the service and then recreated the volume and now it works.
20:15 medhu i agree but inorder to test and see how things work with glusterfs i am lacking some thing
20:15 medhu i have gone throught the document and administrative handbook as well
20:16 medhu i have done mount -t glusterfs localhost:/<vol-name> /mountpoint
20:16 semiosis kkeithley: xdr file errors?
20:16 medhu touch /mountpoint/{a,b,c...}
20:16 medhu i have mounted the same way on the other node as well before touching
20:18 medhu on both my nodes my volumes are mounted as /vol
20:19 medhu does this matter? should the names differ in each node?
20:19 medhu gluster volume create www replica 2 transport tcp server1:/vol server2:/vol
20:19 medhu this ^ is what i am doing
20:20 semiosis medhu: i follow this ,,(brick naming) scheme
20:20 glusterbot medhu: http://goo.gl/l3iIj
20:20 semiosis medhu: check iptables, maybe?  glusterfs needs some ,,(ports)
20:20 glusterbot medhu: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
20:21 medhu i have opened 24007 and 24008.
20:21 semiosis medhu: another thing, you should not write to the brick directories, instead all writes (really all operations) should be done throuhg a client mount point
20:22 semiosis medhu: you will need more ports open, each brick runs a process that exports the brick directory over a port.  see ,,(processes)
20:22 glusterbot information.
20:22 glusterbot medhu: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more
20:22 medhu deleted volumes do not reset counter meaning they use other ports above 24009 ?
20:24 semiosis larsks: "Makes me wonder how exactly Linux responds to a failed disk on a mounted filesystem." <-- failed disk could mean lots of things, from bit rot to a fire in the raid array, and everything in between.
20:24 kkeithley semiosis: jason_sage was getting errors. E.g. 'cannot open /run/rpcbind/rpcbins.xdr no such file'
20:24 kkeithley but he's gone now
20:25 Staples84 joined #gluster
20:27 gbrand_ joined #gluster
20:27 semiosis medhu: every brick you make on a server increments a counter, which starts at 24009 and goes up from there.  for example, if a server has one brick in each of two volumes, the counter goes up twice.  then if one volume is deleted, and another created with a different brick, it will get 24011, because the counter is never decremented
20:27 semiosis even though the server never had more than two bricks at a time, it has had a total of three over its lifetime
20:28 semiosis kkeithley: thanks for clarifying, i have no idea what that file does.  jason_sage had lots of problems :(
20:28 medhu semiosis: thanks for your help. in the link about naming bricks you have shared, i have bit of confusion
20:29 medhu my each server has an external volume (ebs-amazon) attached
20:29 medhu and that volume is mounted to /vol
20:29 rwheeler joined #gluster
20:30 medhu one server has 100GB of volume attached and the other server has 500 GB volume attached
20:30 medhu both have xfs on them
20:30 medhu and each of these volumes are mounted on /vol on the respective servers
20:31 semiosis it's a good idea to have equal sized bricks.  that's very easy with ebs
20:31 medhu now i have done replica 2 transport tcp server1:/vol and server2:/vol
20:32 medhu from what i have done, wont i acheive a replicated+distributed environment?
20:32 medhu ofcourse i will go back and try with equal sized disks now
20:33 jbrooks joined #gluster
20:36 lpabon joined #gluster
20:37 larsks semiosis: certainly, those are all possibilities.  I hope from context it was somewhat obvious to what I was referring...
20:38 semiosis medhu: if you have replica 2, and only two bricks, then you have pure replicate, not distributed-replicated.
20:39 medhu what to do for achieving replicated+distributed with 2nodes?
20:39 semiosis larsks: i take the view that what isn't there can't break.  i like a simpler architecture.  see my ,,(ebs rant) which applies pretty much the same to physical raid arrays as it does to md/lvm arrays
20:39 glusterbot larsks: I do not know about 'ebs rant', but I do know about these similar topics: 'canned ebs rant'
20:40 semiosis ,,(canned ebs rant)
20:40 glusterbot http://goo.gl/GJzYu
20:41 hateya joined #gluster
20:41 semiosis in summary, only reasons I consider valid for combining disks into arrays are 1) increased single-thread performance, and 2) storing extremely large files*
20:42 semiosis * - for some definition of "extremely large" which depends on your disk size and total number of files to be stored
20:43 medhu semiosis: could get it
20:43 semiosis but i'm biased since in EC2 it's easier to just replace a whole "datacenter" than deal with one failed component
20:43 medhu opening higher ports solved the issue
20:44 semiosis medhu: great!
20:44 medhu but i see the whole size as 100Gig
20:44 medhu thanks a lot semiosis
20:44 semiosis medhu: so if you want to make a distributed replicated volume you need to provide a multiple of the replica size, for example replica 2, you would need to provide 4, or 6, or 8, or ... bricks
20:45 semiosis lets say you give replica 2 and 6 bricks, this is called a 3 x 2 distributed-replicated volume.  your files will be distributed evenly over 3 pairs of replicas.
20:46 medhu ohk! but how can i make 6 bricks out of 2 servers
20:46 semiosis so if all bricks were 100GB then you would have total space of 300GB.
20:46 medhu each server has only one volume which is mounted to a single mount point
20:46 semiosis medhu: server1:/vol/brick1 server2:/vol/brick1 server1:/vol/brick2 server2:/vol/brick2 ...
20:47 semiosis medhu: attach more ebs volumes to your servers
20:47 medhu semiosis: this seems to be great
20:47 medhu so i can attach two ebs volumes to each node
20:48 semiosis you can attach a whole ebs volume to a partition number, for example, /dev/sdf1 .. /dev/sdf6, each of those looks like a partition of /dev/sdf but really they are all entire ebs volumes
20:48 medhu and make a 2 x 2 replica, distributed
20:48 semiosis sure
20:48 semiosis another tip is to make the brick directory a subdirectory of your mount point, for example...
20:49 semiosis attach an ebs volume to /dev/sdf1, mount that as /bricks/brick1, then in your volume create, specify a subdirectory of it, like server1:/bricks/brick1/data
20:49 elyograg I use the name of the volume as the subdir.  I can put more than one volume on the same set of bricks.
20:50 semiosis +1
20:50 semiosis forgot about that :)
20:50 medhu this is really a great tip
20:51 semiosis i dedicate an ebs volume per brick, and i name the bricks for volumes, so for a volume called myvol, i would actually mount the brick at /bricks/myvol1
20:51 semiosis for example
20:52 hateya joined #gluster
20:54 medhu semiosis: elyograg: I was fiddling around for quite long. you guys helped me a lot. thanks again
20:54 semiosis you're welcome
21:05 jskinner_ joined #gluster
21:14 gbrand__ joined #gluster
21:25 hateya joined #gluster
21:27 leoboiko joined #gluster
21:31 leoboiko hi. I'm trying to understand gluster quotas. they don't talk at all to the standard Unix quota tools? (edquota, /usr/bin/quota etc)
21:33 tqrst joined #gluster
21:33 tqrst is there a way to make glusterfs not eat ~20G memory on every node when rebalancing?
21:35 elyograg another question to the pile: is the S3 integration via swift supposed to handle subdirectories? It didn't seem to do so on gluster-swift based on swift 1.4.8, I have the 1.7.4-based version now but I haven't tried it yet on that.
21:39 glusterbot New news from newglusterbugs: [Bug 895528] 3.4 Alpha Tracker <http://goo.gl/hZmy9>
21:42 tryggvil joined #gluster
22:00 lh joined #gluster
22:15 badone joined #gluster
22:15 johnmark elyograg: you can store in subdirs, and you can read a file in a subdir, if you know it's path
22:15 johnmark but you can't read the contents of a subdir
22:15 johnmark er list the contents...
22:17 elyograg johnmark: is this the case for any s3, even the official amazon?
22:18 Mo___ joined #gluster
22:18 JoeJulian tqrst: Does it eat 20G on your clients during a rebalance too???
22:24 disarone joined #gluster
22:29 johnmark elyograg: not sure. all I know is that it is a Swift API limitation
22:29 johnmark I don't know if S3 can do nested containers
22:30 tqrst JoeJulian: fsd, sorry. Clients are ok.
22:33 tqrst annnd segfaults again
22:33 tqrst I give up
22:35 JoeJulian This is 3.3.1?
22:36 tqrst yes
22:36 JoeJulian Hmm, 3.3.1's been the first version I've been able to get to rebalance successfully.
22:37 tqrst I've never seen a rebalance complete in the past year we've been using gluster
22:37 tqrst we usually just gave up after a week or five
22:37 JoeJulian Maybe try lowering performance.cache-size? I run at 8MB but that's because I run 60 bricks per server.
22:38 JoeJulian I'm also running performance.io-cache: off on some volumes.... I don't remember why...
22:39 tqrst hm, cache-size is unset (default is 32M I think?), with 2-3 bricks per server
22:39 JoeJulian And I don't do nfs, so nfs.disable: on for me.
22:39 tqrst same
22:39 JoeJulian I'm not saying any of those will help, but just that I have those set and have seen successful rebalances.
22:39 semiosis elyograg: the structure of S3 is a simple key value store, there is no hierarchy except in your own naming conventions.  a bucket has keys, like "foo" or "foo:bar" or "foo/bar" and keys have values which can be any data
22:41 tqrst heh, one of my servers has 5213606563767386112 failures in rebalance status
22:43 elyograg that's definitely not using a 32-bit counter. ;)
23:14 Humble joined #gluster
23:27 fidevo joined #gluster
23:36 JoeJulian tqrst: What are the errors? Did you check the logs? Pretty-much the only errors I've had are where it refuses to move a file from a less-full brick to a more-full one.
23:36 JoeJulian That's overcome by adding "force" to the start command.
23:40 tqrst JoeJulian: hold on, anonymizing the last part of my rebalance logs before I gave up and stopped it
23:43 tqrst http://pastie.org/pastes/6363037​/text?key=b5kpnzuvb857uscfqgzpja
23:43 glusterbot <http://goo.gl/yIE4C> (at pastie.org)
23:50 JoeJulian filetype differs on subvolumes? That's only a warning?
23:53 xian2 joined #gluster
23:56 xian1 joined #gluster
23:58 xian2 joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary