Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-12

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:16 tryggvil joined #gluster
00:38 wushudoin joined #gluster
01:05 peter_ joined #gluster
01:13 peter_ hi, anyone here so I can ask a newbie question
01:13 andreask left #gluster
01:15 yinyin joined #gluster
01:16 semiosis hello
01:16 glusterbot semiosis: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
01:16 semiosis peter_: ^^^^
01:17 gm__ joined #gluster
02:01 quillo joined #gluster
02:07 plarsen joined #gluster
02:22 theron joined #gluster
02:29 balunasj joined #gluster
02:43 kevein joined #gluster
03:21 yinyin joined #gluster
03:24 avati joined #gluster
03:26 bharata joined #gluster
03:31 andreiw joined #gluster
03:32 andreiw hiya
03:32 andreiw too bad the LISA'12 BoF got cancelled...are you guys intending on doing it some other day or is that it?
03:34 andreiw Anyway...I noticed the native protol is XDR-based, and was wondering if there is protocol documentation akin to the NFS RFCs...
03:34 yinyin joined #gluster
03:53 zwu joined #gluster
04:00 JoeJulian ndevos might have something, maybe.
04:01 JoeJulian Y
04:07 sripathi joined #gluster
04:30 ramkrsna joined #gluster
04:36 sgowda joined #gluster
04:50 rastar joined #gluster
04:59 yinyin joined #gluster
05:00 hagarth joined #gluster
05:00 vpshastry joined #gluster
05:00 overclk joined #gluster
05:04 avati joined #gluster
05:05 hchiramm_ joined #gluster
05:10 lng joined #gluster
05:11 lng Hi! I'd like to turn Distributed Replicated Volume to Distributed. How do I do it?
05:12 lng Should I remove 2 of 4 nodes and edit configs?
05:14 bulde joined #gluster
05:15 kanagaraj joined #gluster
05:17 kanagaraj joined #gluster
05:21 Humble joined #gluster
05:24 shylesh joined #gluster
05:27 lng Is it okay to mount volume by hostname which resolves to multiple IP returned by round robin?
05:29 elyograg lng: that should be fine for a fuse mount.  the host/ip you give it during mounting is only used to get the volume information, after that it contacts all the servers in the volume directly. i don't think i'd try it for an nfs mount.
05:31 lng elyograg: I use NGC
05:31 lng native gluster client
05:33 elyograg that's fuse.  the round-robin dns thing is i have heard recommended as a way to do things.
05:33 lng so I don't need to touch it then... okay
05:34 lng I have a lot of split brain files from time to time... I don't know what to do
05:34 lng maybe I have something to change in my setup
05:43 lng is it possible to see network partition happened by gluster logs?
05:53 raghu joined #gluster
06:06 mohankumar joined #gluster
06:06 rwheeler joined #gluster
06:09 shylesh joined #gluster
06:20 puebele1 joined #gluster
06:23 vimal joined #gluster
06:24 glusterbot New news from resolvedglusterbugs: [Bug 764762] non-root owned files/dirs not shown <http://goo.gl/7dVMX>
06:44 yinyin joined #gluster
06:48 shylesh joined #gluster
06:53 rgustafs joined #gluster
06:54 glusterbot New news from resolvedglusterbugs: [Bug 861306] Stopping the volume does not clear the locks <http://goo.gl/rHyBd> || [Bug 765236] KVM migration fails : permission denied <http://goo.gl/zQr2o>
06:59 ngoswami joined #gluster
07:09 mdarade joined #gluster
07:21 kwevers joined #gluster
07:23 mdarade left #gluster
07:24 kwevers joined #gluster
07:30 Nevan joined #gluster
07:31 dobber joined #gluster
07:32 shylesh joined #gluster
07:50 kwevers joined #gluster
07:52 ekuric joined #gluster
08:04 nixpanic andreiw: no RFCs for the protocol, the best I have is http://people.redhat.com/ndevos/t​alks/gluster-wireshark-201211.pdf
08:04 glusterbot <http://goo.gl/QAzbm> (at people.redhat.com)
08:05 * ndevos switches irc clients
08:06 ndevos andreiw: wireshark-1.8 and newer should have a pretty complete support, but that does not describe the procedures and interactions :-/
08:06 ndevos and, if you find an issue with a capture, send it my way and I'll have a look
08:09 Humble joined #gluster
08:18 guigui3 joined #gluster
08:19 hchiramm_ joined #gluster
08:19 masterzen joined #gluster
08:20 mooperd joined #gluster
08:21 badone joined #gluster
08:29 gbrand_ joined #gluster
08:29 nissim joined #gluster
08:30 gbrand_ joined #gluster
08:38 andreask joined #gluster
08:45 rwheeler joined #gluster
08:45 Humble joined #gluster
08:53 sripathi joined #gluster
09:00 duerF joined #gluster
09:04 Humble joined #gluster
09:07 mooperd joined #gluster
09:08 sripathi1 joined #gluster
09:15 passie joined #gluster
09:17 passie I'm trying the following command according to the quick howto readme, but I'm getting a error
09:17 passie gluster volume create gv0 replica 2 10.248.101.55:/export/brick1 10.248.101.56:/export/brick1
09:17 passie Brick: 10.248.101.55:/export/brick1, 10.248.101.56:/export/brick1 one of the bricks contain the other
09:18 passie this has something todo with getfattr and setfattr, but i'm not sure howto fix this
09:21 x4rlos joined #gluster
09:23 x4rlos Hi all. Im looking for gluster to host a shared filesystem across two machines in different buildings for data protection. Is gluster used for this type of requirement? Or are there better tools? Thanks.
09:23 Humble joined #gluster
09:25 balunasj|mtg joined #gluster
09:27 bauruine joined #gluster
09:29 ndevos passie: do your peers have different UUIDs? check 'gluster peer status' on both servers
09:30 DaveS joined #gluster
09:30 passie Hmmm they don't.
09:30 passie Hostname: 10.248.101.56
09:30 passie Uuid: a105cef2-207b-4526-978c-57d19d8f1ad3
09:31 passie how to correct this ?
09:33 passie oke just format would do it, I thnk
09:34 ndevos the uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time
09:34 ndevos it's a common issue when servers are cloned...
09:34 deepakcs joined #gluster
09:35 ndevos passie: you can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again
09:35 badone joined #gluster
09:36 ndevos JoeJulian: could you add "one of the bricks contain the other" to the glusterbot faq?
09:37 Humble joined #gluster
09:40 mooperd joined #gluster
09:40 mooperd joined #gluster
09:48 Humble joined #gluster
09:49 passie ndevos: the uuid are corrected.
09:49 passie Creation of volume gv0 has been successful. Please start the volume to access data.
09:49 passie thank you
09:49 JoeJulian @help factoid
09:49 glusterbot JoeJulian: Error: There is no command "factoid".
09:49 JoeJulian @list factoid
09:49 glusterbot JoeJulian: Error: 'factoid' is not a valid plugin.
09:49 JoeJulian @list factoids
09:49 glusterbot JoeJulian: alias, change, forget, info, learn, lock, random, rank, search, unlock, and whatis
09:49 JoeJulian @help learn
09:49 glusterbot JoeJulian: (learn [<channel>] <key> as <value>) -- Associates <key> with <value>. <channel> is only necessary if the message isn't sent on the channel itself. The word 'as' is necessary to separate the key from the value. It can be changed to another word via the learnSeparator registry value.
09:50 JoeJulian Or did you mean as an automatic response?
09:50 ndevos yes, like the 'or a prefix' <something> does
09:51 ndevos passie: good to know, thanks for confirming!
09:51 passie On to the next hurdle , always fun to learn something new
09:52 Humble joined #gluster
09:53 ndevos passie: its all about learning and sharing :)
09:55 JoeJulian @learn cloned servers as Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again.
09:55 glusterbot JoeJulian: The operation succeeded.
09:56 JoeJulian @mp add "one of the bricks contain the other" "echo $nick: [whatis cloned servers]"
09:56 glusterbot JoeJulian: The operation succeeded.
09:56 JoeJulian one of the bricks contain the other
09:56 glusterbot JoeJulian: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe
09:56 glusterbot again.
09:58 ndevos JoeJulian: thanks! just a pity the "again." lands on a 2nd line, but oh well
09:59 JoeJulian Yeah. If you want to play with the wording, just have it forget and learn "cloned servers" over again.
10:01 tru_tru joined #gluster
10:01 hchiramm_ joined #gluster
10:03 hchiramm_ joined #gluster
10:15 toruonu joined #gluster
10:15 toruonu So, just saw the 3.4.0-qa5 announcement, did the ext4 patch make it there?
10:15 guest2012 joined #gluster
10:23 vpshastry joined #gluster
10:25 guigui3 joined #gluster
10:31 guest2012 Hello, does anyone know where can I find GlusterFS 3.3.1 packages for Ubuntu Lucid?
10:32 JoeJulian toruonu: I'd be very surprised.
10:33 JoeJulian guest2012: Are there none at download.gluster.org?
10:35 guest2012 JoeJulian, uhm, there I can see the semiosis' PPA reference only
10:35 guest2012 and on the PPA the 3.3.0 version for lucid only, not the 3.3.1
10:36 toruonu JoeJulian: so what does it take to get the ext4 fix into a release. I'd consider it quite a major problem as it blocks kernel upgrades
10:36 * guest2012 having the bad intention to install precise packages on lucid nodes :(
10:36 JoeJulian Hmm, we need to get someone building for that I guess.
10:37 toruonu it would be great to get it into 3.4.0 with all the qemu stuff etc. And it would be perfect before March :) then we could actually build a lot of our infrastructure on top of that
10:37 JoeJulian toruonu: Apparently it's complicated. bulde told me yesterday that he really wants it in 3.4.0 and 3.3.2, so I would imagine he's trying to get that managed.
10:38 toruonu as I'm quite new to gluster, what's the usual release cycle?
10:39 toruonu i.e. approximate timeframes for 3.3.2 and 3.4.0
10:39 guest2012 toruonu, that's a high priority since... 2012-08-17. Unfortunately I haven't the skill to help
10:39 * bulde agrees with JoeJulian
10:39 JoeJulian Tries to be every 3 months, pending approval of course.
10:39 bulde toruonu: soon we will fix it..
10:44 toruonu also, what are plans for the native mount to have all the nice benefits like the NFS one or moving to NFSv4 etc (afaik right now we have to mount with vers=3)
10:45 JoeJulian toruonu: That's a question for the fuse developers. FSCache is a kernel thing.
10:46 ndevos toruonu: the native nfs-server and fuse don't work nicely together, that is a complex fuse problem and I do not think a solution is worked on
10:47 JoeJulian And my answer is assuming you're referring to the FSCache as the "nice benefits". Personally, I don't like stale data.
10:47 toruonu well I don't like missing negative lookups and horribly slow directory listings etc :)
10:47 toruonu didn't fuse get dirstatplus in 3.7?
10:48 JoeJulian I guess I just don't list directories all that often.
10:48 ndevos fscache and fuse could be made to work together, that has been requested as bug 883785
10:48 glusterbot Bug http://goo.gl/FLkUA medium, medium, ---, aavati, NEW , RFE: Make glusterfs work with FSCache tools
10:48 toruonu *readdirplus
10:48 toruonu well it's not listing dirs
10:48 toruonu it's the negative lookups of code in python etc
10:48 toruonu that's what cache helps A LOT :)
10:49 toruonu as you remember from our debugging the code execution times were orders of magnitude different
10:49 toruonu 10s to 10 minutes
10:49 niv joined #gluster
10:49 ndevos fscache only caches contents, not directory entries for all I know
10:49 toruonu well it would be interesting how nfs can list a 30k directory in seconds while fuse based one does it in minutes
10:49 toruonu s/30k/3k/
10:49 glusterbot What toruonu meant to say was: well it would be interesting how nfs can list a 3k directory in seconds while fuse based one does it in minutes
10:50 toruonu won't even think what the 30k directory would do on fuse :)
10:50 toruonu also … don't think I see this as an issue on HDFS, lemme check
10:50 toruonu it's also mounted through fuse
10:50 toruonu [root@wn-d-117 4abb869c2c9729b82d5a6bb08525e8d2]# time ls|wc -l
10:50 toruonu 3122
10:50 toruonu real0m0.298s
10:51 toruonu 3k files in 0.3s and it's a fuse mount
10:51 greylurk joined #gluster
10:52 zoldar how to avoid situation, where filesystem/disk error resulting in device being disconnected for one of the bricks renders whole volume unavailable for clients?
10:53 JoeJulian repliate
10:53 JoeJulian replicate
10:53 vimal joined #gluster
10:53 zoldar this actually is replicated volume, there are 2 bricks
10:54 zoldar however until I disable/reboot the faulty node, the mounts do not work on clients
10:54 JoeJulian @hostnames
10:54 glusterbot JoeJulian: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
10:54 JoeJulian Then you can use ,,(rrdns)
10:54 glusterbot You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
10:55 zoldar so either way I need independent mechanism monitoring underlying filesystem and triggering such failover when needed?
10:55 JoeJulian You're asking two different questions.
10:56 JoeJulian 1. How do I keep my clients connected if a server goes down. 2. How can I *mount* a volume if the server I chose to mount from has gone down.
10:56 JoeJulian So for 1, replication does that automatically.
10:57 JoeJulian Your (fuse) client is always connected to every brick in the volume.
10:57 JoeJulian For 2, rrdns is the easiest. There's some other command line option to specify a failover server too, but I forget what it is.
10:58 hchiramm_ joined #gluster
10:58 toruonu btw I'd want to use NFS due to all the caching that we just discussed, but I'd like it to be gracefully failing over in case there are issues. I'm guessing setting a rrDNS for NFS mount would make sure that the mount can be established independent of wether the initial node is down, but how can I keep it going if the node that was connected to goes away?
10:58 JoeJulian That's typically done with a floating ip, like ucarp.
10:59 toruonu well that's problematic as the same servers do other stuff as well, not just gluster and the client connecting to it will cause issues if the ip moves around as gluster and hdfs aren't quite the same :)
11:00 toruonu and I'd rather not add a specific Ip just for gluster connection
11:00 JoeJulian No, you would specify a unique address that would be used for nfs.
11:00 zoldar JoeJulian: ok, so as I understand it, I meant 1st problem. To clarify. I have a replicated volume consisting of 2 bricks on 2 separate nodes. There's a fuse client connected to them. Then disk on one of the nodes gets detached due to soft/hard error. From this moment on, the fuse client can't see the data until I reboot the node with failed filesystem.
11:01 guest2012 JoeJulian, have you had any news about the destiny of the replace-brick command (w/ reference to the plan not to support it anymore)?
11:02 JoeJulian zoldar: I haven't seen that behavior (and I've had that situation happen more than once). Check your logs for clues. Maybe use ,,(wireshark) to do some further diagnostics.
11:02 glusterbot zoldar: Nixpanic has done some work on a wireshark decoder for glusterfs: http://goo.gl/gXXQ4 and http://goo.gl/8ZrVP
11:02 JoeJulian guest2012: I haven't, but I'm against it.
11:03 nixpanic "some work"
11:03 JoeJulian Yeah, that's kind-of old, isn't it...
11:04 JoeJulian Probably should just be one link now too, huh.
11:04 ndevos @forget wireshark
11:04 glusterbot ndevos: The operation succeeded.
11:04 * guest2012 humbly adding +1 to keep supporting the replace-brick command
11:06 ndevos @learn wireshark as With version 1.8 Wireshark contains close-to-complete support for the Gluster RPC protocols, see http://goo.gl/5Q7Xk for more details.
11:06 glusterbot ndevos: The operation succeeded.
11:08 ndevos ah, wrong link
11:08 ndevos @forget wireshark
11:08 glusterbot ndevos: The operation succeeded.
11:08 JoeJulian Put the long link. glusterbot will shorten it.
11:08 ndevos @learn wireshark as With version 1.8 Wireshark contains close-to-complete support for the Gluster RPC protocols, see http://goo.gl/npWsQ for more details.
11:08 glusterbot ndevos: The operation succeeded.
11:12 guigui3 left #gluster
11:21 rudimeyer_ joined #gluster
11:23 rudimeyer_ When setting up a replica 2 volume, with 2 servers, 2 bricks i get the following error: "one of the bricks contain the other". If i change the brick paths so they are not identical I have no problem. The bricks are newly created with mkfs.xfs so no old stuff is lying around. According to the docs its recommended to have the same naming on all servers? Am I missing something?
11:23 glusterbot rudimeyer_: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe
11:23 glusterbot again.
11:24 rudimeyer_ hehe
11:24 inodb_ joined #gluster
11:26 rudimeyer_ Problem solved, thank you glusterbot :)
11:27 ndevos wow, just added and now first hit already!
11:27 JoeJulian You must be psychic
11:28 toruonu what on the wireshark stuff?
11:28 toruonu I clicked it :p if that's what we're talking about :P
11:29 hagarth1 joined #gluster
11:29 ndevos no, the "cloned servers" bit
11:35 toruonu ah :)
11:50 zoldar I've compiled a couple of log excerpts from the failed brick and from fuse client: http://dpaste.com/844637/ . Does it give any hint as to what is causing such behavior?
11:50 glusterbot Title: dpaste: #844637 (at dpaste.com)
11:52 JoeJulian Aha, zoldar, I see the problem. You were under the impression that the "disk on one of the nodes gets detached due to soft/hard error" but it doesn't seem to. Those errors suggest that the failed disk is still attached causing all sorts of hate and discontent.
11:54 zoldar JoeJulian: the detach actually happend, but from what I understand gluster isn't aware of it - that's just some place in the directory structure. Of course I could terribly misconfigure something in that regard
11:56 zoldar anyway shouldn't IO errors trigger gluster to exclude the faulty brick?
11:56 JoeJulian If the drive was unmounted (and your brick is in a subdirectory of that drive mount) you just get a "brick directory not found" error and glusterfsd ends.
11:57 JoeJulian ... and yes. There's an open feature request for that as well.
11:57 zoldar guess you are right
11:57 zoldar mount was still there, the disk went away
11:57 vpshastry joined #gluster
11:57 JoeJulian Are you using xfs?
11:58 zoldar yes
11:58 pepe123 joined #gluster
11:58 ransoho joined #gluster
11:58 JoeJulian hmm, interesting. xfs unmounted itself automatically in Centos6.3 when my drive failed. I wonder if there's a setting for that.
11:59 ransoho hi, critical probelm anybody here?
11:59 JoeJulian hello
11:59 glusterbot JoeJulian: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
12:00 JoeJulian ransoho: ^^
12:00 ransoho ok, much aprucitated
12:01 ransoho i had split brain and permission problem so decided to run setfattr -x trusted.glusterfs.VOL1 /data/vid1 on the 2 bricks(not the mount)
12:01 ransoho now i cant see the folder form gluster mounts
12:01 ransoho im realy out of options here
12:01 JoeJulian stat vid1 from the client.
12:02 ransoho and its realy big folder so moving it to tmp directory then back into gluster is not an option
12:02 vimal joined #gluster
12:02 zoldar JoeJulian: http://dpaste.com/844642/ seems that in my case it didn't. I don't know what's the use of fs shutdown while leaving it mounted
12:02 glusterbot Title: dpaste: #844642 (at dpaste.com)
12:03 ransoho hi jpe, what do you meam stat vid1?
12:03 JoeJulian vid1 is the directory you can't see, right?
12:03 ransoho yes
12:03 JoeJulian in the client mount, type "stat vid1"
12:04 ransoho it say no suck file or directory
12:04 ransoho ls -iR /mnt/gluster/vid1/
12:04 JoeJulian Have you tried unmounting and mounting again?
12:05 ransoho yes. didnt help ither, gluster cant see the folder anymore
12:05 JoeJulian Can you stop and start the volume?
12:05 JoeJulian which version is this?
12:05 ransoho ill try again, i think i restared it half hour ago
12:06 ransoho only 1 volume / VOL1
12:09 ransoho it looks like it helped
12:09 JoeJulian awesome. I'm going to sleep then. It's 4am.
12:11 ransoho :) many thanks, just to be sure. does trusted.gfid need to be diffrent on the 2 nodes / underling filesystem
12:12 JoeJulian absolutely not
12:12 ransoho its diffrent
12:12 JoeJulian Then you're split brain.
12:12 ransoho trusted.gfid=0sp8D8EipXRMuCctplsQFZNA== & trusted.gfid=0sp8D8EipXRMuCctplsQFZNA==
12:12 ransoho whats does it mean
12:13 JoeJulian Add "-e hex" to the options.
12:13 JoeJulian 'cause that looks identical from here.
12:14 ransoho :) it does, my mistake
12:14 JoeJulian And I thought I was the one who was tired.
12:14 ransoho LOL
12:15 hagarth joined #gluster
12:15 JoeJulian The answer to your question is that the gfid is the inode number used throughout the cluster.
12:16 JoeJulian Also see the the article about the .glusterfs directory on my blog at http://joejulian.name
12:16 glusterbot Title: JoeJulian.name (at joejulian.name)
12:16 ransoho i see many split brains on many big folders and realy dont know how it can happen in our way of working
12:16 * JoeJulian goes to bed.
12:16 JoeJulian @split-brain
12:16 glusterbot JoeJulian: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
12:18 ransoho great ill have a look tnx
12:18 edward1 joined #gluster
12:20 ransoho not good, i see again  0-VOL1-replicate-0: Unable to self-heal permissions/ownership of ........
12:21 ransoho real strange, i reseted the attributes and did full self heal
12:21 ransoho (possible split-brain). Please fix the file on all backend volumes
12:22 ransoho what else can be done to fully fix the folder?
12:24 ransoho i dont get it, on both nodes the gfid of the file is trusted.gfid=0sRivnfCfrQrqcW3ffkDkTyg==
12:24 ransoho but still split brain errors on log
12:27 Humble joined #gluster
12:29 Humble joined #gluster
12:45 puebele1 joined #gluster
12:45 ransoho does anybody know what is the best short way to fix afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 0-VOL1-replicate-0: Unable to self-heal permissions/ownership of
12:46 ransoho Please fix the file on all backend volumes
12:47 ransoho iv allready deleted the folder from one backend then it selfed heal but didnt help then reseted setfattr -x trusted.glusterfs.VOL1 on bothe backends restarted VOL1 but no use, same log errors
12:48 ransoho help anyone?
12:51 ransoho anyone?  Unable to self-heal permissions/ownership of
12:53 spn joined #gluster
12:53 spn joined #gluster
12:54 nhm ransoho: might want to wait a bit, it's pretty early in the US right now..
12:54 nullck joined #gluster
12:55 puebele joined #gluster
12:56 rastar left #gluster
12:58 theron joined #gluster
13:00 Humble joined #gluster
13:03 ransoho anybody know how to remove split brain flag of a folder?
13:04 torbjorn__ joined #gluster
13:04 ransoho with setfattr
13:04 toruonu shoot one of them :P sorry, couldn't resist, no idea though as too young with gluster
13:05 nissim joined #gluster
13:06 Humble joined #gluster
13:07 ndevos ransoho: see http://review.gluster.org/#patch,unified,4216,1,doc/split-brain.txt
13:07 glusterbot <http://goo.gl/hKjRs> (at review.gluster.org)
13:11 Humble joined #gluster
13:16 hchiramm_ joined #gluster
13:16 hagarth joined #gluster
13:26 puebele1 joined #gluster
13:27 Humble joined #gluster
13:27 cyberbootje joined #gluster
13:28 shylesh joined #gluster
13:28 hchiramm_ joined #gluster
13:29 ekuric joined #gluster
13:33 Humble joined #gluster
13:35 spn joined #gluster
13:39 aliguori joined #gluster
13:40 passie joined #gluster
13:42 ngoswami joined #gluster
13:44 Nr18 joined #gluster
13:45 guest2012 Hello, while trying to replace a brick (yes, it's the third attempt) I am seeing a flood on the source server of messages like the following: E [afr-self-heal-common.c:1087:afr​_sh_common_lookup_resp_handler] 0-storage-pump: path [long file path removed] on subvolume storage-replace-brick => -1 (No such file or directory)
13:45 guest2012 is that normal?
13:48 guest2012 (the file obviously exists on the source brick)
13:52 cbehm joined #gluster
13:53 rastar joined #gluster
13:54 chirino joined #gluster
13:55 lh joined #gluster
13:55 lh joined #gluster
13:55 manik joined #gluster
13:56 hagarth @channelstats
13:56 glusterbot hagarth: On #gluster there have been 57433 messages, containing 2553687 characters, 426214 words, 1739 smileys, and 232 frowns; 435 of those messages were ACTIONs. There have been 19815 joins, 720 parts, 19159 quits, 5 kicks, 34 mode changes, and 5 topic changes. There are currently 175 users and the channel has peaked at 188 users.
14:02 Humble joined #gluster
14:07 ctria joined #gluster
14:12 ekuric joined #gluster
14:22 nueces joined #gluster
14:24 rwheeler joined #gluster
14:24 Humble joined #gluster
14:24 x4rlos I just ran gluster for first time. Set up server1 and server2. Server1 hosted 2 partitions on gluster (gv0 and gv1). Mounted server1:/mnt/brick1 onto a test client. Mounted server2:/mnt/brick2 onto same client. Wrote to both mounts fine.
14:25 x4rlos Killed server 1 and then took a few seconds but the gluster moved over to server 2 for both mountpoints. Im really impressed with this :-))
14:27 x4rlos How can i tell if a client has been lost from the volumes? I run a gluster volume info and it just shows as it always does.
14:28 guest2012 not sure about the client, but if a brick is missing you should not see it in gluster volume status
14:30 guest2012 I remember a bug in which the brick was not listed in 'gluster volume status', instead of showing 'Online: N' as one might expect
14:30 guest2012 don't know if the bug still exists
14:30 guigui1 joined #gluster
14:30 Humble joined #gluster
14:38 x4rlos I can see in gluster peer status :-)
14:38 x4rlos Just not in volume status.
14:38 x4rlos (glusterfs 3.2.7 built on Aug  3 2012 22:04:55)
14:38 guest2012 ah, ok. I thought you were on 3.3.x
14:39 x4rlos backports from debian. Just playing with it, apologies.
14:41 x4rlos As the writes aren't synchronous (?) how can i tell where the primary write occurs? So not i have started my server1 back up again, will it continue to write to server2 as primary for both partitions? How is this controlled/shown? Thanks if anyone can help/
14:45 guest2012 I'm no expert, but I think that in 3.3.x the 'primary' brick depends on the order in the creation command line
14:46 ctria Hey all, I have an issue running nagios on gluster volume. I get an error that main configuration is able to be read, while when i do strace to the command i see that open is successful and fstafs64 returns 0. Moving the config / executable to any other FS works fine. Any ideas?
14:46 ctria This is a fuse mount with default mount options
14:48 ndevos ctria: fstatfs returns 0 on success, I doubt that is related
14:49 ctria ndevos, http://fpaste.org/s2Bf/
14:49 glusterbot Title: Viewing Paste #259702 (at fpaste.org)
14:50 ctria could it be any type of locking?
14:51 ndevos ctria: no idea, but the strace show open=ok, fstatfs64=ok, close=ok and write=ok - just the message written is weird
14:53 ndevos ctria: you'll need to find the piece of code that causes that, probably something checks the "struct statfs" and errors out on that
14:54 stopbit joined #gluster
14:54 ctria hm... the executable is 32bit
14:56 ndevos ctria: okay, maybe it puts a value from "struct statfs" in a 32-bit int?
14:57 ctria this is what i'm thinking... I just found that another "nagios" execution seems to work
14:57 ctria but this is 32bit too
14:57 gbrand__ joined #gluster
15:00 x4rlos guest2012: thanks for that.
15:03 rudimeyer_ joined #gluster
15:06 theron joined #gluster
15:11 mdarade joined #gluster
15:13 mdarade left #gluster
15:17 x4rlos When an ls is performed ont he glustered directory - it consults both the gluster servers. Interesting.
15:17 x4rlos (from what i can see from tcpdump oputput).
15:18 mdarade1 joined #gluster
15:21 robinr joined #gluster
15:22 guest2012 x4rlos, that's why http://joejulian.name/blog​/dht-misses-are-expensive/
15:22 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
15:22 robinr Hi, I've got error of "Self-heal daemon is not running. Check self-heal daemon log file". The log file is at: http://pastebin.com/nftAUsV1 The error I got is "[2012-12-12 10:17:07.204112] E [afr-self-heald.c:1128:afr_start_crawl] 0-RedhawkScratch-replicate-0: Could not create the task for 0 ret -1" and "[2012-12-12 10:17:45.393700] W [dict.c:2339:dict_unserialize] (-->/lib64/libc.so.6() [0x381c843630] (-->/usr/lib64/libglusterf​s.so.0(synctask_wrap+0x12)
15:22 robinr [0x381d84b862] (-->/usr/sbin/glusterfs(gluster​fs_handle_translator_op+0x16f) [0x408e5f]))) 0-dict: buf is null!"
15:22 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
15:23 mdarade1 left #gluster
15:28 x4rlos guest2012: Thanks for this. Just wrote content into server1 and then tried to read it and saw the it update too.
15:28 wushudoin joined #gluster
15:28 Norky joined #gluster
15:31 x4rlos guest2012: Interesting, when i change the content on server2 and then check again from client, i get input/output error. eek :-)
15:32 x4rlos Writing back from server1 has broken it :-)
15:32 jbrooks joined #gluster
15:32 guest2012 on _server2_, thus touching the brick _directly_?
15:33 guest2012 _never_ touch the brick directly
15:34 guest2012 only touch data through the NFS or FUSE client
15:34 x4rlos ah cool. I will engrave this into my mind :-)
15:35 x4rlos I dont seem to see any 'heartbeat'-like traffic going from server1->server2. How does it know if the other is dead or not? (maybe i am just missing the messages).
15:35 puebele joined #gluster
15:37 guest2012 x4rlos, dunno
15:38 semiosis someone wanted 3.3.1 on ubuntu lucid?
15:38 semiosis would grumbling about updating to precise have any effect?
15:42 guest2012 semiosis, here I am
15:43 m0zes x4rlos: server1 doesn't care if server2 is dead. the client cares, as it is what writes to both.
15:43 guest2012 update to precise already planned, certainly, but somewhat scared about the possibility that Ubuntu could backport the ext4-killing patch into precise 3.2.x kernel
15:44 wica ext4-killing patch?
15:44 guest2012 wica, http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
15:44 glusterbot <http://goo.gl/PEBQU> (at joejulian.name)
15:44 wica Thnx
15:46 guest2012 let's not forget that lucid is going to be supported until half-2015, being an LTS
15:52 dbruhn joined #gluster
15:55 nissim joined #gluster
15:56 x4rlos m0zes: So to get this right, the client checks the peer group/status and then writes the output to both the servers?
15:57 x4rlos So when i manually (i know i shouldn't) changed the file on server1, cat'd it on the client - it noticed that server2 had different content and wrote it back to server2?
15:58 m0zes x4rlos: don't touch the bricks directly. when you change a file on a brick, you don't update the xattrs that glusterfs uses to keep track of metadata, like status of the file on other peers.
15:59 m0zes it will assume something really went wrong (i.e. split-brain) and the file will not be readable through the client. it waits for you to solve the split-brain situation manually.
16:01 m0zes the only time it can be "safe" to touch the brick directly is in the case of a split-brain. the way to fix that is to identify which brick has the "good" file and delete the file from the "bad" brick.
16:01 dbruhn You need to update the meta file when you delete that file from the bad brick don't you? on 3.3.0 and up anyway?
16:02 * m0zes doesn't run 3.3 and x4rlos is testing with 3.2.7
16:03 m0zes @split-brain
16:03 glusterbot m0zes: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
16:05 wica btw, nice website
16:05 guest2012 you may want to check: https://github.com/jdarcy/​glusterfs/tree/heal-script
16:05 glusterbot <http://goo.gl/Vvrqf> (at github.com)
16:06 guest2012 under extras/heal_script
16:09 balunasj joined #gluster
16:12 nissim hi
16:12 glusterbot nissim: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:13 nissim I am running fedora17 with openstack essex + glusterfs 3.3.1-1 and can't mount the filesystem although from gluster side everything looks fine
16:13 nissim volume is started
16:13 nissim peers are all connected
16:14 nissim my nfs log is full of disconnect msgs and connection refused and I dont really understand why
16:15 nissim my setup includes infiniband equipment and I am trying to make things work over rdma
16:15 nissim can anyone help?
16:15 guest2012 rdma not fully supported on 3.3.1?
16:15 guest2012 is that still true?
16:16 nissim had problems with rdma even with 3.2.7
16:16 nissim tried to fall back to tcp and still have the same problems
16:16 kkeithley only because it didn't get the full QE love that was originally intended. The rdma code has been in there a while and people are using it.
16:17 gbrand_ joined #gluster
16:17 nymous joined #gluster
16:17 nymous hello
16:17 glusterbot nymous: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:17 nissim firewall is disabled, selinux disabled , rpc services up, fuse/rdma modules loaded
16:17 nissim but still same result
16:18 nymous is there any solutions for gluster on top of nas/san?
16:18 mdarade1 joined #gluster
16:18 mdarade1 left #gluster
16:18 kkeithley nissim: how does it work with tcp?
16:18 nymous i have an IBM TotalStorage NAS and several hosts attached to it with fibre channel
16:19 dbruhn nymous: gluster is a NAS
16:19 nissim well, it doesnt, same result although gluster vol info and peer status are ok
16:19 nissim I am knocking my head on the wall with this
16:19 dbruhn if you are using a san as your storage below your hosts you can use gluster on those hosts to present a single name space via gluster
16:19 nymous i have created several logical disk on it and trying to use them with gluster
16:20 x4rlos m0zes: Thanks all. Im still looking at it :-)
16:20 nymous dbruhn: yes, i'm willing to do so... but all my "bricks" are accessible through any of this host. can i have something like gluster multipathing?
16:20 dbruhn What OS are you running and how are you attacking the same to the servers
16:20 guest2012 kkeithley: "full QE love", that's not what one would generally intend as "not supported". I would intend "does not work", for example :)
16:20 kkeithley nymous: yes, I have a few volumes on a SAN that I use for brick volumes.
16:21 nissim anyone?
16:21 nissim anyone using RDMA ???
16:21 nymous i.e. any gluster server will have access to all brick and if one fails, another server will caught up other's bricks
16:22 guest2012 nymous, don't really see the point in having gluster aggregating what is already aggregated. Check also a simple DLM, maybe ocfs2?
16:22 nymous dbruhn: i have 2 servers with opensuse on them, volumes are attached via qla2xxx fc with multipathing
16:23 kkeithley guest2012: I didn't write the docs. In RHS the rdma support is in tech preview, which means, generally, that it works, and you can try it, but Red Hat isn't going to bend over backwards to make it work if it doesn't work for you.
16:23 nymous guest2012: i'm willing to have fast shared storage with load balancing and high availability
16:24 guest2012 nymous, aren't lb and ha provided by the san itself?
16:24 kkeithley Kinda the same story for the community release. I don't love the words, but like I said, I didn't write them.
16:24 nueces joined #gluster
16:24 nymous guest2012: i cannot export same brick on both servers... so if one fails, all attached bricks will fail too
16:26 dbruhn Nymous, doesn't your NAS interface on your storage solution already provide you with a global file system?
16:26 dbruhn or does it not actually have a NAS interface
16:26 nymous moreover, fc has two 2gbit ports, so i think i can get more speed aggregating 2 servers
16:27 nymous dbruhn: it doesn't, it just exports raid array as is as a scsi lun
16:28 guest2012 kkeithley, as you say, the same is valid for community release, so I don't see the point in adding "not supported" specifically on rdma. I'm saying this just in case someone is listening
16:28 mdarade1 joined #gluster
16:28 mdarade1 left #gluster
16:29 mohankumar joined #gluster
16:40 nymous so?
16:40 nymous no multipathing? no chances to export same brick through several servers?
16:41 dbruhn nymous: The gluster server is going to need access, and if you wanted to have redundancy through them, you would need to use the replication, which doubles your storage needs.
16:41 kkeithley gluster is not a shared-disk file system like, e.g. ocfs2.
16:42 nymous i don't wanna replicate something that is already replicated
16:42 dbruhn you could turn off your SAN level replication and provision that storage for your gluster nodes and use the internal replication
16:42 SpeeR joined #gluster
16:42 dbruhn if not, the gluster system will lose a node, and until that node is replaced you will not have access to the data stored on that server
16:43 dbruhn s/node/server
16:43 kkeithley sharing the same brick through several servers would be bad
16:43 nymous not very well for enterprise...
16:43 nymous kkeithley: it's multipathing, it works for gpfs for example
16:44 dbruhn nymous, the system isn't intended to work trying to compete with SAN features, there are plenty of enterprises using it
16:44 dbruhn you just need to decide which technology you want to use, instead of bandaiding a couple things together.
16:45 nymous ok, any performance tuning options for san bricks?
16:46 nymous i've already tried cluster with 2 bricks exported 1 per node, it doesn't perform very well
16:47 dbruhn What kind of volume had you set up
16:47 nymous raid 5 over fc
16:47 dbruhn Sorry, I meant what kind of Gluster volume?
16:48 dbruhn distributed,striped,replicated?
16:48 nymous i've tried distributed and striped
16:48 dbruhn striped is going to be slower than distributed and if you lose a node the whole system will fail in your instance.
16:49 nymous yes, i know
16:49 dbruhn There are no SAN specific tuning options, but in the 3.3.0 admin manual there is a whole section outlining tunings.
16:49 nymous i just willing to have all my nodes loaded...
16:49 dbruhn How are your clients accessing the file system, NFS, native client, SMB?
16:49 nymous tried native client
16:50 dbruhn and what is your network backend between the two gluster servers?
16:50 nymous i have 7 more nodes without a storage attachment and willing to make a shared storage for whole cluster
16:50 nymous it's gigabit ethernet
16:51 dbruhn I am not sure if this will help you, as I don't know what kernel version you are on. http://git.kernel.org/?p=linux/kernel​/git/mszeredi/fuse.git;a=commit;h=81a​ecda8796572e490336680530d53a10771c2a9
16:51 glusterbot <http://goo.gl/T7EUC> (at git.kernel.org)
16:51 nymous i'm on 3.6.5
16:51 dbruhn That might help, you can adjust your small file cache sizes
16:51 dbruhn I think you need to be on 3.7 for that patch
16:54 nymous i have very poor latency times
16:54 dbruhn I am not sure if anyone has attempted RDMA over ethernet, that might be something to look into. It might also be a waste of your time as I am not sure it works for this.
16:54 nymous several seconds on creating files etc
16:57 nymous anyways, it is very bad if gluster doesn't support brick failover
16:57 lh joined #gluster
16:57 itamarjp joined #gluster
16:57 itamarjp joined #gluster
16:58 dbruhn when using gluster replication, it does support a failure
16:58 dbruhn Check into RDMA, I know it does wonders over infiniband, just not sure if it supported over ethernet
16:59 dbruhn and look into going to a 3.7 kernel for that patch and see if it takes care of your latency issues.
16:59 dbruhn Obviously one step at a time to measure the difference in change, and see if it corrects it enough for you
17:04 ransoho joined #gluster
17:05 ransoho joined #gluster
17:10 theron joined #gluster
17:11 Norky joined #gluster
17:12 hagarth joined #gluster
17:19 bauruine joined #gluster
17:23 zaitcev joined #gluster
17:23 hchiramm_ joined #gluster
17:25 x4rlos N00b question: Can gluster apply access constraints like nfs does in /etc/exports?
17:28 Humble joined #gluster
17:32 Mo_ joined #gluster
17:33 dbruhn What kind of constraints? Like limit access to an IP range?
17:34 x4rlos yeah. So if i want to save my database stuff in there.
17:34 x4rlos I could just firewall off the ports, but thought there would be something on the gluster side of things.
17:35 eightyeight what concerns are there for setting up a 3-way replica, where two servers are 3.2.7 and one is 3.2.5?
17:35 dbruhn If you are serving the file system out via NFS you can for sure, I am looking on the native client side
17:35 eightyeight the two are debian testing, and the third is ubuntu 12.04
17:35 m0zes eightyeight: don't do it. your glusterfs servers/clients should all be the same version.
17:36 eightyeight m0zes: what concerns are there with the minor version being off?
17:36 x4rlos dbruhn: I am new to this, and probably very simple.
17:36 mdarade joined #gluster
17:37 dbruhn x4rlos: starts on page 31 of this guide http://www.gluster.org/wp-content/up​loads/2012/05/Gluster_File_System-3.​3.0-Administration_Guide-en-US.pdf
17:37 glusterbot <http://goo.gl/bzF5B> (at www.gluster.org)
17:37 m0zes eightyeight: that has been the advice from every dev when a new version comes out, don't mix your versions. there could be changes in the rpc communication, or just bug fixes that need to be applied everywhere...
17:37 x4rlos dbruhn: aah, i have that open - sorry to be jumping the gun.
17:37 dbruhn looks like the auth.allow option is what you want to set
17:39 x4rlos yes, yes it is. Sorry, i must have skipped over that page somehow. I'm stupid. Thanks.
17:40 rastar joined #gluster
17:40 eightyeight m0zes: understood. i'm just curious of the specifics
17:40 eightyeight m0zes: however, it appears we are going to add the 3.3.1 debian repository, and standardize on that
17:40 eightyeight this is just for a sandbox, and not production data anyway
17:41 eightyeight speaking of which, can you cluster over ipv6, or is that not yet supported?
17:42 lh joined #gluster
17:42 lh joined #gluster
17:44 x4rlos dbruhn: Hmmm. So i can specify an ip range for access, but i guess i have to reject them all, then add the accepted ones?
17:44 x4rlos How can i see these permissions after i set them?
17:46 dbruhn Honestly, that I am not sure. I believe if you create an authorized list it should reject all others. I might be wrong about that
17:46 x4rlos no probs, thankyou. I will have a play and likely continue testing tomorrow.
17:46 errstr joined #gluster
17:46 x4rlos fyi: volume info does display.
17:47 daMaestro joined #gluster
17:47 dbruhn Awesome, I was just about to check that and post the something
17:48 dbruhn s/something/ same thing
17:49 _NiC How much performance-drop should I expect if I run KVM VM's on top of gluster? From what I've found on the web, it's quite significant?
17:49 x4rlos I'm running an old version (3.2.7) But i have restricted the access and attempted mount on the client (over glusterfs) and its let me mount, and even tells me its rw - but now trying to ls the mountpoint and stops.
17:51 m0zes _NiC: depends. with KVM images on a local filesystem, I wouldn't expect better than 20MB/s io. on a glusterfs one, I get ~12MB/s io.
17:52 m0zes imo, the qemu disk i/o stack isn't very fast.
17:53 _NiC m0zes, yeah. We have a stack that runs clvm but that has a san, and for this other stack we'll only have local disks in each server.
17:53 _NiC would 12MB/s be .. "enough" for a handful of lamp-servers?
17:53 _NiC Hard to answer I guess. :-)
17:54 m0zes _NiC: for significant i/o, I mount a glusterfs volume into the vm. it performs wonderfully :)
17:54 _NiC m0zes, I was considering that actually, have just the OS-stuff in the qemu disk, and have stuff like /home mounted over glusterfs.. :)
17:55 _NiC and try to limit all io-stuff such as syslog etc to keep it to a minimum
17:56 m0zes yep, that has been the most performant in my experience, and then in the event of something catastrophic, you don't have to try recovering data from big binary disk image files.
17:57 _NiC Do you have a suggestion for a good method of measuring io speed that I can use to compare?
17:59 JoeJulian need caffeine...
18:01 JoeJulian _NiC: Nothing like the real world to determine if it meet's ,,(Joe's performance metric). ;)
18:01 glusterbot _NiC: nobody complains.
18:02 _NiC :)
18:02 dbruhn I am assuming you can use iozone to do testing
18:03 _NiC yeah, I'll give that a try.
18:03 JoeJulian Can? yes. Is it going to, in any way, model your real usage? What's it going to tell you about any package you test against it? That the package was designed to pass a performance test or that it's treating your data safely?
18:04 _NiC It probably won't reflect real usage, but at least it's a pointer.
18:04 JoeJulian jdarcy has read the source of some of the other packages and is usually disgusted with their lack of regard for data safety.
18:06 JoeJulian My preference (if I had a choice) would be to set up the development system and apply the traffic (scaled to match the scale of the dev system vs production) that you're designing against and measure the results.
18:07 JoeJulian I don't have a choice, so it's straight to production and damned the torpedoes!
18:07 _NiC hehe
18:08 _NiC Can I set up gluster on a single brick and expand to more bricks later?
18:08 JoeJulian yes
18:10 eightyeight so, with my 3-way cluster, i would like to replicate the data twice, using 'distributed replicated' volumes. because it requires the number of bricks to be a multple of the replicate, should i setup two bricks on each peer?
18:11 eightyeight this would be 6 bricks- two on each peer in the cluster
18:11 eightyeight is this sufficient?
18:12 JoeJulian eightyeight: That's essentially what I do.
18:12 elyograg eightyeight: each pair of bricks that makes up a replica set should live on different hardware.  You can set that up on 3 servers the way that you suggest, but you'll have a much easier time with an even number of servers.
18:13 JoeJulian I have a 2 -> 3 replica 2 migration plan on my blog. The ending result sounds like what you're interested in.
18:13 eightyeight elyograg: this is a sandbox for us to learn the tech, and possibly create proof-of-work concepts higher up
18:14 errstr does the order of the command determine the pairing in eightyeight's senario?
18:14 elyograg errstr: yes.
18:14 eightyeight JoeJulian: it's for storing vm images on zfs. figured we could use '/pool/vmsa' and '/pool/vmsb' on each peer as the bricks
18:14 errstr interesting
18:15 deckid joined #gluster
18:17 theron joined #gluster
18:17 elyograg eightyeight: this is how I would do it: server1:/pool/vmsa/vol server2:/pool/vmsa/vol server2:/pool/vmsb/vol server3:/pool/vmsb/vol server3:/pool/vmsc/vol server1:/pool/vmsc/vol
18:18 eightyeight elyograg: why?
18:18 elyograg the additional /vol is protective - if one of the mount points were to disappear and you don't do that, gluser will happily replicate onto your root filesystem.
18:19 elyograg eightyeight: why on which part?
18:19 eightyeight elyograg: the order of the bricks in that way
18:19 sjoeboo quick question: Can I mount a sub directory of a gluster volume?
18:20 semiosis afaik you can with nfs but not with the fuse client
18:20 elyograg eightyeight: if you're doing replica N, each sequence of N bricks on the commandline specifies a replica set.
18:20 sjoeboo crap
18:20 semiosis sjoeboo: however you can mount the volume one place & bind mount a subdir somewhere else
18:21 sjoeboo yeah...hrmmm
18:21 JoeJulian elyograg, eightyeight: I would do server1:/pool/vmsa/vol server2:/pool/vmsa/vol server2:/pool/vmsb/vol server3:/pool/vmsb/vol server3:/pool/vmsa/vol server1:/pool/vmsb/vol
18:22 eightyeight JoeJulian: why that order of bricks? just curious.
18:22 eightyeight oh, i see
18:22 elyograg JoeJulian: that would certainly work, but you lose the "at a glance" knowledge that vmsb on server2 and vmsb on server1 have no connection to each other.
18:23 JoeJulian When you add your 4th server and migrate everything all over again that's still going to be confusing anyway.
18:24 rastar left #gluster
18:27 elyograg JoeJulian: true enough.  I'll be avoiding the whole problem by never adding one server at a time.
18:30 mohankumar joined #gluster
18:36 theron_ joined #gluster
18:36 copec Does gluster work with RoCE or the soft-RoCE rdma?
18:42 JoeJulian Not sure, and getting conversation out of Raghavendra about rdma specifics isn't the easiest.
18:42 JoeJulian He means well, just is one of those type of people that replies to the first line in any email.
18:45 hchiramm_ joined #gluster
18:47 Alpinist joined #gluster
18:55 dstywho joined #gluster
18:56 y4m4 joined #gluster
18:57 puebele1 joined #gluster
18:59 andreiw left #gluster
19:04 deckid I'm running into an issue where a new ditstribute-replicate mount is duplicating directories.  I search around the irc logs and community with no luck.  Was hoping someone here had a direction they could point me.
19:05 deckid I'm seeing even "." and ".." four times.
19:06 deckid I verified that the host servers are running the same gluster version as the client.  Just not sure where to start on this one.  Any file that is created is only created once.  Any directory shows up 4 times.
19:08 elyograg deckid: I had this problem when I first set up a volume. I had done something wrong on my second node.  I no longer remember what I did wrong, but when I fixed it, the problem went away.
19:09 deckid OK, I'll try to recreate the volume.  Maybe I just fat fingered something.
19:26 bauruine_ joined #gluster
19:34 bauruine joined #gluster
19:35 hchiramm_ joined #gluster
19:39 noob2 joined #gluster
19:39 noob2 anyone see that pnfs 4.1 was merged into the 3.7 kernel?
19:42 noob2 nvm, it requires metadata servers. no longer excited about it haha :)
19:42 noob2 gluster rules
19:45 blubberdi joined #gluster
19:47 _NiC Is there any particular filesystem I should use for gluster?
19:47 semiosis xfs
19:47 semiosis with inode size 512 (or 1024 if using UFO)
19:48 semiosis thats the last recommendation i remember hearing from devs
19:48 _NiC what's UFO?
19:48 kkeithley _NiC: xfs. If you're feeling brave, btrfs. Otherwise ext4 will do
19:48 kkeithley UFO is Unified File and Object
19:48 daMaestro hey kkeithley
19:48 kkeithley basically it's OpenStack Swift + gluster
19:49 semiosis kkeithley: ext4, really?
19:49 semiosis ,,(ext4)
19:49 glusterbot Read about the ext4 problem at http://goo.gl/PEBQU
19:49 kkeithley yeah, yeah, that's a kernel problem. ;-)
19:49 _NiC :)
19:49 kkeithley Notice I said xfs and btrfs first
19:50 _NiC I've never actually used xfs before. Is it scary? :-)
19:50 semiosis only around halloween when it dresses up like zfs
19:50 kkeithley reiser is the only scary fs IMO
19:50 daMaestro we've had major issues with xfs on EL5
19:50 daMaestro had to go back to ext3
19:51 kkeithley el5 I can believe. It's a lot better in newer distributions
19:53 elyograg _NiC: in the past I have seen problems with xfs filesystems getting major corruption with power outages, especially on databases.  now that barrier support exists, or if you have a hard drive controller that's got nonvolatile (or battery backed) memory, this should be far less of a problem.
19:55 semiosis or... glusterfs replication?
19:55 * semiosis hopes
19:58 * toruonu sighs… mellanox just asked tcpdumps from ALL machines simultaneously. I have 200 of them…
19:59 semiosis parallel-ssh ftw
19:59 toruonu out of curiosity (trying to avoid taking the dumps), anyone know of a good way to understand what the f*** floods my network? :)
20:00 toruonu all mellanox got from the switch dumps is that something does it and that interfers with traffic even within same switch intra port… instead of 10G I get sometimes nice 9.5G sometimes variations from 10Mbit/s to 3Gbit/s (nodes don't do anything else). So something spams the net…
20:00 gbrand_ joined #gluster
20:02 semiosis toruonu: you could add an iptables rule that would not affect traffic flow but would count matching packets, then query iptables on your machines to find out which one sent the most during the storm
20:02 semiosis i'd go with an output chain rule for that
20:03 semiosis again, parallel-ssh for adding the rule to all machines & also later for querying iptables
20:03 JoeJulian cssh is what I use.
20:03 semiosis but if you have 200 machines you probably already know about parallel-ssh & similar tools
20:04 toruonu semiosis: indeed pssh is indeed a blessing :)
20:05 toruonu then again not all nodes have similar configs and allow host/key based login guess I need to add those
20:05 semiosis toruonu: the iptables counter is probably the most lightweight to get some insight, but something like collectd would probably give you much more useful data
20:08 semiosis i saw a packetstorm once caused by a bad linux workstation, idk if it was a bad nic or some bad software combination but the result was this dev's workstation flooding the network with a tcp ACK packet
20:08 toruonu how'd you track it down?
20:09 toruonu I'm guessing it's probably one or two nodes spamming somehow, but it's a bitch to find out...
20:09 semiosis jumped on the office firewall to see who was flooding our T1, saw the dev's IP
20:09 semiosis that was easy to track down
20:10 mooperd joined #gluster
20:10 toruonu ah … we don't use firewalls as we need high throughput … the fabric we've built should work at least at 160Gbit/s, with some localization even higher. But right now we're seeing utter crap performance where per stream copies of 7GB fail in 1h etc ...
20:11 toruonu ok, will contemplate those hints. For tonight living in EU I'm off to sleep, will continue tracking this in the morning
20:31 kai_ joined #gluster
20:33 kai_ Hello, I'm seeing a weird probe behavior, here is what I did. I installed glusterfs-server-3.3 on Ubuntu 12.04 / AWS instance. Bake an image from that instance, then spin up 4 servers from that image.
20:34 semiosis you copied the glusterfs server uuid in /var/lib/glusterd/glusterd.info which must be unique on all servers
20:34 semiosis if it's missing glusterd will generate a new random one at service start
20:34 kai_ so before creating the image, I should delete /var/lib/glusterd/glusterd.info?
20:36 semiosis i guess so
20:36 semiosis also are you using the latest glusterfs, 3.3.1, from the ubuntu-glusterfs-3.3 repo?
20:36 kai_ thanks. That explains. Yes, thanks for your Ubuntu PPA and build
20:37 semiosis yw
20:42 nhm anyone seen jdarcy lately?
20:42 semiosis nhm: https://twitter.com/Obdurodo​n/status/278739191137984513
20:42 glusterbot <http://goo.gl/L2pR7> (at twitter.com)
20:43 _NiC How can I create a replicated volume on a single node with a single brick (but that will later be expanded)?
20:43 nhm semiosis: aaah, ok.  He's normally busy but usually responds sooner or later. :)
20:48 SteveCooling i'm a bit stuck here. got a brand new gluster with one volume. mounts fine, shows up in "df", but "ls" freezes. RHEL6 and glusterfs 3.3.2. mounted via fuse.
20:48 JoeJulian @ext4
20:48 glusterbot JoeJulian: Read about the ext4 problem at http://goo.gl/PEBQU
20:49 semiosis nhm: tweet him :)
20:50 SteveCooling JoeJulian: ouch
20:50 JoeJulian Damned kernel devs. ;)
20:50 semiosis 3.3.2 is out?
20:50 semiosis @latest
20:50 glusterbot semiosis: The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
20:51 JoeJulian I was wondering about that too. I didn't see a release from kkeithley...
20:51 SteveCooling 3.3.1...
20:51 SteveCooling sorry, typo
20:51 semiosis :)
20:51 nhm semiosis: I'm so in the stone age, I signed up for twitter because our marketting guys made me. ;)
20:53 SteveCooling one of my 4 SuperMicro FatTwin nodes died during RHEL6 install. and then ext4 breaks between staging and production install
20:53 JoeJulian Ugh!
20:54 JoeJulian Well, just put xfs on your bricks or roll back the kernel and you'll be fine.
20:54 SteveCooling i think i'll go the xfs route...
20:54 SteveCooling thanks
20:57 peterlin joined #gluster
20:58 peterlin Hello, I seem to have a situation after a brick went up and down where a file seems to be locked, and libvirtd doesn't work. How can I find out where the lock is held, and clear it?
20:58 peterlin This is on version 3.3.0
20:59 nissim joined #gluster
21:08 andreask joined #gluster
21:17 nhm SteveCooling: I always wanted to try gluster on ZFS.
21:18 _NiC cool, I have a replica volume up and running! :-) Two bricks on the same server, but still..
21:18 SteveCooling nhm: and now I notice xfs needs extra entitlements on RHEL6... gonna sleep on whether i'm switching to Centos in the morning
21:19 blubberdi Hi, I've set up a gluster replicated volume with two bricks. But if I now start my application it take 6-7 minutes to start instead of 10 seconds. Here is the paste of `gluster volume profile www info` http://pastie.org/private/8qmtun9tgaadqzpfouwdpq . As far as I can tell the problem are the much "LOOKUP" Fop's. Is there any way I can improve the performance?
21:19 glusterbot <http://goo.gl/cRpxX> (at pastie.org)
21:19 nhm SteveCooling: Wow, that's nuts.  I'm lucky enough that I get to mostly work in ubuntu or debian.
21:20 _NiC I don't really see any reason to not go with centos, assuming you have skills/resources to fix problems without support from RH
21:22 nhm _NiC: it's tough.  On one hand it's all open source, on the other hand, it's nice to get paid for all the work that goes into RHEL.  Ethically it's a challenging problem.
21:24 _NiC that's true.
21:24 SteveCooling we have not needed RH support since we moved to RHEL almost 6 years ago. but I'm sure good old murphy will find a way
21:24 _NiC we have a bunch of rhel servers still, but most of our new stuff is centos.
21:25 _NiC we used to be 1-2 people doing this stuff though, so we didn't have time to dive into any problems that might show up (not that we had very many)
21:25 er|c joined #gluster
21:25 er|c joined #gluster
21:25 SteveCooling run centos on all non-critical
21:25 _NiC now we're 8-9, so we're a bit  more self-fixing
21:25 SteveCooling i'm all alone on this here :-/
21:26 nhm The shop I used to be in originally used RHEL, then SLES, then moved to CentOS.
21:27 nhm Now it's primarily ubuntu which is a lot nicer for my needs anyway.
21:27 semiosis nhm: what do you use on your desktop?
21:27 nhm semiosis: Ubuntu
21:27 nhm thinking about switching to Mint.
21:27 semiosis switch to kubuntu :)
21:28 nhm Naw, I'm an old gnome2 diehard. :)
21:29 nhm I'm probably ending up on some unity/redhat hitlist right now. ;)
21:29 * m0zes runs gentoo on all his servers :D
21:30 nhm m0zes: one of the guys in our irc channel was espousing the benefits of arch last night.  :)
21:31 _NiC We had issues with ubuntu on servers :-\
21:32 _NiC is it upstart they call it? That was our problem
21:32 peterlin I really need a hand with my deadlock issue :/. What can I do to resolve it? the cluster is currently self-healing the files i think the problem is with.
21:33 _NiC m0zes, heh, we had a guy who ran gentoo on his desktop. He was sent a .doc once, and he said he'd read it in a couple of days once openoffice had compiled .. :-)
21:33 peterlin Libvirtd hangs when i try for instance "virsh list"
21:41 badone joined #gluster
21:44 peterlin How can I remove the blocked locks? I can remove the files completely from the bricks if need be, if I can find the entry in .glusterfs also I guess
21:46 peterlin If I try to rm the file from the mountpoint, rm hangs
21:48 mooperd left #gluster
21:50 _NiC I'd love to help, but I'm clueless, sorry.
21:50 nhm As am I. :(
21:50 nhm I mostly provide comic relief.
21:52 JoeJulian peterlin: ,,(ext4)?
21:52 glusterbot peterlin: Read about the ext4 problem at http://goo.gl/PEBQU
21:53 peterlin Haven't changed kernel, and it was working right before I started the second brick again..
21:54 _NiC humm. I just rebooted my server and when I now mount the gluster volume, it's 2GB instead of 50GB? :-\
21:54 JoeJulian _NiC: Sounds like your brick didn't mount.
21:55 _NiC ah. that sounds correct.
21:56 _NiC yes. much better now. :-)
21:56 peterlin I would just like to completely remove the offenting file, (the only one with BLOCKED locks listed in the statedump), or clear the locks somehow
21:56 JoeJulian peterlin: There's clear-locks commands in 3.3
21:57 blubberdi JoeJulian: (looks like you back from idle and know gluster :) ) Do you have an idea for my problem? (~ 50 line above)
21:57 peterlin I'm trying to use it, but I don't know how to :/
21:59 JoeJulian ~php | blubberdi
21:59 glusterbot blubberdi: php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details.
22:00 JoeJulian blubberdi: Also check out http://joejulian.name/blog​/dht-misses-are-expensive/
22:00 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
22:01 blubberdi JoeJulian: Thank you! I read it.
22:02 peterlin What would happen if I just remove the file from both bricks?
22:03 JoeJulian peterlin: I don't know either. When I tried using it once (successfully btw), I just ran them all. volume clear-locks myvol /lockedfile1.img kind all inode, then replacing the last word with "entry" then "posix"
22:03 JoeJulian And removing the file won't clear the locks.
22:03 JoeJulian They're stored in memory on all the brick servers.
22:04 peterlin When I run that i get: clear-locks getxattr command failed. Reason: Transport endpoint is not connected
22:04 JoeJulian restart glusterd on your servers
22:07 peterlin Can I do that safely? I have the filesystem mounted and lots of VM's running from it, just no way to control it since libvirtd won't start :/
22:07 JoeJulian Yes, glusterd is just the management service
22:09 peterlin JoeJulian: Restarted the processes on both peers, but the command still fails :/
22:12 JoeJulian Still transport endpoint not connected?
22:13 peterlin Yes.
22:14 peterlin But the mount-point on both machines are working, and it seems to be self-healing still.
22:14 _NiC I've added _netdev in /etc/fstab on my mountline for the volume, but it still doesn't mount on boot.. what am I missing?
22:15 _NiC err.. never mind.
22:15 _NiC typo :)
22:15 kai_ I have a question about recovering a lost drive/server. Say I have a distributed replica 2 x 2, and one of the server is gone for whatever reason. Is it possible to have a brand new server to replace its position and recover? I tried volume replace command but it just hangs since the server is gone.
22:19 JoeJulian peterlin: Check the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file on your servers and see if it has any clues. ENOTCONN seems odd.
22:20 hattenator joined #gluster
22:20 JoeJulian kai_: Yes, just do the replace-brick command and add the word "force" to the end.
22:20 _NiC hm, no, with _netdev in the correct place it still didn't work.
22:20 _NiC when I do mount -a, it says unknown option _netdev (ignored)
22:20 peterlin JoeJulian: I have this for you: 0-glusterfsd: /var/lib/glusterd/vols/vmstora​ge/trusted-vmstorage-fuse.vol: No such file or directory
22:20 _NiC (but it gets mounted)
22:20 peterlin JoeJulian: My volume was once upgraded from 3.2.6, perhaps I missed something?
22:22 JoeJulian peterlin: Is this 3.3.1?
22:22 JoeJulian now
22:22 peterlin No, 3.3.0
22:23 JoeJulian Ah, ok. There's your problem.
22:23 peterlin I'm assuming there is a solution? :)
22:23 JoeJulian besides upgrade? ;)
22:24 peterlin Well, I'd rather just get everything working as soon as I can :/. What is that missing file doing and what can I do to fix it?
22:24 JoeJulian I reported bugs against that stuff that all seems to have been fixed in 3.3.1. Kill glusterd on your servers. Pick on that has a good /var/lib/glusterd/vols tree and rsync it to the other server(s). Start glusterd.
22:25 peterlin I have a vmstorage-fuse.vol, but no trusted-vmstorge-fuse.vol nowhere
22:25 mooperd joined #gluster
22:26 peterlin I remember something about setting a password before that I did not do?
22:26 JoeJulian Oh, I wonder if that's...
22:27 JoeJulian glusterd --xlator-option *.upgrade=on -N
22:28 peterlin That command was executed when I upgraded, just like in the guide here: http://vbellur.wordpress.com/2012/​05/31/upgrading-to-glusterfs-3-3/
22:28 glusterbot <http://goo.gl/qOiO7> (at vbellur.wordpress.com)
22:29 peterlin what is supposed to be in the trusted-vmstorage-fuse.vol file?
22:32 JoeJulian looks to be exactly the same as the fuse.vol without "trusted-"
22:35 peterlin All right, i'll just make a copy of that, then...
22:37 JoeJulian brb... gotta go change a diaper. :S
22:44 peterlin JoeJulian: Mine's asleep already :). All right, that allowed me to clear the locks on that file, but libvirtd still won't start :/
22:47 JoeJulian peterlin: Do you have any zombies on that machine?
22:48 peterlin Yes, the libvirtd process that stopped working (and i tried /etc/init.d/libvirtd-bin stop)
22:48 JoeJulian Damn, I knew it...
22:50 peterlin The original process seems to have crashed like that when I started the brick that wen't offline...
22:51 peterlin If a lock is aquired on a file as it is being healed?
22:53 peterlin Or the fact that i'm in sweden and its already midnight and babies are sound asleep?
23:07 manik joined #gluster
23:10 peterlin JoeJulian: ?
23:15 chacken1 joined #gluster
23:27 daMaestro joined #gluster
23:38 TSM joined #gluster
23:38 JoeJulian Sorry peterlin, my daughter pulled me away.
23:40 peterlin I know what it's like. I have a 1 year old myself. When they pull they pull :)
23:41 nhm 5 year old son and 3 year old daugher.  She was just climbin on my back shouting "go elephant go!"
23:41 JoeJulian So just libvirtd is zombied, huh? No qemu processes?
23:42 peterlin Yes, all the kvm processes are running fine (although slow, because of the self-heal in progress on that filesystem)
23:44 JoeJulian I wonder if we could kill it with sysrq
23:45 peterlin I'm not sure what the libvirtd process does, but it hangs as soon as I do a virsh list or try to connect using virt-manager. Before I brought the missing brick online again it worked fine
23:45 yinyin joined #gluster
23:45 JoeJulian Yeah, once one's zombied, starting another will always do that.
23:46 peterlin I have killed it before, when I cleared those locks before a few zombie processes went away..
23:46 JoeJulian Maybe there's more locks?
23:46 peterlin let me see
23:47 peterlin No, no more locks in BLOCKED state.
23:49 peterlin Ah, there was some on the other brick?
23:49 JoeJulian Could be.
23:49 JoeJulian theoretically, iirc, you should be able to find the pid of the zombie in the dump file.
23:54 y4m4 is 'Eco Wilson' around here?
23:55 peterlin JoeJulian: I clearned locks I found on another brick, but when I make a new statedump the locks show up as blocked again almost right away.
23:55 peterlin This is on files currently being healed.
23:57 semiosis @later tell Technicool y4m4 was looking for you
23:57 glusterbot semiosis: The operation succeeded.
23:58 semiosis y4m4: sometimes, but apparently not right now

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary