Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-15

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:32 mricon yes, if it needs to execute commands as root, it would be silly to do anything else.
00:36 mricon looks like arcfour is the fastest cipher to use. I'll try with that.
00:55 mohankumar joined #gluster
00:59 hybrid5122 joined #gluster
01:00 chlunde joined #gluster
01:05 copec joined #gluster
01:06 yinyin joined #gluster
01:07 jdarcy joined #gluster
01:09 _pol joined #gluster
01:22 plarsen joined #gluster
01:37 jules_ joined #gluster
01:49 kevein joined #gluster
01:49 kevein_ joined #gluster
01:50 kevein joined #gluster
01:58 m0zes if you can mount the *remote* geo-rep volume from a box in your primary trusted pool, you can use something like gluster://<ip>:volname as your uri for geo-rep.
01:58 jules_ joined #gluster
01:58 yinyin hi. if enable eager lock but without write-behind, how to avoid overlapping write?
02:07 lkthomas joined #gluster
02:07 lkthomas hey guys
02:08 lkthomas does gluster be able to breakdown big files into smaller pieces ?
02:09 m0zes lkthomas: with the ,,(stripe) translator
02:09 glusterbot lkthomas: Please see http://goo.gl/5ohqd about stripe volumes.
02:12 lkthomas m0zes: so basically I need to create a mirror gluster then stipe it ?
02:12 lkthomas stripe
02:13 m0zes lkthomas: one of the *few* things stripe gains you is splitting files into multiple chunks on the backend seemlessly to the client. I haven't used it, but for most it isn't what they want. you don't *have* to mirror, but I'd recommend it.
02:14 m0zes lkthomas: why do you want this?
02:15 lkthomas actually I am planning to use gluster distributed advantage to store backup file
02:15 lkthomas the backup is huge, like 500GB each
02:15 lkthomas not sure if it will evenly distribute on the storage pool
02:17 m0zes fair enough. there are oddities from what I understand. the first stripe of a file tends to be on the first brick in of your fs. this may cause issues for you. you'll have to test and find out.
02:17 disarone joined #gluster
02:17 lkthomas m0zes: so gluster does not break down files ?
02:19 m0zes it does with the stripe translator.
02:19 lkthomas what am I going to see on the backend file system?
02:20 lkthomas m0zes: do you know what's the breakdown size ?
02:21 m0zes lkthomas: it is configurable http://gluster.org/community/documentation/index​.php/Gluster_3.2:_Setting_Volume_Options#cluster.stripe-block-size
02:21 glusterbot <http://goo.gl/m7cZ5> (at gluster.org)
02:21 m0zes lkthomas: like I said, though. you'll need to test to figure out how it will behave.
02:21 lkthomas ok
02:22 m0zes the way I understand it is that it makes the files sparse on the backend, and fills the relevant portions with the striped data
02:23 lkthomas m0zes: does gluster fs ever corrupt at all ?
02:24 m0zes lkthomas: I've not seen it happen, but that doesn't mean it can't.
02:24 lkthomas LOL, ok
02:27 lkthomas m0zes: something I still confusing
02:27 lkthomas if I use mirror, I lost 50% of capacity
02:27 lkthomas if I use stripe, I lost redundancy
02:28 lkthomas anything could fall in middle ?
02:28 lkthomas like this:
02:28 lkthomas disk 1: 1-2-3-4
02:28 lkthomas disk 2: 3-4-5-6
02:28 lkthomas disk 3: 5-6-7-8
02:28 lkthomas ...etc
02:28 m0zes not really. redundancy in this instance is replication. there is no raid-5 equivalent in glusterfs.
02:29 lkthomas so it doesn't do partial replication
02:29 m0zes no.
02:30 kkeithley1 joined #gluster
02:30 lkthomas so basically you guys is losing 50% capacity
02:37 lkthomas_ joined #gluster
02:38 lkthomas joined #gluster
02:38 lkthomas sorry, press the wrong button :P
02:40 jclift joined #gluster
02:41 lkthomas I see, so other than mirror, stipe, there is something call "none"
02:44 lkthomas m0zes: basically, if I use none, file will distribute across volume, when the hard disk is dead, I only lost one part of data, correct ?
02:45 Humble joined #gluster
02:46 m0zes with a purely distribute volume, you would lose the files on *that* brick if your brick dies. I use RAID-5 for my bricks, and use mostly distributed volumes.
02:47 lkthomas m0zes: so basically you suggest to go for RAID-5 first then use stripe or none
02:47 lkthomas cost is too high on mirror
02:48 m0zes that is what *I* use, I cannot say that it is or isn't right for your use case. I am just trying to provide you with the facts of what the tradeoffs are. with a distributed volume, you files *must* be smaller than your individual bricks.
02:50 lkthomas files must be smaller than bricks <- does it apply to stripe ?
02:51 m0zes that doesn't apply to stripe, due to the way it splits the data into chunks on each brick.
02:51 lkthomas m0zes: ok, I got the concept now. thanks
03:04 Humble joined #gluster
03:25 mooperd joined #gluster
03:27 lpabon joined #gluster
03:43 mooperd joined #gluster
03:51 sjoeboo joined #gluster
03:51 _pol joined #gluster
03:59 mohankumar joined #gluster
03:59 bharata joined #gluster
04:04 bulde joined #gluster
04:06 saurabh joined #gluster
04:09 aravindavk joined #gluster
04:09 aravindavk joined #gluster
04:10 Humble joined #gluster
04:18 aravindavk joined #gluster
04:21 shylesh joined #gluster
04:22 sgowda joined #gluster
04:34 mohankumar joined #gluster
04:38 harshpb joined #gluster
04:40 shireesh joined #gluster
05:03 Ryan_Lane joined #gluster
05:03 vex can anyone tell me what 'E [afr-self-heald.c:685:_link_inode_update_loc] 0-storage-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)'
05:03 vex errors mean, in my glustershd.log file?
05:03 sripathi joined #gluster
05:03 vex thousands and thousands streaming by
05:12 bala joined #gluster
05:16 f0urtyfive anyone know how I can get configure to enable georeplication on solaris?
05:17 sgowda joined #gluster
05:22 lalatenduM joined #gluster
05:24 sahina joined #gluster
05:26 satheesh joined #gluster
05:31 bulde joined #gluster
05:34 sjoeboo joined #gluster
05:35 vshankar joined #gluster
05:40 raghu joined #gluster
05:45 sripathi joined #gluster
05:51 bala joined #gluster
05:56 shireesh joined #gluster
05:56 sgowda joined #gluster
05:59 lalatenduM joined #gluster
06:03 sripathi joined #gluster
06:03 rastar joined #gluster
06:03 bala joined #gluster
06:04 sripathi1 joined #gluster
06:06 f0urtyfive I'm getting a compile error on solaris
06:06 f0urtyfive first error is "nlm4-xdr.h:95: error: syntax error before "u_int32_t""
06:07 satheesh joined #gluster
06:15 harshpb joined #gluster
06:17 timothy joined #gluster
06:19 bulde joined #gluster
06:23 bharata joined #gluster
06:31 20WAB4F92 joined #gluster
06:32 f0urtyfive so what, When redhat "sponsored" the project they decided it shouldnt be usable on solaris anymore?
06:37 samppah f0urtyfive: not really, i just think there isn't any active solaris development going on
06:38 vimal joined #gluster
06:45 f0urtyfive samppah: https://bugzilla.redhat.com/show_bug.cgi?id=812488
06:45 glusterbot <http://goo.gl/1L0xO> (at bugzilla.redhat.com)
06:45 glusterbot Bug 812488: medium, low, ---, rajesh, ASSIGNED , Compile error on nlm4 section
06:50 kkeithley_blr IIRC correctly, it didn't build on Solaris even before — long before — Red Hat sponsored the project, i.e. before it bought Gluster, Inc.
06:50 timothy joined #gluster
06:53 kkeithley_blr About once every two years someone comes along and asks about Solaris. What should we conclude from that about where to spend our time?
06:54 f0urtyfive errr?
06:54 f0urtyfive Why does this exist then? http://gluster.org/community/docum​entation//index.php/Gluster_3.1:_I​nstalling_GlusterFS_on_OpenSolaris
06:54 glusterbot <http://goo.gl/k5OLn> (at gluster.org)
06:55 kkeithley_blr Seriously? 3.1 is how old? We shipped 3.3 over six months ago and we're about to ship 3.4 in a few weeks.
06:57 satheesh joined #gluster
06:57 kkeithley_blr It's up to the community to make it work, to keep it working on things like Solaris.
06:57 f0urtyfive no idea how old it is
06:58 rotbeard joined #gluster
06:58 f0urtyfive kkeithley_blr: while that is a valid point, why is a bug that breaks the build on a previously working OS considered "low" just because that OS isnt Redhat?
06:59 f0urtyfive fyi, I found this, which looks like it will provide the necessary fixes: http://nanohikari.blogspot.com/2012/07/in​stalling-glusterfs-33-on-openindiana.html
06:59 glusterbot <http://goo.gl/khlBw> (at nanohikari.blogspot.com)
06:59 stickyboy joined #gluster
07:00 f0urtyfive I dont even like solaris, but ZFS is worth it
07:01 kkeithley_blr We fix bugs all the time on non-Red Hat OSes. File a bug. Ask nanohikari why he didn't file a bug with his fixes.
07:01 glusterbot http://goo.gl/UUuCq
07:01 displaynone joined #gluster
07:01 f0urtyfive hm? I linked to the bug above
07:02 f0urtyfive I'm building it now to see if those changes worked, if they do I'll stick something in that existing bug about it
07:02 kkeithley_blr That would be great
07:02 hagarth joined #gluster
07:03 Nevan joined #gluster
07:17 ngoswami joined #gluster
07:23 jtux joined #gluster
07:38 stickyboy I think I have a recursive loop or something in my gluster volume.
07:38 stickyboy ls over fuse seems to hang, but if I `strace -p` I can see that it's listing the same files over and over.
07:41 samppah stickyboy: anything in the log files?
07:42 shireesh joined #gluster
07:42 Ryan_Lane joined #gluster
07:45 stickyboy samppah: Nope... :\
07:46 stickyboy samppah: But strace on the hung `ls` process looks like this:   http://pastie.org/pastes/6508794​/text?key=ekc9pkvtvhjjsjot5fpy4w
07:46 glusterbot <http://goo.gl/3NtrG> (at pastie.org)
07:46 ctria joined #gluster
07:57 harshpb joined #gluster
07:59 ekuric joined #gluster
08:17 tjikkun_work joined #gluster
08:19 stickyboy samppah: Ah, I just mounted my volume via NFS and I see this crazy mirror-in-a-mirror effect:
08:19 stickyboy http://pastie.org/pastes/6509133​/text?key=exxoijeqvy2epkm81jvocw
08:19 glusterbot <http://goo.gl/kb04O> (at pastie.org)
08:21 sripathi joined #gluster
08:30 samppah stickyboy: uhh.. can you check on backend filesystem if there is anything obvious that is causing it?
08:33 stickyboy samppah: Sure.  I can ls -l the backend mount points to see if permissions are weird maybe?
08:34 sripathi1 joined #gluster
08:35 bharata joined #gluster
08:40 stickyboy samppah: Is there any problem with mounting the fuse mount points on the storage servers themselves?
08:41 samppah stickyboy: not that i know and i haven't experienced any issues with that.. with nfs you might run into some issues
08:42 samppah stickyboy: just to make sure, your backend filesystem and glusterfs filesysten are mounted to different mount directories?
08:43 stickyboy samppah: Yeah, my raw ext4 mounts are somewhere like /mnt/gluster/server1/sda1/ and my client mounts are generally to places like /home
08:44 stickyboy Interesting.  I just mounted a new volume as NFSv3 and copied over data, now I can ls it just fine.
08:45 samppah stickyboy: oh.. are you aware of ext4 bug?
08:45 mooperd joined #gluster
08:48 Staples84 joined #gluster
08:48 stickyboy samppah: I'm aware of it, yeah.  I heard that GlusterFS 3.4 will have an appropriate fix to mitigate it.
08:49 stickyboy samppah: Interesting, do you think the ext4 bug might be involved here?
08:50 samppah stickyboy: sorry i have no idea how it affects :/
08:50 andreask joined #gluster
08:51 stickyboy samppah: Ok.
08:51 stickyboy Ah, actually... I just noticed in my NFS mount point that there are a bunch of duplicates.
08:52 stickyboy Lemme look up the ext4 issue... see if it had something to do with this.
08:53 stickyboy samppah: "Since the dictionary entry did not have a cached offset, it would try to create one again and would end up in an endless loop."
08:53 stickyboy http://www.gluster.org/2012/08/glus​terfs-bit-by-ext4-structure-change/
08:53 glusterbot <http://goo.gl/86YVi> (at www.gluster.org)
08:53 stickyboy Ah!
08:53 Norky joined #gluster
08:53 stickyboy It must be this... damn.
09:01 harshpb joined #gluster
09:02 harshpb joined #gluster
09:07 tjikkun_work joined #gluster
09:21 dobber_ joined #gluster
09:28 sripathi joined #gluster
09:29 tryggvil_ joined #gluster
09:32 tryggvil__ joined #gluster
09:36 hateya joined #gluster
09:37 Skunnyk Hello
09:37 glusterbot Skunnyk: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:37 Skunnyk grumpf
09:40 Skunnyk well, I have launch a gluster volumee heal full 48hours ago after replaced a crashed server by a new one, and replication is still not finish :/
09:53 statix_ joined #gluster
09:59 tryggvil__ joined #gluster
10:07 hybrid5121 joined #gluster
10:08 glusterbot New news from newglusterbugs: [Bug 920434] Crash in index_forget <http://goo.gl/mH2ks> || [Bug 917686] Self-healing does not physically replicate content of new file/dir <http://goo.gl/O8NMq> || [Bug 913699] Conservative merge fails on client3_1_mknod_cbk <http://goo.gl/ThGYk>
10:14 Nagilum hmm, why do I have a /var/lib/glusterd/vols/gv01/rb_mount mount? Is that normal when doing a heal?
10:18 tryggvil__ joined #gluster
10:19 shireesh joined #gluster
10:19 Nagilum hmm, glusterd hanging again
10:19 Nagilum http://pastebin.com/NFaeRQyF
10:19 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
10:21 Nagilum http://dpaste.org/avw7T/
10:21 glusterbot Title: dpaste.de: Snippet #221518 (at dpaste.org)
10:22 tryggvil___ joined #gluster
10:28 hybrid5121 joined #gluster
10:30 sripathi joined #gluster
10:36 mooperd joined #gluster
10:38 glusterbot New news from newglusterbugs: [Bug 921942] iobuf: Add a function iobref_clear <http://goo.gl/678wn>
10:42 davis_ joined #gluster
10:44 manik joined #gluster
10:54 timothy joined #gluster
11:00 hybrid512 joined #gluster
11:03 hybrid5121 joined #gluster
11:07 hybrid512 joined #gluster
11:08 glusterbot New news from newglusterbugs: [Bug 830497] [FEAT] geo-replication failover/failback <http://goo.gl/XkT0F> || [Bug 847842] [FEAT] Active-Active geo-replication <http://goo.gl/Z41og>
11:10 glusterbot New news from resolvedglusterbugs: [Bug 851954] [FEAT] Multi Master Geo Replication <http://goo.gl/C1v4d>
11:18 raghug joined #gluster
11:20 raghug @aravindavk: were you looking for me?
11:20 jclift joined #gluster
11:21 _pol joined #gluster
11:21 harshpb joined #gluster
11:22 harshpb joined #gluster
11:23 displaynone joined #gluster
11:30 satheesh joined #gluster
11:33 manik joined #gluster
11:36 jdarcy joined #gluster
11:38 glusterbot New news from newglusterbugs: [Bug 908518] [FEAT] There is no ability to check peer status from the fuse client (or maybe I don't know how to do it) <http://goo.gl/TnrZT> || [Bug 915996] [FEAT] Cascading Geo-Replication Weighted Routes <http://goo.gl/eyX0V>
11:40 glusterbot New news from resolvedglusterbugs: [Bug 764758] add-brick to somehow sync the change in pathinfo map in the client mount <http://goo.gl/naqzK>
11:43 balunasj joined #gluster
11:44 isomorphic joined #gluster
11:45 timothy joined #gluster
11:51 bala joined #gluster
11:51 harshpb joined #gluster
11:55 dobber_ joined #gluster
11:59 ctria joined #gluster
12:11 sjoeboo joined #gluster
12:11 harshpb joined #gluster
12:12 tryggvil__ joined #gluster
12:13 tryggvil___ joined #gluster
12:13 aravindavk joined #gluster
12:16 timothy joined #gluster
12:25 andreask joined #gluster
12:26 yinyin joined #gluster
12:32 deepakcs joined #gluster
12:33 tryggvil__ joined #gluster
12:34 tryggvil___ joined #gluster
12:37 HaraldJensas joined #gluster
12:44 jdarcy joined #gluster
12:45 rcheleguini joined #gluster
12:52 H__ yay, End, ./testlab-upgrade-gluster.sh ran for 61 seconds # That's on 5 machines (2 clients and 3 servers)
12:54 aliguori joined #gluster
13:01 Staples84 joined #gluster
13:04 lpabon joined #gluster
13:06 GabrieleV joined #gluster
13:08 robo joined #gluster
13:09 bennyturns joined #gluster
13:13 Humble joined #gluster
13:13 rwheeler joined #gluster
13:14 jtux joined #gluster
13:15 dustint joined #gluster
13:20 edward1 joined #gluster
13:28 zetheroo joined #gluster
13:29 zetheroo if you have 3 servers which you would like to have a replicated gluster across ... do I use a command like this one?
13:29 zetheroo gluster volume create gvola replica 2 transport tcp gnode1:/mnt/bricks/gnode1a gnode2:/mnt/bricks/gnode2a gnode3:/mnt/bricks/gnode3a
13:29 ctria joined #gluster
13:33 jtux joined #gluster
13:36 zetheroo is there replica 1, replica 2 and replica 3 ? .... how does this work? what is the significance of the number 1, 2 or 3 ?
13:37 semiosis there is no replica 1
13:37 H__ FYI doing a gluster 3.3.1 replace-brick on my test servers, I see 25 MiB/s, 200 tps read, 110 tps write. and most importantly : stable so far ! (This broke on 3.2.5)
13:38 semiosis replica N (N >= 2) means that every file will be replicated on N bricks, and that there will be MxN bricks in the volume, where M is a the number of replica sets.  files will be distributed evenly over the replica sets.
13:38 jdarcy zetheroo: The replica number is the number of copies you'll have for each file, so you can survive replica_number-1 failures.
13:38 jdarcy Alternatively, what he said.  ;)
13:40 zetheroo so I have 3 servers each with a xfs volume (LVM) mounted at /mnt/gluster ... I want each servers xfs volume to have the identical data
13:41 zetheroo just one copy on each server
13:41 semiosis volume create replica 3 server1:/mnt/gluster server2:/mnt/gluster server3:/mnt/gluster
13:41 zetheroo ok ...
13:42 jdarcy Right.  That will work, at least in 3.4, but you should be aware that performance will be lower than non-replicated.
13:42 jdarcy Write performance, that is.  Read performance scales (approximately) with the number of replicas.
13:42 zetheroo well I am not using 3.4
13:42 zetheroo 3.3 here
13:43 jdarcy There were some issues with replica 3 in 3.3 (I should know because I'm the one who fixed them).  Not clear if you'd be likely to hit them, but I like to be conservative when users' data is involved.
13:43 semiosis ?!
13:43 zetheroo oh?
13:44 zetheroo well the idea is to have our running VM images on there ... so do you foresee any issues with that?
13:45 zetheroo do I have to upgrade to 3.4? ...
13:45 zetheroo I thought 3.4 was not out yet ...
13:45 jdarcy AFAIK https://bugzilla.redhat.com/show_bug.cgi?id=802417 existed in 3.3 and the fix hasn't been backported.
13:45 glusterbot <http://goo.gl/V2r2t> (at bugzilla.redhat.com)
13:45 glusterbot Bug 802417: high, high, 3.4.0, jdarcy, ON_QA , [glusterfs-3.3.0qa27]: gfid change in the backend leading to EIO
13:47 sjoeboo joined #gluster
13:47 jdarcy 3.4 is finishing alpha and entering beta.  Personally I'd use it instead of 3.3, but I'm a developer.  I'm well aware that people using code in production and/or not working on the code often have a different perspective.  ;)
13:47 jdarcy All old code looks like crap from a developer's perspective.
13:47 zetheroo these are production machines here so I have to be caredul
13:47 zetheroo careful*
13:48 zetheroo can the fix be applied to v3.3 ?
13:48 stickyboy jdarcy: :D
13:49 zetheroo and if I use replica 2 will that mean that file.vm is only replicated to two bricks in stead of three?
13:50 jdarcy zetheroo: The patch would probably apply to 3.3 (I'd have to try it to be sure), but the result would be something even less thoroughly tested than the 3.4 alpha.
13:50 zetheroo ic]
13:50 jdarcy zetheroo: Right, so you'd have protection vs. one failure, and you'd have to be smart about migrating VMs to the right alternate host.
13:51 zetheroo hmm ... more work  ... and more complication ... more possibility for error ... :P
13:51 zetheroo and the issue that you fixed is this something that I will certainly run into if I use replica 3 ?
13:52 plarsen joined #gluster
13:53 jtux joined #gluster
13:53 zetheroo what are the chances that I would have the same issue? .. :)
13:53 jdarcy zetheroo: I think in many cases it's benign, but there are also cases where it could lead to undetected divergence between replicas.
13:53 cw joined #gluster
13:54 jdarcy zetheroo: We have many users who had been using replica 3, and only this one had problems (which TBH was a bit of a surprise to me since I'd originally found the bug by code inspection).
13:55 zetheroo if I go with replica 3 and this issue occurs is there a symptom to alert me to it?
13:57 jdarcy Well, in this case the user got EIO on opening some files, with "gfid differs on subvolume" in the logs.  Not sure if there are other ways this could manifest.
13:57 zetheroo ok, so it's something that you would have to be on the lookout for ...
13:57 semiosis jdarcy: iirc that's come up a few times here in channel lately, i didnt know what to make of it
13:58 semiosis brb
13:58 jdarcy zetheroo: Or at least able to recognize, yeah.
13:59 ninkotech_ joined #gluster
13:59 zetheroo hmm ...  I am really not sure now ... I don't want our backup images to be corrupted ... that would be painful
13:59 lpabon joined #gluster
13:59 jdarcy I hate telling people that replica 3 can't be trusted, but with a history like this and without it being fully tested (it's not a supported feature in RHS and we have no regular tests set up for it) I think it would be irresponsible to say otherwise.
14:00 H__ ewwww
14:01 jdarcy If it were me, I'd go with replica 2 and deal with the cases where the surviving replica isn't on all of the other hosts.
14:01 jdarcy Replica 2 gets shaken out pretty thoroughly, all the time.
14:01 semiosis and still use quorum, especially since you're going to have client mounts on your servers
14:01 zetheroo ok I did "gluster volume create kvmgluster replica 3 saturn:/mnt/gluster mars:/mnt/gluster neptune:/mnt/gluster"
14:01 rwheeler joined #gluster
14:01 jdarcy BTW, you can do replica 2 across three servers.  You just have to create multiple bricks per server.
14:02 zetheroo and: "gluster volume start kvmgluster"
14:02 tqrst unrelated: is the rebalanced-files counter supposed to be accurate in the rebalance status? According to this, one of my servers rebalanced 8.5 *billion* files.
14:02 semiosis hmm maybe quorum wouldnt work for you.  what a dilemma
14:02 jdarcy gluster volume create xyz replica 3 server1:/a server2:/a server3:/b server1:/b server2:/c server3:/c
14:03 jdarcy semiosis: Server-side quorum would work, but that's also new in 3.4
14:03 zetheroo so now ... where do I go from here ... do I start moving the VM images onto the xfs drives ?
14:04 zetheroo and whatever I put on one drive will be replicated to the other 2 ?
14:04 semiosis zetheroo: i explained that to you the other day, let me find the log
14:04 jdarcy zetheroo: I don't have enough information to answer that.  It depends on how you place your VMs, how acceptable it would be to have a VM run on a host where all data is remote (after a failure), etc.
14:05 zetheroo jdarcy: I just need file VM1.img to be on all three servers
14:05 jdarcy Generally you do *not* place data directly into the directories you're using for bricks.  You place data only through the GlusterFS mount point, which will then put data on the bricks itself.
14:05 zetheroo ah ok
14:06 zetheroo so I got to mount the gluster volume on one of the servers ?
14:06 semiosis zetheroo: http://irclog.perlgeek.de/g​luster/2013-03-13#i_6583657
14:06 glusterbot <http://goo.gl/jqqnS> (at irclog.perlgeek.de)
14:06 jdarcy zetheroo: Do you mean you need it to be equally visible on all three servers, or you need it to be *physically present* on all three servers?
14:06 zetheroo I am trying to follow this  http://kaivanov.blogspot.ch/20​12/07/deploying-glusterfs.html
14:06 glusterbot <http://goo.gl/gXxM0> (at kaivanov.blogspot.ch)
14:07 zetheroo jdarcy: i need it to be physically present on all three servers
14:07 zetheroo in case a host dies I need the image to be on another host
14:08 zetheroo ok, so I will need to mount the gluster volume on one of the three servers ...
14:08 zetheroo does ti matter which one?
14:09 jdarcy zetheroo: So running a VM on one host with the data on another would be unacceptable performance-wise?
14:09 zetheroo yes, for now
14:09 semiosis have you even tried?
14:09 zetheroo maybe later on if we setup network aggregation or something it may be ok
14:10 jdarcy zetheroo: Is this manual migration, or under some cloud infrastructure?
14:10 zetheroo what would be the point of having the VM running off the data on another host? ... if that host dies you got nothing but an empty shell of a VM ...
14:10 zetheroo no this is all manual
14:11 zetheroo just an in-house kvm setup to replace an ancient VMware one
14:12 zetheroo thing is that due to the work that is done on some of the VM's they need to run as fast as possible ... weather modeling and such is done on them ...
14:12 semiosis fast, cheap, reliable -- pick two :)
14:12 zetheroo haha
14:12 zetheroo I recall that
14:13 jdarcy zetheroo: If it's manual, why not use replica 2 (which is more thoroughly tested etc.) and just be careful to restart a VM after a failure on the surviving host that actually has the data?  It's not hard to tell which one that is.
14:13 zetheroo ok, well we just got to give this a go and see how it runs I guess
14:13 semiosis +1
14:13 zetheroo ok, but I need help with this ... :)
14:13 foster joined #gluster
14:13 jdarcy getfattr -d -e text -n trusted.glusterfs.pathinfo /vm_images/whatever.img
14:13 zetheroo how do I remove the gluster volume I made with replica 3 ?
14:14 zetheroo what the blazes is that!?
14:14 semiosis stop it then delete it
14:14 semiosis "gluster volume help", or ,,(rtfm)
14:14 glusterbot Read the fairly-adequate manual at http://goo.gl/E3Jis
14:14 jdarcy zetheroo: That's a command that will show you information about where the file actually is.
14:14 zetheroo ok I stopped it
14:15 semiosis jdarcy: ,,(pathinfo)
14:15 glusterbot jdarcy: I do not know about 'pathinfo', but I do know about these similar topics: 'pasteinfo'
14:15 jdarcy Hm.  I think a branch just fell outside.  Gotta go check for damage.
14:15 semiosis oops thought i made a factoid for that.
14:15 semiosis @learn pathinfo as find out which brick holds a file with this command on the client mount point: getfattr -d -e text -n trusted.glusterfs.pathinfo /client/mount/path/to.file
14:15 glusterbot semiosis: The operation succeeded.
14:16 zetheroo gluster volume remove  kvmgluster  ?
14:16 zetheroo hmm ... did not work ... do I have to remove each brick from the volume?
14:17 semiosis "did not work" is not helpful.  help us help you... what was the actual error?  maybe pastebin some logs
14:18 zetheroo ok I did:  gluster volume remove-brick kvmgluster replica 3 saturn:/mnt/gluster mars:/mnt/gluster neptune:/mnt/gluster
14:18 zetheroo and that worked
14:18 zetheroo or not!?
14:18 zetheroo "Removing bricks from replicate configuration is not allowed without reducing replica count explicitly."
14:19 semiosis you dont want to remove bricks, you want to delete the volume.  if you ran "gluster volume help" you would see that...
14:19 semiosis volume delete <VOLNAME> - delete volume specified by <VOLNAME>
14:19 semiosis as i told you above, "stop it, then delete it"
14:20 zetheroo Deleting volume kvmgluster has been successful
14:20 zetheroo ok
14:20 zetheroo so now I will make it again with replica 2
14:21 zetheroo I did :  gluster volume create replica 2 saturn:/mnt/gluster mars:/mnt/gluster neptune:/mnt/gluster
14:21 zetheroo and I get:   Wrong brick type: 2, use <HOSTNAME>:<export-dir-abs-path>
14:22 semiosis zetheroo: do yourself a favor and read the "gluster volume help" output
14:22 elyograg zetheroo: you said replica 2, but you gave it three bricks.
14:22 semiosis elyograg: and no volume name!
14:22 elyograg semiosis: that too, didn't notice.
14:23 zetheroo oh so you can only do replica 2 over two bricks?
14:23 semiosis zetheroo: for replica N you need MxN bricks
14:23 semiosis a multiple of the replica count
14:23 elyograg zetheroo: 2 exactly, or multiples of 2, for a distributed-replicated volume.
14:24 zetheroo so if I have 3 bricks ... I cannot do replica 2 ...
14:25 semiosis zetheroo: jdarcy explained how to do replica 2 with three servers above... two bricks per server.
14:25 stickyboy left #gluster
14:25 zetheroo yeah - but I already setup the bricks and we don't want to have 2 x 3TB bricks on each server
14:26 zetheroo right now on each server the 2 x 3TB drives are configured with LVM into one 6TB lv
14:27 zetheroo and the goal was to have each 6TB lv identical on all three servers
14:27 zetheroo but now this is not possible it seems ... unless we add a fourth server to the lot
14:27 zetheroo which I actually could do
14:28 zetheroo so I am just going to get a gluster going between saturn and mars for now ... (don't you just love my server names :P  )
14:29 zetheroo so I did :  gluster volume create kvmgluster replica 2 saturn:/mnt/gluster mars:/mnt/gluster
14:29 zetheroo and I get:   /mnt/gluster or a prefix of it is already part of a volume
14:29 glusterbot zetheroo: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
14:29 wushudoin joined #gluster
14:30 bstansell_ joined #gluster
14:32 zetheroo what is the "trusted.glusterfs.volume-id" or how does one find it?
14:34 manik joined #gluster
14:34 jbrooks joined #gluster
14:37 _pol joined #gluster
14:39 zetheroo ok I did this on both server saturn and mars:
14:39 zetheroo setfattr -x trusted.glusterfs.volume-id /mnt/gluster/
14:39 zetheroo and I am still getting this message when trying to make a new gluster volume
14:39 zetheroo /mnt/gluster or a prefix of it is already part of a volume
14:39 glusterbot zetheroo: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
14:39 zetheroo thanks glusterbot :P
14:41 rwheeler joined #gluster
14:42 jdarcy joined #gluster
14:42 lpabon_ joined #gluster
14:42 Humble joined #gluster
14:42 sjoeboo joined #gluster
14:43 zetheroo how do you restart glusterd ?
14:43 lpabon_ joined #gluster
14:45 lpabon joined #gluster
14:46 bennyturns joined #gluster
14:49 zetheroo when I do: getfattr -m- -d /mnt/gluster/
14:49 zetheroo I get:
14:49 zetheroo getfattr: Removing leading '/' from absolute path names
14:49 zetheroo # file: mnt/gluster/
14:49 zetheroo trusted.gfid=0sAAAAAAAAAAAAAAAAAAAAAQ==
14:51 _benoit_ left #gluster
14:53 zetheroo I even tried deleting the bricks and recreating them ... and it's the same ...
14:53 zetheroo bloody hell !!!
14:55 jclift zetheroo: setfattr -x trusted.gfid /mnt/gluster     ?
14:56 manik joined #gluster
14:57 jclift zetheroo: Just to be sure... /mnt/gluster is the brick name on your system isn't it?  It's not where you're wanting to mount the end result Gluster volume.  ?
14:58 zetheroo yes - thanks that worked!!
14:58 jclift zetheroo: Cool. :)
14:58 jclift me+1 :)
14:58 zetheroo even deleting the /mnt/gluster brick and creating a new brick called /mnt/brick did not work
14:58 zetheroo :P
14:59 zetheroo ok, so I got my replica 2 volume - yay
14:59 jclift Hmmm... I wonder if the extended attribute might have been stored in the same inode as the /mnt/gluster directory's directory entry.
15:00 * jclift is still learning things
15:00 jclift zetheroo: Do you remember if you nuked the actual "/mnt/gluster" directory entry when you did that, and recreated it?
15:01 * jclift is just kind of wondering where the extended attribute stuff is stored.
15:01 jclift I should go look that up later. :D
15:01 zetheroo I did rm -rf /mnt/gluster
15:01 robos joined #gluster
15:01 jclift Interesting.
15:01 jclift Yep, I should *really* go look that up later then. :D
15:03 zetheroo heh ;)
15:04 zetheroo well I mounted my gluster volume on one of the two servers ... do I have to mount it also on the 2nd server?
15:05 daMaestro joined #gluster
15:06 tjstansell jdarcy: you still around?
15:06 zetheroo last I saw of jdarcy was:  (03:22:04 PM) jdarcy left the room (quit: Ping timeout: 245 seconds).
15:09 displaynone joined #gluster
15:22 zetheroo trusted.glusterfs.pathinfo="(<​REPLICATE:gluster-replicate-0> <POSIX(/mnt/brick):mars:/mnt/b​rick/images/theta2-disk1.img> <POSIX(/mnt/brick):saturn:/mnt/b​rick/images/theta2-disk1.img>)"
15:23 zetheroo does this mean that the theta2 image file is now located on both mars and saturn ?
15:27 lalatenduM joined #gluster
15:32 Alpinist joined #gluster
15:33 hagarth joined #gluster
15:38 jclift zetheroo: What happens if you look at the brick filesystem on both servers?  Is it there?
15:39 jclift zetheroo: Then, what happens if you diff them?  Are they the same? :D
15:39 jclift i.e. take a look.  Heh
15:39 zetheroo ah nice
15:39 zetheroo thanks ... easy way to do it ;)
15:39 jclift Just for awareness, it's possible for you to have a file that "shows up" on multiple bricks, but that isn't actually the same.
15:40 jclift For example, if your setup is set to _stripe_ the files instead of replicate, then you'll have 1/2 the file contents on each of 2 servers.
15:40 jclift (or 1/3 on 3, etc)
15:40 jclift Think sparse files, etc.
15:40 jclift Thus the "diff it to check" thing.
15:40 jclift Anyway, I've got to get some sleep.  Good luck zetheroo. :D
15:41 zetheroo thanks - it seems to be working :D
15:41 zetheroo have a good rest ;)
15:41 jclift :)
15:41 tryggvil__ joined #gluster
15:51 shylesh joined #gluster
15:52 zetheroo do bricks have to be identical in size?
15:53 semiosis no, but it is considered a best practice
15:53 semiosis you really dont want bricks to fill up -- no filesystem handles that well
15:53 hagarth1 joined #gluster
15:54 zetheroo hmm ... thing is that the fourth server only has 2.5" bays ... so if I was going to try to match one of the 3rd servers 3TB drives I may have to get 3 x 1TB drives for the fourth server ...
15:55 zetheroo and I don't know if 3 x 1TB drives give the identical amount of space as a single 3TB drive
15:56 manik joined #gluster
15:57 semiosis imho if the bricks are roughly the same size that's enough
15:57 semiosis again, it really only matters if they fill up, which is bad, so you dont want one smaller brick to fill up when the others still have space left
15:58 zetheroo ah right ... a brick being 1 or more HDD's ...
15:59 zetheroo I have to keep getting my head around this stuff :)
16:02 zetheroo maybe its worth waiting for 3.4 to come out with replica 3
16:02 zetheroo any idea on how close 3.4 is to being released?
16:03 tjstansell semiosis: is there any plan to support bricks of different sizes?  even if it's "pretent all disks are the same size as the smallest brick in the system" for capacity planning, reporting, etc?
16:04 nueces joined #gluster
16:05 semiosis tjstansell: uhhh, what?
16:06 tjstansell well, it sounds like you're saying if you have bricks of say, 1TB, 2TB, 3TB, it'll write data to all fo them evenly and fill up the 1TB brick while the whole capacity is still around 50%, right?
16:06 tjstansell i guess it partly depends on if you're distributing vs replicate though
16:07 mooperd_ joined #gluster
16:07 zetheroo so something like there being a space limit defined by the smallest brick!?
16:08 tjstansell in just a simple distributed scenario, if you have 5TB of data, what happens in that 1,2,3TB brick config?
16:08 tjstansell does it fill up the first brick entirely?
16:08 ekuric left #gluster
16:08 tjstansell or know that it's pretty close and then start putting the rest in the 2TB and 3TB bricks?
16:08 tjstansell without necessarily having the 1TB brick hit 100% capacity
16:08 _pol joined #gluster
16:10 bstansell joined #gluster
16:11 tjstansell either that, or it treats all of them as 1TB in size (the smallest of the bricks) when you ask for space used, etc.
16:12 tjstansell so if you're monitoring your gluster volumes capacity, you don't accidentally fill one brick before realizing it.
16:12 semiosis i think it will try to allocate new files onto the bricks that aren't near capacity.  there's a config option to set what "near" means
16:12 semiosis however if you have files which continue to grow, glusterfs wont move them around afaik
16:14 semiosis with replication the smallest brick in a replica set determines the capacity of that set, obviously.  so if you have a pure replicate volume with two bricks, 1TB & 3TB, the capacity is 1TB.
16:14 tjstansell ok. that's good ;)
16:15 zetheroo nice
16:15 vshankar joined #gluster
16:15 zetheroo is there a release date for 3.4 ?
16:15 semiosis no
16:15 zetheroo ok
16:16 zetheroo is there such a thing as replica 4 ?
16:16 semiosis you can do that, but it's even less common (and thus less tested) than replica 3
16:17 zetheroo I see
16:17 semiosis bbiab
16:17 zetheroo ok, well my replica 2 seems to be working a treat ... so it's been a good day :D
16:17 zetheroo take care all and thanks for your help!!
16:18 zetheroo left #gluster
16:25 semiosis yw
16:27 bstansell_ joined #gluster
16:29 Alpinist joined #gluster
16:41 redsolar_office joined #gluster
16:41 Mo___ joined #gluster
16:41 lh joined #gluster
16:41 lh joined #gluster
16:44 dustint joined #gluster
16:46 Guest43573 joined #gluster
16:47 Guest43573 left #gluster
16:47 gbrand_ joined #gluster
17:00 _pol joined #gluster
17:01 _pol joined #gluster
17:01 GabrieleV joined #gluster
17:13 lalatenduM joined #gluster
17:23 Nagilum hmm, gluster volume replace-brick gives me an error in the logfile: Unable to get lock for uuid: 66fad515-f35c-492f-baf4-62b633622af8, lock held by: 66fad515-f35c-492f-baf4-62b633622af8
17:25 bulde joined #gluster
17:25 puebele joined #gluster
17:28 Ryan_Lane joined #gluster
17:30 semiosis weird, i'd try restarting glusterd on all your servers.
17:32 Nagilum I wish it would at least provide enough information to manually clear the lock..
17:33 semiosis i think that would be very complicated
17:33 JoeJulian I suppose a peer probe would tell you which server is that uuid and you could just try restarting glusterd on that one.
17:34 semiosis peer status -- if you dont see it in the list, you're on it
17:34 Nagilum JoeJulian: already did that
17:34 JoeJulian er, yeah. Still haven't had my coffee...
17:36 Nagilum zeg: restarting glusterd on the node 66fad515-f35c-492f-baf4-62b633622af8
17:36 Nagilum *hrm*
17:37 Nagilum maybe it clears up after 30min
17:37 Nagilum lets wait n see
17:38 balunasj joined #gluster
17:39 JoeJulian 30 min is a likely timeframe for an incomplete frame. That is the timeout.
17:48 theron joined #gluster
17:52 mooperd joined #gluster
17:53 ctria joined #gluster
17:53 timothy joined #gluster
17:56 Nagilum hmm, no change :/
17:56 lh joined #gluster
18:03 mricon how atomic is geo-replication?
18:04 mricon is there a way to do something like rsync's --delete-after?
18:19 wN joined #gluster
18:34 piotrektt_ joined #gluster
18:34 piotrektt__ joined #gluster
18:36 cyberbootje1 joined #gluster
18:41 manik joined #gluster
18:44 rob__ joined #gluster
18:45 cyberbootje joined #gluster
18:45 manik joined #gluster
18:45 cyberbootje joined #gluster
18:45 manik1 joined #gluster
18:47 manik joined #gluster
18:49 lh joined #gluster
18:50 mooperd_ joined #gluster
18:50 mooperd_ joined #gluster
18:59 lh joined #gluster
19:07 wN joined #gluster
19:10 rotbeard joined #gluster
19:25 GreyFoxx joined #gluster
19:28 wN joined #gluster
19:30 y4m4 joined #gluster
19:33 Alpinist joined #gluster
19:35 joehoyle joined #gluster
19:37 cw joined #gluster
19:41 nonsenso joined #gluster
19:45 eiki joined #gluster
19:45 hateya joined #gluster
19:56 _pol_ joined #gluster
19:58 _pol joined #gluster
20:12 nueces joined #gluster
20:13 nonsenso i'm new to running glusterfs and have just deployed it in a development environment in ec2.
20:13 nonsenso i was wondering what the usual patterns are to backup volumes.
20:14 nonsenso ideally, i'd like to run regular snapshots of the bricks at the ebs level but i'm not sure how to ensure data consistency.
20:15 nonsenso any pointers/advice would be much appreciated.  :)
20:20 hattenator joined #gluster
20:27 ProT-0-TypE joined #gluster
20:28 semiosis nonsenso: my advice: use ebs, don't use lvm (unless you have to for file size or single thread throughput), use the CreateImage EC2 API call to snapshot whole servers at once, use dns
20:29 semiosis nonsenso: the only way to ensure data consistency is to pause your application in a consistent state before taking a snapshot
20:30 H__ is snapshotting in the 3.5 roadmap ?
20:30 semiosis H__: what do you have in mind?  you can already stop the volume, take a snapshot, then restart the volume.  what more would you have glusterfs do?
20:35 disarone joined #gluster
20:36 mricon I'm constantly seeing this traceback http://paste.fedoraproject.org/5163/79790136/
20:36 glusterbot Title: #5163 Fedora Project Pastebin (at paste.fedoraproject.org)
20:36 mricon during geo-replication
20:36 mricon it pretty much prevents it from working
20:37 mooperd left #gluster
20:38 mricon rather, it works but constantly breaks
20:38 mricon can't imagine it's very good for performance
20:45 mricon restarting glusterd on the geo-replication slave seems to fix it... for now
20:51 mricon still crashes with the same traceback, but more rarely now
20:52 bstansell joined #gluster
20:57 mricon crashing every 20 seconds now. :)
21:11 mricon ok, this fixes it http://review.gluster.org/#patch,sidebyside,4233,2,xlators/features/​marker/utils/syncdaemon/resource.py
21:11 glusterbot <http://goo.gl/aQ6fx> (at review.gluster.org)
21:20 disarone joined #gluster
21:20 disarone joined #gluster
21:24 jdarcy joined #gluster
21:25 nonsenso semiosis: do most people usually do filesystem level backups?
21:27 semiosis idk about most people, i use ebs snapshots, created by calling CreateImage on my servers (with the reboot option disabled)
21:27 nonsenso how do you "freeze" the filesystem?
21:27 semiosis my restore procedure is basically launching a new instance from the snpashot, and updating dns to point to the new instance
21:28 semiosis nonsenso: i dont need to freeze the filesystem
21:28 nonsenso you just block your/the app from writing to the fs?
21:29 _pol_ joined #gluster
21:29 semiosis not even
21:31 nonsenso do you have multiple bricks per volume?
21:31 semiosis yes
21:32 nonsenso huh.  perhaps i'm being overtly paranoid about inconsistencies during backups?
21:32 semiosis depends on your use case
21:34 ajm nonsenso: it definitely depends, but for a lot of use cases its perfectly fine to do that
21:34 ajm you'll just probably need to fsck (or do innodb recovery or whatever) the volume if/when you restore
21:34 ajm and/or fsck after snapshot finishes
21:34 semiosis no fsck with xfs
21:35 semiosis although yes with ext you would need to fsck
21:35 ajm xfs_check to verify?
21:35 semiosis i suppose so
21:35 ajm I don't xfs anything anymore so, can't speak to it
21:36 semiosis although lets consider the ec2 use case... replicating between datacenters.  one instance dies, its replica remains online... when you restore the dead instance from the last backup of it, anything corrupt/lost will just get resynced from the surviving replica(s)
21:37 semiosis if you're concerned about an entire ec2 region falling into the sea then you need to take other measures, but for "simple" server failure this may be enough for some people
21:37 nonsenso well you can just ship the snapshots around
21:37 nonsenso between regions, even.
21:37 semiosis nonsenso: how would you do that?
21:37 nonsenso it's a new feature aws recently added
21:38 semiosis awesome!  istr seeing something about that
21:38 nonsenso you can copy snapshots between regions
21:38 semiosis \o/
21:38 nonsenso but it's not incremental
21:38 nonsenso so
21:38 nonsenso it has some way to go yet.
21:39 semiosis so, stagger your snapshots, that way two snapshots will be able to sync each other on restore :)
21:40 nonsenso so far my setup is pretty light and simple.  3 replica sets across 3 azs.
21:42 semiosis i recommend using dedicated hostnames (e.g. gluster1.my.domain.net) which CNAME to the public-ipv4 of the glusterfs server instances, and mapping a server's own hostname to 127.0.0.1 in its own /etc/hosts file
21:42 nonsenso at some point i suspect we'll introduce another region or two.
21:42 nonsenso i'm using only vpc
21:43 nonsenso so that doesn't seem to be an issue.  the first POC i stood up was in "ec2-classic" and yeah, i had to mess with that w/ EIPs and blah.
21:43 nonsenso it cracks me up the public cloud there is now called "classic"
21:44 * semiosis needs to catch up on the latest aws news!
21:44 semiosis havent had time to keep up on everthing, just caught a few headlines
21:45 cyberbootje1 joined #gluster
21:45 nonsenso basically, you should start planning on moving to vpc.  :)
21:46 semiosis ok
21:47 semiosis s/planning on//
21:47 glusterbot semiosis: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
21:47 semiosis glusterbot: meh
21:47 glusterbot semiosis: I'm not happy about it either
21:47 nonsenso haha.
21:47 nonsenso how many instances do you have in ec2?
21:48 semiosis not a huge amount, but many other priorities keep me away from systems stuff
21:48 nonsenso totally.
21:50 nonsenso thanks for the pointers, btw.
21:51 Humble joined #gluster
21:52 semiosis yw
22:12 tryggvil__ joined #gluster
22:28 _pol joined #gluster
22:39 Humble joined #gluster
22:45 lh joined #gluster
22:46 H__ semiosis: right . It won't be feasible by just gluster itself as it is built on top of other filesystems. They need the snapshotting capability to expose it to gluster in the first place.
22:47 mricon various references I find say that geo-replication is faster if the remote slave is a gluster volume itself.
22:48 mricon is that pretty accurate?
22:49 H__ why would that be so ?
22:51 mricon I'm just asking if that's so. :)
22:56 ProT-0-TypE joined #gluster
22:58 Ryan_Lane joined #gluster
23:02 H__ it may be, but i want to know why
23:12 kbsingh joined #gluster
23:12 kbsingh johnmark: ping

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary