Camelia, the Perl 6 bug

IRC log for #gluster, 2013-07-01

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:07 xymox_ joined #gluster
00:15 mooperd joined #gluster
00:42 robo joined #gluster
00:56 htrmeira left #gluster
01:39 kevein joined #gluster
02:14 kevein joined #gluster
02:18 bulde joined #gluster
02:28 kevein joined #gluster
02:29 harish joined #gluster
02:41 bharata joined #gluster
02:48 harish joined #gluster
02:51 sjoeboo joined #gluster
02:52 recidive joined #gluster
03:14 mmalesa joined #gluster
04:02 andreask joined #gluster
04:05 shylesh joined #gluster
04:15 harish joined #gluster
04:20 anands joined #gluster
04:50 hagarth joined #gluster
05:05 kshlm joined #gluster
05:06 bulde joined #gluster
05:09 kevein joined #gluster
05:09 vpshastry joined #gluster
05:15 mohankumar__ joined #gluster
05:34 guigui3 joined #gluster
05:36 Ravitezu joined #gluster
05:38 psharma joined #gluster
05:39 Ravitezu Hi all,
05:39 Ravitezu we chose to use oVirt for our environment and i'd like to know how Gluster can help us when use it with oVirt.
05:39 Ravitezu could some one help me?
05:40 Ravitezu we dont have any shared storage.. we use local storage for storing virtual machine images.
05:40 Ravitezu on the servers which host virt. machines
05:41 saurabh joined #gluster
05:49 samppah Ravitezu: you can use GlusterFs to create shared storage
05:50 samppah either using dedicated servers for Gluster or share disks from oVirt nodes
05:51 Ravitezu samppah: thanks for the reply.
05:51 Ravitezu let's say i have 2 oVirt nodes with ~2TB storage on each, So we chose to use the Local storage available on oVirt nodes rather than going for external storage(like Netapp Filer)
05:53 Ravitezu how GlusterFs can help in this scenario ? Can it help me in combining the local storages(2TB + 2TB) and use as a Shared storage?
05:54 samppah Ravitezu: i'm not sure if it's possible to use Local storage (as stated in oVirt) and Glusterfs at same time.. how ever you are running oVirt on Centos for example you can install GlusterFS server on oVirt nodes and create gluster volume using local disks
05:57 ricky-ticky joined #gluster
05:57 Ravitezu oVirt’s web admin console includes a graphical tool for configuring Gluster Volumes.
06:00 samppah yeh, i've never tried it actually.. gluster CLI is enough :)
06:02 Ravitezu samppah: I'm new to GlusterFs. I'd like to know what is the exact use of GlusterFs?
06:02 Ravitezu Do you have any links?
06:09 raghu joined #gluster
06:12 vimal joined #gluster
06:15 syoyo joined #gluster
06:22 vshankar joined #gluster
06:23 jtux joined #gluster
06:26 sgowda joined #gluster
06:26 kshlm joined #gluster
06:29 satheesh1 joined #gluster
06:29 satheesh joined #gluster
06:39 bala joined #gluster
06:42 kevein_ joined #gluster
06:42 kevein_ joined #gluster
06:44 kevein_ joined #gluster
06:48 ekuric joined #gluster
06:50 dobber joined #gluster
06:53 puebele joined #gluster
06:53 ramkrsna joined #gluster
06:54 ctria joined #gluster
07:09 harish joined #gluster
07:11 puebele joined #gluster
07:12 alinux joined #gluster
07:17 fidevo joined #gluster
07:24 ujjain joined #gluster
07:25 samppah does anyone know what's the status of bug 922183
07:25 glusterbot Bug http://goo.gl/ZD3FO is not accessible.
07:28 jord-eye1 joined #gluster
07:30 jord-eye1 Hello everyone. Anybody have dealed with glusterfs + nginx? I'm facing some problems with nginx D state processes....
07:30 harish joined #gluster
07:33 mmalesa joined #gluster
07:45 tziOm joined #gluster
07:48 mooperd joined #gluster
07:53 ngoswami joined #gluster
07:59 dobber joined #gluster
08:04 satheesh joined #gluster
08:05 mmalesa joined #gluster
08:06 Norky joined #gluster
08:12 X3NQ joined #gluster
08:22 alinux joined #gluster
08:25 kevein joined #gluster
08:32 ndevos samppah: it seems that it requires bug 913051 (patch, most important) and bug 959075 (no patch yet)
08:32 glusterbot Bug http://goo.gl/esXVh unspecified, unspecified, ---, pkarampu, MODIFIED , Use anonymous fds if the file is not opened in one of replica subvolumes
08:32 glusterbot Bug http://goo.gl/Esx2G medium, unspecified, ---, sgowda, NEW , dht migration-  open not sent on a cached subvol if open done on different fd once cached changes
08:34 samppah ndevos: thank you.. could you confirm that it's also related to bug 953887?
08:34 glusterbot Bug http://goo.gl/tw8oW high, high, ---, pkarampu, MODIFIED , [RHEV-RHS]: VM moved to paused status due to unknown storage error while self heal and rebalance was in progress
08:36 samppah if i undestand correcly patch metioned in 953887 should be included in beta4 but the issue still exists
08:37 ndevos samppah: from a quick glance one is the RHEV-RHS version, the other is for glusterfs-upstream
08:37 rajesh joined #gluster
08:40 ndevos samppah: hmm, I think you also need the patch from bug 913051, maybe you can check if you have that one in release-3.4 too? the bug only lists a merge in master
08:40 glusterbot Bug http://goo.gl/esXVh unspecified, unspecified, ---, pkarampu, MODIFIED , Use anonymous fds if the file is not opened in one of replica subvolumes
08:41 ndevos samppah: but anyway, I think it would be good to leave a note in the bugreport, if others hit that problem too, they should know it's not completely working yet
08:46 samppah ndevos: i'll do that! patch from 913051 seems to be included in beta4 aswell
08:49 glusterbot New news from newglusterbugs: [Bug 844584] logging: Stale NFS messages <http://goo.gl/z72b6>
08:50 ndevos samppah: thanks! in case there are more dependencies, someone should point those out in the bug, or maybe the patches are not sufficient and more work is needed
08:55 guigui3 left #gluster
08:55 guigui1 joined #gluster
09:08 mmalesa joined #gluster
09:15 morse joined #gluster
09:20 mmalesa_ joined #gluster
09:25 ramkrsna joined #gluster
09:25 ramkrsna joined #gluster
09:32 mooperd_ joined #gluster
09:34 ngoswami joined #gluster
09:58 harish joined #gluster
09:59 CheRi joined #gluster
10:03 venkat joined #gluster
10:05 ramkrsna joined #gluster
10:11 FilipeMaia joined #gluster
10:12 FilipeMaia I'm trying to use NFS mounting from macosx but I'm getting the following error:
10:12 FilipeMaia sudo mount -t nfs -o intr,tcp,nolock,vers=3 davinci.icm.uu.se:/gv0 /tmp/gluster/
10:12 FilipeMaia mount_nfs: can't mount /gv0 from davinci.icm.uu.se onto /private/tmp/gluster: No such file or directory
10:12 FilipeMaia What could be the problem
10:13 jord-eye1 FilipeMaia: have you tried without the slash? davinci.icm.uu.se:gv0
10:13 FilipeMaia I think I did but i'll try again
10:13 FilipeMaia mount_nfs: can't mount gv0 from davinci.icm.uu.se onto /private/tmp/gluster: No such file or directory
10:14 FilipeMaia BTW /private/tmp/gluster exists
10:14 jord-eye1 you're trying to mount to /tmp/gluster, but it says /private/tmp/gluster ...
10:14 jord-eye1 yes, just as you said
10:14 FilipeMaia symlinks
10:15 jord-eye1 try first without symlinks
10:15 FilipeMaia sudo mount -t nfs -o intr,tcp,nolock,vers=3 davinci.icm.uu.se:gv0 /private/tmp/gluster/
10:15 FilipeMaia mount_nfs: can't mount gv0 from davinci.icm.uu.se onto /private/tmp/gluster: No such file or directory
10:15 piotrektt joined #gluster
10:15 piotrektt joined #gluster
10:15 FilipeMaia ls /private/tmp/gluster/
10:15 FilipeMaia <empty>
10:15 jord-eye1 "ls -l"
10:15 FilipeMaia empty again
10:16 jord-eye1 can you try to create a dir in /tmp and mount there?
10:16 FilipeMaia /tmp is a symlink
10:16 jord-eye1 is "private" an encripted dir?
10:16 FilipeMaia I can choose another dir
10:16 jord-eye1 yes
10:17 FilipeMaia sudo mount -t nfs -o intr,tcp,nolock,vers=3 davinci.icm.uu.se:gv0 mnt
10:17 FilipeMaia mount_nfs: can't mount gv0 from davinci.icm.uu.se onto /Users/filipe/mnt: No such file or directory
10:17 jord-eye1 is NFS enabled on the volume?
10:17 FilipeMaia ls -la mnt
10:17 jord-eye1 I have to ask
10:17 FilipeMaia I did gluster volume set gv0 nfs.disable off
10:17 FilipeMaia but it seems to be the default
10:17 jord-eye1 ok
10:17 jord-eye1 I don't know then
10:18 FilipeMaia gluster volume status shows
10:18 ccha showmount -e davinci.icm.uu.se
10:18 FilipeMaia NFS Server on localhost38467Y381
10:18 FilipeMaia Exports list on davinci.icm.uu.se:
10:18 FilipeMaia /gv1                                *
10:19 jord-eye1 what says: netstat -lpon | grep 38467
10:19 ccha so it's gv1 and not gv0
10:19 FilipeMaia gluster volume info does not show any gv1
10:19 FilipeMaia http://paste.debian.net/13580/
10:19 glusterbot Title: debian Pastezone (at paste.debian.net)
10:20 FilipeMaia I might have had a gv1 in the past
10:20 spider_fingers joined #gluster
10:20 spider_fingers left #gluster
10:21 ccha can you stop the volume ?
10:21 FilipeMaia gv1?
10:21 ccha gv0
10:21 ccha and check again showmount
10:21 FilipeMaia [root@login gluster]# gluster volume stop gv1
10:21 FilipeMaia Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
10:21 FilipeMaia Volume gv1 does not exist
10:21 ccha stop gv0
10:22 FilipeMaia But gv0 is mounted in many places
10:22 ccha ah
10:22 ccha what happended if you mount gv1 ?
10:22 bala joined #gluster
10:22 FilipeMaia using nfs?
10:22 ccha yes
10:22 FilipeMaia sudo mount -t nfs -o intr,tcp,nolock,vers=3 davinci.icm.uu.se:/gv1 mnt
10:22 FilipeMaia mount_nfs: can't mount /gv1 from davinci.icm.uu.se onto /Users/filipe/mnt: Invalid argument
10:23 FilipeMaia I tried with and without the /, same error
10:23 ccha your showmount is strange
10:23 FilipeMaia Does the fact that there is a normal nfs server running a problem?
10:23 ccha what about showmount -e davinci.icm.uu.se on another client ?
10:24 ccha does it show gv0 or gv1
10:24 FilipeMaia what packages includes showmount?
10:24 ccha nfs-common
10:24 FilipeMaia oh as root of course
10:25 FilipeMaia gv1 from other clients
10:25 FilipeMaia BTW other bricks also show gv1
10:25 FilipeMaia (with the gluster volume mounted)
10:26 ccha oh davinci.icm.uu.se is one your hostanme brick ?
10:26 FilipeMaia BTW who are the NFS servers? Those who mount the glusterfs volume?
10:26 FilipeMaia ccha: actually it's the only one who's not a brich
10:26 FilipeMaia brick
10:26 ccha nfs server should be one of gluster server
10:26 FilipeMaia ccha: aren't all bricks gluster servers?
10:27 ccha what about showmount -e a001 ?
10:27 FilipeMaia [root@a001 ~]# showmount -e a001
10:27 FilipeMaia Export list for a001:
10:27 FilipeMaia /gv1 *
10:28 ccha hum I don't know... something weird about volume should gv0 and exported as gv1
10:29 FilipeMaia BTW I went to /var/lib/glusterd
10:29 FilipeMaia and did grep -R gv1 *
10:29 FilipeMaia and it come out empty
10:29 samppah can you send output of gluster volume info to pastie.org?
10:31 FilipeMaia http://paste.debian.net/13581/
10:31 glusterbot Title: debian Pastezone (at paste.debian.net)
10:31 piotrektt joined #gluster
10:31 piotrektt joined #gluster
10:33 FilipeMaia It's a mistery
10:34 FilipeMaia BTW there's a couple of mistakes in http://gluster.org/community/docum​entation/index.php/Gluster_3.1:_Ma​nually_Mounting_Volumes_Using_NFS
10:34 glusterbot <http://goo.gl/83ZPo> (at gluster.org)
10:34 FilipeMaia (like extra ':') and inconsistent options (mountproto)
10:36 FilipeMaia Is it possible to mount glusterfs on macosx using fuse directly?
10:36 ngoswami joined #gluster
10:44 samppah FilipeMaia: i think there is no gluster fuse client for mac currently
10:47 hagarth joined #gluster
11:03 harish joined #gluster
11:13 CheRi joined #gluster
11:14 jord-eye1 Just in case no one has seen my question this morning, I try again: Anybody have dealed with glusterfs + nginx? I'm facing some problems with nginx D state processes....
11:16 shylesh joined #gluster
11:20 venkat joined #gluster
11:27 samppah jord-eye1: haven't seen that myself.. does it happen often?
11:27 rwheeler joined #gluster
11:30 jord-eye1 samppah: yes, almost every three days
11:31 jord-eye1 I have nginx running with 4 workers, and they are getting D state one by one, on different ours. When one gets D state load average sticks to 1, when another one gets D state, load avg sitcks to 2, and so on. this morning I have load avg to 4 and nginx wasn't working
11:31 jord-eye1 it all began when I moved to glusterfs
11:31 jord-eye1 I was using nfs before
11:31 jord-eye1 (native NFS, no gluster)
11:32 Nagilum jord-eye1: what version are you running?
11:32 Nagilum ..I have the same issue here, it's not nginx related
11:32 ollivera joined #gluster
11:32 jord-eye1 3.3.1-1
11:32 Nagilum k, then I can tell you that 3.3.1-15 has the same issue
11:33 Nagilum there is however also 3.3.2qa3 available
11:33 Nagilum http://bits.gluster.com/pub/​gluster/glusterfs/3.3.2qa3/
11:33 glusterbot <http://goo.gl/miEUH> (at bits.gluster.com)
11:33 jord-eye1 Nagilum: I thought it too, but I have several apache using the same gluster volumes and they don't have this problem (AFAIK)
11:34 Nagilum I have Apaches, vsftpd and cherokees running into the same issue
11:34 jord-eye1 It only happens with nginx. What do you use?
11:34 jord-eye1 ok
11:34 jord-eye1 have you tried 3.3.2?
11:34 Nagilum not yet
11:34 jord-eye1 I don't feel comfortable with beta/alpha versions in production
11:34 jord-eye1 obviously
11:35 Nagilum yeah, hence my hesitation..
11:35 jord-eye1 jejeje, I won't try it first :P
11:36 Nagilum although the gitlog doesn't show big changes: http://git.gluster.org/?p=glusterfs.​git;a=shortlog;h=refs/tags/v3.3.2qa3
11:36 glusterbot <http://goo.gl/IctJs> (at git.gluster.org)
11:36 Nagilum just fixes..
11:38 jord-eye1 I have done some diagnostics on the machines affected when this happens. lsof does not show any open file from gluster mountpoint. strace of course is freeze, because the D state. Unmounting the volume is not possible because "open files" error, and killing glusterfsd does not help either. Nginx persists in its D state. Only reboot works
11:39 Nagilum no
11:39 Nagilum you can salvage the situation
11:39 jord-eye1 how?
11:39 Nagilum if lsof for the glusterfs mount is clean, you can umount
11:39 jord-eye1 it does not work for me
11:40 Nagilum right
11:40 Nagilum usually the processes hanging wont die
11:40 Nagilum even after -9
11:40 Nagilum but all others using the gfs will go away
11:40 Nagilum then you can umount -f
11:40 Nagilum you may have to run it two or three times
11:41 Nagilum but it usually works
11:41 Nagilum then the hung processes are also gone
11:41 jord-eye1 see, when I try to umount:
11:41 jord-eye1 # umount /mnt/static-content
11:41 jord-eye1 umount: /mnt/static-content: device is busy.
11:41 jord-eye1 (In some cases useful info about processes that use
11:41 jord-eye1 the device is found by lsof(8) or fuser(1))
11:41 Nagilum umount -f /mnt/static-content
11:41 jord-eye1 hmmm...
11:41 jord-eye1 I see. I'll try next time
11:41 jord-eye1 but the problem is still there
11:42 jord-eye1 thanks for the tip Nagilum
11:46 rcheleguini joined #gluster
11:52 kkeithley Nagilum: BTW, generally I've only doing RPMs for alpha, beta, and GA releases. (And I saw that semiosis pointed you at the el6 RPMs on bits.gluster.org.)
11:52 Nagilum GA?
11:53 kkeithley General Availability, i.e. the release.
11:53 Nagilum ah
11:53 Nagilum thx
11:53 guigui5 joined #gluster
11:55 manik joined #gluster
11:55 manik left #gluster
11:57 venkat joined #gluster
12:01 bfoster joined #gluster
12:06 manik joined #gluster
12:08 edward1 joined #gluster
12:11 balunasj joined #gluster
12:13 FilipeMaia joined #gluster
12:13 rgustafs joined #gluster
12:13 recidive joined #gluster
12:17 FilipeMaia joined #gluster
12:20 anands joined #gluster
12:21 ekuric joined #gluster
12:24 CheRi joined #gluster
12:26 plarsen joined #gluster
12:30 mooperd_ joined #gluster
12:37 theron joined #gluster
12:37 vpshastry joined #gluster
12:47 aliguori joined #gluster
12:49 joelwallis joined #gluster
12:56 andreask joined #gluster
12:57 bala joined #gluster
13:08 m0zes joined #gluster
13:09 robo joined #gluster
13:20 vpshastry joined #gluster
13:21 bennyturns joined #gluster
13:23 JoeJulian Nagilum, jord-eye1: I've been using nginx and haven't seen that problem. Next time it does that it would be good to get a state dump of the client (kill -USR1) and attach that to a bug report.
13:29 Nagilum JoeJulian: by client you mean the "glusterfs" process?
13:31 kkeithley Nagilum: yes
13:31 Nagilum k
13:31 jord-eye1 JoeJulian: ok
13:33 mohankumar__ joined #gluster
13:49 vpshastry joined #gluster
13:50 jthorne joined #gluster
13:51 zaitcev joined #gluster
13:52 bugs_ joined #gluster
13:53 hagarth joined #gluster
13:53 duerF joined #gluster
13:54 rwheeler joined #gluster
13:54 kkeithley @ext34
13:54 glusterbot kkeithley: I do not know about 'ext34', but I do know about these similar topics: 'ext4'
13:54 kkeithley @ext4
13:54 glusterbot kkeithley: (#1) Read about the ext4 problem at http://goo.gl/xPEYQ or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
13:55 kkeithley JoeJulian: 404 error! ^^^
13:57 kkeithley @forget ext4
13:57 glusterbot kkeithley: Error: 2 factoids have that key. Please specify which one to remove, or use * to designate all of them.
13:58 kkeithley @forget ext4
13:58 glusterbot kkeithley: Error: 2 factoids have that key. Please specify which one to remove, or use * to designate all of them.
13:58 kkeithley huh
13:58 jord-eye2 joined #gluster
13:58 kkeithley @forget ext4*
13:58 glusterbot kkeithley: Error: There is no such factoid.
13:58 kkeithley @list
13:58 glusterbot kkeithley: Admin, Alias, Anonymous, Bugzilla, Channel, ChannelStats, Conditional, Config, Dict, Factoids, Google, Herald, Later, MessageParser, Misc, Network, NickCapture, Note, Owner, Plugin, RSS, Reply, Seen, Services, ShrinkUrl, String, Topic, URL, User, Utilities, and Web
13:58 kkeithley @list Factoids
13:58 glusterbot kkeithley: alias, change, forget, info, learn, lock, random, rank, search, unlock, and whatis
13:59 kkeithley @search ext4
13:59 glusterbot kkeithley: There were no matching configuration variables.
13:59 kkeithley @info
13:59 glusterbot kkeithley: Error: The command "info" is available in the Factoids, MessageParser, and RSS plugins. Please specify the plugin whose command you wish to call by using its name as a command before "info".
13:59 kkeithley gah
13:59 * ndevos *blink*
14:00 ndevos @forget ext4 Read about
14:00 glusterbot ndevos: Error: There is no such factoid.
14:00 ndevos @forget ext4 #1
14:00 glusterbot ndevos: Error: There is no such factoid.
14:00 * ndevos gives up
14:04 hagarth @factoids
14:07 deepakcs joined #gluster
14:07 kkeithley @whatais ext4
14:07 kkeithley @whatis ext4
14:07 glusterbot kkeithley: (#1) Read about the ext4 problem at http://goo.gl/xPEYQ or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:08 GabrieleV joined #gluster
14:08 kkeithley @change ext4#1 Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:08 glusterbot kkeithley: Error: The command "change" is available in the Factoids, Herald, and Topic plugins. Please specify the plugin whose command you wish to call by using its name as a command before "change".
14:09 kkeithley @change factoid ext4#1 Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:09 glusterbot kkeithley: Error: The command "change" is available in the Factoids, Herald, and Topic plugins. Please specify the plugin whose command you wish to call by using its name as a command before "change".
14:09 kkeithley @forget ext4#1
14:09 glusterbot kkeithley: Error: There is no such factoid.
14:09 kkeithley @forget ext4 #1
14:09 glusterbot kkeithley: Error: There is no such factoid.
14:09 kkeithley @forget #1 ext4
14:09 glusterbot kkeithley: Error: There is no such factoid.
14:10 kkeithley @forget #1
14:10 glusterbot kkeithley: (forget [<channel>] <key> [<number>|*]) -- Removes a key-fact relationship for key <key> from the factoids database. If there is more than one such relationship for this key, a number is necessary to determine which one should be removed. A * can be used to remove all relationships for <key>. If as a result, the key (factoid) remains without any relationships to a factoid (key), it
14:10 glusterbot kkeithley: shall be removed from the database. <channel> is only necessary if the message isn't sent in the channel itself.
14:10 kkeithley @forget ext4 1
14:10 glusterbot kkeithley: The operation succeeded.
14:10 ndevos hah!
14:10 kkeithley @learn ext4 Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:10 glusterbot kkeithley: (learn [<channel>] <key> as <value>) -- Associates <key> with <value>. <channel> is only necessary if the message isn't sent on the channel itself. The word 'as' is necessary to separate the key from the value. It can be changed to another word via the learnSeparator registry value.
14:10 kkeithley @learn ext4 1 Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:10 glusterbot kkeithley: (learn [<channel>] <key> as <value>) -- Associates <key> with <value>. <channel> is only necessary if the message isn't sent on the channel itself. The word 'as' is necessary to separate the key from the value. It can be changed to another word via the learnSeparator registry value.
14:11 kkeithley @learn ext4 1 as Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:11 glusterbot kkeithley: The operation succeeded.
14:11 kkeithley @whatis ext4
14:11 glusterbot kkeithley: Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:11 Guest41201 joined #gluster
14:11 kkeithley @ext4
14:11 glusterbot kkeithley: Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:11 kkeithley close but no cigiar
14:12 kkeithley learn ext4 #1 as Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:12 glusterbot <http://goo.gl/PEBQU> (at joejulian.name)
14:12 kkeithley @whatis ext4
14:12 glusterbot kkeithley: Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:13 kkeithley @forget ext4 1
14:13 glusterbot kkeithley: The operation succeeded.
14:13 kkeithley @whatis ext4
14:13 glusterbot kkeithley: I do not know about 'ext4', but I do know about these similar topics: 'ext4 1'
14:13 kkeithley @forget ext4 1
14:13 glusterbot kkeithley: Error: There is no such factoid.
14:13 hagarth @whatis ext4 1
14:13 glusterbot hagarth: I do not know about 'ext4', but I do know about these similar topics: 'ext4 1'
14:13 hagarth @whatis 'ext4 1'
14:13 glusterbot hagarth: Error: No factoid matches that key.
14:13 hagarth i give up
14:13 kkeithley @forget 'ext4 1'
14:13 glusterbot kkeithley: Error: There is no such factoid.
14:13 ndevos @ext4 1
14:13 glusterbot ndevos: Read about the ext4 problem at http://goo.gl/PEBQU
14:14 kkeithley @forget ext4
14:14 glusterbot kkeithley: Error: There is no such factoid.
14:14 kkeithley @forget ext4 *
14:14 glusterbot kkeithley: Error: There is no such factoid.
14:14 semiosis what's the problem?
14:14 semiosis @ext4
14:14 glusterbot semiosis: I do not know about 'ext4', but I do know about these similar topics: 'ext4 1'
14:14 semiosis @ext4 1
14:14 glusterbot semiosis: Read about the ext4 problem at http://goo.gl/PEBQU
14:14 ndevos it's like nfs4.1?
14:14 semiosis @nfs4.1
14:15 glusterbot semiosis: I do not know about 'nfs4.1', but I do know about these similar topics: 'nfs'
14:15 ndevos haha
14:15 semiosis you want it to be ext4 instead of ext4 1?
14:16 semiosis @forget ext4 1
14:16 glusterbot semiosis: Error: There is no such factoid.
14:16 kkeithley there were originally two factoids keyed by ext4, one the link to JoeJulians blog (which was 404'd) and the other was track the bug
14:16 kkeithley I was trying to fix the link to JoeJulian's blog
14:16 semiosis oh ok
14:16 semiosis @forget "ext4 1"
14:16 glusterbot semiosis: The operation succeeded.
14:17 kkeithley @learn ext4 as Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
14:17 glusterbot kkeithley: The operation succeeded.
14:17 kkeithley @learn ext4 as Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:17 glusterbot kkeithley: The operation succeeded.
14:17 kkeithley @whatis ext4
14:17 glusterbot kkeithley: (#1) Read about the ext4 problem at http://goo.gl/xPEYQ or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:17 semiosis that's the ticket
14:19 kkeithley gah, but that goo.gl link is borken
14:19 kkeithley @forget ext4 1
14:19 glusterbot kkeithley: The operation succeeded.
14:19 kkeithley @learn ext4 as http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change
14:19 glusterbot kkeithley: The operation succeeded.
14:19 kkeithley @ext4
14:19 glusterbot kkeithley: (#1) Track the ext4 bugzilla report at http://goo.gl/CO1VZ, or (#2) http://goo.gl/Jytba
14:20 kkeithley @whatis ext4
14:20 glusterbot kkeithley: (#1) Track the ext4 bugzilla report at http://goo.gl/CO1VZ, or (#2) http://goo.gl/Jytba
14:20 semiosis @forget ext4 1
14:20 glusterbot semiosis: The operation succeeded.
14:20 semiosis @ext4
14:20 glusterbot semiosis: http://goo.gl/Jytba
14:20 semiosis oops i just broke your fix
14:20 kkeithley @forget ext4
14:20 glusterbot kkeithley: The operation succeeded.
14:20 kkeithley @whatis ext4
14:20 glusterbot kkeithley: I do not know about 'ext4', but I do know about these similar topics: 'ebs'
14:21 kkeithley @learn ext4 as Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change
14:21 glusterbot kkeithley: The operation succeeded.
14:21 kkeithley @ext4
14:21 glusterbot kkeithley: Read about the ext4 problem at http://goo.gl/Jytba
14:21 semiosis looks good
14:21 kkeithley @learn ext4 as Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:21 glusterbot kkeithley: The operation succeeded.
14:21 balunasj joined #gluster
14:21 kkeithley @whatis ext4
14:21 glusterbot kkeithley: (#1) Read about the ext4 problem at http://goo.gl/zfH6d or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:21 kkeithley whew
14:22 semiosis #1 is broken again
14:22 kkeithley @ext4
14:22 glusterbot kkeithley: (#1) Read about the ext4 problem at http://goo.gl/zfH6d or (#2) Track the ext4 bugzilla report at http://goo.gl/CO1VZ
14:22 semiosis the goo.gl link changed
14:22 kkeithley wtf, it worked for me a minute ago, now it's borked again
14:23 semiosis @replace
14:23 glusterbot semiosis: (replace [<channel>] <number> <topic>) -- Replaces topic <number> with <topic>.
14:23 semiosis @update
14:23 semiosis hm
14:23 kkeithley why is it adding %2C on the end?
14:23 semiosis i have an idea...
14:23 semiosis @forget ext4 *
14:23 glusterbot semiosis: The operation succeeded.
14:23 semiosis @ext4
14:23 glusterbot semiosis: I do not know about 'ext4', but I do know about these similar topics: 'ebs'
14:23 RangerRick7 joined #gluster
14:24 semiosis @learn ext4 as Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change or follow the bug report here https://bugzilla.redhat.com/show_bug.cgi?id=838784
14:24 glusterbot semiosis: The operation succeeded.
14:24 glusterbot Bug 838784: high, high, ---, sgowda, ON_QA , DHT: readdirp goes into a infinite loop with ext4
14:24 semiosis @ext4
14:24 glusterbot semiosis: Read about the ext4 problem at http://goo.gl/Jytba or follow the bug report here http://goo.gl/CO1VZ
14:24 semiosis @ext4
14:24 glusterbot semiosis: Read about the ext4 problem at http://goo.gl/Jytba or follow the bug report here http://goo.gl/CO1VZ
14:25 semiosis adding another factoid under the same key seems to have caused glusterbot to re-shorten the first link
14:26 semiosis @seen chirino
14:26 glusterbot semiosis: chirino was last seen in #gluster 3 days, 0 hours, 19 minutes, and 11 seconds ago: <chirino> :)
14:35 chirino joined #gluster
14:35 robo joined #gluster
14:37 RangerRick8 joined #gluster
14:41 jiku joined #gluster
14:46 robo joined #gluster
14:50 guigui5 left #gluster
14:50 guigui3 joined #gluster
14:56 jtriley joined #gluster
14:56 y4m4 joined #gluster
15:09 daMaestro joined #gluster
15:14 kaptk2 joined #gluster
15:15 bstr Hey guys, i have a two node replicated brick, and im trying to increase the brick size locally (xfs grow) without adding a third node to the mix. Will i need to re-balance after i grow the filesystem locally?
15:20 NuxRo bstr: i dont think so, there is nowhere you can balance files since all you have is 2 bricks
15:23 Technicool joined #gluster
15:25 neofob joined #gluster
15:31 jclift_ joined #gluster
15:33 rwheeler_ joined #gluster
15:33 bstr NuxRo : is there any risk of loosing data once i bring gluster back up and remount?
15:33 bstr this is a replica volume not striped..
15:37 NuxRo bstr: well, you should always calculate for worst case scenario, but i do not see why it should not work, filesystem resizes are not that exotic anymore
15:39 jag3773 joined #gluster
15:42 bstr NuxRo : Thanks for the help :)
15:44 NuxRo let me know if it works :D
15:44 NuxRo remember, if it goes wrong we never talked :)
15:50 plarsen joined #gluster
15:52 saurabh joined #gluster
15:53 bsaggy joined #gluster
15:55 manik joined #gluster
15:56 bugs_ joined #gluster
15:57 nightwalk Over the past couple days, I've ran into a couple interesting behaviors in glusterfs 3.3.1 (Centos 6.3 rpm build)
15:58 nightwalk It seems traversing directories and setting large numbers of xattrs has caused the backing daemon to chew 15-20% of a core for in excess of 24 hours now
16:00 nightwalk The script that's setting the xattrs basically goes through and checks for freshness of the file checksums in various ways (mostly involving info from stat). It also seems to randomly think some files don't have up-to-date checksums when I'm pretty sure they did
16:02 nightwalk I haven't traced the process yet to see what the former is about (might be totally normal for all I know), but I'm guessing the latter is because of reading stale data from the other brick or something and shouldn't happen (?)
16:06 guigui3 joined #gluster
16:07 nightwalk The CPU usage caused the vm's nfs server to partially go out to lunch during the heaviest setattr/getfattr activity and somehow caused accesses to the exported directories (none of which were on gluster mounts) from other hosts to freeze, but that probably says more about linux' in-kernel nfs server code than anything
16:10 nightwalk Anyway, this isn't much of a problem for me since only the initial run is ever going to be this heavy, and I can always run it on new data/bricks prior to making the gluster mount if need be. I just thought it might be worthwhile to report since I could see it being an edge case that might not have been covered thoroughly
16:11 semiosis chirino: ping
16:15 chirino pong
16:16 semiosis hi, got a few questions for you
16:16 semiosis what IDE do you use for java development?
16:16 chirino IDEA
16:16 semiosis cool
16:17 semiosis i switched libgfapi-jni over to testng so I could order the test execution
16:17 semiosis how bad is that?
16:17 chirino sounds fine.
16:18 semiosis hehe great
16:18 chirino testng does not have as great IDE integration.. that's usually why I stick /w junit
16:18 chirino but testng does have some nice features.
16:18 semiosis the only way I've been able to get the tests to run OK is with mvn install
16:18 semiosis but i'm ok with that at least for now
16:19 semiosis last of all, i came across JNA on friday and curious what your opinion is of that technology
16:19 semiosis if you're at all familiar with it
16:20 chirino yeah you have to adjust the env of the test case in the IDE
16:20 chirino to get the jni lib to load right
16:20 semiosis makes sense, i'll mess around with that
16:20 chirino JNA is similar to hawtjni, but you loose compile time type checking.
16:20 chirino constant loading
16:21 chirino there is also a the FFI overhead.
16:21 chirino but, it's great in that you don't need to compile the JNI module.
16:22 chirino so it's just a tradeoff.
16:22 bstr NuxRo : looks to have been successful, stopping gluster volume, extending filesystem and restarting volume
16:22 semiosis interesting, i wonder if that would make any difference to licensing
16:24 manik joined #gluster
16:26 semiosis thx chirino
16:26 kushnir-nlm nightwalk: I've seen similar behavior under many different scenarios.
16:27 semiosis btw i started this project which will be the high-level consumer of libgfapi-jni https://github.com/semiosis​/glusterfs-java-filesystem
16:27 glusterbot <http://goo.gl/KNsBZ> (at github.com)
16:27 tziOm joined #gluster
16:27 semiosis digging in to the nio.2 filesystems stuff
16:27 chirino semiosis: well technically LGPL is supposed to allow users of the lib to license under something different.
16:27 chirino and leveldbjni is really juse a suer
16:27 chirino user
16:28 nightwalk kushnir-nlm: any common theme behind them?
16:29 bulde joined #gluster
16:49 manik joined #gluster
16:51 kushnir-nlm nightwalk: Common theme seems to be anything with lots of crawling activity ends up using lots of cpu and goes on forever... I'm trying out 3.3.1-15, and it seems to work better.
16:52 nightwalk kushnir-nlm: and the cpu activity continues for hours or days after the crawling activities have stopped?
16:52 Mo_ joined #gluster
16:52 kushnir-nlm nihgtwalk: Days.
16:54 atrius_ joined #gluster
16:55 kushnir-nlm My build is 3 servers, 8 SSD bricks per server. Distributed with 3 replicas. Servers are VMs on HyperV, 4 vCPU, 16 GB RAM, 8 Crucial M4 or Vertex 4 disks in passthrough. 10GbE. TCP transport. RHEL 6 in the VMs. Gluster 3.3.1-1 (3.3.1-15 in testing). Stuff like adding a replica takes forever. If I force replication with stats of a tree, CPU goes through the roof forever... I've also seen lots
16:55 kushnir-nlm of page allocation errors in the logs leading me to set zone_reclaim_mode to 1.
16:56 nightwalk the main thing I've observed is a lot of stopping/starting messages like this: "I [afr-self-heald.c:1082:afr_dir_exclusive_crawl] 0-data-kernel-replicate-0: Another crawl is in progress for data-kernel-client-0"
16:56 kushnir-nlm Yes, that also.
16:57 nightwalk Ironically, none of this comes anywhere close to being as much of a problem as memory usage is
16:57 kushnir-nlm I think 2 bricks per CPU was a mistake. I'm testing out 4 vCPU and 4 brick servers.
16:58 kushnir-nlm What are you seeing in terms of RAM use? I've never seen much of an issue?
16:58 nightwalk glusterfs has over 2G of resident memory out of the 4G that's allocated for eachvm
16:58 kushnir-nlm 4G might be a little skimpy
16:59 nightwalk if resource-hungry zfs can get by on 4G, glusterfs surely should be able to as well
16:59 nightwalk especially in this use case, since it amounts to nothing more than an rsyncfs
17:00 kushnir-nlm I know nothing about ZFS. :)
17:01 nightwalk well, let's just say that you'd want a minimum of 8G of ram if you ever wanted to use it. 16G if the system is busy at all :)
17:01 kushnir-nlm Red hat storage appliance specs at one point recommended 4CPU and 16GB of RAM, so that's what I went for. :)
17:01 nightwalk I probably wouldn't build a baremetal system with less than 32G these days
17:03 nightwalk the core count doesn't matter nearly as much as large amounts of ram for caching and buffering and a fast interconnect
17:04 kushnir-nlm IDK... I see CPU maxed out during replications, heavy writes, etc...
17:05 bala joined #gluster
17:05 kushnir-nlm Each brick results in a glusterfsd process, and they all start maxing out CPU during high load scenarios... I.e. 20 node cluster writes of millions of ~100K files over fuse mount.
17:06 nightwalk I was talking about NAS servers in general.
17:06 kushnir-nlm Ahh
17:06 alinux joined #gluster
17:07 nightwalk even though context switches aren't nearly as expensive as they used to be, I imagine highly active FUSE filesystems like gluster incur part of their overhead there
17:07 nightwalk FUSE is also kind of isolated from a lot of the kernel facilities that might reduce some of the other processing overhead, so there's also that
17:09 nightwalk you also have to consider that part of the CPU busy work COULD be from too little ram, having to swap out pages, evict cache, and so forth
17:10 nightwalk (that sort of thing will normally show up as additional system load in top)
17:17 \_pol joined #gluster
17:22 JonnyNomad joined #gluster
17:32 lpabon joined #gluster
17:44 FilipeMaia joined #gluster
18:01 robos joined #gluster
18:22 nightwalk oh, then there's the matter of operations on multi-access filesystems being synchronous and blocking for safety. and hw raid normally being set to write through vs writeback because of lack of bbu, bad bbu, etc
18:22 nightwalk all of which increase io wait, which ups load average (and I'm assuming that what you meant by cpu usage -- the two aren't the same thing)
18:39 alinux joined #gluster
18:41 daMaestro joined #gluster
18:41 bstr joined #gluster
18:44 andreask joined #gluster
18:46 mooperd joined #gluster
18:48 \_pol joined #gluster
18:54 piotrektt joined #gluster
19:05 ricky-ticky joined #gluster
19:13 rwheeler joined #gluster
19:22 FilipeMaia joined #gluster
19:27 RangerRick8 joined #gluster
19:35 \_pol joined #gluster
19:35 kaptk2 joined #gluster
19:36 bugs_ joined #gluster
20:12 kushnir-nlm Nightwalk: Usage and load. Load from the 8 processes with dozens of threads on 4 CPUs, and CPU usage. I get what you're saying. Most of my stack is set for write-through, swap is not used. Disks are SSDs... IO wait is actually minimal throughout it all.
20:35 nightwalk kushnir-nlm: hmm...interesting. could very well be the same behavior then. I'm coming up blank with any possible explanations though, so hopefully it all means something to the few here who are in-the-know
20:42 JoeJulian nightwalk, kushnir-nlm: Though you mentioned nfs once, I think I'm reading this as you're doing these operations through fuse?
20:43 nightwalk JoeJulian: no. gluster happens to be in the same vm as the nfs kernel server, but that's as closely related as it gets in my case
20:45 nightwalk gluster taking up a bunch of cpu cycles probably just exposed a bug in the nfs server
20:45 nightwalk or perhaps the linux nfs client on the other end. not sure
20:46 JoeJulian ok, now I'm less clear. When you're walking the tree setting xattrs, is that through fuse or nfs?
20:47 nightwalk that is through fuse via a loopback gluster mount
20:48 nightwalk I'm using glusterfs for multi-master replication purposes, but only one of the two nodes actually has to be doing anything to trigger this behavior (and it happens only on THAT node)
20:49 nightwalk there's a dead brick still in most of the affected volumes, if that matters. It would take too much time and energy to blow away the glusterfs data and re-create the volumes, so I just left it as-is ever since the third replica became defunct
20:56 JoeJulian Ouch... the third defunct replica requires that every write or inode update tags the file as needing healed since a replica is offline. Doing a "remove-brick ... replica 2" would probably do wonders for that performance.
20:57 nightwalk I believe I tried that once upon a time and it gave me static
20:59 nightwalk actually, I've had some difficulty creating bricks and so forth too. I don't have my notes handy, but as I recall, a simple 'gluster volume stop' and 'gluster volume start' wasn't enough to kick a new volume into operation
20:59 nightwalk instead, I had to kill the relevant processes (kill -9) *then* volume stop/start
21:00 nightwalk I guess it can't hurt to try again just to be sure and refresh my memory on why remove failed though
21:00 JoeJulian Hmm, probably due to the peer being missing. I thought removal would work though...
21:01 nightwalk actually, we're supposed to be able to *add* bricks on-the-fly from what I read too. Never worked for me. Had to destroy the volume and re-create with a different replica count
21:01 JoeJulian I know for a fact that adding bricks works.
21:02 JoeJulian But I'm not sure if adding bricks works with a missing peer.
21:02 nightwalk probably my mistake with the cli then. I remember trying everything I could think of, but I ultimately just blew away the important volumes and re-created since it was easier
21:03 nightwalk turned out to be a major mistake time-wise though -- crawling all of the files on each side and doing all of the checksumming et al is ungodly slow
21:04 nightwalk JoeJulian: ah, here's the reason:
21:04 nightwalk "gluster volume remove-brick data-kernel replica 2 storage1.lan:/data/.kernel storage2:/data/.kernel"
21:04 nightwalk "Removing bricks from replicate configuration is not allowed without reducing replica count explicitly."
21:05 nightwalk ^ it WAS specified explicitly though -- "replica 2"
21:05 JoeJulian ~pasteinfo | nightwalk
21:05 glusterbot nightwalk: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:08 nightwalk Interesting...I did 'gluster volume info' on the volume listed above and it only shows the two remaining bricks now, despite the remove-brick command having failed
21:08 JoeJulian Hmm
21:08 nightwalk I tried remove-brick on another volume and it reported "Remove Brick commit force successful"
21:08 foster joined #gluster
21:08 nightwalk however, 'gluster volume info' on that one still shows all three :/
21:09 nightwalk ugh...I tried the same remove-brick command a second time, and now the ONLY brick listed is the dead one :(
21:10 JoeJulian .... which brick was the dead one?
21:10 nightwalk storage3
21:10 JoeJulian Then why did you try to remove storage1.lan and storage2?
21:10 * nightwalk facepalm
21:11 nightwalk I guess because I was thinking in terms of add-brick
21:12 JoeJulian That's why it was complaining (and someone needs to file a bug report on the fact that it did it anyway). When you changed the replica count to 2, but tried to reduce the volume to 1 brick, those didn't match.
21:12 glusterbot http://goo.gl/UUuCq
21:12 nightwalk disregard the thing about that volume though. It appears the volume I tried that on was an artifact that I'd forgotten to clean up. oops...
21:13 nightwalk ok...when I try it on the RIGHT volume, it says "Remove Brick commit force successful"
21:14 nightwalk it being "gluster volume remove-brick repos replica 2 storage3.lan:/data/repos/.mirrored"
21:14 JoeJulian Sounds good.
21:14 nightwalk gluster volume info shows the right two bricks (storage1 and storage2) still in the volume, so it looks like that works and I *was* just doing something silly :)
21:15 JoeJulian I'll have to transfer my silly trophy to you.
21:15 nightwalk yay!
21:16 nightwalk Hmm...I forgot about having already removed storage3 from the original volume and tried to remove it again and got, "number of bricks provided (1) is not valid. need at least 2 (or 2xN)" from the remove-brick command
21:16 nightwalk The behavior is proper, but the error messages are damn-confusing sometimes :/
21:17 JoeJulian agreed
21:19 nightwalk I just tried purging the dead brick from a couple other volumes and it worked nicely
21:19 nightwalk JoeJulian: so thanks for helping me get that much squared away anyway :)
21:23 nightwalk ok. Just manually removed the dead brick from all eight of my gluster volumes on this particular replica set, so now I just need to point my script at a gluster mount again and see if the issue still exists (I suspect it will)
21:26 badone joined #gluster
21:29 JoeJulian You're welcome. Glad to help.
21:30 bsaggy_ joined #gluster
21:32 nightwalk Here's the script I was referencing if anyone is interested: http://paste.ubuntu.com/5834379/
21:32 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
21:33 badone joined #gluster
21:35 FilipeMaia joined #gluster
21:35 nightwalk It's just a fairly basic proof-of-concept script that records various file attributes as xattrs along with the checksum so that future iterations on the same directory/directories can determine whether or not anything has changed without having to spend time doing potentially heavy checksums on every file each time
21:37 nightwalk It's part of a larger home-built file-level dedup and indexing solution, but as I discovered, it's also kind of handy for stress-testing filesystems' xattr support :)
21:37 JoeJulian I would like to see this as a python script that uses glupy to communicate directly, or a java app that uses semiosis' new jni.
21:37 semiosis don't hold your breath
21:37 nightwalk you will probably NEVER see me make ANY python code -- I hate python
21:38 JoeJulian lol
21:38 nightwalk java is on my todo list to learn, but I haven't had a chance yet ;)
21:38 JoeJulian What's your language of preference (other than bash)?
21:38 nightwalk C or PHP, mainly
21:38 nightwalk I'm not opposed to ruby or other C-like languages, though
21:39 JoeJulian Well, libgfapi is ready for your C coding. :D
21:39 nightwalk It's just that pythons' idiotic, backward use of spaces as more than a delimiting character stirs up bad memories of languages past :/
21:39 JoeJulian I have actually started looking at how to add support to php as well. That's as far as I've gotten though.
21:40 semiosis ruby is hardly c-like
21:40 JoeJulian ruby is more perl-like.
21:41 nightwalk semiosis: I meant that in that you can embed code from other languages into it
21:42 semiosis jruby will be able to use libgfapi-jni
21:42 semiosis however you could also connect MRI via FFI
21:42 nightwalk not having to care so much where the domain of one language begins and another ends has a few distinct advantages ;)
21:42 semiosis PDQ IYKWIM
21:43 nightwalk plus, I really didn't have a choice but to dabble in ruby since that's what puppet uses
21:43 JoeJulian I would not want to be the "next guy" trying to debug that.
21:44 nightwalk It shouldn't be difficult for any computer scientist worth his salt *shrug*
21:44 JoeJulian salt's python.
21:45 nightwalk but the spaces...they aren't used as DELIMITING characters, and no braces to hug my code :(
21:45 JoeJulian Hehe
21:46 nightwalk fwiw, I don't really like makefiles either, but at least they can use the "legacy support" card there
21:47 nightwalk and makefiles are mostly generated automatically by other code these days anyway
21:50 nightwalk Hmm...interesting. The memory usage of 2G+ for gluster has dropped to a couple hundred megs of resident memory usage after removing the failed brick. It was storing something like dirty pages or the gluster equivalent for a write intent bitmap I'm assuming...?
21:53 brosner joined #gluster
21:55 JoeJulian Raghav just posted an email this morning with some information (and a change rfc) about that. https://lists.gnu.org/archive/html/​gluster-devel/2013-07/msg00000.html
21:55 glusterbot <http://goo.gl/Qeg9L> (at lists.gnu.org)
21:57 bsaggy joined #gluster
21:59 nightwalk That's actually some pretty good info. It gave me answers to a few other things that I've wondered about as well
21:59 JoeJulian Me too
22:02 nightwalk I use the same kind of process with the dedup solution I talked about earlier. One of the other scripts crawls through select directories and adds a hardlink for new files in the parent mount's .dedup directory, for example
22:03 nightwalk That makes it easier to detect how many duplicates of a given file that I have spread across all of the machines I run it on.
22:04 nightwalk I have other things I intend to build on it eventually, but that's about as far as I've gotten so far. Still trying to figure out what file sharing method(s) make the most sense in the context of a networked file-level dedup solution
22:05 nightwalk That post gave me a few more things to think about on THAT front now too...
22:06 JoeJulian Now if you could just turn that effort into a translator, you would probably have more fans than I do.
22:13 nightwalk possibly
22:13 nightwalk but the bar for entry is way higher for that sort of thing since I'd first have to get intimate with gluster's internals
22:14 nightwalk It also wouldn't be portable that way, obviously
22:14 JoeJulian Do you have intimacy issues? ;)
22:14 nightwalk I needed this to work not only on linux, but also on OI/solaris and *BSD as well
22:15 nightwalk Which, just a few small tweaks to what I have and it *should*
22:16 nightwalk current versions of gluster don't build on *BSD and it seems to be a pain on solaris as well
22:19 nightwalk JoeJulian: anytime python is involved, there's a good chance of "intimacy issues", yes :)
22:20 * jclift_ was briefly thinking about how to do a de-dup translator the other day too
22:21 jclift_ (and also a compression one, in the hope it would be easier)
22:21 jclift_ Compressing and decompressing files -> pita
22:21 nightwalk jclift_: it wouldn't be too difficult except for the multi-access part. the server would really have to be the one to do it, and even then, only ONE could do it
22:21 jclift_ Well, for dedup, it just needs to do a lookup of what it's after, and then appropriate store/retrieve
22:21 jclift_ For non-compressed, it's not too weird
22:22 nightwalk otherwise you'd be looking at having massive amounts of ram being used on each system for the DDT (dedup table) plus all of the cache consistency issues of sql and various other cluster & HA technologies
22:23 nightwalk it's actually not weird even when compressed. you just have to dedup before you compress
22:23 * jclift_ cheating solution would be to md5sum (or similar acceptable hash) each new incoming block, then store the hash + block offset in an external service.  Redis or PG or something
22:24 nightwalk the script I posted earlier does the checksumming at the file level, which is the most resource-efficient form
22:24 nightwalk large amounts of block checksums leads to large DDTs in a hurry. zfs' dedup demostrates that pretty well
22:25 jclift_ Meh, give them to an external service that can be centralised just for ie
22:25 jclift_ it
22:25 jclift_ eg off host redis or something
22:25 jclift_ let it take care of that problem :)
22:25 nightwalk I don't particularly like using the user namespace for the xattrs, but it's the most portable scope to use
22:26 nightwalk No, no, no. offsite hosting is majorly bad without high grade encryption at the client (something else we don't have a translator for, if I'm not mistaken)
22:26 jclift_ off host, not off site
22:27 nightwalk I dunno. sounds like an SQL server revisited to me
22:27 jclift_ eg have the redis store be on different boxes, so you're not having ram problems on the gluster boxes
22:27 jclift_ Sure.  Similar, but more lopside performance for a specific use case
22:28 jclift_ as far as I understand so far, redis is kind of optimised for key/value stuff, and pretty much only that
22:28 nightwalk ok...let's take your example and say we use SQL as the backend. Then we have large amounts of latency while querying, so we have to add memcached. What does memcached do? It stores things locally in ram where possible
22:28 jclift_ Why the latency assumption?
22:28 JoeJulian Do compression/deduplication as a post-process. Writes go to a file as they do now. glustershd comes along, checks the compressed attribute. Finding none, it creates a new compressed file and sticky-links to it.
22:28 jclift_ Yeah, that's a useful thought too
22:29 nightwalk jclift_: because you can't dedup unless you know whether a block is unique or not
22:29 jclift_ Running "hardlinks" on a filesystem is an (easy) form of dedup isn't it?
22:29 nightwalk JoeJulian: that's what we concluded in my discussion with bettering zfs' dedup as well
22:30 nightwalk However, if you're going to essentially be doing batch dedup anyway, it may as well be it's own thing rather than being intimately tied to one specific stack
22:30 JoeJulian It has all the advantages of journalling and unless you REALLY need those bytes RIGHT NOW then it's a good tradeoff.
22:30 nightwalk jclift_: yes. that's what I do atm
22:30 * jclift_ doesn't understand why having an external thing == "Then we have large amounts of latency while querying"  ?
22:31 jclift_ Using ethernet maybe, but that's the low end scenario
22:31 nightwalk jclift_: because NICs are slow unless you're doing something like 10GbE /w rdma or ib/iser
22:31 * jclift_ would call RDMA an assumption
22:31 jclift_ IB being pretty cheap and all
22:31 nightwalk maybe, maybe not *shrug*
22:32 jclift_ Heh, k
22:32 nightwalk I'd still venture to guess that even 10GbE would be painful to operate over
22:32 nightwalk DDR ib maybe. certainly QDR
22:32 jclift_ Yeah, I like IB.  Personal preference tho
22:32 nightwalk and yes, the adapters are cheap, but the cables and switches are still ungodly expensive
22:32 jclift_ Which reminds me, I'm getting distracted.
22:33 * jclift_ gets back to RDMA stuff
22:33 jclift_ QDR yeah
22:33 jclift_ I havent found a decent source of not-stupid-expensive QDR/FDR cables yet :(
22:34 jclift_ DDR are ~US$14 tho, which is pretty workable in comparison
22:35 jclift_ nightwalk: Anyway, if you do ever get some dedup style translator code running, there are plenty of people around who'd be willing to test it :D
22:37 nightwalk I bet there are. Probably a bunch of people like me waiting for the dragonflybsd people to make good on their HAMMER2 promise :)
22:41 \_pol joined #gluster
22:59 andreask joined #gluster
23:04 brosner joined #gluster
23:15 tc00per joined #gluster
23:19 ninkotech_ joined #gluster
23:19 ninkotech joined #gluster
23:20 fidevo joined #gluster
23:22 tc00per Anybody seen a whitepaper or blog post on moving your bricks to a subdir from your mountpoint WITHOUT requiring a complete rebuild? With only 263G/5.5T used in a 2x2 setup would it just be easier/faster to blow the gluster volume away and restore from backup? Thoughts/opinions welcome...
23:23 JoeJulian I've done that.
23:23 tc00per @JoeJulian... you've done which?
23:26 JoeJulian I killed the brick process (glusterfsd), unmounted the brick, did a "replace-brick ... force" to replace the old brick with the new brick. This starts glusterfsd on the new brick, but we haven't mounted yet so kill it again. rm -rf  everything under the mountpoint. mount the brick again. move everything to the new subdir. "getfattr -m . -d -e hex" the old brick root. setfattr all the attributes on the new brick root. clear all the xattr on the
23:26 JoeJulian old brick root. volume start ... force.
23:27 JoeJulian I can do a much longer writeup tomorrow if that's not clear.
23:30 tc00per So you leave the volume 'running' the whole time then...?
23:47 aknapp joined #gluster
23:47 aknapp hello
23:47 glusterbot aknapp: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
23:47 ninkotech_ joined #gluster
23:47 aknapp I"m having a problem with Gluster 3.3  I have it setup between two machienes on the same VLAN.  I've created a volume with 2 bricks.  However, replciation is not working
23:47 ninkotech joined #gluster
23:48 aknapp it's set up as a replicated volume
23:48 aknapp looking for someone to point me in the right direction
23:49 aknapp I've read everything I can find on the web and I'm out of options on how to troubleshoot this
23:50 aknapp running gluster on Debian Wheezy
23:55 JoeJulian aknapp: Don't write directly to the bricks. You have to mount the volume using a client. That's the usual thing that people do to get that.
23:55 JoeJulian I'm leaving the office. I'll check back in when I get home if that's not the problem.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary