Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-06-18

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 _dist or did I misread that?
00:02 Matthaeus You misread.  I was using dd directly.
00:02 Matthaeus If you graph dd's performance with ever-increasing block sizes, you'll notice that larger blocks are far, FAR more efficient.
00:02 Matthaeus nc effectively forces a 1k block size.
00:03 Matthaeus You can specify the size of the TCP receive buffer, which may or may not have an effect.
00:03 _dist alright I'm getting nearly 920 with the 1k bs. Do you think I'm loosing the last 3gbps on packet overhead?
00:05 Matthaeus 920 MByte?
00:05 _dist yes
00:05 _dist well, that's where I'm topping it's not consistent.
00:05 _dist it's within 50Mbytes of that per test
00:05 Matthaeus I think there are still inefficiencies with using a block size that small.
00:06 Matthaeus But all of this is an artificial benchmark anyway.
00:07 _dist yeap, do you know another tool I can use to send data for the same test case? I want to compare /dev/null receiver to my local disk receiver
00:07 Matthaeus dd to a file should work, right?
00:08 _dist yeah, but then I'm either forcing 1k writes on my FS, or not. 1k writes on FS are slow, bigger on NC is slow :)
00:08 Matthaeus I think you misunderstood something.
00:08 Matthaeus nc FORCES 1k blocks.
00:08 Matthaeus dd might be able to buffer, though.
00:09 Matthaeus so nc -lp 5000 | dd of=/tmp/test.file bs=1M
00:09 _dist ah
00:10 Matthaeus Mathematically, you can expect to lose 10% to tcp overhead, btw.
00:11 _dist awesome, changing the output bs through DD was the trick I needed
00:12 Matthaeus Getting happier throughput now?
00:13 _dist well NC still doesn't hit 10gbps to /dev/null, but I am getting about 550Mbytes when the NC is routed to my FS
00:13 _dist which is about 37% slower than local write
00:14 _dist (with the same block size)
00:17 _dist but, I feel comfortable that I have a baseline for data over network transfer to disk that's applicable to my environement. Thanks for your help!
00:19 Matthaeus Anytime!
00:31 coredump joined #gluster
00:35 _dist time for supper, home :) ttyl
00:38 vpshastry joined #gluster
01:02 vpshastry joined #gluster
01:03 ceddybu joined #gluster
01:04 ceddybu Do i lose failover functionality when using the NFS client instead of FUSE ?
01:09 bala joined #gluster
01:14 gildub joined #gluster
01:20 harish joined #gluster
01:45 Paul-C joined #gluster
01:47 ilbot3 joined #gluster
01:47 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:56 harish joined #gluster
02:02 huleboer joined #gluster
02:04 GabrieleV joined #gluster
02:10 semiosis ceddybu: yes. nfs client doesnt have automatic failover like fuse client does. people usually set up a virtual IP to workaround that
02:18 semiosis JoeJulian: around?
02:25 semiosis joined #gluster
02:39 semiosis JoeJulian: https://launchpad.net/~semiosis/+arch​ive/ubuntu-glusterfs-3.4qa/+packages <-- there you go
02:39 glusterbot Title: Packages in “ubuntu-glusterfs-3.4qa” : ubuntu-glusterfs-3.4qa : semiosis (at launchpad.net)
02:40 semiosis JoeJulian: btw, curious how you're deploying packages... do you have your own PPA?  your own apt repo?
02:40 semiosis i'll most likely see your answer (if any) tmrw, heading out shortly.
02:41 semiosis s/heading out/afk/
02:41 semiosis glusterbot: you're slow
02:41 glusterbot What semiosis meant to say was: i'll most likely see your answer (if any) tmrw, afk shortly.
02:44 DV__ joined #gluster
02:45 haomaiwang joined #gluster
02:46 vpshastry joined #gluster
02:57 ceddybu joined #gluster
02:57 ceddybu semiosis: Thanks for the information. is this virtual ip added to each glusterfs brick node?
03:01 ceddybu or is it a 'floating' ip that moves around the nodes....
03:04 ceddybu https://access.redhat.com/site/docume​ntation/en-US/Red_Hat_Storage/2.0/htm​l/Administration_Guide/ch09s04.html
03:04 glusterbot Title: 9.4. Configuring Automated IP Failover for NFS and SMB (at access.redhat.com)
03:07 ceddybu1 joined #gluster
03:09 gildub joined #gluster
03:14 ceddybu joined #gluster
03:20 huleboer joined #gluster
03:24 plarsen joined #gluster
03:31 ceddybu1 joined #gluster
03:40 bstr joined #gluster
03:43 RameshN joined #gluster
03:45 kanagaraj joined #gluster
03:49 ndarshan joined #gluster
03:49 itisravi joined #gluster
03:50 shubhendu joined #gluster
04:01 raxdigital Is it normal for /etc/exports to be blank when volume info shows 'nfs.export-volumes: on'
04:05 primechu_ joined #gluster
04:06 nthomas joined #gluster
04:11 RameshN joined #gluster
04:16 raxdigital !nfs
04:18 haomaiwa_ joined #gluster
04:18 hagarth joined #gluster
04:21 ndarshan joined #gluster
04:22 kdhananjay joined #gluster
04:27 haomaiw__ joined #gluster
04:29 ppai joined #gluster
04:31 kdhananjay joined #gluster
04:32 ceddybu joined #gluster
04:37 nbalachandran joined #gluster
04:45 psharma joined #gluster
04:56 RameshN joined #gluster
04:57 rastar joined #gluster
04:57 sjm joined #gluster
04:58 kdhananjay1 joined #gluster
05:04 vpshastry joined #gluster
05:07 kshlm joined #gluster
05:08 dusmant joined #gluster
05:09 koguma joined #gluster
05:15 edong23 joined #gluster
05:16 deepakcs joined #gluster
05:16 rjoseph joined #gluster
05:17 prasanthp joined #gluster
05:25 aravindavk joined #gluster
05:26 Philambdo joined #gluster
05:33 ppai joined #gluster
05:47 davinder14 joined #gluster
05:48 aravindavk joined #gluster
05:49 lalatenduM joined #gluster
05:54 hagarth joined #gluster
05:54 dusmant joined #gluster
05:54 rjoseph joined #gluster
05:56 rgustafs joined #gluster
05:57 shylesh__ joined #gluster
05:59 saurabh joined #gluster
06:04 bala1 joined #gluster
06:11 raghu joined #gluster
06:15 jtux joined #gluster
06:19 jcsp joined #gluster
06:23 jtux joined #gluster
06:23 Philambdo joined #gluster
06:23 psharma joined #gluster
06:23 gildub joined #gluster
06:23 harish joined #gluster
06:23 jbrooks joined #gluster
06:23 T0aD joined #gluster
06:23 hchiramm_ joined #gluster
06:23 ghenry joined #gluster
06:23 a2 joined #gluster
06:25 glusterbot New news from newglusterbugs: [Bug 1024465] Dist-geo-rep: Crawling + processing for 14 million pre-existing files take very long time <https://bugzilla.redhat.co​m/show_bug.cgi?id=1024465>
06:25 spajus joined #gluster
06:25 spajus_ joined #gluster
06:25 spajus joined #gluster
06:27 mbukatov joined #gluster
06:30 XpineX joined #gluster
06:32 nfsnobody joined #gluster
06:33 nfsnobody left #gluster
06:36 ceddybu joined #gluster
06:36 ekuric joined #gluster
06:43 spandit joined #gluster
06:47 ctria joined #gluster
06:56 Rydekull joined #gluster
07:02 eseyman joined #gluster
07:04 rjoseph joined #gluster
07:05 necrogami joined #gluster
07:08 ricky-ti1 joined #gluster
07:08 aravindavk joined #gluster
07:14 hagarth joined #gluster
07:15 kdhananjay joined #gluster
07:17 keytab joined #gluster
07:20 ThatGraemeGuy joined #gluster
07:20 dusmant joined #gluster
07:21 hybrid512 joined #gluster
07:22 ktosiek joined #gluster
07:22 nshaikh joined #gluster
07:24 ProT-0-TypE joined #gluster
07:28 aravindavk joined #gluster
07:29 glusterbot New news from resolvedglusterbugs: [Bug 874554] cluster.min-free-disk not having an effect on new files <https://bugzilla.redhat.com/show_bug.cgi?id=874554>
07:43 deepakcs joined #gluster
07:55 rotbeard joined #gluster
07:59 glusterbot New news from resolvedglusterbugs: [Bug 878663] mount_ip and remote_cluster in fs.conf are redundant <https://bugzilla.redhat.com/show_bug.cgi?id=878663> || [Bug 951661] Avoid calling fallocate(), even when supported, since it turns off XFS's delayed-allocation optimization <https://bugzilla.redhat.com/show_bug.cgi?id=951661> || [Bug 959887] clang static src analysis of glusterfs <https://bugzilla.redhat.com/show_bug.cgi?i
08:00 Nightshader2 joined #gluster
08:03 aravindavk joined #gluster
08:12 vimal joined #gluster
08:21 liquidat joined #gluster
08:30 ppai joined #gluster
08:35 karnan joined #gluster
08:49 ndarshan joined #gluster
08:50 ThatGraemeGuy joined #gluster
08:51 bala1 joined #gluster
08:53 dusmant joined #gluster
08:59 glusterbot New news from resolvedglusterbugs: [Bug 1020154] RC bug: Total size of master greater than slave <https://bugzilla.redhat.co​m/show_bug.cgi?id=1020154>
09:00 ThatGraemeGuy joined #gluster
09:02 spajus joined #gluster
09:05 jtux joined #gluster
09:06 hchiramm__ joined #gluster
09:08 Nightshader joined #gluster
09:12 RameshN joined #gluster
09:21 _polto_ joined #gluster
09:45 Sunghost joined #gluster
09:46 Slashman joined #gluster
09:47 ndarshan joined #gluster
09:47 Sunghost @Jsemiosis @JoeJulian @partner - Hello, i installed now 3.5.1beta2 on my Debian and have 100% CPU Load on brick running rebalance - any idea why
09:49 Sunghost rebalance is still in progress, but it seems that nothing happend - volume log doesnt show any new activity
09:49 bala1 joined #gluster
09:51 dusmant joined #gluster
09:52 rgustafs joined #gluster
10:01 Sunghost Now both bricks failed for rebalancing
10:02 Sunghost [rpc-clnt.c:208:call_bail] 0-vol1-client-0: bailing out frame type(GlusterFS 3.3) op(LOOKUP(27)) xid = 0x8209
10:03 deepakcs joined #gluster
10:11 Sunghost from kernel.log -> Pid: 3184, comm: glusterfsd Not tainted 3.2.0-4-amd64 #1 Debian 3.2.57-3+deb7u2 Gigabyte Technology Co., Ltd. GA-A75M-S2V/GA-A75M-S2
10:11 Sunghost perhabs a problem with kernel or hardware?
10:11 Sunghost JoeJulian ???
10:17 tryggvil joined #gluster
10:20 nthomas joined #gluster
10:21 tryggvil joined #gluster
10:23 dusmant joined #gluster
10:24 rjoseph joined #gluster
10:24 partner i wish i could reproduce that but don't have time to try it out
10:25 aravindavk joined #gluster
10:27 SpComb gurrr the ubuntu 14.04 glusterfs-client init scripts stall on the glusterfs mounts *before* bringing up the network on boot
10:30 Slashman joined #gluster
10:31 SpComb and huh, mount fails to mount it based on /etc/fstab even though the equivalent `mount -t glusterfs ...` works fine
10:33 XpineX_ joined #gluster
10:34 spajus joined #gluster
10:37 Sunghost strange is that i can stop  the rebalance via cli but not continue - if i say start again he say is started, but status says stopped
10:37 SpComb nas:/scratch    /srv/scratch    glusterfs       nosuid,nodev    0       0
10:37 SpComb is that incorrect somehow?
10:37 SpComb seems like the nosuid,nodev options screw up glusterfs
10:39 Sunghost i cant restart the rebalance - i will restart both bricks ;(
10:43 SpComb yeah, dropped the nosuid,nodev options (replaced with just "defaults") in /etc/fstab and the system booted up again
10:43 SpComb weird, I though nosuid/nodev were generic VFS options that should work with fuse?
10:45 davinder14 joined #gluster
10:55 partner should it have some _netdev option there to hint it needs networking? not sure about ubuntu but seen plenty of these mount issues, Joe wrote artice about it last time we discussed this
10:59 tdasilva_ joined #gluster
11:00 tdasilva joined #gluster
11:02 rjoseph joined #gluster
11:04 fraggeln joined #gluster
11:05 tty00 joined #gluster
11:05 fraggeln is there any way to specify the port that gluster-client should use?
11:07 nishanth joined #gluster
11:09 partner i would guess no as the ports are somewhat dynamically assigned for each brick
11:10 tty00 and why did you change the base-port from 24009 to 49152?
11:10 tty00 partner: no, glusterfs uses a range that can be calculated, so it's more like semi-dynamic
11:11 Sunghost how can i stop rebalance - actually nothing happend no process is running, but brick is online, vol is online and rebalance status says in progerss
11:11 edward1 joined #gluster
11:11 hchiramm_ tty00, https://bugzilla.redhat.com/show_bug.cgi?id=824233
11:11 glusterbot Bug 824233: high, medium, ---, amarts, CLOSED CURRENTRELEASE, Stick to IANA standard while allocating brick ports
11:11 partner each brick gets new port so you can't really predict unless you keep some track of them.. or query from gluster
11:15 dusmant joined #gluster
11:15 tty00 hchiramm_: tnx :)
11:15 hchiramm_ tty00, np! :)
11:17 eshy joined #gluster
11:20 morse joined #gluster
11:23 jiffe98 joined #gluster
11:31 deepakcs joined #gluster
11:37 aravindavk joined #gluster
11:39 harish joined #gluster
11:44 kdhananjay joined #gluster
11:47 romero joined #gluster
11:54 dusmant joined #gluster
11:54 harish joined #gluster
11:57 Nightshader joined #gluster
11:59 ppai joined #gluster
12:01 ctria joined #gluster
12:03 dusmant joined #gluster
12:04 Sunghost how can i downgrade from glusterfs 3.5 to 3.4.4?
12:05 fraggeln since I ryn 3.5 myself, why do you want to downgrade?
12:09 mbukatov joined #gluster
12:09 partner due to issues we've been trying to figure out lately and it was all working with 3.4
12:11 hagarth joined #gluster
12:14 shubhendu joined #gluster
12:18 DV__ joined #gluster
12:20 ndarshan joined #gluster
12:21 tdasilva_ joined #gluster
12:35 kanagaraj joined #gluster
12:35 qdk joined #gluster
12:39 coredump joined #gluster
12:40 Sunghost i removed brick2 and the peer from pool and installed glustfs new
12:40 Sunghost now it seems i run into this bug https://bugzilla.redhat.co​m/show_bug.cgi?id=1055928
12:40 glusterbot Bug 1055928: urgent, urgent, RHS 3.1.0, rhs-bugs, NEW , Volume creation fails with error "host is not in 'Peer in Cluster' state"
12:41 Sunghost peer status is ok but i cant add-brick ;(
12:41 Magne left #gluster
12:43 partner i already hinted yesterday to check that area. check possible /etc/hosts entries, dns and everything
12:44 Sunghost i have done and all is fine
12:44 _polto_ joined #gluster
12:44 partner especially faulty hosts entry might screw things up but could be some firewall too not allowing the brick ports
12:45 partner sure, i trust you, just pointing out potential places to check still
12:46 Sunghost i recheck, have no firewall between in lan and installation is fresh on both systems - names are resolveable
12:50 Sunghost see that on brick2 the glusterd didnt start on reboot - i will check logs
12:51 sroy joined #gluster
12:51 julim joined #gluster
12:51 Sunghost compared with brick1 - syslog says nothing about glusterfsd
12:56 glusterbot New news from newglusterbugs: [Bug 1110777] glusterfsd OOM - using all memory when quota is enabled <https://bugzilla.redhat.co​m/show_bug.cgi?id=1110777>
12:57 msciciel joined #gluster
13:04 marbu joined #gluster
13:05 lpabon joined #gluster
13:05 Sunghost now i have a problem with old brick
13:06 Sunghost add-brick from brick1 says "volume add-brick: failed: Staging failed on clusternode02. Error: vol1 is already part of a volume"
13:06 japuzzo joined #gluster
13:07 jdarcy joined #gluster
13:07 Sunghost remove-brick from brick2 says "volume remove-brick start: failed: Incorrect brick vol1"
13:07 Sunghost volume info shows only brick1
13:07 partner if you removed it there are still attributes stating it is/was part of volume, clean them up?
13:08 Sunghost how
13:08 brad_mssw joined #gluster
13:09 Sunghost i remove-brick and detach brick2 before like the howto from redhat
13:09 LebedevRI joined #gluster
13:10 Sunghost then i restart the server and do peer probe and add-brick
13:10 Sunghost what else should be cleaned up between the steps?
13:13 aravinda_ joined #gluster
13:13 RameshN_ joined #gluster
13:13 kaushal_ joined #gluster
13:13 shubhendu_ joined #gluster
13:13 kanagaraj_ joined #gluster
13:13 deepakcs_ joined #gluster
13:13 dusmantkp_ joined #gluster
13:13 rastar_ joined #gluster
13:13 prasanth|afk joined #gluster
13:13 vikumar joined #gluster
13:14 psharma_ joined #gluster
13:14 navid__ joined #gluster
13:14 sac`away` joined #gluster
13:14 hagarth1 joined #gluster
13:16 bala2 joined #gluster
13:17 bennyturns joined #gluster
13:17 hchiramm_ joined #gluster
13:18 ctria joined #gluster
13:18 RameshN__ joined #gluster
13:18 vikumar__ joined #gluster
13:18 prasanth|brb joined #gluster
13:18 deepakcs__ joined #gluster
13:19 psharma__ joined #gluster
13:19 aravinda__ joined #gluster
13:19 kshlm joined #gluster
13:19 rtalur__ joined #gluster
13:19 sac`away joined #gluster
13:19 partner you did all this: http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
13:19 glusterbot Title: GlusterFS: {path} or a prefix of it is already part of a volume (at joejulian.name)
13:19 nshaikh joined #gluster
13:20 hagarth joined #gluster
13:20 partner ie. you are re-adding the brick as far as i understood
13:21 bala1 joined #gluster
13:21 Sunghost yes i tried - because of the problems above
13:22 shyam joined #gluster
13:22 Sunghost should i run this on the brick1 or brick2? brick2 has no more gluster directories
13:22 partner the one you're (re-)adding
13:22 hchiramm_ joined #gluster
13:22 diegows joined #gluster
13:23 Sunghost yes brick2 but i have delete after detach and remove-brick all folders
13:23 Sunghost whould it be simplier to create a new vol2 and move all files from vol1 to vol2?
13:24 Sunghost on mounted volume of course
13:24 Nightshader Hi guys, are there snapshot possibilites available for Gluster 3.5?
13:27 Sunghost whould say yes - but not tested - see here http://community.redhat.com/blog​/2014/05/glusterfs-3-5-unveiled/
13:27 glusterbot Title: GlusterFS 3.5 Unveiled Red Hat Open Source Community (at community.redhat.com)
13:29 Nightshader Sunghost: You really should use mkfs to create a fresh filesystem for your brick2 - my guess is you have some hidden attributes on it
13:30 pkoro joined #gluster
13:32 Sunghost that whould be bad, because its a fresh install with raid6 and xfs and cost some time
13:32 partner that takes like few seconds to do the xfs again?
13:32 Sunghost to reinstall, or ... ok that should the question
13:32 partner for the brick
13:33 partner that's where the hidden attributes are, on .glusterfs dir that ocntain for example the info if it ever was part of any volume. and if it was it needs to be cleaned up
13:34 Sunghost i deleted that directory after remove-brick and detach peer - i though it whould not needed - should it be deleted by gluster in further versions?
13:34 partner nothing is ever deleted by gluster, that would be bad
13:34 Sunghost i actually have a fresh brick2 with only installed latest beta of gluster for debian
13:36 Sunghost ok how should i do that? actual the partition is mounted on /media/node02 <- can i simply run mkfs.xfs /media/node2 ?
13:36 mjsmith2 joined #gluster
13:36 partner umount it and then do the mkfs and then mount it back
13:36 partner i assume its "empty" with no real data and stuff
13:37 Sunghost you are right with gluster deleted anything, but the attributes or what ever was used in the volume or trusted pool
13:37 Sunghost yes its empty
13:37 partner as it reads on Joes block, it is intentional to leave those there
13:37 Sunghost i created it as i installed debian via the installer - have i gave mkfs any parameter or things like that
13:38 partner alright, i'm close to having my own 3.5.0 testing up with two servers
13:38 Sunghost yes but that leads to those failures you can find lot of it in the net and if i remove a brick and the peer from trusted pool why should i have those attr? if i rejoin it could be created new and all should be find
13:38 Sunghost fin
13:38 Sunghost sore fine
13:39 Sunghost ;) sorry fine
13:40 partner because if you join an old brick with whatever data to existing volume nobody knows what will happen, probably something gets screwed up
13:41 davinder15 joined #gluster
13:41 partner its simple to fix and is way safer this way
13:42 Sunghost but as i understand if i remove brick and perhaps detach from pool i have the problem i have and why not create if i join the pool and brick automaticly a new id or attr. e.g. to make a clean join
13:43 Sunghost back to my - should i mkfs.xfs on unmounted partition whitout parameter or have i set one?
13:44 partner because that is something you don't want to happen to your brick, to join any volumes automatically, possibly with wrong or corrupted data, you are now in control and removing few attributes is not that big of a deal
13:45 partner i've always used the 512 inode size but it was recently discussed here its not that bad if you go with 256, only might affect the performance a bit if any
13:46 Sunghost ok - there must be reason for it and iam no expert in glusterfs - so  its ok
13:46 Sunghost ok i deleted the .glusterfs directory and the attributes are still there? the only way now to fix is to reformat the partition - right?
13:47 Nightshader mkfs.xfs -i size=512 /dev/<yourraid6device> -f
13:47 partner it uses those internally. if the attributes don't fit into the given size it will use yet another inode for the file causing performance penalty
13:48 Sunghost can i delete these internal ids??
13:49 partner just mkfs and we're done, i've made the same operation already 8 times during this conversation to my old and new test boxes to make some bricks available
13:49 partner the info is there with the brick, not elsewhere
13:50 lalatenduM joined #gluster
13:52 bene2 joined #gluster
13:52 Sunghost i want to check current inode size and run tune2fs -l /dev/md2 but got unknow magic number and cant find filesystemsuperblock
13:53 Sunghost mh ok same on brick1 runs only on real devices?
13:56 plarsen joined #gluster
13:57 partner if its lvm then point the tune2fs to logical volume
13:57 Sunghost no lvm
13:57 Sunghost sorry for questions but i want to be sure all is correct an i have no new failure in xfs or gluster
13:58 jph98 joined #gluster
13:59 Sunghost ok find xfs_info and that say that md2 has isize=256 and sectsz=4096 - so it must be a simple mkfs.xfs on md2 right
13:59 partner hard to say, pointing it to /dev/md0 works for me (or whatever other device)
13:59 jph98 hi all, when using gluster for storage of transient files is there a built in mechanism to cleanup files after a specific retention period?
14:00 gmcwhistler joined #gluster
14:00 partner 16:47 < Nightshader> mkfs.xfs -i size=512 /dev/<yourraid6device> -f
14:00 partner just umount if first if its mounted..
14:00 jph98 or is it a case of writing a cleanup script...
14:02 jcsp_ joined #gluster
14:08 LebedevRI joined #gluster
14:11 XpineX_ joined #gluster
14:14 ndk joined #gluster
14:14 XpineX__ joined #gluster
14:19 Sunghost ok its simpler
14:20 Sunghost the volume path was/is node02:/media/node2/vol1/....many foldes and .glusterfs folder
14:20 Sunghost i deleted the .glustefs folder and backup some data from the folders
14:20 Sunghost i hold the vol1 directory
14:21 Sunghost now i deleted the vol1 and create it new and now i can add the brick
14:21 Sunghost peer status and volume info seems to be good - nativ mount is ok and size semms good to - i now will copy some files onto it
14:21 daMaestro joined #gluster
14:22 partner alright, good stuff
14:22 Sunghost next problem
14:22 haomaiwa_ joined #gluster
14:23 wushudoin joined #gluster
14:24 Sunghost nativ mount on brick1 with hostname brick1 is good but on brick2 with name brick2 doesnt work
14:24 Sunghost sh: 0: getcwd() failed: No such file or directory
14:25 partner i'm not following...
14:26 jobewan joined #gluster
14:27 partner i usually have say: server1:brick1 and server2:brick1 and so forth
14:32 Sunghost mh strange same command and now its mounted
14:32 Sunghost timeing problem?
14:32 P0w3r3d joined #gluster
14:33 partner i din't have remotest idea what you're doing but if its progressing the good :)
14:33 Sunghost your question -> if i add a brick i have to say the full path to the directory which will part of the volume
14:33 vpshastry joined #gluster
14:33 Sunghost if i mount the volume i only use the volume name e.g. vol1
14:33 partner that's how it goes yes
14:34 partner volume is just the name of the volume and gluster knows all the details. for any brick you will need to explicitly define host and path to the point where you have your brick
14:38 Sunghost yes thats how i understand to
14:38 Sunghost too
14:40 partner heh, 512 MB memory is not enough for 100 bricks..
14:42 Sunghost 512MB ?
14:43 Sunghost ok now i have the same problem like before in 4.4 it fills the first brick and say unsuficent space
14:43 mbukatov joined #gluster
14:44 partner so, you know have again distributed 2 brick system?
14:44 shyam left #gluster
14:44 jdarcy Is there supposed to be a community meeting today?
14:44 Sunghost i have to run reblance
14:45 Sunghost rebalance and he will make ~50:50 on both bricks - its not how i thing it should work but its ok
14:45 partner hmm fix-layout might be needed to hint gluster its ok to write there, not exactly sure of 3.5 behaviour
14:46 Sunghost i had the same problem in 4.4 as i add a 3brick and i have to run rebalance - before i tested it with fix-layout but finaly only rebalance work
14:46 partner jdarcy: so it seems according to etherpad
14:48 jdarcy Fix-layout will just cause a proportional share of *new* files to be placed on the new brick(s).  To move *old* files you need a full rebalance.
14:48 marbu joined #gluster
14:48 partner yeah but i recall to get any new files there that's at least what you might need to run (given the rebalance issues currently out there)
14:49 partner according to their hashes, that is
14:49 marbu joined #gluster
14:49 partner 10.10.2.94:test-volume  431G   90G  319G  22% /mnt/test-volume
14:49 partner quickly tested i have no issues with wheezy and 3.5.0
14:49 Sunghost ok and xfs
14:49 partner Number of Bricks: 99 x 2 = 198
14:50 Sunghost ;) nice
14:50 Sunghost any known problems with xfs
14:50 Sunghost brick1 reports again the same xfs trace like at the beginning
14:50 partner probably, i haven't had any issues, i have shortly 500M files on top of xfs and gluster
14:51 Sunghost log for volume stand now still, before its very fast scrolling
14:51 jdarcy It should also be possible to set up layouts that are weighted by *free* space so that most allocations go to the newest/emptiest bricks, but that's still a work in progress - 3.7 at least.
14:51 BradLsys joined #gluster
14:51 Sunghost sounds good jdarcy
14:52 partner jdarcy: my weighting is done so that i set the min.disk.free and once its reached the writes go elsewhere :)
14:52 Pupeno joined #gluster
14:52 Sunghost is set min.disk free to 100mb but it didnt write to the free brick
14:52 Sunghost Pid: 3547, comm: glusterfsd Not tainted 3.2.0-4-amd64 #1 Debian 3.2.57-3+deb7u2 Gigabyte Technology Co., Ltd. GA-A75M-S2V/GA-A75M-S2
14:52 Sunghost from brick1
14:52 Sunghost kernel problem? faulty drive?
14:53 partner i don't think that does anything there with your distributed volume
14:53 Sunghost cpu is up to 100% for glusterfsd
14:53 jdarcy I wouldn't rely on that too much, TBH.  It's a little less efficient and tends to fill new bricks one at a time instead of evenly, plus it's not the best-tested part of the DHT code.
14:53 kshlm joined #gluster
14:54 partner Sunghost: i've got such message moment ago as my memory was maxed out, check yours?
14:54 primechuck joined #gluster
14:54 partner jdarcy: i don't have any options..
14:54 Sunghost atop says 1.5 total and ree 11.1
14:55 Sunghost no swap in use and 11G free
14:55 partner rebalance leaks so it cannot be run long enough, this has worked for 1,5 or so without issues
14:55 jdarcy partner: Yes, I understand and apologize.
14:55 Sunghost ? didnt understand - its a known problem?
14:55 brad[] Hi all, I have a remedial quesiton - what is the functional difference between a replicated volume and a distributed replicated volume, when multiple hosts are involved?
14:55 partner i would like to see a graph of fragmentation of that volume, i think its baaaad but at the moment nothing i could do to fix it
14:56 partner jdarcy: heh, not sure what you apologize :) not complaining or nothing, just saying how i'm doing stuff here (and why)
14:56 partner but as its in the news, the 3.4.4-2 should fix many so really waiting for it
14:57 jag3773 joined #gluster
14:57 bene3 joined #gluster
14:57 jdarcy partner: I apologize because I feel that we as a group should be doing more to address exactly these kinds of issues so that workarounds aren't necessary.
14:58 partner jdarcy: nah, its a risk i took with the known pros and cons
14:58 Sunghost cant follow - what should i do? downgrade
14:58 partner you're all doing great job and the project is going forward fast
14:59 partner Sunghost: different thing which won't affect you with those file amounts
14:59 hagarth joined #gluster
14:59 jdarcy Thanks for that.  :)  Sometimes all we see is the icky parts, it's good to know some folks appreciate the good parts as well.
14:59 partner sure, credit deserved, this has helped us a lot to scale the storage compared to previous approach
14:59 JustinClift *** Weekly Gluster Community Meeting is starting now in #gluster-meeting on irc.freenode.net ***
15:00 partner i'm limited giving help to the community but trying whenever i can
15:03 jdarcy What was the previous approach?
15:03 partner we basically set up servers and installed in-house service to store and serve the files from that particular node. it required lots of library updates and stuff to have all the clients aware of new storage (http traffic)
15:04 partner now we have frontends doing stuff and backends with the gluster mounts, quite many levels but also gives great flexibility
15:04 jdarcy Yep, a lot of our users seem to come from home-grown sharding.  Might be more common than single NAS or other DFS.
15:05 tdasilva joined #gluster
15:05 partner we didn't have anything this fancy 5+ years ago, everything "clustered" was more close to fsck'd...
15:05 raghug joined #gluster
15:06 partner brad[]: sorry, did not ignore you, was just looking for some pictures..
15:06 zaitcev joined #gluster
15:06 partner brad[]: would this explain anything more: https://access.redhat.com/site/documentation/e​n-US/Red_Hat_Storage/2.1/html/Administration_G​uide/images/Distributed_Replicated_Volume.png
15:07 partner when talking with 2 servers each having something, the distributed has 50/50 of data on both. loose other node and you are left with 50% of data
15:08 partner replicated in same setup is 100/100, loose one and you still have all your data
15:08 brad[] partner: No problem - how would the picture change if the two servers had all 4 bricks in a replicated volume? All data on all bricks?
15:08 brad[] ah, ok.
15:09 partner dist-repl is just combined from both, maybe you have 4 bricks on both servers , each brick replicated between and then dist making them all sum up into one larger volume
15:10 jdarcy The most important thing is that "replica N" in a distribute/replicate volume allows you to survive any N-1 failures, regardless of the total brick count.
15:10 jag3773 joined #gluster
15:10 vpshastry joined #gluster
15:11 jdarcy We create as many N-way replica sets as we can given the total bricks, then distribute across those sets.
15:11 hagarth @channelstats
15:11 glusterbot hagarth: On #gluster there have been 319019 messages, containing 12624997 characters, 2086512 words, 7697 smileys, and 1049 frowns; 1591 of those messages were ACTIONs. There have been 139121 joins, 3933 parts, 135342 quits, 24 kicks, 475 mode changes, and 7 topic changes. There are currently 241 users and the channel has peaked at 249 users.
15:12 hagarth wow.. I didn't notice that we peaked close to 250
15:12 primechuck joined #gluster
15:13 Sunghost @partner - i actually cant say where the problem is - what is the result of all above?
15:14 partner Sunghost: i lost track on your case, are you rebalancing now or did it fail?
15:14 _dist joined #gluster
15:15 Sunghost actually it runs and both bricks reported as in progress, but on brick1 the cpu runs with 100% <- is that normal?
15:16 Sunghost rebalance status seems to stand still no new size no new scanned no failures no skipped
15:16 _dist Just for everyone's info, I added a third replica to my volume yesterday (about 700GB of small-medium files and it did not tank my volume, everything was fine.
15:18 Sunghost could it be an xfs problem?
15:18 _dist Sunghost: gluster (for us anyway) takes up a significant amount of CPU usage if doing anything outisde normal day to day operations. A volume heal will usually consume up to 6/12 cores fully.
15:19 Sunghost mh - ok, but i didnt noticed it before - ok i will wait
15:19 partner yeah its "normal", i have 100% cpu per brick on every server
15:20 Sunghost while rebalance or in normal operation ;)?
15:20 partner rebalance
15:20 partner on normal operation my cpu is almost idling
15:21 Sunghost ok i am reassured
15:21 Sunghost should i run after the rebalance is successfull a fix-layout?
15:22 partner i have graphs from my boxes and at the point bricks doing almost 100% disk utilization the cpu stays idle, with some iowait on the very peak
15:22 partner not needed
15:22 partner fix-layout is lighter version of rebalance which purpose is to create the dir structure to new bricks allowing writes
15:23 Sunghost ok so like in my case - brick1 is full and i add new brick2 or not?
15:23 partner thought i see dirs coming even without that..
15:23 partner "Fixing the layout is necessary because the layout structure is static for a given directory. In a scenario where new bricks have been added to the existing volume, newly created files in existing directories will still be distributed only among the old bricks."
15:24 partner you need to do either rebalance or rebalance fix-layout
15:24 kdhananjay joined #gluster
15:24 Sunghost ok understand - sorry i am for 5min away
15:24 partner as stated earlier, fix-layout does NOT move any files, its just "mkdir" for the dirs of your existing brick1
15:24 partner to allow writes to new brick whenever the hash points them belonging there
15:25 _dist I'm wondering if anyone knows when gluster _can't_ replicate
15:26 _dist Is there some type of file lock etc that prevents it, or that it waits to be released?
15:28 _dist "failed to erase pending xattrs on volume" <-- anyone seen this error before?
15:28 jdarcy One of my goals is to make fix-layout unnecessary.  When bricks are added or removed, clients should automatically detect that their layouts are out of date and fetch/apply new ones from a canonical source.
15:28 partner yeah, that would be nice
15:28 jdarcy A lot of the infrastructure already exists for http://review.gluster.org/#/c/7702/ and just needs to be adapted.
15:28 glusterbot Title: Gerrit Code Review (at review.gluster.org)
15:29 partner i have huge directory structure so it takes insane amount of time to "fix" while the write could do it on the same go..
15:29 hagarth jdarcy:  I wonder if we should deprecate rebalance and bring in self-balancing semantics (migrate data on access or something like that)
15:29 jdarcy _dist: Something wrong with xattrs on one of your bricks?  Maybe something exhausted in the local FS?
15:30 jdarcy hagarth: I'd be really hesitant about *moving data* on access, for the same reason we stopped doing self-heal that way.  But layout fixes are much lighter weight.
15:31 _dist jdarcy: could be, I'm concerned, data seems to be "fine" ext4, what should I look into?
15:31 hagarth jdarcy: without self-heal data redundancy was at risk, that was probably one of the bigger reasons to move away from self-healing upon access.
15:31 jdarcy _dist: I think there are some ext4 tools to show consumption of various internal resources (e.g. inodes).  Something might turn up there.
15:32 jdarcy hagarth: Agreed.  The problem is that a rebalance I/O rate that's "under the radar" for some people in terms of performance disruption is also too slow (rebalance takes too long) for others.
15:33 _dist jdarcy: from "I need to look into this" to "let's drop all bricks but one" how concerned should I be about this?
15:33 jdarcy _dist: If GlusterFS is having trouble updating replication status, there's a potential - a *potential* - for lost data.
15:33 partner i'd be happy to be able to somehow tune the rebalance speed.. it does take forever with larger amount of data
15:34 jdarcy _dist: You're basically running in a state where you're unprotected from the next failure.
15:34 _dist jdarcy: how would I notice that kind of data loss?
15:35 jdarcy _dist: You'd probably see EIO errors and files reported as being in split-brain (though that's not exactly what it is).
15:36 jdarcy hagarth: To put it another way, I think we'll always need explicit migrate-data to get it over with, and if we have on-demand as well then we have to deal with the contention/coordination issues.
15:38 partner i've been running rebalance now for 10 servers around 24 hours, i've got moved max 57 GB from the semi-newest bricks further to newest ones
15:38 hagarth jdarcy: yeah, need to think through this. rebalancing at scale with the current approach will be very painful.
15:38 partner thought lots of failures too as the previous bricks are full so they cannot be balanced to proper places
15:39 elico joined #gluster
15:39 _dist jdarcy: I don't know what glusters requirements are, and until I checked thought everything was cool. My inode size is 256, I have tons left in that fs.
15:39 halfinhalfout joined #gluster
15:39 partner the server having 57 GB moved has 15 TB data which mostly should move forward from those bricks (due to hash)
15:40 brad[] partner: thanks btw
15:41 jdarcy _dist: It might also be worth checking for other errors related to xattrs.  Other than that, I don't know.  It doesn't remind me of any common failure mode.
15:41 partner np
15:42 hagarth _dist: also, check gluster volume status to see if all gluster bricks are online
15:44 _dist hagarth: everything is up, healthy, no split-brain just lots of healing going on, lots of heal-failed and that entry
15:46 Sunghost short reply - both bricks still in progress i will wait - no healt is necessary as i understand - many thanks so far for great team and product - good job - have to get the train ;) see you soon
15:47 hagarth1 joined #gluster
15:49 _dist I feel like this is somehow (based on timing of the error) related to me adding a 3rd replica, I think I'm going to drop that brick and see if that fixes the issue
15:52 haomaiwa_ joined #gluster
15:52 sjm joined #gluster
15:56 halfinhalfout looks like S56glusterd-geo-rep-create-post.sh is missing from the 3.5 ubuntu PPA packages
15:57 halfinhalfout is that intentional or a known issue?
15:58 jdarcy _dist: You're doing three-way replication?
15:58 MacWinner joined #gluster
15:58 _dist jdarcy: yes, would those xattr errors be "normal" for when it's building the third replica?
15:58 * jdarcy has tested three-way replication *from scratch*, but isn't sure how much mileage there has been on migrating from two-way to three-way.
15:59 jdarcy It does bring up some additional possibilities I hadn't considered.
15:59 _dist I have pinned down that the moment I changed the replica from 2-3 in glusterd.log these errors started in glustershd.log
16:00 jdarcy I think this should go in Bugzilla at this point.  There's probably an issue - maybe just a spurious error message - to do with adding/updating the pending flags in that particular case.
16:00 _dist what should I do? should I remove the newly created 3rd brick? This data is, beyond important so yes it's backed up, rolling back means loosing 12 hours though
16:01 _dist I'll make a bug for sure, but my primary concern right now is the data :)
16:01 ndk joined #gluster
16:01 jdarcy If you were already at replica two, I don't think the risk of data loss is very large.  It just means you'll be lacking that *second* level of protection until we sort things out.
16:02 _dist ok, only the _old_ bricks are comaplining in their logs, the new one thinks it's fine
16:02 bene3 joined #gluster
16:02 jdarcy _dist: Yeah, each brick has information about the state *on other bricks*, because we can't get state on brick X when X is down.  ;)
16:03 zerick joined #gluster
16:03 jdarcy _dist: My guess is that the operation to update the flags on $oldbrick for the state on $newbrick is failing because that xattr doesn't exist yet.
16:04 _dist jdarcy: that would explain this behaviour 100%
16:04 jdarcy _dist: But now that we know it's a two-way-to-three-way thing, it should be possible to reproduce and/or diagnose further.
16:05 _dist jdarcy: I'm willing to do that in a test setup I did tell myeslf (as I pressed enter) that I should really simulate the add brick before doing it)
16:05 jdarcy I might try to reproduce it this afternoon if I can clear my "desk"
16:05 bala joined #gluster
16:06 _dist so in the meantime, I don't need to panic. Is there any way we can quickly verify your hypothesis?
16:07 _dist I suppose a newly created file, would never show this beahviour. But, I have no idea how to track that, I'd rather verify that something is missing from an old file if possible
16:08 haomaiwang joined #gluster
16:09 jdarcy _dist: It should be possible to create an R=2 volume, write a few files (possibly just one), add-brick moving to R=3, access the file, and see if the same messages pop out.
16:10 _dist jdarcy: that would only show me that the same problem happens, not that the cause is missing xattr data for node 3s stuff on node 1/2.
16:11 kaushal_ joined #gluster
16:12 _dist (sorry if I'm being picky, but loosing any of this data would be pretty unacceptable)
16:12 jdarcy _dist: I'm doing that experiment now.  Once I see exactly what's in the logs I might be able to figure out whether that's it.
16:13 _dist Ok, I really appreciate it I'll wait right here :)
16:14 in joined #gluster
16:14 chirino joined #gluster
16:16 jdarcy Argh.  What version of GlusterFS are you running?
16:16 mjsmith2 joined #gluster
16:16 _dist 3.4.2 january build
16:17 partner darn, just upgraded from that to test 3.5.0 stuff
16:17 jdarcy OK, I don't have that handy right now, and there were some major changes to AFR (replication) since then.  Good news: the problem doesn't appear in master.
16:17 jdarcy Bad news: it's going to take longer to check this out.
16:18 halfinhalfout semiosis: appears http://s56glusterd-geo-rep-create-post.sh/ is missing from the 3.5 ubuntu PPA packages. is that known or intended?
16:18 _dist if your guess is correct, I expect dropping the 3rd brick will fix this issue rather than doing some manual repair
16:18 jdarcy If you could run the experiment I described and send me the logs, that would probably be helpful.
16:18 semiosis halfinhalfout: unknown, unintended
16:18 JoeJulian My guess is that it's not in master because the fix for another bug fixed that... just a sec and I'll see if I can find the one I'm thinking of.
16:18 _dist ok
16:20 * jdarcy hopes that the fix wasn't one of his, because that would be awkward.
16:22 halfinhalfout semiosis: thx. how do I submit request to get missing file looked at?
16:23 semiosis you just did :)
16:23 halfinhalfout :-) thx
16:23 semiosis yw
16:24 jdarcy My main contribution to three-way was http://review.gluster.org/#/c/4070/ back in November 2012, so that should already be in 3.4
16:24 glusterbot Title: Gerrit Code Review (at review.gluster.org)
16:25 _dist so if I read the log correctly, and your guess it right. It's complaining that it can't clear xattr (probably set as !unhealthy) for a file on the new node because it isn't there to begin with ?
16:26 jdarcy Pretty much.  Do you have an actual function and file/line number from the log?
16:26 _dist "[2014-06-18 16:22:17.342486] W [client-rpc-fops.c:1822:client3_3_fxattrop_cbk] 0-datashare_volume-client-1: remote operation failed: No such file or directory
16:26 _dist [2014-06-18 16:22:17.342507] I [afr-self-heal-entry.c:127:af​r_sh_entry_erase_pending_cbk] 0-datashare_volume-replicate-0: <gfid:8fcb6c51-945d-4e22-b011-999124931531>: failed to erase pending xattrs on datashare_volume-client-1 (No such file or directory"
16:30 Mo_ joined #gluster
16:32 jdarcy _dist: Anything in the logs for the old bricks, perhaps from _posix_handle_xattr_keyvalue_pair?
16:32 _dist well, that log is off an old brick, you want me to look in shd for that ?
16:33 jdarcy _dist: Are you sure?  Those look like entries I'd expect in the client or self-heal daemon, not the actual brick log (in /var/log/glusterfs/bricks/$munged_path.log)
16:33 _dist this "cat /var/log/glusterfs/glustershd.log | grep _xattr" on all three yields nothing
16:34 _dist oh ok, looking now
16:35 [o__o] joined #gluster
16:37 _dist last few lines from new brick's log --> https://dpaste.de/AYgV
16:37 glusterbot Title: dpaste.de: Snippet #271982 (at dpaste.de)
16:37 _dist a grep for _xattr showed nothing
16:38 _dist the very first lines after the volume was created are the same as well
16:38 JoeJulian ugh.. bugzilla is running slower than ever!
16:38 _dist sorry I mean brick was added
16:38 _dist (not volume created, its' months old)
16:38 jdarcy Hm, then it might not be exactly what I thought.  Those messages might be harmless BTW.  We really need to get some of the current AFR experts on this.
16:39 _dist in the meantime, beyond reproducing it via a test, is there anything I can/should do ?
16:40 B21956 joined #gluster
16:41 JoeJulian bug 1104861
16:41 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1104861 unspecified, unspecified, ---, pkarampu, NEW , AFR: self-heal metadata can be corrupted with remove-brick
16:42 jdarcy _dist: At this point it's hard to tell whether rolling forward or backward is better.  I'd say wait until someone smarter than me sees and responds to the bug report.
16:43 haomaiwang joined #gluster
16:44 _dist hmmm, well I suspect the safest thing would be to stop all shares (it's a fileshare), let my rsync backup run and use it as a basis for a brand new volume. I can't afford to wait if there is damage being done to data.
16:45 _dist and while I assume that bug JoeJulian just posted may not exist on 3.4.2 it makes me concerned about trying to remove the brick, I'll try that after I know I have a solid backup.
16:46 jdarcy I don't think that's likely.  This definitely seems to involve missing *metadata* for the third replica, not the data itself.  Still, better to have a plan B for when bigmouth (me) is wrong.
16:46 JoeJulian Oh, it's been around since 3.2
16:46 JoeJulian I just never removed bricks before.
16:47 _dist I'm going to go to lunch, this will pretty much be all I work on this afternoon. I'll post a bug when I get back after reproducing it, and try to plan out how to recover my prod if the worst happens. But you're right, it seems to just be metadata
16:47 _dist but bad metadata could cause a false split-brain, maybe it won't (hasn't yet)
16:47 _dist bbiab
16:47 jdarcy Correct.
17:03 Matthaeus joined #gluster
17:07 _dist back, starting test now, shouldn't take too long
17:12 plarsen joined #gluster
17:13 kiwikrisp joined #gluster
17:14 _dist single file did not reproduce issue, but I was lazy and the fs isn't identical
17:16 _dist so it is, unsafe to remove the new brick because remove-brick from replica 3-2 has an issue in 3.4.2?
17:24 tdasilva joined #gluster
17:25 lmickh joined #gluster
17:27 hagarth joined #gluster
17:28 mjsmith2 joined #gluster
17:30 mjsmith2_ joined #gluster
17:32 coredump joined #gluster
17:32 _polto_ joined #gluster
17:33 _dist Not sure where to go next, I can't reproduce it, it might require a certain amount of data, or it could be completely isolated to our setup
17:34 doekia joined #gluster
17:34 rotbeard joined #gluster
17:35 lalatenduM joined #gluster
17:36 Pupeno joined #gluster
17:38 kiwikrisp anybody upgraded to 3.5 and seen a huge performance drop off? Using replica 2 between 2 machines acting as nfs storage for xencenter. I upgraded from 3.4 and my performance died w/ processors over 100% utilization.
17:38 kiwikrisp sorry, storage for xenserver 6.2sp1
17:39 JoeJulian semiosis: 3.4.4-ubuntu1~precise2?
17:40 JoeJulian kiwikrisp: That doesn't sound very fun. I haven't heard about anything like that but it sure sounds like something's not right. Are you on 3.5.1beta2 or 3.5.0?
17:41 Pupeno joined #gluster
17:41 _dist maybe I did something wrong? With the volume online yesterday I ran "gluster volume add-brick volume replica x+1 new-brick" <-- was that the right way to do it?
17:42 kiwikrisp joeJulian: glusterfs 3.5.0 built on Apr 23 2014 12:53:55
17:44 ctria joined #gluster
17:44 haomaiwang joined #gluster
17:50 _dist JoeJulian: While watching the logs, it seems likely that only the new replica has xattr data for the health of all 3 bricks. What will the shd do? Brick 1&2 will always think 3 is un-healed, and brick 3 won't
18:03 _dist jdarcy: do you know who I should talk to for my issue? Does it make sense to file a bug if I can't reproduce it? Thx
18:03 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
18:07 halfinhalfout anyone know when Bug 1077452 - Unable to setup/use non-root Geo-replication; commits will be released? https://bugzilla.redhat.co​m/show_bug.cgi?id=1077452
18:07 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1077452 high, high, ---, vshankar, MODIFIED , Unable to setup/use non-root Geo-replication
18:07 glusterbot Bug 1077452: high, high, ---, vshankar, MODIFIED , Unable to setup/use non-root Geo-replication
18:15 kiwikrisp JoeJulian: is there anything I can check or optimization tweaks that you know of? Did the hardware requirements change dramatically, etc?
18:19 lalatenduM joined #gluster
18:20 semiosis JoeJulian: the patch version is ubuntu2~precise1
18:21 semiosis https://launchpad.net/~semiosis/+arch​ive/ubuntu-glusterfs-3.4qa/+packages
18:21 glusterbot Title: Packages in “ubuntu-glusterfs-3.4qa” : ubuntu-glusterfs-3.4qa : semiosis (at launchpad.net)
18:26 _dist I think it might be deleting files, but there is a lot of activity on the server so it's really hard to tell
18:27 _dist at the top of this, https://dpaste.de/ZLyQ <-- does that mean it deleted a file on other replicas because it was missing on one?
18:27 glusterbot Title: dpaste.de: Snippet #271999 (at dpaste.de)
18:29 _dist I can't have it doing that, it'll make fixing this a nighmare.
18:30 bet_ joined #gluster
18:33 Pupeno joined #gluster
18:38 brad_mssw joined #gluster
18:46 JoeJulian @ semiosis++
18:46 glusterbot JoeJulian: semiosis's karma is now 2
18:46 JoeJulian @ semiosis+=50000
18:47 _dist Ok, I added a this https://bugzilla.redhat.co​m/show_bug.cgi?id=1110914 <-- based on compares it doesn't look like I'm currently loosing data.
18:47 glusterbot Bug 1110914: urgent, unspecified, ---, pkarampu, NEW , Missing xattr data after adding replica brick
18:49 ramteid joined #gluster
18:50 Matthaeus joined #gluster
18:50 LebedevRI joined #gluster
18:56 theron joined #gluster
18:56 gmcwhist_ joined #gluster
18:57 glusterbot New news from newglusterbugs: [Bug 1110916] fix remaining *printf format warnings on 32-bit <https://bugzilla.redhat.co​m/show_bug.cgi?id=1110916> || [Bug 1110914] Missing xattr data after adding replica brick <https://bugzilla.redhat.co​m/show_bug.cgi?id=1110914>
18:59 Pupeno joined #gluster
19:08 semiosis JoeJulian: how do you distribute the debs?  do you have a ppa?  an apt repo?
19:12 JoeJulian We mirror your ppa
19:14 B21956 joined #gluster
19:14 semiosis cool
19:16 _dist Is there anyway I can stop healing attemps on my new brick, cancel all pending and the delete it so the remove-brick bug doesn't affect me?
19:23 gmcwhistler joined #gluster
19:28 edong23 joined #gluster
19:37 jcsp_ joined #gluster
19:39 * _dist prepares to rebuild volume from backup after business hours
19:51 julim joined #gluster
19:53 diegows joined #gluster
20:02 Pupeno joined #gluster
20:10 jason_ joined #gluster
20:25 jcsp__ joined #gluster
20:36 sjm left #gluster
20:39 Matthaeus joined #gluster
20:54 Matthaeus joined #gluster
20:54 theron joined #gluster
21:01 partner hmm, i forgot i was rebalancing and peer probed two new servers, i stopped the rebalance but the new peers keep insisting running the rebalance even if they aren't yet providing any bricks.. version 3.3.2, stop/status reports stopped but the two are yet in progress and prevents add-brick operation
21:02 partner just wondering should i this or that..?
21:03 partner nvm, detached the peers
21:06 partner hmm, talked with jdarcy about the need (or lack of it) of fix-layout earlier today but again my directory structure gets built even without any actions to the new brick..
21:08 partner not sure if the docs are up to date but i read from the behaviour i don't need to anything, there's already gigabyte of material stored.. :o
21:10 Pupeno joined #gluster
21:13 mql joined #gluster
21:15 partner hmph
21:18 diegows joined #gluster
21:21 partner crap, its scanning the files
21:27 mql joined #gluster
21:28 _dist so I rebuilt my entire volume as a replica 3 to start, and am currently sending over data via rsync
21:28 _dist once that's done I can final sync, turn off old and mount new as prod
21:43 cmtime joined #gluster
21:49 cmtime I am having a problem with glusterfs and small files 1-5MB.  When writing we are seeing write speeds of 1-2MB/s with small files but 50MB/s with large files.  Any ideas?
21:49 cmtime Type: Distributed-Replicate
21:49 cmtime Number of Bricks: 6 x 2 = 12.
21:49 cmtime performance.io-thread-count: 64
21:49 cmtime performance.flush-behind: OFF
21:49 cmtime performance.cache-size: 256MB
21:49 cmtime diagnostics.client-log-level: CRITICAL
21:49 cmtime diagnostics.brick-log-level: ERROR
21:49 cmtime cluster.min-free-disk: 30GB
21:49 cmtime storage.linux-aio: on
21:49 cmtime performance.write-behind-window-size: 512MB
22:00 Matthaeus cmtime: What's the network link between your servers like?
22:02 cmtime infinband on the bricks.  I also have a two share servers with ctdb/samba that mount gluster over infiniband as gluster clients.   All tcp over infiniband till RDMA works better.
22:03 Matthaeus Underlying FS and disks?
22:03 cmtime XFS
22:03 cmtime /dev/md4                /brick0                 xfs     defaults,noatime,nodiratime,log​bufs=8,largeio,swalloc,inode64
22:03 cmtime md4 : active raid0 md3[1] md2[0]
22:03 cmtime 78137742848 blocks super 1.2 256k chunks
22:03 cmtime
22:03 cmtime md3 : active raid5 sdy[12] sdz[11](S) sdx[9] sdw[8] sdv[7] sdu[6] sdt[5] sds[4] sdr[3] sdq[2] sdp[1] sdo[0]
22:03 cmtime 39068871680 blocks super 1.2 level 5, 256k chunk, algorithm 2 [11/11] [UUUUUUUUUUU]
22:04 cmtime bitmap: 0/30 pages [0KB], 65536KB chunk
22:04 cmtime md2 : active raid5 sdm[12] sdn[11](S) sdl[9] sdk[8] sdj[7] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdd[1] sdc[0]
22:04 cmtime 39068871680 blocks super 1.2 level 5, 256k chunk, algorithm 2 [11/11] [UUUUUUUUUUU]
22:04 cmtime bitmap: 0/30 pages [0KB], 65536KB chunk
22:04 Matthaeus Have you tested what kind of small file performance you get when writing to the FS raw, without gluster?
22:05 cmtime At the time of setup it was fine.
22:05 cmtime But randomly this week small file performance went from good to slow.
22:06 huleboer joined #gluster
22:07 cmtime With zero changes done on gluster.  I am going to try and schedule a restart in the next day or so but that is kinda a pain with my staff that use it.
22:08 lyang0 joined #gluster
22:09 primechuck joined #gluster
22:16 halfinhalfout joined #gluster
22:30 semiosis cmtime: please use pastie.org for multiline pastes
22:30 cmtime okay =)
22:33 semiosis small is a relative term of course, but usually the "small file" performance hit is down below 100kb
22:33 semiosis particularly for db-like workloads where you have writes of a few bytes or a few hundred bytes
22:33 semiosis lots of round trips
22:33 semiosis for little data
22:33 cmtime this is all video and image files HD
22:33 semiosis interesting
22:34 cmtime but I did just notice when I am copying files from inside the directory that I share out over samba I am seeing the same performance hit on gluster be it large or small files
22:35 semiosis a ha!
22:35 cmtime but doing a cat /dev/zero > xxx is writing at 50MB/s
22:35 semiosis neat
22:35 cmtime so I am getting conflicting data lol on the same mount point
22:36 semiosis most typical way to hit the small write penalty is something like dd if=/dev/zero of=/gluster/client/path count=1000 bs=100
22:36 semiosis vs bs=1M
22:38 semiosis good luck
22:38 * semiosis afk
22:38 cmtime ty
22:58 xymox joined #gluster
22:58 ProT-0-TypE joined #gluster
23:02 fidevo joined #gluster
23:03 primechuck joined #gluster
23:40 dtrainor hey JoeJulian, that brick finished sync'ing.  not sure how/where/when/why but it did

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary