Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-12-17

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:13 _Bryan_ joined #gluster
00:27 sputnik13 joined #gluster
00:28 sputnik13 noob question, what's the drawback of using ext4 over xfs?
00:31 JoeJulian performance is pretty equal. Ext resize and fsck take much longer, affecting maintenance and recovery needs. According to those in the know, the ext code is a lot messier than the xfs code, with the latter getting smaller and the former getting bigger and stringier.
00:35 sputnik13 joined #gluster
00:36 sputnik13 JoeJulian: cool, thanks
00:36 Paul-C joined #gluster
00:44 sputnik13 joined #gluster
00:55 devilspgd partner_, I've had glusterfs chew a lot more than that. It's semi-random, at one point there was no workload on the machine at all, but glusterfs was still busy with "something"
00:56 devilspgd I'm a gluster n00b and still really learning though, might be something I screwed up :)
00:58 MugginsM joined #gluster
01:07 ttkg joined #gluster
01:07 theron joined #gluster
01:18 theron joined #gluster
01:19 jaank joined #gluster
01:22 gildub joined #gluster
01:27 bala joined #gluster
01:32 theron_ joined #gluster
01:47 loki_ Why is gluster-server not part of epel ?
01:47 loki_ Is there a repo i can add ?
01:48 loki_ got it .. http://download.gluster.org/pub/gluster/glust​erfs/3.4/3.4.0/EPEL.repo/glusterfs-epel.repo
02:07 haomaiwa_ joined #gluster
02:17 daxatlas joined #gluster
02:38 jbrooks joined #gluster
02:40 gildub joined #gluster
03:01 _pol joined #gluster
03:16 cleo_ joined #gluster
03:17 meghanam__ joined #gluster
03:18 cleo_ Hi,
03:18 cleo_ How many gluster developers is in the world?
03:19 cleo_ i want to know the specific number
03:20 M28 joined #gluster
03:21 rejy joined #gluster
03:22 KHARE joined #gluster
03:24 rjoseph joined #gluster
03:30 ninkotech joined #gluster
03:30 ninkotech_ joined #gluster
03:33 DV joined #gluster
03:33 meghanam joined #gluster
03:35 kanagaraj joined #gluster
03:40 RameshN joined #gluster
03:54 nshaikh joined #gluster
04:01 atinmu joined #gluster
04:06 itisravi joined #gluster
04:11 shubhendu joined #gluster
04:21 hagarth cleo_: any reason behind wanting to know that number?
04:28 harish joined #gluster
04:30 spandit joined #gluster
04:30 nbalacha joined #gluster
04:31 plarsen joined #gluster
04:31 nishanth joined #gluster
04:41 cleo_ I trying to compare the distributed file systems and i think that gluster is the most best distributed file system
04:41 lalatenduM joined #gluster
04:41 cleo_ and i want to show that by making certain reasons
04:44 DV joined #gluster
04:49 atalur joined #gluster
04:50 saurabh joined #gluster
04:51 ppai joined #gluster
04:52 jiffin joined #gluster
04:53 rafi joined #gluster
04:53 Rafi_kc joined #gluster
04:53 ndarshan joined #gluster
04:54 anoopcs joined #gluster
04:56 eightyeight joined #gluster
04:56 xrsa joined #gluster
04:59 nshaikh joined #gluster
05:00 tomased joined #gluster
05:03 al joined #gluster
05:05 soumya_ joined #gluster
05:06 DV joined #gluster
05:07 gildub joined #gluster
05:09 quydo joined #gluster
05:17 prasanth_ joined #gluster
05:18 sac_ joined #gluster
05:23 hagarth cleo_: it is hard to quantify but you may find some approximate stats here - http://bitergia.com/projects/red​hat-glusterfs-dashboard/browser/
05:25 kdhananjay joined #gluster
05:26 Anuradha joined #gluster
05:30 jbrooks joined #gluster
05:37 aravindavk joined #gluster
05:52 overclk joined #gluster
05:55 meghanam__ joined #gluster
05:55 meghanam joined #gluster
05:55 soumya_ joined #gluster
06:01 nshaikh joined #gluster
06:05 raghu joined #gluster
06:06 kshlm joined #gluster
06:08 ricky-ticky1 joined #gluster
06:19 SOLDIERz joined #gluster
06:26 JoeJulian loki_: Of course, that will only install the 3.4 version. I would recommend you try 3.5 or 3.6 if you're just getting started.
06:35 DV joined #gluster
06:40 anil joined #gluster
06:44 kovshenin joined #gluster
06:51 bala joined #gluster
06:54 ppai joined #gluster
06:56 kdhananjay joined #gluster
07:00 sputnik13 joined #gluster
07:03 shubhendu joined #gluster
07:14 Paul-C joined #gluster
07:16 pcaruana joined #gluster
07:21 ekuric joined #gluster
07:22 ctria joined #gluster
07:23 sputnik13 joined #gluster
07:25 jtux joined #gluster
07:28 ghenry joined #gluster
07:35 ndarshan joined #gluster
07:38 rgustafs joined #gluster
07:39 rgustafs joined #gluster
07:39 Philambdo joined #gluster
07:40 badone joined #gluster
07:41 shubhendu joined #gluster
07:51 morsik joined #gluster
07:53 ndarshan joined #gluster
07:55 ndevos @later tell cleo_ openhub (renamed ohloh) has statistics for many projects, including gluster: https://www.openhub.net/p/glusterfs
07:55 glusterbot ndevos: The operation succeeded.
08:01 marcoceppi joined #gluster
08:10 haomaiwa_ joined #gluster
08:10 deniszh joined #gluster
08:17 fsimonce joined #gluster
08:32 Fen1 joined #gluster
08:38 DV joined #gluster
08:41 Fen1 joined #gluster
08:42 bala joined #gluster
08:54 lyang0 joined #gluster
09:03 Debloper joined #gluster
09:03 ndarshan joined #gluster
09:07 LebedevRI joined #gluster
09:15 ppai joined #gluster
09:16 Gorian joined #gluster
09:18 ghenry joined #gluster
09:21 bala joined #gluster
09:23 liquidat joined #gluster
09:37 gildub joined #gluster
09:38 karnan joined #gluster
09:50 mbukatov joined #gluster
09:53 bala joined #gluster
09:59 [Enrico] joined #gluster
10:03 jaank joined #gluster
10:04 nshaikh joined #gluster
10:10 Slashman joined #gluster
10:12 elico joined #gluster
10:20 rgustafs joined #gluster
10:20 cultavix joined #gluster
10:20 cultavix joined #gluster
10:24 karnan joined #gluster
10:26 diegows joined #gluster
10:30 tvb|afk joined #gluster
10:30 tvb Hi guys, I need some help with removing a brick so I can reinstall the server
10:31 mator go ahead
10:31 tvb assuming it is best to remove the brick before reinstalling is the correct way
10:31 gildub joined #gluster
10:32 nshaikh joined #gluster
10:34 tvb mator: you want to help me?
10:40 mator not sure, since you've not asked any question yet
10:40 Fen1 why do you want to reinstall ?
10:43 tvb sorry here goes
10:44 tvb I am having issues with a two peer cluster
10:44 tvb I am unable to fix a split-brain
10:44 tvb I have one 'good' peer
10:44 tvb I want to reinstall the bad peer to start over
10:46 ndevos maybe base your actions on http://gluster.org/community/documen​tation/index.php/Gluster_3.4:_Brick_​Restoration_-_Replace_Crashed_Server ?
10:47 tvb jep read that page
10:47 tvb but that assume my node is already installed
10:47 tvb I have now stopped glusterd on the bad server
10:48 tvb Can I leave the peer disconnected and start reinstalling
10:48 tvb Or do I need to remove the bricks etc completely?
10:48 glusterbot News from newglusterbugs: [Bug 1171954] [RFE] Rebalance Performance Improvements <https://bugzilla.redhat.co​m/show_bug.cgi?id=1171954>
10:49 ndevos well, you can completely re-install the server, including re-formatting the bricks... is that what you want to do?
10:52 marcoceppi joined #gluster
10:53 nishanth joined #gluster
10:55 tvb ndevos: well yeah
10:55 tvb and then add a 3rd node as well
11:02 calum_ joined #gluster
11:04 tvb my good node has glusterd 3.3.0
11:04 tvb can I still connect when I update my bad node to the latest available version?
11:05 ndevos oh, I'm not sure about that...
11:05 Norky joined #gluster
11:06 tvb hmmm so what would be the beste route here?
11:06 tvb reinstall to 3.3.0 as well and then upgrade?
11:06 ndevos I would first make sure everything is working correctly with 3.3, and then upgrade to 3.4, 3.5 and maybe even to 3.6 - but one step at the time
11:07 tvb ndevos: ok
11:08 ndevos upgrading from 3.3 to 3.4 may require some steps in offline mode, but you should read the release notes to be sure
11:08 ndevos @upgrade notes
11:08 tvb or maybe add the 3rd node first?
11:08 glusterbot ndevos: I do not know about 'upgrade notes', but I do know about these similar topics: '3.3 upgrade notes', '3.4 upgrade notes'
11:08 ndevos @3.3 upgrade notes
11:08 glusterbot ndevos: http://vbellur.wordpress.com/2012/​05/31/upgrading-to-glusterfs-3-3/
11:09 ndevos hmm, okay, then ,,(3.4 upgrade notes)
11:09 glusterbot http://vbellur.wordpress.com/2013/​07/15/upgrading-to-glusterfs-3-4/
11:13 ndevos and more release notes for 3.4 here: https://github.com/gluster/glusterfs​/tree/release-3.4/doc/release-notes
11:31 tvb|afk joined #gluster
11:31 tvb|afk back
11:31 nshaikh joined #gluster
11:35 SOLDIERz joined #gluster
11:45 SOLDIERz joined #gluster
11:47 SmithyUK joined #gluster
11:49 ppai joined #gluster
11:52 calum_ joined #gluster
11:54 anil joined #gluster
11:55 nishanth joined #gluster
11:59 smohan joined #gluster
12:00 lpabon joined #gluster
12:02 harish joined #gluster
12:03 jdarcy joined #gluster
12:07 kanagaraj joined #gluster
12:16 T3 joined #gluster
12:28 tvb ndevos: W: Failed to fetch http://ppa.launchpad.net/semiosis/​ubuntu-glusterfs-3.3/ubuntu/dists/​trusty/main/binary-amd64/Packages  404  Not Found
12:28 tvb don't think I can install 3.3 anymore
12:29 tvb oh
12:29 tvb it is looking for trusty
12:29 tvb hmmm
12:30 tvb shall I try with 3.4 instead?
12:31 tvb bah
12:31 tvb peer probe: failed: Peer file1 does not support required op-version
12:32 tvb oh the other way around works though
12:33 B21956 joined #gluster
12:33 Kins joined #gluster
12:35 ira joined #gluster
12:38 edwardm61 joined #gluster
12:38 tvb ndevos: does this command MOVE the brick or duplicates it
12:38 tvb gluster volume replace-brick $volname $from $to start
12:41 tvb or perhaps I need gluster volume replace-brick <VOL> brick1 brick2 commit force
12:42 tvb anyone who can assist me here? I don't want to fuckup the working brick
12:46 SOLDIERz joined #gluster
12:48 edward1 joined #gluster
12:48 tvb hmm cant restart the glusterd daemon on the new server anymore
12:48 tvb "readv failed (No data available)"
12:50 ira joined #gluster
12:51 kanagaraj joined #gluster
12:55 RobertLaptop joined #gluster
12:55 anil joined #gluster
12:57 TvL2386 joined #gluster
12:57 SOLDIERz joined #gluster
12:58 Fen1 joined #gluster
12:58 Gorian joined #gluster
13:00 mator tvb, what is your last problem? i didn't followed channel
13:01 tvb mator: hi
13:01 tvb ok im back to two peers, one Connected the other disconnected on purpose
13:01 tvb Now I have setup a 3rd node and want to connect that
13:01 tvb how to proceed?
13:05 SOLDIERz joined #gluster
13:05 tvb mator: ?
13:09 bennyturns joined #gluster
13:09 fandi joined #gluster
13:10 calum_ joined #gluster
13:10 tvb ok I probed the 3rd node
13:10 tvb State: Accepted peer request (Connected)
13:11 tvb hmm
13:11 tvb Peer 192.168.0.200 does not support required op-version
13:11 lalatenduM joined #gluster
13:11 tvb ok Im going to reinstall with ubuntu 12.04 en use 3.3
13:11 anoopcs joined #gluster
13:15 mator what version of glusterfs you're trying to connect? are they the same?
13:16 Gorian joined #gluster
13:18 tvb file1 is 3.3.x
13:18 tvb file3 (new one) was 3.4
13:19 tvb but im reinstalling with 3.3 now
13:19 mator ok
13:22 tvb ok installed
13:22 tvb now could you help me with adding it to the cluster correctly?
13:24 SOLDIERz joined #gluster
13:26 tvb mator: got a few min for me?
13:28 hchiramm joined #gluster
13:28 tvb ok State: Peer in Cluster (Connected)
13:28 tvb now I need to add the file3 as brick?
13:29 Slashman joined #gluster
13:35 SOLDIERz joined #gluster
13:35 tvb meh gluster volume add-brick STATIC-DATA replica 3 file3:/data/export3/ is resulting in Host file3 not a friend
13:38 theron joined #gluster
13:38 theron joined #gluster
13:40 calisto joined #gluster
13:41 M28 joined #gluster
13:43 tvb anyone who can help me with Host file3.<domain> not a friend
13:46 deniszh left #gluster
13:47 deniszh joined #gluster
13:49 SOLDIERz joined #gluster
13:49 vimal joined #gluster
13:54 vimal joined #gluster
14:01 tvb ok almost there
14:01 tvb i have succesfully added file3 as brick
14:04 tvb now how do I force a sync?
14:04 prasanth_ joined #gluster
14:06 tvb oh im guessing it is already running
14:06 tvb no active sinks for performing self-heal on file
14:07 julim joined #gluster
14:10 T3 joined #gluster
14:13 virusuy joined #gluster
14:13 virusuy joined #gluster
14:15 calisto joined #gluster
14:15 tvb or not
14:15 tvb not sure
14:15 tvb how can I very if the data is being synced
14:16 tvb I can see the folders, but the size of the files is staying 0
14:23 nbalacha joined #gluster
14:24 tvb hmm I guess some files are fucked
14:24 tvb Skipping entry self-heal because of gfid absence
14:26 leo__ joined #gluster
14:27 raghu joined #gluster
14:28 leo__ joined #gluster
14:35 marcoceppi joined #gluster
14:36 bene joined #gluster
14:36 dberry joined #gluster
14:38 nshaikh joined #gluster
14:38 shubhendu joined #gluster
14:39 jdarcy joined #gluster
14:44 sysadmin-di2e joined #gluster
14:45 sysadmin-di2e Has anyone experienced duplicate disks and files after upgrading to 3.6.1?
14:45 strata duplicate bricks? no. duplicate files, i'd hope so.
14:47 sysadmin-di2e sorry, we are running vm's on our gluster mount point which is replicated to two bricks.  But, on each brick, we are seeing duplicate files(ie. two disk files, two console.log)
14:49 SOLDIERz joined #gluster
14:50 sysadmin-di2e That is one of our issues,  The other issue is significant slow down to a crawl by some of the vm's and some are running normally.  I've check the disk utilization, disk read and write and io utilization and and each glusternode and hypervisor were not doing anything.
14:53 klaxa|work joined #gluster
14:56 klaxa|work i'm having some trouble using qemu with gfapi, here's some info: https://gist.github.com/klaxa/c6891413a5323cf9b43c
14:59 coredump joined #gluster
15:01 sysadmin-di2e Is doing a rebalance after upgrading to 3.6.1 suggested?  I did one right before the upgrade on 3.3.1.
15:04 wushudoin joined #gluster
15:07 lalatenduM joined #gluster
15:07 klaxa|work joined #gluster
15:08 klaxa|work i already enabled server.allow-insecure and rpc-auth-allow-insecure in /etc/glusterfs/glusterd.vol
15:08 RameshN joined #gluster
15:18 ndevos klaxa|work: did you do a 'gluster volume stop $VOLUME ; gluster volume start $VOLUME' after changing the server.allow-insecure option?
15:19 klaxa|work restarting glusterd is not equivalent i guess? i will try that
15:20 ndevos no, restarting glusterd is different, the brick processes need an updated configuration, and that configuration gets newly generated upon a volume stop/start
15:21 klaxa|work i hoped i could avoid that, but i will do that once the virtual machines are shut down
15:21 klaxa|work thanks!
15:21 ndevos restarting glusterd is needed for the change in /etc/glusterfs/glusterd.vol, but that does not affect the brick processes or their configuration
15:22 klaxa|work i just restarted the volume and i get the same error
15:22 ndevos changing server.allow-insecure should be possible without the stop/start with the latest (3.7-devel?) version, but I have not checked that
15:23 ndevos hmm, not sure what could be the issue then...
15:24 klaxa|work i'll try it out on the other machine with the other brick, maybe it's specific to that host
15:26 klaxa|work nope, that yields the same error
15:26 daxatlas joined #gluster
15:27 tdasilva joined #gluster
15:30 plarsen joined #gluster
15:31 klaxa|work funnily enough it works in the test-environment
15:31 klaxa|work i'll check for differences
15:32 lalatenduM joined #gluster
15:37 lmickh joined #gluster
15:37 plarsen joined #gluster
15:40 soumya_ joined #gluster
15:42 ndevos klaxa|work: what version of libgfapi is used on that server? http://review.gluster.org/5995 should be included in 3.5.0 and that is one thing that could cause issues with qemu
15:43 klaxa|work qemu should be linked against whatever is included in the 3.5.3 debian package
15:44 klaxa|work i'll check that, also it appears that server.allow-insecure was not set on the volume although it was specified, maybe i didn't do that correctly
15:46 ndevos klaxa|work: you can check the log of the brick, there should be a protocol/server part in the section where the xlator stack is printed, that section should have the allow-insecure option
15:49 ndevos klaxa|work: also, the hostname for the server is "vsh04.ib", right? would that happen to be "ib" for InfiniBand? things might still be a little unstable in 3.5 for that
15:49 klaxa|work yes, it's just the ip address, we're running ip over infiniband
15:49 klaxa|work so it's tcp after all
15:50 klaxa|work okay, now i get permission denied errors, i think that's easier to fix
15:50 klaxa|work https://gist.github.com/klaxa/cc3896b6339c414d59d8
15:50 jobewan joined #gluster
15:50 klaxa|work apparently server.allow-insecure was not set correctly
15:52 klaxa|work do i really have to set the volume ownership to the user running the virtual machine? is there no way around that?
15:53 klaxa|work what other implications does it have?
15:53 ndevos yes, permission denied... maybe qemu runs as a non-root user?
15:53 ndevos create the files and chown/chgrp them works too
15:54 klaxa|work ah, right i remember running into that problem in the test environment too
15:54 ndevos or have the files created in a directory where your qemu process can write/create files
15:55 klaxa|work changing the ownership of the files in question solved it, thanks for your help, the machine is running now :)
15:56 sysadmin-di2e Would I be correct in thinking that a rebalance w/ fix layout could potentially fix slow vm read write access?
15:56 ndevos sysadmin-di2e: no, not really, those are pretty unrelated
15:57 sysadmin-di2e What could I look at to resolve slow vm performance?
15:57 sysadmin-di2e All I did was upgrade from glusterfs 3.3.1 to 3.6.1
15:58 sysadmin-di2e I did a rebalance before the upgrade.
15:59 sysadmin-di2e Well I did add two bricks to bring the total up to 20.
15:59 ira joined #gluster
15:59 gkleiman_ joined #gluster
16:00 ndevos sysadmin-di2e: I'd start with http://www.gluster.org/community/docu​mentation/index.php/Virt-store-usecas​e#Applying_the_Tunables_on_the_volume
16:01 sysadmin-di2e would that still appy if some of our vm's are performing just fine and others are slow to the point where any command takes about 1-2 mins to complete?
16:02 kmai007 joined #gluster
16:03 kmai007 can glusterfs now export a directory path of a volume and mount it via fuse-client ?
16:03 kmai007 i know u can do that on the client side with NFS mounts
16:04 ndevos kmai007: no, that is not available yet
16:04 kmai007 thanks ndevos
16:04 kmai007 can i pose a question of how i achieved it? can somebody tell me if its a bad idea?
16:05 kmai007 i mounted the glusterfs volume via fuse
16:05 kmai007 on a client
16:05 kmai007 then i bind mounted the directory path off the root of the volume fuse mount
16:06 ndevos sysadmin-di2e: different VMs probably have different workloads, I do not think a performance issue can be generalized as easy as that
16:06 kmai007 like so http://fpaste.org/160638/88323891/
16:07 ndevos kmai007: yes, I think that is a relatively common way to solve it
16:08 sysadmin-di2e ndevos: On the slow VMs, I was noticing io wait times of over 50% and when I look at the hypervisor, they were at close to 0%.
16:09 kmai007 thanks ndevos
16:10 kmai007 ndevos do you know of addtional logging of glusterfs3.5.3-1 ?
16:10 kmai007 i upgraded from 3.4.3 and my storage server is logging alot more
16:10 kmai007 tho' everything appears to be operational
16:10 ndevos sysadmin-di2e: I really would not know about that
16:11 partner_ my upgrade from 3.3 to 3.4 did the same, logged clients to death
16:11 kmai007 partner_: how did u resolve it?
16:11 kmai007 log = /dev/null ?
16:11 kmai007 in /etc/fstab
16:11 ndevos kmai007: hmm, maybe, it can be that some messages have a WARNING/INFO level, but should be DEBUG instead (INFO is logged by default, DEBUG not)
16:12 _pol joined #gluster
16:12 ndevos kmai007: you can add a mount option to reduce logging, like log-level=WARNING
16:12 sysadmin-di2e Thanks for your heap ndevos
16:12 kmai007 can you point me to doc. how to modify my logging
16:12 sysadmin-di2e help
16:12 partner_ kmai007: we set the log-level=WARNING
16:12 kmai007 but that is for client side
16:12 kmai007 how about storage side of /var/log/glusterfs/bricks/<VOL>
16:13 sysadmin-di2e has anyone run into this issue? http://gluster.org/community/documentation/i​ndex.php/Gluster_3.2:_Setting_Volume_Options
16:13 sysadmin-di2e sorry wrong link http://www.gluster.org/pipermail/glu​ster-users/2014-October/019317.html
16:13 ndevos yes, you can set that on the bricks like: gluster volume set $VOLUME diagnostics.brick-log-level WARNING (or something like that)
16:14 ndevos check with 'gluster volume set help' for the exact name of the option
16:14 partner_ yeah i had issues on the client side, unfortunately there is no remount -option available so change is "big"
16:15 squizzi joined #gluster
16:15 kmai007 thanks ndevos
16:15 partner_ maybe on the server side too but at least there's plenty more disk for logs.. this whole logging topic should be seen into, its been on the discussions quite often lately
16:16 bene joined #gluster
16:16 bennyturns joined #gluster
16:18 kmai007 partner_: are you on 3.5.3 yet?
16:19 sysadmin-di2e Kmai007: what version are you on?
16:19 kmai007 glusterfs-3.5.3-1
16:19 kmai007 from 3.4.3
16:19 kmai007 it was a challenege
16:20 kmai007 was -> is
16:20 elico joined #gluster
16:20 sysadmin-di2e oh okay.  What challenges did you encounter?
16:20 RameshN joined #gluster
16:20 kmai007 i have quotas enabled, and i didn't know there was "prep" work to getting it carried over to the newer version of glusterfs
16:21 kmai007 and i was use to seeing the quota-limits displayed on the client
16:21 kmai007 you have to enable that feature on 3.5.3 now
16:21 kmai007 to see that limit on the client, else it will be the full size of the brick
16:21 sysadmin-di2e Oh okay.  Good info to know.  Thanks
16:22 kmai007 its not a bad release
16:22 kmai007 just things i did not expect to run into
16:22 sysadmin-di2e yeah, we went from 3.3.1 to 3.6.1.
16:24 daMaestro joined #gluster
16:25 kmai007 how is 3.6.1
16:25 kmai007 that is a big jump
16:27 sysadmin-di2e It was a big jump. Overall, it seems pretty good.  We are experiencing a few hiccups here and there.  The bad part about trying a new release is that documentation is lacking.
16:28 kmai007 agreed. its everywhere, from github, to gluster.org, to redhat storage 3.0 doc
16:29 wushudoin left #gluster
16:34 partner_ kmai007: not yet, 3.4.5 at this point
16:34 partner_ thought thinking should i put the newest to the new boxes and just run with couple of versions there. probably will setup a new volume anyways so shouldn't matter
16:35 partner_ lots of options and plenty of ways to screw things up
16:35 partner_ well, "lots" but some, we're solving it now with own software that is aware of the locations of the files and what not
16:37 kmai007 replication/distribution seems fine for me,
16:37 T3 joined #gluster
16:37 kmai007 the memory management also looks better on 3.5.3
16:38 partner_ most busy currently so don't think i have any time even think upgrading until february
16:38 saurabh joined #gluster
16:44 vimal joined #gluster
16:44 sysadmin-di2e Will a rebalance on a newer version fix IO errors from a previous rebalance from a different version?
16:45 partner_ hmm i can't find any traces of how/where the debian packaging is done.. was about to check if the logrotation issues will get fixed there aswell on the next version
16:47 jobewan joined #gluster
16:55 John_HPC joined #gluster
16:56 John_HPC Whats the proper way to restore the trusted.glusterfs.dht value?
17:29 partner_ ah ok i think i got it, forgot they were elsewhere
17:34 sac_ joined #gluster
17:34 Telsin joined #gluster
17:34 rwheeler joined #gluster
17:42 sputnik13 joined #gluster
17:43 side_control joined #gluster
17:46 sysadmin-di2e partner_: what is the avg speed that you are getting when transferring files between two bricks?
17:47 Telsin left #gluster
17:52 diegows joined #gluster
17:58 plarsen joined #gluster
18:15 TrDS joined #gluster
18:20 PeterA joined #gluster
18:27 anoopcs joined #gluster
18:28 siel joined #gluster
18:30 John_HPC OK. I think I figured it out
18:30 John_HPC http://paste.ubuntu.com/9552944/
18:30 John_HPC It apperas a "testclient" was setup
18:31 John_HPC can I just delete those?
18:32 JoeJulian John_HPC: rebalance fix-layout
18:32 John_HPC ok!
18:33 _pol joined #gluster
18:35 sysadmin-di2e if I do a netstat -tap | grep glusterfsd from one of the gluster nodes, should I see all of the of other nodes connected?  I was unde the impression that I should only see the replica pair.
18:37 JoeJulian Yes, you shuold. The self-heal daemon, the nfs service, and any client will connect to every brick in the volume.
18:38 sysadmin-di2e Thank you Joe
18:38 JoeJulian You're welcome.
18:38 sysadmin-di2e Can you point me in the right direction to troubleshoot my VM performance issues?
18:39 JoeJulian libgfapi ?
18:39 sysadmin-di2e I've upgraded our 10x2 gluster setup to 3.6.1 and now some of the VM's are extremely slow and some work just fine
18:39 JoeJulian How is it supposed to perform compared to what you're actually experiencing?
18:40 JoeJulian Ah
18:40 JoeJulian Does gluster volume heal $vol info show anything wrt those slow images?
18:43 sysadmin-di2e There is an at least 4 disk on each node that are being healed and one has 16
18:44 JoeJulian I would guess that's the problem and it should resolve when the heals are finished.
18:45 sysadmin-di2e But the crazy part isnt any stress being put on the hypervisor's.  I would think the iowait would be high but their not.  Most of the gluster nodes and hypervisors are sitting idle
18:45 sysadmin-di2e does it take a long time for these to heal?  This has been like this for the past 8 hours.
18:46 cmtime Any advice on small ish files. When writing image files jpg performance is 9-12MB/s with large files I am seeing 120MB/s. Number of Bricks: 6 x 2 = 12
18:47 sysadmin-di2e I was under the impression that everytime a change occurs in a VM image, a heal would need to occur.  Am I correct?
18:53 plarsen joined #gluster
18:55 JoeJulian No. The client handles updating the replicas. No healing necessary.
18:56 JoeJulian As long as the client can connect to all the bricks.
18:57 sysadmin-di2e so the clients are able to connect to all of the bricks, but there is long lag when trying to traverse through the mounted directories
18:58 sysadmin-di2e like it will take about 30 secs to run an ls command on the mounted directory where all of the instances are stored
19:01 sysadmin-di2e so I have the gluster mount point in the fstab of the clients
19:04 hagarth joined #gluster
19:07 partner_ sysadmin-di2e: you mean transferring between two volumes?
19:07 partner_ ref to your earlier question for me
19:08 sysadmin-di2e partner_: transferring between two bricks in the same volume
19:08 partner_ no idea, i've probably moved a brick only once ever
19:08 sysadmin-di2e oh okay.
19:09 partner_ thought i'm going to do such operation probably shortly, would be interested to know in advance how much its supposed to take..
19:09 sijis joined #gluster
19:09 sijis ,,services
19:10 sysadmin-di2e I was asking because I wasn't getting great perfomance on small files, but the there was a performance boost on larger files.
19:10 sysadmin-di2e But that is referenced in one of the blogs I was reading.
19:11 sijis sysadmin-di2e: my experience.. if you do a ls in a dir with a lot of files.. its slow
19:11 partner_ we have loads of files on directories but never need to list them, we know the path and filenames and access them always straight
19:11 sijis however, if you do it from brick .. the response is, at expected
19:12 sysadmin-di2e Thanks sijis
19:12 partner_ its a murder to do it via client mount :o
19:12 sijis we basically dropped gluster because of the ls thing. the application worked as expected (it know the path to the files)
19:12 sijis but from an application management perspective, our guys just coudln't deal with it
19:12 sijis partner_: ya.. completely agree
19:13 sysadmin-di2e yeah, we are thinking about moving to ceph instead
19:13 partner_ i can think several use cases where ls or similar kinds of things would be a must and slow performance on that major annoyance
19:13 sijis they had some processes to sync these volumes, via tar .. and it would be 40 min on nfs .. to 2hrs with gluster :(
19:14 sysadmin-di2e so sijis, you had nfs enabled on gluster?  Ours is turned off because I'm under the assumption that it will not work well with failover.
19:18 partner_ dodih, just finished up installing some gluster boxes, 280 drives, that should hold couple of files again
19:19 sysadmin-di2e so why is it that 'ls' is super slow on clients and runs just fine on bricks?
19:19 sijis sysadmin-di2e: no.. a typical nfs server not within gluster
19:20 sysadmin-di2e oh okay
19:20 sijis we were comparing typical nfs setup with a gluster setup. main reason to look at gluster was for redunancy.. if our nfs server needed maintenance (patches, reboot, etc) we'd have to bring down our apps
19:20 B21956 left #gluster
19:21 sysadmin-di2e oh okay.
19:22 sijis sysadmin-di2e: i never found out why the discrepancy. most of things i've read talked about gluster wasn't setup for that sort of task (paraphrasing)
19:22 sijis i do recall seeing some recent updates that should hvae improved some of that stuff but i don't konw for sure
19:24 sysadmin-di2e @sijis, I hope so.  The ls commands is not a big deal to me, but running our VM's are.
19:24 sysadmin-di2e Did you have issues with your applications that were hosted  on the glusterfs?
19:24 sysadmin-di2e what are y'all using instead of gluster?
19:25 sijis sysadmin-di2e: no issues on the application performance. none. and rebooting a gfs node (we have a 2 node replication setup)
19:25 sijis nobody noticed anything
19:26 sijis it truly is an awesome tech
19:26 sysadmin-di2e That is what we have, but after we added four more bricks and upgrade to 3.6.1, our VM performance has suffered.
19:26 sysadmin-di2e we were using 3.3.1 before.
19:29 sijis we were on 3.4.2
19:29 sysadmin-di2e yeah, I agree the technology is great.  I just wish the documentation was more organized.
19:30 sijis there was a blog with a few performance tweaks. they helped a little with the ls thing for us
19:30 sysadmin-di2e do you happen to remember the url?
19:31 sijis http://www.jamescoyle.net/how-to/​559-glusterfs-performance-tuning
19:31 sijis i honestly didn't notice much diff with noatime and nodirtime.. but the performache.cache-size i upped to 1G and that helped
19:32 sijis i can't recall the % improviement.. but it was maybe 10-15%. i didn't touch any of the other tweaks mentioned
19:34 sysadmin-di2e sounds good.  About to  try that now.
19:34 iherbig joined #gluster
19:37 T3 as per I understand of gluster so far, lvm is completely not needed on a default glusterfs setup, right?
19:40 sijis sysadmin-di2e: are you using gluster as an nfs target for vmware/xen?
19:40 sijis meaning.. putting vms on the volume
19:42 sysadmin-di2e sijis: we are not.  We are using openstack.  We are using ext4.
19:45 partner_ T3: correct but IMO it brings great flexibility
19:45 sijis sysadmin-di2e: ahh. that's cool.
19:46 strata sysadmin-di2e: in your journey of versions, have you found anything that says whether or not 3.6.x is stable or not? I'm on 3.6.x too and i'm beginning to wonder if i should downgrade to 3.4.x or something. there's no indication on gluster website saying which versions are devel/stable/unstable/willcrashyourshit
19:48 sijis sysadmin-di2e: never heard of ceph.. but i'll have to check that out
19:49 sijis strata: they have that noted somewhere.. i've seen it before
19:49 deniszh joined #gluster
19:50 strata i've been untar'ing a 255GB file containing ~55 million jpegs since Dec 14 into a distributed gluster volume.
19:50 * strata wonders if 3.4.x is faster
19:51 dgagnon joined #gluster
19:51 sysadmin-di2e strata: I have not seen anything as of yet to say that 3.6.x isn't stable.  We have just been trying to resolve the couple of issues that we've come across since installing it.
19:52 strata sysadmin-di2e: ah. your setup sounds a little different than mine :)
19:52 sijis strata: https://www.mail-archive.com/glust​er-users@gluster.org/msg17906.html
19:53 sijis that seems to give a good overall scheme of thigns
19:53 T3 partner_, would you recommend having something like hardware >> lvm >> gluster ?
19:53 sijis i thought there was something more 'official'
19:54 strata sijis: seems ok. though it doesn't really matter in my setup.
19:55 strata sijis: i'm using gluster for more space. i/o and HA isn't important.
19:55 sijis strata: you aren't replicating i assume?
19:55 sijis i can't think of the term for it.
19:56 strata sijis: nope. zfs -> kvm instance w/xfs -> gluster
19:56 strata distribute
19:58 sijis that sounds right
20:02 sysadmin-di2e Can you verify something for me?  I'm noticing on the the bricks, that it copies the path of every file from every brick, but it only has copies files that it is responsible for?
20:04 strata sysadmin-di2e: in a distributed setup?
20:04 sysadmin-di2e yes
20:05 strata [root@storage srv]# ls -al passwd
20:05 strata -rw-r--r--. 1 root root 1177 Dec 17 15:05 passwd
20:05 partner_ T3: that is how i've done my setups
20:05 glusterbot strata: -rw-r--r's karma is now -16
20:05 strata ^ just copied that to the gluster volume
20:05 T3 partner_, awesome, thanks
20:06 strata sysadmin-di2e: i only see it on one brick
20:06 partner_ T3: the HW layer brings you the basic hotswap ability, the LVM gives you flexibility to increase the bricks sizes as needed (ie. start with small and increase when needed)
20:06 sijis replica you will see copy on multiple bricks. it depends on how many bricks
20:07 T3 yeah, that's what I was thinking
20:07 partner_ T3: and lastly the gluster gives you the abilities you already know, move the whole brick away for example. kind of vmware storage vmotion for posix :o
20:07 T3 haha
20:07 T3 nice analogy, partner_
20:07 sysadmin-di2e So lets say that that I had brick 1 2 3 4.  I had the path /test/test1/test1 /test/test2/test2 /test/test3/test3 /test/test4/test4.  I'm seeing that on bricks 1 and 2, its shows the path /test/testX/testX for all of the paths, but only have files test 1 and test2.  Bricks 3 and 4 will have the paths /test/testX/testX and have files test3 and test4
20:07 partner_ T3: some people skip the HW (and lvm) and just use disks straight as bricks (ie. JBOD) and have gluster to do the fault tolerancy on top of it (ie. replicas)
20:08 partner_ haven't tried that myself, there might be some need to kill the processes for failed bricks or they might (?) block/affect rest of the volume
20:08 T3 I'm on the cloud.. don't even know which disks I have
20:09 partner_ ok, one layer less
20:09 T3 heh
20:09 partner_ but most remain still thought mmaybe you can skip lvm and scale the storage by adding more instances
20:09 kovshenin joined #gluster
20:09 T3 I like the idea of underlying lvm
20:09 T3 hmm
20:10 T3 you have a point
20:10 partner_ and if your volume needs to shrink then you just remove some bricks ie. get the data migrated away and then kill that instance
20:10 partner_ maybe some additional steps of course such as detach the peer but talking on high level here anyways
20:11 T3 do you think lvm would add any performance penalty?
20:12 T3 shrinking is not something we need to care about at this point
20:12 _pol joined #gluster
20:12 partner_ i don't think so
20:12 partner_ in fact, how else would you spread your brick on top of several separate raid arrays ie. more spindles?
20:13 partner_ anyways the bottleneck is most likely somewhere on the networking side, YMMV
20:13 T3 right
20:13 T3 the less I rely on networking, the better
20:13 T3 on my scenario that is critical
20:14 T3 so lvm comes handy
20:14 T3 I can just keep adding disks, and if someday that's not possible, then gluster can give me more on different instances
20:15 partner_ that is one option
20:15 partner_ its not too many commands to add a new "physical" disk to existing logical volume
20:16 partner_ maybe 3 :)
20:18 T3 heh yeah
20:20 daMaestro joined #gluster
20:20 partner_ but without lvm OR glusterfs, good luck expanding your volumes :o
20:21 partner_ possible of course but anyways out of the scope here, and harder anyways
20:25 theron joined #gluster
20:27 sysadmin-di2e strata: So lets say that that I had brick 1 2 3 4.  I had the path /test/test1/test1 /test/test2/test2 /test/test3/test3 /test/test4/test4.  I'm seeing that on bricks 1 and 2, its shows the path /test/testX/testX for all of the paths, but only have files test 1 and test2.  Bricks 3 and 4 will have the paths /test/testX/testX and have files test3 and test4
20:27 calisto joined #gluster
20:27 M28 joined #gluster
20:28 Bosse joined #gluster
20:30 T3 no way, partner_
20:40 strata sysadmin-di2e: so you're distribute+replica
20:43 sijis left #gluster
20:44 sysadmin-di2e strata: yes
20:50 daMaestro joined #gluster
21:32 squizzi_ joined #gluster
21:32 kovshenin joined #gluster
21:35 calisto joined #gluster
21:44 sysadmin-di2e JoeJulian: are you still available?
21:46 partner_ has he even woken up yet :o
21:46 JoeJulian Wrong timezone partner_
21:47 JoeJulian It's a quarter to two in the afternoon.
21:47 JoeJulian sysadmin-di2e: That's normal.
21:52 partner_ JoeJulian: hows that guarantee you're awake?-)
21:52 calisto joined #gluster
21:52 JoeJulian Heh, I'm a responsible adult. ;P
21:52 partner_ muah
21:55 partner_ sysadmin-di2e: the files belonging to the missing bricks will be simply unavailable and as Joe said its normal, rest of the files available on the volume will be served for the clients
21:57 n-st joined #gluster
21:58 partner_ one of the first tests i did when getting onboard with glusterfs, couple of VMs and setup volume there and see what happens when one gets rebooted
22:20 badone joined #gluster
22:36 sysadmin-di2e JoeJulian: So I 've narrowed it down to two bricks.  All the VM's on that pair are very sluggish.  We had to rebalance our volume 3 times to even out the disk space.  I added 4 more bricks and those are fine.  What would you suggest I do?
22:36 dberry joined #gluster
22:37 sysadmin-di2e JoeJulian: we have 10 replicated pairs
23:11 TrDS left #gluster
23:11 RicardoSSP joined #gluster
23:26 l0uis Has anyone seen or heard of any bugs that sound like this: I have a job running on a cluster that has numerous tasks, before each task runs they 'mkdir -p /gluster/scratchdir' then change to that directory. In some number of cases I'm seeing subsequent open() calls fail, but not always.
23:27 l0uis In the gluster logs on the client I have some log messages with warnings about the directory and one indicating the mkdir() failed: http://pastebin.com/45va6sHe
23:27 l0uis but the directory exists, so it clearly didn't fail everywhere... and then some entries about self-heal on the directory
23:27 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
23:28 l0uis http://fpaste.org/160820/85888414/
23:28 l0uis running 3.5.2 on ubuntu ... 5 nodes, 10 bricks, replica 2

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary