Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-07-27

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:01 ultrabizweb joined #gluster
01:04 xoritor joined #gluster
01:05 xoritor any pointers on how to speed up network throughput and reduce cpu load when copying VM images on a 2 node cluster after both nodes have been rebooted?
01:05 xoritor cpu is at 400% and network traffic is about .5 mbit/s
01:06 xoritor taking freaking FOREVER to copy the files... at this rate it would be faster to rsync them
01:07 xoritor i have glusterfs working very well until this happens (not very often mind you), but when it does it is PAINFULLY slow
01:14 ultrabizweb joined #gluster
01:19 _zerick_ joined #gluster
01:22 _zerick_ joined #gluster
01:26 JoeJulian If you're experiencing pain, please leave the IRC channel and contact your physician.
01:28 JoeJulian cpu is at 400% because the self-heal is doing a comparison to see what blocks of the image need transferred. Consequently, less network is used for the same purpose.
01:30 xoritor I have already taken my sumatriptan and about 800mg of motrin as per the doctors orders when the migraine kicks in...
01:30 xoritor i figured it was something like that, but is there anything i can do to help move the process along?  anything at all?
01:31 JoeJulian check "gluster volume set help". You can try setting the self-heal method to "full" and see if that works better for you. There's been mixed feedback on that one.
01:31 xoritor so yes, it litterally does pretty much cause me enough stress waiting for it to finish to trigger a migraine
01:32 xoritor hey thanks
01:32 JoeJulian you need bigger clusters.
01:32 JoeJulian When it takes weeks to heal, you get much more xen.
01:33 xoritor yea, well... if it took weeks, it would not be ok
01:33 xoritor i can xfer the images in minutes with scp
01:34 xoritor gluster just may not be the right tool for the job
01:34 JoeJulian but scp starts at the beginning, reads 4k, copies that 4k over, reads the next 4k, etc.
01:34 xoritor still faster than this
01:35 JoeJulian Now, make a change to the filesystem while that's happening.
01:35 xoritor right
01:35 JoeJulian that change is lost.
01:35 xoritor exactly the reason i am using it
01:35 xoritor well.. using glusterfs
01:35 JoeJulian so you're comparing apples to oranges.
01:35 xoritor but when it does go to pot it would be faster just to copy the image files over and recreate the entire cluster from ground zero
01:36 xoritor litterally to start over would be faster than to let it heal
01:37 xoritor i could rebuild the hosts and restore from a back up faster than the heal
01:37 JoeJulian anyway...
01:37 xoritor but it is BEAUTIFUL for most things
01:37 xoritor the day to day with libgfapi works AWESOME
01:38 xoritor the day to day is so fast users have no clue its virtualized
01:38 JoeJulian You can try that, you can try changing the number of simultaneous heals, I've been toying with the idea of changing io-threads and seeing if that makes a difference.
01:38 JoeJulian That's awesome to hear. I'm working toward that myself.
01:39 JoeJulian Without libgfapi, there seems to be a hard-limit of about 1337Mbit through our fuse mounts.
01:39 xoritor i am trying to move everything over to the BD xlator
01:39 xoritor yea
01:39 xoritor its only on recovery
01:40 xoritor arrrrgh!!!
01:40 xoritor anyway....
01:40 xoritor so i am in process of setting up the bd xlator stuff, you know if it will replicate?
01:40 xoritor i cant find any docs so i was going to test it first
01:41 JoeJulian I'm afraid I don't know. It's pretty new and I haven't had any feedback in channel.
01:41 JoeJulian That either means nobody's using it, or it works beautifully.
01:41 xoritor no one using it
01:41 xoritor no docs on it
01:41 xoritor :-/
01:41 xoritor plus it is very non standard right now in its use
01:42 xoritor however.... if it does work and i can set up all the replication, it SHOULD be even faster
01:45 xoritor and i will be adding another system in as well
01:45 xoritor quad bonded gigE using openvswitch  to all of them
01:45 JoeJulian The upcoming "new style replication" (NSR) looks very promising as well.
01:45 xoritor yes...
01:45 JoeJulian Good luck with the bonding. Not all the switch vendors do it right, and even seem to break it from time to time between firmware versions.
01:45 ultrabizweb joined #gluster
01:45 xoritor i even have users home dirs sitting on a VM and using roaming profiles across our lan...  they are not even aware of it
01:45 xoritor yea, lacp bonding and you... got to make sure you get the right switches and that it works right
01:45 xoritor i have had zero problems so far
01:45 JoeJulian https://github.com/gluster/glusterfs/blob/master/doc/features/bd-xlator.md
01:45 glusterbot Title: glusterfs/doc/features/bd-xlator.md at master · gluster/glusterfs · GitHub (at github.com)
01:46 xoritor yea, read that one
01:46 xoritor ;-)
01:47 xoritor wait
01:47 xoritor not that one... thats new
01:47 JoeJulian If you find deficiencies in that, any chance I could con you in to becoming a contributor and enhancing it some?
01:47 xoritor thanks!
01:47 xoritor hmm... dunno i hate writing docs ;-/
01:48 JoeJulian It's written... It's just up to editing now.
01:48 xoritor ill read through it
01:48 xoritor and let you know if there are any issues for sure
01:48 JoeJulian If it was in a web interface where you could just click "edit" and make your changes right there, would that make it more probable?
01:49 JoeJulian I'm beating up the web designers.
01:49 xoritor LOL
01:49 xoritor are they RH IT people?
01:49 JoeJulian yes
01:49 xoritor good luck
01:49 JoeJulian hehe, I have my ways...
01:50 xoritor been waiting for them to fix things with my account for over a YEAR
01:50 xoritor sorry... a bit bitter
01:50 xoritor not really
01:50 JoeJulian Ah, I'm not a subscriber, I'm part of the upstream community. That entices a whole different set of people.
01:51 xoritor they just have to merge my old rhce and cert numbers to my new one in the database
01:51 xoritor and i cant get them to do it
01:52 zerick joined #gluster
01:52 xoritor hmm... maybe
01:53 xoritor i already have access to somethings internal anyway
01:53 xoritor some jira stuff
01:53 xoritor etc...
01:53 * xoritor is a contractor
01:53 ultrabizweb joined #gluster
01:53 xoritor i _used_ to work for the hat
01:54 xoritor now i just play with it
01:54 xoritor ;-)
01:54 JoeJulian :)
01:55 xoritor i teach some classes now, as well as do IT for a small contrating firm
01:56 xoritor it is the small contracting firm that has the small amount of servers
01:56 JoeJulian Sounds like it should be less migrane prone.
01:56 xoritor adding more would be overkill
01:56 xoritor it generally is... until this hits
01:56 xoritor ;-)
01:57 xoritor which is not often
01:57 xoritor and i cant find any reliable docs on how to "help it along" without blowing it all up
01:57 ultrabizweb joined #gluster
01:58 xoritor right now i think i am cpu bound more than anything else
01:58 JoeJulian The Q&D route is to mount the client on the server that needs healed, and do a find on that client mount.
01:59 JoeJulian and I think you're right. If you have more network than cpu, perhaps the full heal method might be better.
02:00 xoritor any way i can set that as default?
02:01 JoeJulian not like a default when you create new volumes, but once set, it's set.
02:01 xoritor i found it think
02:01 xoritor gluster volume set cluster.data-self-heal-algorithm full
02:02 JoeJulian that's the ticket
02:07 mkzero joined #gluster
02:14 xoritor now we are talking!
02:14 xoritor woo hoo
02:14 xoritor 408324.58 Kbps
02:15 xoritor not perfect, but much better than it was
02:16 xoritor cpu is at 7%
02:16 xoritor IO wait is 0%
02:17 xoritor well... 0-30%
02:18 xoritor cpu 5-10% ,  io wait 0-30% , and network using at least 50% of one link .... not perfect, but MUCH better
02:18 xoritor JoeJulian, thanks a TON
02:19 xoritor a super uber huge ton
02:19 JoeJulian You're welcome. Glad I could help.
02:19 xoritor you should be knighted
02:20 JoeJulian lol
02:21 xoritor do you happen to have a list of tunables for the 3.5.x branch?
02:22 JoeJulian Just what's in "gluster volume set". I think they've made them all visible except for the ones that truly don't work.
02:22 xoritor cool
02:22 xoritor thanks
02:23 xoritor oh... one thing missing from that page in the bd-xlator doc
02:23 xoritor how to remove an lv once its made, is it just lvremove?   or do you have to truncate it to zero
02:23 xoritor or... or... or...
02:24 xoritor what would be the correct procedure to remove a logical volume from the volume group once the volume group is a "brick"
02:24 JoeJulian ╯°□°)╯︵ ┻━┻
02:24 xoritor lol
02:25 JoeJulian I would expect the normal "remove-brick"
02:25 xoritor no it would remove the volume group not the logical volume
02:26 xoritor ie... the "brick" is the vg
02:26 JoeJulian Ah
02:26 xoritor when you make an lv in a non standard way.... inside the vg how do you get rid of it
02:26 xoritor setfattr -n "user.glusterfs.bd" -v "lv" /media/image/lv1
02:27 xoritor that creates a lv
02:27 xoritor but nothing ever mentions how to remove it
02:28 xoritor i removed it with lvchange -an /dev/vgname/lvname && lvremove /dev/vgname/lvname
02:28 xoritor but that may not be the right way when it is on many hosts
02:28 xoritor it worked on ONE...
02:28 xoritor ;-)
02:28 JoeJulian afaict, you should be able to rm from the mount
02:30 xoritor hmm.... possibly
02:30 xoritor once i really get it setup i will test it
02:31 brettnem joined #gluster
02:55 ThatGraemeGuy joined #gluster
03:35 marcoceppi joined #gluster
03:58 cjhanks joined #gluster
04:06 cjhanks_ joined #gluster
05:15 cjhanks joined #gluster
05:22 RioS2 joined #gluster
05:58 ricky-ti1 joined #gluster
06:11 nullck joined #gluster
06:14 LebedevRI joined #gluster
07:11 ramteid joined #gluster
07:28 ghenry joined #gluster
07:50 Humble joined #gluster
09:02 clyons joined #gluster
09:11 kumar joined #gluster
10:22 Philambdo joined #gluster
12:10 diegows joined #gluster
12:25 an joined #gluster
12:40 an_ joined #gluster
12:46 an joined #gluster
12:50 an_ joined #gluster
13:26 shubhendu joined #gluster
13:37 Maya__ Hey guys, I'm still in the process of trying to self-heal 80GB worth of files; does anyone know what "Skipping entry self-heal because of gfid absence" means in the glustershd.log file? I'm just getting reams and reams of this message...
14:10 RioS2 joined #gluster
15:19 qdk joined #gluster
15:28 MacWinner joined #gluster
16:10 ricky-ti1 joined #gluster
16:14 plarsen joined #gluster
16:42 kumar joined #gluster
17:33 rotbeard joined #gluster
17:57 mjrosenb joined #gluster
18:08 leebrooks joined #gluster
18:11 leebrooks Good evening guys. I've mistakingly deleted a heap of files directly from a brick (non distributed) rather than through a client and so my disk space is being reported incorrectly by around 20%. The hidden .glusterfs directory is showing the larger usage - will stopping the volume and removing .glusterfs remove my data as well? I've Google'd around but have found conflicting answers.
18:12 LebedevRI joined #gluster
18:23 leebrooks Can anyone offer any insight?
18:36 cfeller joined #gluster
18:40 JoeJulian leebrooks: Assuming by "non distributed" that your volume is replicated, "gluster volume $vol heal full" will repair the damage. Then you can delete through the client.
18:41 leebrooks Hi Julian. Actually I'm running no replication at the moment. It's a standalone brick which is why I'm having troubles
18:41 leebrooks It was actually your writings that helped me narrow it down to the hardlinks being the problem
18:42 JoeJulian Maya__: It means the file was not added through the client. I'm not sure of the process in 3.5 for handling those. Hopefully it will add a gfid to it to later allow it to heal.
18:43 JoeJulian leebrooks: find $brickdir/.glusterfs -type f -links 1 -exec /bin/rm {} \;
18:44 JoeJulian leebrooks: each file should have a link count of 2. If a gfid file does not have 2 links, it's an orphan.
18:44 Maya__ JoeJulian: Got it, thanks again!
18:44 JoeJulian Maya__: You can check by looking at the ,,(extended attributes) of one of the files you're concerned with.
18:44 glusterbot Maya__: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
18:45 leebrooks Right. So I've ran that command (minus the rm, I used ls instead) and it appears to be showing links to files that do still exist, hence my hesitation
18:49 leebrooks I suspect down to the fact that when we migrated the brick from one server to the current server, the ".glusterfs" files would have come across too. Not too sure what the implications of that are but everything has seemed to be working until now.
18:49 leebrooks So my question is, will removing ".glusterfs" also delete data, or is that safe to do and Gluster can then rebuild the indexes?
18:50 JoeJulian yes, you can do that. Just run a heal...full after.
18:51 leebrooks Well since it's not part of replica, heal isn't applicable?
18:56 JoeJulian hmm... right. Sorry, so used to having everything replicated.
18:57 JoeJulian Just mount the volume and do a find $client_mount | xargs stat >/dev/null
18:58 leebrooks Ok thanks
18:58 qdk joined #gluster
18:59 leebrooks So an rm -rf .glusterfs won't remove anything I need? Just want to be 110% sure as you can imagine :)
19:02 JoeJulian If there's a file you didn't intend to delete that *only* exists as a uuid-named gfid file, then yes. You would lose that.
19:03 leebrooks That's fine
19:03 leebrooks Thanks again
19:49 plarsen joined #gluster
20:24 ilbot3 joined #gluster
20:24 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
21:09 Maya___ joined #gluster
22:06 basso joined #gluster
22:07 a1 joined #gluster
22:41 caiozanolla is there any way to speed up a full heal? im aware gluster will run heal in slow ptiority to avoid disturbing clients and the machine, but there are no clients mounting the volume, even though network throughput is very small. i wanted to speed things up as a 3.5TB volume replicating at 25Mb/s will take ages to complete!
22:42 JoeJulian Do the files that are being healed exist?
22:43 caiozanolla they exist on one side
22:44 JoeJulian Sorry, it's Sunday, I'm not supposed to be thinking today. The question was intended to mean do the files exist on the side being healed.
22:45 caiozanolla JoeJulian, its a 2 node replicated setup, 14 bricks. 2nd node failed last week. Now its up but 3.5TB behind.
22:45 caiozanolla so, no, the files do not exist on the receiver
22:47 caiozanolla now i see, load on the machine holding the files is very high. fired up some "ls" that triggered self healing, after that I started manual full heal.
22:47 JoeJulian Ah, ok... the two most common possibilities: the files on the target are sparser than the source; the self-heal daemon is checking for differences in files that need healed so you're only seeing data transfer for the parts where they differ (you may be CPU bound if that's the case).
22:48 JoeJulian For the latter case, volume set $volume cluster.data-self-heal-algorithm full
22:48 JoeJulian Then you'll have lots of network traffic, which often seems to satisfy peoples expectations.
22:49 caiozanolla seeing lots of "remote operation failed: No such file or directory. Path: <gfid:bed91799-cf37-49d9-a30f-70182dd3ceba>"
22:49 caiozanolla because of the ls i suppose
22:49 caiozanolla set $volume cluster.data-self-heal-algorithm full <- how am i supposed to run this?
22:50 caiozanolla at the shell?
22:50 caiozanolla just like that?
22:50 JoeJulian Ah, yes. That's another option. The self-heal algorithm keeps an index of files that have changed when a brick goes offline. If the brick was replaced, you'll need a heal...full to repair the files that were not changed.
22:51 JoeJulian You should be able to set volume options. Just a sec and I'll find that manual section.
22:52 caiozanolla lots of afr-self-heal-common.c:2146:afr_sh_post_nb_entrylk_missing_entry_sh_cbk] 0-FileServerCacheVolume0-replicate-2: Non blocking entrylks failed.
22:53 JoeJulian https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_managing_volumes.md#tuning-options
22:53 glusterbot Title: glusterfs/doc/admin-guide/en-US/markdown/admin_managing_volumes.md at master · gluster/glusterfs · GitHub (at github.com)
22:54 caiozanolla whats the best option to go fast and clean? keeping in mind that i have triggered auto heal by "ls" and also forced a full heal with "gluster volume heal <vol> full"? (btw, there will be no clients using the volume)
22:55 JoeJulian What I would do is mount the client on the destination server then "find $client_mount | xargs stat >/dev/null
22:56 JoeJulian That's how we used to *have* to do it.
22:56 JoeJulian Plus, then you know when it's done.
22:58 caiozanolla how is that different than firing a full heal? (besides being cpu bound on files that DO exist)
22:58 JoeJulian It's not.
23:01 caiozanolla ok, so after it goes through the files that exists I should see an increase on network speed as it will be mostly file transfer instead of file compare?
23:02 caiozanolla s/network speed/network throughput
23:37 JoeJulian caiozanolla: That would be my expectation, yes.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary