Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-01-28

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 JoeJulian At least I didn't give you a lmgtfy.com link. ;)
00:00 xoritor yea
00:00 xoritor https://github.com/gluster/glusterfs/blob/release-3.7/doc/features/afr-arbiter-volumes.md
00:00 glusterbot Title: glusterfs/afr-arbiter-volumes.md at release-3.7 · gluster/glusterfs · GitHub (at github.com)
00:01 xoritor what about that?
00:01 xoritor can you do it with more than 2 "real" briks and an arbiter?
00:01 haomaiwa_ joined #gluster
00:02 xoritor it says it _always_ takes brick 3 as the arbiter but does not say anything about brick4+
00:02 JoeJulian No, that's all about maintaining quorum.
00:02 zhangjn joined #gluster
00:02 xoritor it says it provides the same level of consistency as a normal replica 3 volume
00:03 xoritor not sure how
00:03 xoritor ok
00:03 necrogami joined #gluster
00:03 xoritor so my options are to have smaller file sizes and do it right, or sharding (which is still experimental)
00:03 JoeJulian I'm not sure how either.... I'll have to explore that statement and disprove it.
00:03 xoritor or the hack of just using 3 dirs on 1 fs
00:04 xoritor bwahahaha
00:04 JoeJulian Sharding not experimental, just doesn't have a history of proof.
00:05 xoritor http://www.linux-kvm.org/images/5/51/03x04-Martin_Sivak-oVirt_and_gluster_hyperconverged.pdf
00:05 xoritor thats what i am trying to do "hyperconverged glusterfs hosted oVirt"
00:05 xoritor lol
00:05 xoritor bingo!
00:06 JoeJulian I wish I could take the guy that coinded that term out behind the woodshed.
00:06 xoritor can i come along?
00:06 JoeJulian *coined
00:06 xoritor i want to stab him in the eye
00:06 * xoritor hids
00:06 xoritor hides even
00:06 * xoritor snickers like muttley
00:06 JoeJulian hehe
00:07 xoritor ok so the LARGEST vm image i have is 331GB
00:08 xoritor so it would just BARELY fit
00:08 JoeJulian Yeah, but it's ugly if there's any sparseness to it.
00:09 xoritor why is that/
00:09 xoritor i MAY be able to trim that down some... not sure
00:09 xoritor oh you mean if its only showing the "used" size
00:09 JoeJulian Because it doesn't fill up the brick, other files are put on the brick, it grows and boom. Full disk and no room to write the pending blocks on the guest FS.
00:11 xoritor yea true
00:11 xoritor ok... dinner time
00:11 xoritor bbiab
00:12 pranithk joined #gluster
00:12 JoeJulian Then, it's even more fun, when the person who's using your service adds sparse images that actually exceed the size of the brick. Or stripes zfs across multiple sparse images that also exceed the size of the brick(s) that they're on and you have to tell their salesman that you can guarantee they'll lose all their data.
00:31 om Hi all.  Is there something wrong with this syntax?
00:31 om - unless: {% win_system.get_domain_workgroup %} = {{ salt['pillar.get']('windows_join_domain:correct_windows_domain:domain', '') }}
00:31 om I get this error
00:31 om Rendering SLS 'development:windows_join_domain' failed: Jinja syntax error: expected token 'end of print statement', got ':';
00:32 om wpp[s
00:32 om wrong IRC!
00:32 om sorry
00:36 14WAAT92E joined #gluster
00:40 xoritor JoeJulian, thats just scary!
00:42 xoritor ok one more question on order of the bricks
00:43 xoritor ie.. host1:brick1 host2:brick1 host3:brick1 host4:brick1 host5:brick1 host1:brick2 host2:brick2 etc...
00:44 xoritor now what if i start with 4 nodes and add in the third later
00:44 xoritor ie... host1:brick1 host2:brick1 host3:brick1 host4:brick1 host1:brick2 etc...
00:44 xoritor then i add in 3 bricks from host 5
00:45 shyam joined #gluster
00:45 xoritor will that fubar the "order"
00:49 harish joined #gluster
00:58 nathwill joined #gluster
01:01 haomaiwa_ joined #gluster
01:05 calavera joined #gluster
01:15 EinstCrazy joined #gluster
01:19 zhangjn joined #gluster
01:20 xoritor i just realized something
01:20 xoritor i only need 3 hosts to do h/a for the hosted engine
01:21 xoritor i can put them on a different volume
01:21 xoritor !
01:27 nangthang joined #gluster
01:28 EinstCrazy joined #gluster
01:29 Lee1092 joined #gluster
01:30 aravindavk joined #gluster
01:35 EinstCrazy joined #gluster
01:36 zhangjn joined #gluster
01:44 JoeJulian om: Yes, unless statements are run from a shell and the result of the operation determines success or failure. In linux, that would be: - unless: test {{ string1 }} = {{ string2 }}
01:44 JoeJulian I'm not sure what windows has that does that.
01:46 JoeJulian Left side's wrong, too. You should be using {{}} not {% %}
01:46 EinstCrazy joined #gluster
01:48 JoeJulian xoritor: And yes, it would confuse the order. See https://joejulian.name/blog/how-to-expand-glusterfs-replicated-clusters-by-one-server/
01:48 glusterbot Title: How to expand GlusterFS replicated clusters by one server (at joejulian.name)
01:49 xoritor JoeJulian, i thought so
01:49 xoritor JoeJulian, just wanted to make sure
01:49 xoritor JoeJulian, i figured out a way to do it without using my main data drive!
01:49 xoritor heh
01:49 JoeJulian cool
01:49 xoritor i used an lvm based setup for the hosted-engine brick
01:49 xoritor ;-)
01:50 xoritor only did it on 3 machines too
01:50 xoritor so there you go
01:50 JoeJulian cool
01:56 xoritor yea wish me luck ;-)
02:16 harish joined #gluster
02:18 nangthang joined #gluster
02:19 17WABN1T0 joined #gluster
02:45 EinstCrazy joined #gluster
02:55 tom[] joined #gluster
03:01 haomaiwa_ joined #gluster
03:08 zhangjn joined #gluster
03:10 tswartz joined #gluster
03:13 bharata-rao joined #gluster
03:15 baojg joined #gluster
03:26 om joined #gluster
03:35 spalai joined #gluster
03:37 EinstCrazy joined #gluster
03:49 EinstCrazy joined #gluster
03:50 nbalacha joined #gluster
03:51 mowntan joined #gluster
03:51 itisravi joined #gluster
03:57 atinm joined #gluster
04:00 rafi joined #gluster
04:01 haomaiwa_ joined #gluster
04:10 gem joined #gluster
04:12 kanagaraj joined #gluster
04:17 sakshi joined #gluster
04:25 om joined #gluster
04:32 zhangjn joined #gluster
04:32 ndarshan joined #gluster
04:35 shubhendu joined #gluster
04:40 RameshN_ joined #gluster
04:53 hgowtham joined #gluster
04:53 EinstCrazy joined #gluster
05:01 haomaiwa_ joined #gluster
05:04 Bhaskarakiran joined #gluster
05:05 calavera joined #gluster
05:09 kshlm joined #gluster
05:10 EinstCrazy joined #gluster
05:11 RameshN_ joined #gluster
05:15 Apeksha joined #gluster
05:15 Manikandan joined #gluster
05:16 Apeksha joined #gluster
05:16 nehar joined #gluster
05:19 jiffin joined #gluster
05:22 pppp joined #gluster
05:27 zhangjn joined #gluster
05:28 ppai joined #gluster
05:32 calavera joined #gluster
05:33 skoduri joined #gluster
05:37 baojg joined #gluster
05:39 nangthang joined #gluster
05:46 karthikfff joined #gluster
05:47 anil joined #gluster
05:51 harish joined #gluster
05:53 kotreshhr joined #gluster
05:53 bennyturns joined #gluster
05:54 vimal joined #gluster
05:55 ashiq joined #gluster
06:01 haomaiwang joined #gluster
06:03 poornimag joined #gluster
06:04 dusmant joined #gluster
06:07 calavera joined #gluster
06:08 karnan joined #gluster
06:08 vmallika joined #gluster
06:08 kdhananjay joined #gluster
06:09 rwheeler joined #gluster
06:17 ramky joined #gluster
06:18 atalur joined #gluster
06:20 hchiramm_ joined #gluster
06:20 shubhendu joined #gluster
06:24 ppai joined #gluster
06:25 overclk joined #gluster
06:34 arcolife joined #gluster
06:36 EinstCra_ joined #gluster
06:37 kdhananjay joined #gluster
06:38 gowtham joined #gluster
06:57 xoritor joined #gluster
07:01 haomaiwa_ joined #gluster
07:06 harish_ joined #gluster
07:09 b0p joined #gluster
07:09 shubhendu joined #gluster
07:12 SOLDIERz joined #gluster
07:18 dusmant joined #gluster
07:22 jtux joined #gluster
07:24 zhangjn joined #gluster
07:29 pkrishnaswamy joined #gluster
07:29 pkrishnaswamy left #gluster
07:32 itisravi joined #gluster
07:36 kovshenin joined #gluster
07:38 ppai joined #gluster
07:42 b0p joined #gluster
07:42 kdhananjay joined #gluster
07:50 mhulsman joined #gluster
07:52 PatNarciso joined #gluster
08:01 haomaiwang joined #gluster
08:07 [Enrico] joined #gluster
08:15 jwang_ joined #gluster
08:16 ivan_rossi joined #gluster
08:18 rafi1 joined #gluster
08:19 gowtham_ joined #gluster
08:20 gem joined #gluster
08:20 skoduri joined #gluster
08:20 glafouille joined #gluster
08:31 itisravi joined #gluster
08:32 lord4163 joined #gluster
08:39 doekia joined #gluster
08:40 andreask joined #gluster
08:45 ppai joined #gluster
08:56 spalai joined #gluster
08:58 arcolife joined #gluster
09:00 zhangjn joined #gluster
09:01 haomaiwa_ joined #gluster
09:06 ctria joined #gluster
09:09 shubhendu joined #gluster
09:22 dusmant joined #gluster
09:24 aravindavk joined #gluster
09:28 spalai left #gluster
09:31 EinstCrazy joined #gluster
09:40 baojg joined #gluster
09:41 skoduri joined #gluster
09:51 zhangjn joined #gluster
09:53 harish_ joined #gluster
09:57 RameshN_ joined #gluster
10:01 haomaiwang joined #gluster
10:03 Slashman joined #gluster
10:14 harish joined #gluster
10:16 skyrat joined #gluster
10:18 sabansal_ joined #gluster
10:19 skyrat Hi, could anybody explain why the wheezy repo is no longer available and is renamed to wheezy-broken for 3.7.6. Debian release? Thanks?
10:20 skyrat What is actually broken? The entire build?
10:21 hgowtham REMINDER: Gluster Community Bug Triage meeting (Today) in about 100 minutes at #gluster-meeting
10:27 skoduri joined #gluster
10:34 jiffin1 joined #gluster
10:35 skoduri joined #gluster
10:44 tswartz joined #gluster
10:53 luizcpg joined #gluster
10:58 luizcpg_ joined #gluster
11:01 haomaiwa_ joined #gluster
11:06 jmarley joined #gluster
11:09 andreask left #gluster
11:10 ppai joined #gluster
11:11 msciciel_ joined #gluster
11:18 nehar joined #gluster
11:21 dusmant joined #gluster
11:21 shubhendu joined #gluster
11:40 nehar joined #gluster
11:42 ira joined #gluster
11:45 overclk joined #gluster
11:51 nehar joined #gluster
11:54 hgowtham REMINDER: Gluster Community Bug Triage meeting (Today) in about 8 minutes at #gluster-meeting
11:57 sakshi joined #gluster
12:00 hgowtham_ joined #gluster
12:01 64MAAXBWJ joined #gluster
12:04 spalai joined #gluster
12:08 gem joined #gluster
12:09 gem_ joined #gluster
12:18 nishanth joined #gluster
12:18 ppai joined #gluster
12:24 shubhendu joined #gluster
12:24 jiffin1 joined #gluster
12:24 skoduri joined #gluster
12:31 arcolife joined #gluster
12:34 unclemarc joined #gluster
12:46 hgowtham joined #gluster
12:47 chirino joined #gluster
12:47 Akee joined #gluster
12:49 shyam joined #gluster
12:50 skyrat anybody talking here?
12:55 kdhananjay joined #gluster
12:59 ppai joined #gluster
13:01 karnan joined #gluster
13:01 haomaiwang joined #gluster
13:02 zhangjn joined #gluster
13:03 rastar skyrat: https://www.mail-archive.com/gluster-users@gluster.org/msg23185.html
13:03 glusterbot Title: Re: [Gluster-users] Installing 3.7.6 in Debian Wheezy (at www.mail-archive.com)
13:11 skyrat thanks
13:16 poornimag joined #gluster
13:16 kotreshhr left #gluster
13:17 nishanth joined #gluster
13:18 shubhendu joined #gluster
13:20 dusmant joined #gluster
13:23 ppai joined #gluster
13:37 nbalacha joined #gluster
13:39 drankis joined #gluster
13:43 plarsen joined #gluster
13:47 zhangjn joined #gluster
13:54 Bhaskarakiran joined #gluster
14:02 Bhaskarakiran joined #gluster
14:03 shyam joined #gluster
14:11 jmarley joined #gluster
14:11 nehar joined #gluster
14:19 Bhaskarakiran joined #gluster
14:19 plarsen joined #gluster
14:24 haomaiwa_ joined #gluster
14:32 shubhendu joined #gluster
14:36 nishanth joined #gluster
14:42 hamiller joined #gluster
14:46 luizcpg joined #gluster
15:01 haomaiwang joined #gluster
15:01 skylar joined #gluster
15:13 skoduri joined #gluster
15:14 kdhananjay joined #gluster
15:20 julim joined #gluster
15:22 shyam joined #gluster
15:32 Rapture joined #gluster
15:33 bowhunter joined #gluster
15:34 farhoriz_ joined #gluster
15:38 Liquid-- joined #gluster
15:49 dusmant joined #gluster
15:49 theron joined #gluster
15:49 Saravanakmr joined #gluster
16:01 haomaiwa_ joined #gluster
16:05 coredump joined #gluster
16:10 18WABXDDS joined #gluster
16:10 Bhaskarakiran joined #gluster
16:22 shaunm joined #gluster
16:32 nickage_ joined #gluster
16:32 jiffin joined #gluster
16:34 b0p joined #gluster
16:37 jbrooks joined #gluster
16:40 calavera joined #gluster
16:50 theron joined #gluster
16:50 kovshenin joined #gluster
16:57 shubhendu joined #gluster
17:01 77CAAC4HF joined #gluster
17:04 luizcpg joined #gluster
17:09 ivan_rossi left #gluster
17:09 xoritor JoeJulian, you around?
17:13 xoritor anyone have a pros/cons list of using glusterfs on lvm vs partitons?
17:21 ovaistariq joined #gluster
17:22 ovaistar_ joined #gluster
17:24 bluenemo joined #gluster
17:25 jbrooks joined #gluster
17:26 nishanth joined #gluster
17:28 squizzi_ joined #gluster
17:31 arcolife joined #gluster
17:35 gem joined #gluster
17:41 luizcpg joined #gluster
17:51 tswartz xoritor, i don't believe you can do snapshots on partitions
17:53 xoritor tswartz, thanks!
17:53 xoritor tswartz, anything else?
17:53 xoritor i did decide to go ahead and use LVM btw
17:54 tswartz lvm is easily resizable
17:54 xoritor i have usually used partitions, but i see the futre of me expanding the disk sizes, and that is much easier to do in lvm
17:54 xoritor ;-)
17:54 xoritor yea
17:54 tswartz no reason to use partitions that i can think of
17:54 xoritor one less thing to go wrong
17:54 xoritor heh
17:54 tswartz other than simplicity and no desire to learn and us LVM ;)
17:55 xoritor i am very familiar with lvm
17:55 xoritor and it is NOT perfect...
17:55 xoritor although you can easily restore metadata, other things not so much
17:56 xoritor now.. you can also easily migrate to new drives with lvm
17:56 jiffin joined #gluster
17:56 xoritor as you mentioned "flexibility"
18:01 haomaiwa_ joined #gluster
18:02 Rapture joined #gluster
18:06 theron joined #gluster
18:06 julim joined #gluster
18:30 jiffin1 joined #gluster
18:36 jiffin joined #gluster
18:41 bennyturns joined #gluster
18:42 neofob joined #gluster
18:47 vimal joined #gluster
18:59 arcolife joined #gluster
19:01 haomaiwa_ joined #gluster
19:08 rafi joined #gluster
19:11 theron joined #gluster
19:14 ovaistariq joined #gluster
19:15 jmarley joined #gluster
19:21 nickage_ joined #gluster
19:43 Manikandan joined #gluster
19:49 Manikandan joined #gluster
19:54 hchiramm_ joined #gluster
19:56 mhulsman joined #gluster
20:00 wushudoin joined #gluster
20:01 haomaiwa_ joined #gluster
20:22 B21956 joined #gluster
20:27 theron joined #gluster
21:01 haomaiwa_ joined #gluster
21:06 cpetersen_ hey joejulian
21:06 cpetersen_ is there any way to verify the integrity of a replicated gluster volume?
21:06 cpetersen_ I'm looking through the volume commands and can't find anything truly indicative of showing a healthy volume
21:07 JoeJulian If "gluster volume heal $vol info" is clean, that's typically sufficient.
21:08 JoeJulian You can dot a bitrot scrub, now, with 3.7.
21:09 cpetersen_ yeah I couldn't fully understand the output of that
21:09 cpetersen_ Brick file01:/var/run/gluster/shared_storage
21:09 cpetersen_ <gfid:a2bf7659-5c7b-451b-902e-c7a2451be7b1>
21:09 cpetersen_ Number of entries: 2
21:09 cpetersen_ Brick file02:/var/run/gluster/shared_storage
21:09 cpetersen_ Number of entries: 0
21:09 cpetersen_ Brick file03:/var/run/gluster/shared_storage
21:09 cpetersen_ Number of entries: 0
21:09 cpetersen_ does that mean there is an inconsistency?
21:10 JoeJulian yes
21:10 cpetersen_ OK, so I should run a full heal on it?
21:10 gildub joined #gluster
21:10 post-factum any files listed as required to be healed?
21:10 cpetersen_ that was the total output of the command
21:11 JoeJulian Shouldn't have to run a full heal. The fact that it's there suggests the self-heal daemon knows about it and is (theoretically) working on it.
21:11 post-factum sudo gluster volume heal VOLUME info heal-failed
21:12 JoeJulian There's also recently been added "gluster volume heal $vol statistics"
21:12 cpetersen_ heal-failed is not supported
21:12 post-factum @JoeJulian++
21:12 glusterbot post-factum: JoeJulian's karma is now 24
21:12 post-factum cpetersen_: glusterfs version?
21:13 JoeJulian Yeah, heal-failed retired in favor of not failing ;)
21:13 post-factum lol :D
21:14 cpetersen_ 3.7.6-1.el7
21:15 cpetersen_ it looks like it healed a heal failed entry at one point
21:15 cpetersen_ Starting time of crawl: Thu Jan 28 14:05:24 2016
21:15 cpetersen_ Ending time of crawl: Thu Jan 28 14:05:24 2016
21:15 cpetersen_ Type of crawl: INDEX
21:15 cpetersen_ No. of entries healed: 0
21:15 cpetersen_ No. of entries in split-brain: 0
21:15 cpetersen_ No. of heal failed entries: 0
21:15 cpetersen_ that is my status now
21:15 cpetersen_ there were 2 heal failed entries previously
21:16 cpetersen_ so it looks like I'm good?
21:16 post-factum healthy pink boy, yeah
21:16 cpetersen_ as per the only youtube video on ganesha/gluster, I enabled cache-invalidation as well
21:17 post-factum btw, cache-invalidation is subjected to memory leaking in 3.7.6
21:17 cpetersen_ which would make sure that wherever changes are made, no inconsistencies will happen
21:17 cpetersen_ oh rly?
21:17 cpetersen_ how bad?
21:17 EinstCrazy joined #gluster
21:18 cpetersen_ I need it though, right?  replicated volume, nfs-ganesha shared to ESX for VM storage
21:18 post-factum 15G per several days for my specific conditions. I hope we've debugged it, and 3.7.7 will be without this particular leak
21:18 cpetersen_ technically I will never be making changes on a different cluster ip at the same time as the primary is up
21:19 post-factum nfs-ganesha uses glusterfs api, that is subjected to another memory leak fixed in 3.7.7 as well
21:19 post-factum everything leaks here ©
21:19 cpetersen_ when is 3.7.7 proposed to be released?
21:19 cpetersen_ lol
21:19 post-factum in a week or so
21:19 post-factum anyway, there are still leaks. they drive me crazy
21:19 post-factum but we shall fight
21:20 cpetersen_ in your opinion should I need cache-invalidation?
21:20 post-factum till the last blood drop
21:20 cpetersen_ in my situation?
21:21 post-factum cache-invalidation is OK, but I really dunno whether upcall is ready
21:21 post-factum I'm not dev, btw
21:21 cpetersen_ np :)
21:21 post-factum I just enjoy hunting damned leaks
21:21 cpetersen_ so IYO is that it's a high risk?
21:21 post-factum nope
21:22 post-factum the risk is when data loss happens. leaks != data loss, just small inconvenience
21:22 cpetersen_ I have a fairly small setup with only two VMs
21:22 post-factum do you store VM images on gluster?
21:22 cpetersen_ not images, just runtimes
21:23 JoeJulian if I understand its use correctly, cache invalidation is useful when you have other clients updating files. If you're using it for vm hosting, that's not happening anyway.
21:23 cpetersen_ yeah that's what I was thinking
21:23 post-factum what runtime?
21:23 cpetersen_ but it is an extra precaution
21:23 cpetersen_ sorry, by images do you mean clones or snapshots?
21:23 post-factum VM system images that contain VMs themselves
21:24 cpetersen_ sorry, then yes
21:24 nickage__ joined #gluster
21:24 post-factum hmmm. then 2 brief questions if you don't mind: 1) do you use shards? 2) why not ceph?
21:25 cpetersen_ I'm a bit ignorant to what a shard is
21:25 cpetersen_ I looked at Ceph but I took some advice to go with Gluster, Ceph seems to be overkill for my small setup
21:26 cpetersen_ I needed a solution to employ shared storage between three severs with storage on them in order to achieve HA with vcenter
21:26 cpetersen_ only two VMs are required to be redundantly stored on the replicated volume
21:27 post-factum ok, nvm then
21:27 post-factum but consider taking a look at shards, they should improve heal in case of node failure
21:27 JoeJulian glustershd does not give enough information.
21:27 * JoeJulian goes to file a bug
21:27 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
21:28 cpetersen_ jsut reading up a bit on sharding related to databases
21:28 xoritor if i have 4 hosts, each with 2 bricks using a distrepl (replica 2) and add another host with 2 bricks to the glusterfs is there a way to "reorder" them or is it easier to delete the volume and re-add it
21:28 cpetersen_ would this be horizontal partitioning of the volume?
21:28 post-factum also, we've abandoned glusterfs as vm images storage because of horribly slow heal. back to 3.6 no (reliable) shards were available, and we've stopped at using ceph for network block devices and gluster for file storing
21:29 JoeJulian What's "horribly slow"?
21:29 cpetersen_ when you say horribly slow, what kind of bandwidth between bricks is there and how big are they?
21:29 om left #gluster
21:30 post-factum cpetersen_: "shard" in glusterfs is breaking huge files into small pieces on underlying FS
21:30 cpetersen_ oic so that would be like telling vmware to split your vmdks
21:30 JoeJulian More accurately, how do you quanitfy horribly slow. What is your measure?
21:30 xoritor i had the horribly slow issue a long time ago, but fixed it with a different heal policy (is that the right terminiology?)
21:30 JoeJulian (sounds good to me)
21:30 post-factum @JoeJulian: that is ~20 mins to get heal done while guest OSes (50 VMs) go into read-only because heal prevents useful disk activity
21:31 post-factum with 20Gbps interconnect between 2 nodes
21:31 cpetersen_ damn..
21:31 post-factum but that was in 3.6
21:31 post-factum in 3.7 I'd definitely try shards, but do not want to migrate away from ceph rbd, as it does its job well enough
21:32 cpetersen_ ceph requires a couple servers for coordination efforts, does it not?
21:32 EinstCra_ joined #gluster
21:32 post-factum without shards AFAIK heal transfers the whole file to another node. now imagine each VM image is 2 to 3 GB
21:32 JoeJulian I wouldn't do ceph with less than three.
21:32 cpetersen_ and a redundant copy of the data being replicated?
21:32 cpetersen_ well if they're windows they are a lot more than 2 to 3 gb
21:32 cpetersen_ lol
21:32 JoeJulian No, without shards heal *checks* the whole file for differences, and only transfers the blocks that differ.
21:33 post-factum yep, at least replica 2 for ceph with periodic backups
21:33 post-factum @JoeJulian: ahhhh, ok, anyway, checking the whole file is slow
21:33 JoeJulian and 3 monitors
21:33 xoritor ceph scares the hell out of me
21:33 post-factum yep, 3 mons at least
21:33 JoeJulian Some people say that. My hardware doesn't agree. Not sure what the difference is.
21:34 xoritor every time i set it up and use it i feel like i am just waiting to loose my data
21:34 cpetersen_ hrm
21:34 JoeJulian With my hardware, I'm currently healing 20Tb raw images with no noticeable overhead.
21:34 cpetersen_ damn
21:34 JoeJulian loose data sounds icky.
21:35 post-factum you seem to be lucky, JoeJulian, really
21:35 post-factum we hosted vms on ssds and got that crap with healing
21:35 xoritor at least with glusterfs the files are whole (the way i am using it) and i can figure out what needs to be where
21:35 JoeJulian Well, a lot of us are lucky, but a too-large percentage aren't. I haven't seen a pattern emerge yet though.
21:35 post-factum i saw bugreports about vms failing into r/o with heal
21:36 JoeJulian Yeah, the people that don't have that problem don't file bug reports for it. ;)
21:36 cpetersen_ ahhh
21:36 * cpetersen_ closes his ears
21:36 * cpetersen_ errr eyes
21:36 post-factum cpetersen_: make sure you have backups
21:36 post-factum cpetersen_: and then open your eyes
21:36 cpetersen_ heheh
21:37 JoeJulian But what I really want to know is which file is each shd working on, and what offset is it up to.
21:37 cpetersen_ that would be nice
21:38 DV joined #gluster
21:38 JoeJulian I have 20Tb that's healing.... great... I can't answer management when they ask how much longer.
21:38 cpetersen_ lol
21:38 JoeJulian ... well, worst case, about a week.
21:38 post-factum in ceph you definitely can estimate that. i believe ceph to be more convenient solution for network block device
21:38 post-factum unfortunately, not for FS
21:39 JoeJulian Ceph requires a lot more man hours to maintain.
21:39 post-factum oh, we've already talked about that :)
21:39 post-factum it has higher level of entrance
21:39 JoeJulian I'm hoping to address that problem in my next iteration of salt states, but for now there's a whole lot of manual work when a disk fails.
21:40 post-factum yep, shuffling OSDs is a bit tricky :/
21:40 JoeJulian And when you've got 600 disks in your ceph cluster, that happens fairly frequenly.
21:40 post-factum anyway, replacing brick in gluster is tricky too
21:41 JoeJulian Nah, that's dead simple. They replace the disk, we run salt which sees the unformatted disk, formats it, mounts it, runs "gluster volume start $vol force" and the self-heal happens.
21:42 mhulsman joined #gluster
21:42 post-factum use "force", Luke
21:42 JoeJulian force avoids having to create the volume-id xattr.
21:43 post-factum looks a little bit hacky or quirky, isn't it?
21:43 JoeJulian Next iteration I'll do the extended attributes, but this is sufficient for now.
21:44 JoeJulian It's real life in the business world. Get it done on time vs get it done perfectly.
21:45 post-factum what about replace-brick?
21:45 JoeJulian Waste of time unless you're replacing it to a new server.
21:46 theron joined #gluster
21:47 tom[] joined #gluster
21:48 post-factum oh, that's why i use raid1 for each brick :)
21:49 JoeJulian That's a waste of space for us.
21:51 JoeJulian Plus, it would turn 600 drives into 900. That's a couple extra dollars that nobody's willing to spend.
21:52 post-factum for that amount, i agree
21:52 JoeJulian That's why my default answer for any question about best practice is always, "It depends..."
21:56 post-factum cpetersen_: but backups are always important
21:57 cpetersen_ :)
22:01 haomaiwa_ joined #gluster
22:04 cpetersen_ hmmm JoeJulian I still have 2 heal failed entries
22:05 cpetersen_ I was reading the log wrong
22:05 cpetersen_ I was only looking at one of the nodes in the statistics command :P
22:05 cpetersen_ how can I tell if it is actually healing right now?
22:17 cpetersen_ also, should there be a glustershd service installed?
22:19 JoeJulian On a server, "ps ax | grep glustershd". It should be running but the process name is gluster.
22:21 cpetersen_ my glustershd.log has this, should I be concerned?
22:21 cpetersen_ [2016-01-28 22:18:52.865682] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-gluster_shared_storage-replicate-0: performing entry selfheal on a2bf7659-5c7b-451b-902e-c7a2451be7b1
22:21 cpetersen_ [2016-01-28 22:18:52.934991] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-gluster_shared_storage-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
22:22 JoeJulian Well, doing a lookup on null should probably fail. Not sure why it's doing that though.
22:22 cpetersen_ uh oh another bug report lol
22:23 cpetersen_ the time is also showing wrong lol
22:23 cpetersen_ is that in UTC?
22:23 cpetersen_ errr GMT?
22:23 post-factum looks like UTC
22:24 post-factum try to figure out what the hell a2bf7659-5c7b-451b-902e-c7a2451be7b1 is
22:24 post-factum there is some script resolving gfid into path
22:24 cpetersen_ there is another guid with the same message too
22:24 cpetersen_ [2016-01-28 22:04:11.735573] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-gluster_shared_storage-replicate-0: performing entry selfheal on 7417ac59-06c3-432e-a4e5-f3bf2834f9f0
22:24 cpetersen_ [2016-01-28 22:04:11.754700] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-gluster_shared_storage-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
22:25 post-factum 2 failed entried — 2 gfids, i guess
22:25 cpetersen_ true enough
22:25 JoeJulian Yep, it's the gfid. You would find the gfid file for that id at .glusterfs/a2/bf/a2bf7659-5c7b-451b-902e-c7a2451be7b1
22:25 post-factum or https://gluster.readthedocs.org/en/latest/Troubleshooting/gfid-to-path/
22:25 glusterbot Title: gfid to path - Gluster Docs (at gluster.readthedocs.org)
22:25 JoeJulian It should be hardlinked to the same inode that the original filename is.
22:27 JoeJulian One possibility is that the file doesn't exist, but the entry under .glusterfs/indices/xattrop does still exist.
22:35 cpetersen_ [root@file01 17]# getfattr -n trusted.glusterfs.pathinfo -e text 7417ac59-06c3-432e-a4e5-f3bf2834f9f0
22:35 cpetersen_ 7417ac59-06c3-432e-a4e5-f3bf2834f9f0: trusted.glusterfs.pathinfo: No such attribute
22:35 cpetersen_ 7417ac59-06c3-432e-a4e5-f3bf2834f9f0/state: trusted.glusterfs.pathinfo: No such attribute
22:35 cpetersen_ 7417ac59-06c3-432e-a4e5-f3bf2834f9f0/ganesha: trusted.glusterfs.pathinfo: No such attribute
22:35 cpetersen_ 7417ac59-06c3-432e-a4e5-f3bf2834f9f0/statd: trusted.glusterfs.pathinfo: No such attribute
22:36 cpetersen_ same thing for both
22:36 cpetersen_ [root@file01 bf]# getfattr -n trusted.glusterfs.pathinfo -e text a2bf7659-5c7b-451b-902e-c7a2451be7b1
22:36 cpetersen_ a2bf7659-5c7b-451b-902e-c7a2451be7b1: trusted.glusterfs.pathinfo: No such attribute
22:36 cpetersen_ a2bf7659-5c7b-451b-902e-c7a2451be7b1/state: trusted.glusterfs.pathinfo: No such attribute
22:36 cpetersen_ a2bf7659-5c7b-451b-902e-c7a2451be7b1/ganesha: trusted.glusterfs.pathinfo: No such attribute
22:36 cpetersen_ a2bf7659-5c7b-451b-902e-c7a2451be7b1/statd: trusted.glusterfs.pathinfo: No such attribute
22:36 om2 joined #gluster
22:37 om2 joined #gluster
22:37 om2 joined #gluster
22:38 cpetersen_ [root@file01 shared_storage]# getfattr -n glusterfs.gfid.string /var/run/gluster/shared_storage/.glusterfs/a2/bf/a2bf7659-5c7b-451b-902e-c7a2451be7b1
22:38 tswartz ummm
22:38 cpetersen_ operation not supported lol
22:38 cpetersen_ sorry, I will sotp spamming the chat
22:38 tswartz you know about pastebin right?
22:39 cpetersen_ sorry, I do, noted
22:40 JoeJulian Why are you calling pathinfo anyway? It only tells you, on the client, which bricks your file is on.
22:42 JoeJulian You have two gfid entries in heal info. Check the brick that lists them and ensure they exist in the .glusterfs tree.
22:42 cpetersen_ they do, yes
22:43 cpetersen_ I was running all of the commands from the troubleshooting document which included pathinfo
22:44 om2 joined #gluster
22:44 JoeJulian How many links do they have? (The number after the permissions in "ls -l")
22:46 cpetersen_ one
22:46 cpetersen_ lrwxrwxrwx. 1 root root 52 Jan 27 10:11 a2bf7659-5c7b-451b-902e-c7a2451be7b1 -> ../../6c/2b/6c2b8dc9-584c-417b-9e9f-771231aa5f42/nfs
22:46 JoeJulian And there's your problem.
22:46 JoeJulian Oh, wait.
22:46 JoeJulian That's a symlink. It's a directory gfid.
22:49 cpetersen_ yes it is, hmm
22:49 cpetersen_ I'm confused :)
22:50 JoeJulian getfattr -h -m . -d -e hex a2bf7659-5c7b-451b-902e-c7a2451be7b1
22:51 cpetersen_ http://pastebin.com/jnhrcMVM
22:51 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
22:52 post-factum what a clever bot
22:52 cpetersen_ ha
22:52 shyam joined #gluster
22:52 julim joined #gluster
22:52 cpetersen_ http://fpaste.org/315994/02154514/
22:52 glusterbot Title: #315994 Fedora Project Pastebin (at fpaste.org)
22:52 post-factum one more script https://gist.github.com/semiosis/4392640
22:52 glusterbot 'Title: Glusterfs GFID Resolver\r \r Turns a GFID into a real path in the brick \xc2\xb7 GitHub (at gist.github.com)'
22:53 JoeJulian Yeah, but it's kind-of pointless for directories.
22:54 JoeJulian cpetersen_: try that command on the other brick.
22:55 JoeJulian I wonder if that's why directory heals often don't happen.. hmm...
22:55 cpetersen_ same thing
22:56 cpetersen_ I'm gonna check the second gfid
22:56 ctria joined #gluster
22:56 JoeJulian Ok, on the brick that's listing the gfid in heal info, cd ../../6c/2b/6c2b8dc9-584c-417b-9e9f-771231aa5f42/
22:57 JoeJulian Once there, there "getfattr -m . -d -e hex nfs"
22:58 cpetersen_ http://fpaste.org/315997/45402188/
22:58 glusterbot Title: #315997 Fedora Project Pastebin (at fpaste.org)
22:58 cpetersen_ that is for the second guid
22:58 cpetersen_ gfid sorry
22:58 JoeJulian Ah, there we go.
22:59 JoeJulian What's that look like on the other brick?
22:59 cpetersen_ http://fpaste.org/315998/14540219/
22:59 glusterbot Title: #315998 Fedora Project Pastebin (at fpaste.org)
22:59 cpetersen_ that is for a2bf
22:59 cpetersen_ I'll check other node
23:00 cpetersen_ http://fpaste.org/315999/40220041/
23:00 glusterbot Title: #315999 Fedora Project Pastebin (at fpaste.org)
23:00 cpetersen_ ahar
23:01 JoeJulian Odd that both directories are named nfs.
23:01 cpetersen_ that was from the a2bf symlink nfs folder on node2
23:01 haomaiwa_ joined #gluster
23:01 cpetersen_ so what does it all mean?
23:02 JoeJulian It suggests that a metadata change was in progress when a brick was shut down.
23:03 JoeJulian What I usually do with directories that have trusted.afr entries is just delete them, ie. setfattr -x trusted.afr.gluster_shared_storage-client-1 ...
23:03 JoeJulian That's only true for directories, never for files.
23:06 Bhaskarakiran joined #gluster
23:06 cpetersen_ so I would run that on the gfid file, or the nfs folder beyond the symlink?
23:11 plarsen joined #gluster
23:11 JoeJulian The nfs folder.
23:12 Akee joined #gluster
23:13 post-factum could you explain plz why removing xattr works for folders and why never for files?
23:14 cpetersen_ ok so "setfattr -x trusted.afr.gluster_shared_storage-client-1 nfs"
23:15 calavera joined #gluster
23:17 JoeJulian post-factum: I think there's a bug in how gluster handles healing folders that don't actually need healed. Normally, folders don't have trusted.afr attributes so returning them to that state doesn't break anything. For extra comfort, check and see that the times and permissions are correct, but I've never seen them wrong.
23:17 JoeJulian cpetersen_: yes, that and the other trusted.afr entry as well.
23:17 post-factum @JoeJulian: hm, thx
23:18 cpetersen_ well I did chmod the folder after the mount to have more permissions after I instantiated the glsuter volume
23:18 cpetersen_ gluster
23:18 cpetersen_ could that be why?
23:18 JoeJulian Files, on the other hand, always have those. There are now cli tools to remedy every known inconsistency between trusted.afr states in files, so I'd prefer to use those.
23:19 JoeJulian cpetersen_: on the brick? or through the mount?
23:19 cpetersen_ A is my mount
23:19 cpetersen_ B is my brick
23:20 cpetersen_ err
23:20 cpetersen_ "/var/run/A/B"
23:20 cpetersen_ I did chmod 777 on B to give all permission to anyone through the NFS share
23:20 JoeJulian s/mount/client mount/
23:20 glusterbot What JoeJulian meant to say was: cpetersen_: on the brick? or through the client mount?
23:20 cpetersen_ lol
23:20 cpetersen_ that bot is smart
23:20 JoeJulian Ah, yeah, don't f' with bricks.
23:21 cpetersen_ could that have been why?
23:21 JoeJulian That's like editing inodes through dd.
23:21 JoeJulian Probably
23:21 cpetersen_ k
23:21 cpetersen_ :)
23:21 cpetersen_ I'm living and learning, look at me!
23:22 post-factum bricks are read well but not well written to. however, someone told me once he writes to brick, and server-side heal does the thing
23:22 cpetersen_ yikes
23:23 JoeJulian And I'll spit in his beer if I ever see him.
23:24 post-factum meh, noone wants his beer to be spitted in
23:24 JoeJulian Speaking of seeing people, I'll be speaking at Incontro DevOps Italia in Bologna on April 1.
23:26 post-factum have a nice pizza there
23:26 cpetersen_ I ran the command on both NFS folders but it didn't work :(
23:26 cpetersen_ I stopped the volume and restarted glusterd on that node as well
23:26 cpetersen_ then I started a heal
23:27 cpetersen_ no dice
23:27 JoeJulian Check if there's a file under .glusterfs/indices/xattrop
23:27 cpetersen_ should that have eliminated the symlink?
23:27 cpetersen_ k
23:28 cpetersen_ yes they are both there
23:28 chirino joined #gluster
23:30 cpetersen_ do I delete these indices?
23:32 JoeJulian yes
23:33 cpetersen_ k
23:33 nickage__ joined #gluster
23:37 cpetersen_ alright thats cleared up then
23:37 cpetersen_ haha
23:39 cpetersen_ my nfs-ganesha fails properly which is great
23:39 cpetersen_ and the NFS share works great with the hypervisor
23:43 cpetersen_ is the cache-invalidation memory leak major?
23:43 cpetersen_ how much are we talking?
23:43 cpetersen_ constant maintenance requried?
23:43 cpetersen_ required?
23:43 post-factum that is related to FUSE-client
23:44 post-factum glusterfs api wasn't subjected to that AFAIK
23:45 cpetersen_ oic
23:45 post-factum probably, i stray cache invalidation with inode cache invalidation
23:45 post-factum fuse-client is subjected to _inode_ cache invalidation memleak
23:45 cpetersen_ ah yes ok
23:45 post-factum nfs-ganesha leaks too ;)
23:45 cpetersen_ so I should be good then :)
23:45 cpetersen_ oh yah?
23:45 cpetersen_ what amount are we talking?
23:46 post-factum haven't tried to trigger OOM, that bug is already fixed but not released
23:46 post-factum inode cache related as well
23:47 post-factum nothing scary for small scales, i guess
23:47 post-factum probably you do not have several millions of email files withing one volume
23:49 post-factum ok, good night, have fun
23:57 calavera joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary