Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-11-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 Ryan__ joined #gluster
00:14 plarsen joined #gluster
00:18 plarsen joined #gluster
00:22 JoeJulian @brick order
00:22 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
00:41 mpiet_cloud joined #gluster
00:54 Gambit15 joined #gluster
00:56 shdeng joined #gluster
01:10 Ryan__ joined #gluster
01:24 arpu joined #gluster
01:36 hchiramm joined #gluster
01:49 kramdoss_ joined #gluster
02:02 luizcpg joined #gluster
02:15 derjohn_mobi joined #gluster
02:27 haomaiwang joined #gluster
02:47 ilbot3 joined #gluster
02:47 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:59 haomaiwang joined #gluster
03:18 aravindavk joined #gluster
03:19 Lee1092 joined #gluster
03:34 nbalacha joined #gluster
03:37 shubhendu joined #gluster
03:44 atinm joined #gluster
03:50 ndarshan joined #gluster
03:56 satya4ever joined #gluster
03:56 magrawal joined #gluster
04:06 buvanesh_kumar joined #gluster
04:07 shubhendu joined #gluster
04:08 nishanth joined #gluster
04:10 itisravi joined #gluster
04:12 _ndevos joined #gluster
04:12 _ndevos joined #gluster
04:15 masber joined #gluster
04:16 shubhendu joined #gluster
04:30 hgowtham joined #gluster
04:33 Caveat4U joined #gluster
04:35 haomaiwang joined #gluster
04:47 k4n0 joined #gluster
05:05 prasanth joined #gluster
05:09 skoduri joined #gluster
05:13 [diablo] joined #gluster
05:17 karthik_us joined #gluster
05:18 satya4ever joined #gluster
05:30 ankitraj joined #gluster
05:31 kramdoss_ joined #gluster
05:34 karnan joined #gluster
05:35 jiffin joined #gluster
05:49 hchiramm joined #gluster
05:51 Muthu joined #gluster
05:51 shubhendu joined #gluster
05:56 rafi joined #gluster
05:58 kdhananjay joined #gluster
06:03 circ-user-XHlen joined #gluster
06:09 hchiramm joined #gluster
06:10 luizcpg joined #gluster
06:25 jeremyh joined #gluster
06:27 prth joined #gluster
06:32 Saravanakmr joined #gluster
06:34 hackman joined #gluster
06:34 skoduri joined #gluster
06:44 msvbhat joined #gluster
06:45 mhulsman joined #gluster
06:50 Bhaskarakiran joined #gluster
07:04 aravindavk joined #gluster
07:04 sanoj joined #gluster
07:15 devyani7 joined #gluster
07:26 jtux joined #gluster
07:59 jbrooks joined #gluster
07:59 jtux joined #gluster
08:04 jri joined #gluster
08:18 rastar joined #gluster
08:19 gem joined #gluster
08:26 ankitraj joined #gluster
08:31 abyss^ JoeJulian: I do getfattr -m .  -d -e hex on both dirs and both have the same attr :/
08:34 Caveat4U joined #gluster
08:35 HitexLT joined #gluster
08:38 Philambdo joined #gluster
08:39 satya4ever joined #gluster
08:39 Slashman joined #gluster
08:42 flying joined #gluster
08:45 Javezim abyss^ are you getting Input/Output error on a file or something?
08:46 kramdoss_ joined #gluster
08:55 abyss^ Javezim: I have split-brain on dir, on client side getfattr -m .  -d -e hex return nothing. On server side I get the same values. No, I don't get input/outpur error because that error accurs only on files not dirs.
08:56 Javezim Oh directory damn
08:58 Javezim You have a Replica setup?
08:58 mhulsman joined #gluster
08:59 abyss^ ofcourse :D
08:59 nix0ut1aw joined #gluster
09:00 abyss^ Javezim: yeah, damn, all files gluster's tool fixed for me but now I have issue with dirs;)
09:00 Javezim You're running this on the bricks? getfattr -m .  -d -e hex not on the glusterfs mountpoint?
09:02 bluenemo joined #gluster
09:07 riyas joined #gluster
09:07 panina joined #gluster
09:07 mhulsman1 joined #gluster
09:09 nix0ut1aw joined #gluster
09:26 abyss^ Javezim: yes
09:40 Gnomethrower joined #gluster
09:43 abyss^ Javezim: Any idea please?;)
09:51 msvbhat joined #gluster
09:54 buvanesh_kumar joined #gluster
09:55 om2_ joined #gluster
09:56 panina joined #gluster
10:08 Debloper joined #gluster
10:11 devyani7 joined #gluster
10:24 flyingX joined #gluster
10:27 derjohn_mobi joined #gluster
10:30 kotreshhr joined #gluster
10:36 ivan_rossi1 joined #gluster
10:40 social joined #gluster
11:10 Muthu joined #gluster
11:10 flying joined #gluster
11:28 Jacob843 joined #gluster
11:35 arcolife joined #gluster
11:37 kramdoss_ joined #gluster
11:38 kshlm Weekly community meeting will start in ~25 minutes in #gluster-meeting
11:38 panina joined #gluster
11:56 aravindavk joined #gluster
12:06 kshlm Another reminder. The weekly meeting has begun in #gluster-meeting.
12:10 jdarcy joined #gluster
12:13 kxseven joined #gluster
12:22 arcolife joined #gluster
12:28 snixor joined #gluster
12:35 prasanth joined #gluster
12:42 Debloper joined #gluster
12:53 arpu joined #gluster
12:55 luizcpg joined #gluster
13:00 itisravi joined #gluster
13:07 satya4ever joined #gluster
13:10 luizcpg joined #gluster
13:10 kdhananjay joined #gluster
13:11 msvbhat joined #gluster
13:13 johnmilton joined #gluster
13:20 shyam joined #gluster
13:28 jiffin joined #gluster
13:30 social joined #gluster
13:32 k4n0 joined #gluster
13:35 ira joined #gluster
13:43 unclemarc joined #gluster
13:51 mhulsman joined #gluster
13:56 d0nn1e joined #gluster
13:56 shyam joined #gluster
14:06 skoduri joined #gluster
14:06 mhulsman joined #gluster
14:07 ankitraj joined #gluster
14:10 prth joined #gluster
14:14 satya4ever joined #gluster
14:15 mhulsman joined #gluster
14:20 jiffin joined #gluster
14:22 haomaiwang joined #gluster
14:26 shyam joined #gluster
14:33 skylar joined #gluster
14:42 nbalacha joined #gluster
14:47 squizzi joined #gluster
14:49 jobewan joined #gluster
14:52 aravindavk joined #gluster
14:59 bowhunter joined #gluster
15:05 Debloper joined #gluster
15:06 [diablo] joined #gluster
15:07 JoeJulian abyss^: Please share the attributes from all bricks for the directory you're having trouble with via fpaste.org.
15:07 msvbhat joined #gluster
15:10 kpease joined #gluster
15:13 abyss^ JoeJulian: will you be tonight (my time;)). I have to go and I will be here within 4-5 hours. Thank you for your support.
15:13 JoeJulian I'll be here.
15:14 abyss^ Ok, thanks, see you soon:)
15:14 JoeJulian (unless the country suddenly descends in to chaos)
15:14 jiffin joined #gluster
15:23 aravindavk joined #gluster
15:27 farhorizon joined #gluster
15:41 farhoriz_ joined #gluster
15:58 ivan_rossi1 left #gluster
15:59 kpease joined #gluster
16:03 atrius joined #gluster
16:12 Gnomethrower joined #gluster
16:12 farhorizon joined #gluster
16:29 luizcpg joined #gluster
16:30 wushudoin joined #gluster
16:31 Caveat4U joined #gluster
16:35 haomaiwang joined #gluster
16:42 irated https://gist.github.com/pryorda/​3b41f5e85edd961cb7ccab040ce0df06
16:42 glusterbot Title: commands · GitHub (at gist.github.com)
16:42 irated Not sure how to fix the split brian files in that case.
16:48 hackman joined #gluster
17:03 jkroon joined #gluster
17:08 haomaiwang joined #gluster
17:16 hackman joined #gluster
17:17 social joined #gluster
17:24 mhulsman joined #gluster
17:25 mhulsman1 joined #gluster
17:31 jiffin joined #gluster
17:33 Caveat4U joined #gluster
17:47 Caveat4U joined #gluster
17:59 Caveat4U joined #gluster
18:03 Caveat4U_ joined #gluster
18:05 satya4ever joined #gluster
18:09 om2 joined #gluster
18:09 mlhess joined #gluster
18:10 virusuy joined #gluster
18:11 om2_ joined #gluster
18:14 Caveat4U joined #gluster
18:15 Caveat4U joined #gluster
18:16 kpease joined #gluster
18:26 guhcampos joined #gluster
18:30 Gambit15 joined #gluster
18:33 nathwill joined #gluster
18:42 rastar joined #gluster
18:42 plarsen joined #gluster
18:45 ajneil joined #gluster
18:47 pioto joined #gluster
18:47 Caveat4U joined #gluster
18:50 shyam joined #gluster
19:10 msvbhat joined #gluster
19:10 Caveat4U joined #gluster
19:14 irated JoeJulian: what do you think about this? root@prod-datastore1:~# gluster volume heal delta_sftp_prod split-brain source-brick prod-datastore1-data:/datastore/delta_sftp_prod
19:14 irated Healing gfid:f97438a0-4a50-48e1-826e-0f9774669280 failed:Transport endpoint is not connected.
19:16 JoeJulian I think you either have a dead brick process or you have a firewall problem.
19:18 guhcampos joined #gluster
19:25 irated same broadcast domain so not firewall
19:25 irated but the bricks appear online
19:25 * irated goes and checks
19:25 JoeJulian even iptables?
19:25 irated default accept
19:26 irated no denies
19:26 JoeJulian May want to take a look with wireshark then. "Transport endpoint is not connected" comes from the tcp stack.
19:27 irated should i try restarting the process first maybe it "hung"
19:28 JoeJulian It's possible.
19:28 JoeJulian You can also try connecting with telnet/nc
19:31 shyam joined #gluster
19:32 guhcampos joined #gluster
19:32 irated yeah connected
19:34 JoeJulian irated: One other thing, when hunting down connection issues, always check against the bricks at the same timestamp. Sometimes there's clues there.
19:35 haomaiwang joined #gluster
19:37 irated fwiw data is the same
19:38 irated date*
19:38 Caveat4U joined #gluster
19:49 abyss^ JoeJulian: I won't make it today with that split-brain:( My son don't wanna sleep;) Please forgive me. Will you be able tommorow about that time when you firstly write to me?
19:51 JoeJulian I'm around here normally between 16:00-17:00 UTC for most of the following 8 hours.
19:55 JoeJulian irated: Sorry, I only help publicly. This isn't my job so I'm much more interested in helping you learn how to figure out the problems yourself than I am to figure it out for you. I am, however, willing to look at log files to help you identify the lines you should be looking at and try to tell you what they mean.
19:55 irated Fair enough
19:56 JoeJulian Paste up the problem you see in the client log to fpaste.org and if there's any corresponding log entry on the brick logs for the error, paste that too. Include a chunk of lines above and below the error in case there's context that applies.
19:56 irated How would I trace this? Load spike to 15 on the client side, 5 on the serverside, and when i kill the 2nd brick all is happy.
19:57 irated no entries in the logs from what i can tell
19:57 fabiang joined #gluster
19:57 JoeJulian Sounds like it's trying to heal something.
19:57 JoeJulian That's why I turn off client-side heals (always).
19:58 JoeJulian That especially would make sense if there's no load without the replica brick.
19:59 JoeJulian During heals, my servers typically sit around a 60 load but the clients still seem to function normally.
20:00 irated Yeah, our clients arnt functioning normally for some reason
20:00 farhoriz_ joined #gluster
20:00 irated https://gist.github.com/pryorda/​e91b7c6a5f40eb6dac9d21c86194a51d
20:00 glusterbot Title: gist:e91b7c6a5f40eb6dac9d21c86194a51d · GitHub (at gist.github.com)
20:01 Caveat4U joined #gluster
20:02 JoeJulian "crawl is in progress" so that looks like it's doing a full heal.
20:06 irated we have alot of big directories that take a bit to heal
20:06 irated Is it crawling server side or client?
20:07 JoeJulian Pretty sure it's server side, but then again I don't allow clients to participate in the heals so maybe they do.
20:07 irated I have the clients off so this should be strictly server side right?
20:16 Caveat4U joined #gluster
20:16 shaunm joined #gluster
20:18 Caveat4U joined #gluster
20:22 JoeJulian right
20:29 irated JoeJulian: what if i started those after a heal has started?
20:30 Caveat4U joined #gluster
20:30 irated Also, It apears since I have disabled healing completely and restarted healing all is fine.
20:30 irated Does gluster suck at healing large directories?
20:31 JoeJulian No, but when you started a heal on a client, I'm starting to think that disabling heals doesn't actually stop the ongoing heal.
20:31 * irated is starting to agree with you
20:31 irated :)
20:31 JoeJulian In fact, I bet it doesn't even empty the queue (16 by default).
20:32 irated whats the best queue size?
20:32 JoeJulian Doesn't matter if you limit heals to the shd.
20:32 irated ah
20:33 irated So now that i have the setting for the clients to be disabled you think i could restart the shd?
20:33 JoeJulian Or just let it run.
20:33 JoeJulian Or are you saying you disabled the shd?
20:33 irated Yes
20:34 irated I disabled healing altogether to prove that it is infact healing being the poor sport
20:34 JoeJulian Then yes, I would leave shd running.
20:34 irated okay
20:34 JoeJulian You'll see high load. Mostly cpu.
20:35 JoeJulian If you don't have enough cpu to handle differential heals, change the self-heal algorithm to full. That'll use way more network but no cpu.
20:35 irated soooo...
20:35 irated anything with cluster. is client side?
20:36 JoeJulian For self-heal, yes.
20:36 irated okay
20:36 irated what does full do?
20:36 JoeJulian Just copies the whole file.
20:37 irated like meta-data file or?
20:37 JoeJulian diff only copies blocks of the file that are different.
20:37 irated oh
20:37 Caveat4U joined #gluster
20:37 JoeJulian diff locks a chunk of file, calculates a hash at both ends, then compares them. If they're the same, it moves on. If not, it copies that chunk from source to sink.
20:38 JoeJulian That hash calculation can use a lot of cpu.
20:38 irated where do i find the definition of all the settings?
20:38 JoeJulian gluster volume set help
20:38 JoeJulian It's pretty detailed.
20:39 irated set this to full? cluster.data-self-heal-algorithm        (null)
20:40 irated since we have a decent network?
20:40 irated and its mainly small files
20:40 irated 1K -> 100MB
20:41 JoeJulian If that works best for your use case, sure.
20:46 irated testing it out
20:47 irated Do you normally deal with large directories
20:47 irated say 250K+
20:47 JoeJulian No, I try to keep them smaller.
20:48 post-factum irated: if you have such a directory, you are doing something wrong
20:48 post-factum that won't work okay even with local fs
20:48 JoeJulian Typically you should try to tree them down with some sort of logic.
20:49 irated JoeJulian: trust me i know
20:49 irated we were talking about that today..
20:49 post-factum also, granular directory healing was merged, that should speed up things
20:49 irated post-factum: its faster with local dirs then gluster.
20:49 post-factum irated: lol, obviously
20:50 irated :P
20:50 irated gluster solves alot of problems for us
20:50 post-factum and brings new
20:50 irated we are just trying to get it "smooth" and right
20:50 irated cpu load client side jumped again :9
20:50 irated :(
20:50 irated and listings are broked
20:57 JoeJulian Mount a client into an unused directory. Wait for it to have an increased load. fpaste the new log file.
20:59 irated do i need to set the client to debug or anything?
20:59 irated Can I set the self heal threads to 1 or something?
21:00 irated just to see if its a concurrency with listing lots of dirs at once loading things up?
21:01 JoeJulian I don't think it's threaded.
21:03 irated this is only if client side three are enabled right ? cluster.background-self-heal-count
21:04 JoeJulian I'm not even sure if that's threaded or just queued.
21:08 farhorizon joined #gluster
21:09 irated ah
21:12 irated JoeJulian: Option: disperse.background-heals
21:14 JoeJulian That's for dispersing them among shd daemons.
21:20 gem joined #gluster
21:20 irated so....
21:20 irated What triggers a file to be healed?
21:24 post-factum irated: e.g., stat()
21:25 Caveat4U joined #gluster
21:26 Caveat4U joined #gluster
21:29 amye joined #gluster
21:34 irated So does that mean every file is considered a healable once its wrote to the fs?
21:35 irated Also what are the risks with leaving SHD off?
21:36 irated it seems like its creating more issues right now...
21:36 irated Which could be because of our datastructure + # of files
21:49 mhulsman joined #gluster
21:50 farhoriz_ joined #gluster
21:50 mhulsman1 joined #gluster
21:53 JoeJulian Actually, a self-heal is triggered by lookup()
21:54 JoeJulian irated: So with shd and client heals off, you increase the risk of split brain and have the potential for bypassing replication.
21:55 JoeJulian So if the one good disk fails, you've lost data.
21:59 irated so the last one really isnt a "big risk" as we have a san behind our gluster vms
21:59 irated the splitbrain concerns me
22:02 Caveat4U joined #gluster
22:05 irated okay this is interesting the load jumps on the opposite server from the heal as well :/
22:05 snehring irated, is this ovirt/rhev?
22:06 irated nah vmware
22:06 irated @ snehring
22:06 snehring ah k, I had a similar weirdness with ovirt that I never got a satisfying fix for it just 'fixed itself'
22:07 snehring one server would have high load and clients wouldn't be able to ls
22:07 irated snehring: fun
22:07 irated snehring: check if heal got turned off ;)
22:08 snehring heal was on full, it was just really strange
22:09 snehring hopefully you have a bit better luck
22:09 martin_pb joined #gluster
22:11 irated mine is really strange and I have narrowed it down ti healing
22:11 irated just not sure why
22:14 farhorizon joined #gluster
22:24 haomaiwang joined #gluster
22:29 JoeJulian My guess would be that your vms are resource starved.
22:29 Caveat4U joined #gluster
22:33 hackman joined #gluster
22:42 zat joined #gluster
22:44 zat Is there a trick to setup a non-uniform replica scheme? Example: I have replica 4 in a cluster of 20 computers, but I want 2 servers to always have a full copy of everything.
22:50 farhoriz_ joined #gluster
22:52 Caveat4U joined #gluster
22:53 JoeJulian zat: First of all, replica 4 is probably overkill. You could make 2 servers that have their own volume and use geo-replicate to back everything up to them.
22:54 irated JoeJulian: cpu resources are the same on the two different hosts
22:54 irated I thought it was starved as well
22:55 irated and decided to migrate it around
22:55 zat JoeJulian: that was an example though, the thing is that those 20 computers can be shut off at any time, all of them.
22:55 zat and the two servers should always be up
22:55 irated if you access the files inside brick folder structure its uber fast
22:55 irated only 76 more files to heal so im just letting it go
22:56 JoeJulian +1
22:57 Caveat4U joined #gluster
22:57 JoeJulian zat: That's cool. I always recommend you just do the math and ensure that you're not wasting resources to get the SLA you desire. ,,(reliability calculation)
22:57 glusterbot zat: I do not know about 'reliability calculation', but I do know about these similar topics: 'reliability calculations'
22:57 JoeJulian @reliability calculations
22:57 glusterbot JoeJulian: Calculate your system reliability and availability using the calculations found at http://www.eventhelix.com/realtimemantra/faul​thandling/system_reliability_availability.htm . Establish replica counts to provide the parallel systems to meet your SLA requirements.
23:00 zat lol nice
23:00 zat it's for a school lab
23:01 zat I want to start Linux from a Gluster volume, where every computer should be part of the cluster also.
23:14 Caveat4U joined #gluster
23:15 farhorizon joined #gluster
23:42 Caveat4U joined #gluster
23:54 masuberu joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary