Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-11-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:11 zhangjn joined #gluster
00:27 mlhamburg_ joined #gluster
00:56 EinstCrazy joined #gluster
00:56 zhangjn joined #gluster
01:12 vpsspecialists joined #gluster
01:13 zhangjn joined #gluster
01:39 Lee1092 joined #gluster
01:41 haomaiwa_ joined #gluster
01:50 harish joined #gluster
01:59 nangthang joined #gluster
02:01 haomaiwa_ joined #gluster
02:11 mlncn joined #gluster
02:36 kshlm joined #gluster
02:48 ilbot3 joined #gluster
02:48 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:01 haomaiwa_ joined #gluster
03:06 Merlin__ joined #gluster
03:14 Humble joined #gluster
03:22 kdhananjay joined #gluster
03:23 kshlm joined #gluster
03:26 srsc joined #gluster
03:39 sakshi joined #gluster
03:40 skylar joined #gluster
03:49 atinm joined #gluster
03:51 shubhendu joined #gluster
03:56 skylar1 joined #gluster
03:56 bharata-rao joined #gluster
04:01 itisravi joined #gluster
04:01 haomaiwa_ joined #gluster
04:01 nbalacha joined #gluster
04:02 gem joined #gluster
04:03 neha_ joined #gluster
04:05 kdhananjay joined #gluster
04:19 TheSeven joined #gluster
04:20 zhangjn joined #gluster
04:20 ramteid joined #gluster
04:21 jiffin joined #gluster
04:22 ppai joined #gluster
04:22 nishanth joined #gluster
04:28 aravindavk joined #gluster
04:32 ghenry joined #gluster
04:35 vimal joined #gluster
04:38 kotreshhr joined #gluster
04:55 kalzz joined #gluster
05:01 haomaiwang joined #gluster
05:05 ashiq joined #gluster
05:06 R0ok_ joined #gluster
05:07 ndarshan joined #gluster
05:09 RameshN joined #gluster
05:12 ramky joined #gluster
05:14 zhangjn joined #gluster
05:15 Bhaskarakiran joined #gluster
05:16 rafi joined #gluster
05:22 skylar joined #gluster
05:23 Manikandan joined #gluster
05:27 skoduri joined #gluster
05:29 pppp joined #gluster
05:30 nishanth joined #gluster
05:33 poornimag joined #gluster
05:35 hos7ein joined #gluster
05:41 anil_ joined #gluster
05:45 dusmant joined #gluster
05:46 hgowtham joined #gluster
05:47 hgowtham_ joined #gluster
05:59 SOLDIERz joined #gluster
06:01 6A4AA7FHP joined #gluster
06:02 overclk joined #gluster
06:05 kdhananjay joined #gluster
06:07 vmallika joined #gluster
06:08 RameshN joined #gluster
06:10 Saravana_ joined #gluster
06:10 dlambrig_ joined #gluster
06:11 dusmant joined #gluster
06:12 kotreshhr joined #gluster
06:17 Manikandan joined #gluster
06:19 beeradb_ joined #gluster
06:20 zhangjn joined #gluster
06:21 deepakcs joined #gluster
06:21 overclk joined #gluster
06:25 vimal joined #gluster
06:32 vimal joined #gluster
06:33 Norky joined #gluster
06:36 spalai joined #gluster
06:37 Merlin__ joined #gluster
06:39 kotreshhr joined #gluster
06:40 vimal joined #gluster
06:41 Merlin___ joined #gluster
06:43 Manikandan joined #gluster
06:48 dusmant joined #gluster
06:49 mlhamburg1 joined #gluster
06:49 zhangjn joined #gluster
06:51 atalur joined #gluster
07:01 haomaiwa_ joined #gluster
07:10 mlncn joined #gluster
07:12 kovshenin joined #gluster
07:13 [Enrico] joined #gluster
07:15 Humble joined #gluster
07:15 RameshN joined #gluster
07:18 mobaer joined #gluster
07:28 F2Knight joined #gluster
07:28 mobaer left #gluster
07:29 aspandey joined #gluster
07:30 mhulsman joined #gluster
07:32 zhangjn joined #gluster
07:33 jtux joined #gluster
07:35 mobaer joined #gluster
07:50 rafi joined #gluster
07:51 spalai left #gluster
07:53 arcolife joined #gluster
07:59 vmallika joined #gluster
08:00 SOLDIERz joined #gluster
08:01 haomaiwa_ joined #gluster
08:13 deniszh joined #gluster
08:17 plarsen joined #gluster
08:26 Merlin__ joined #gluster
08:26 fsimonce joined #gluster
08:30 Trefex joined #gluster
08:36 haomaiwa_ joined #gluster
08:38 Merlin__ joined #gluster
08:38 Manikandan joined #gluster
08:40 ppai_ joined #gluster
08:44 ivan_rossi joined #gluster
08:50 dusmant joined #gluster
08:51 gildub joined #gluster
08:52 ctria joined #gluster
08:52 gem_ joined #gluster
08:54 ivan_rossi joined #gluster
08:55 a2 joined #gluster
09:01 mhulsman joined #gluster
09:01 haomaiwa_ joined #gluster
09:05 RameshN joined #gluster
09:09 spalai joined #gluster
09:15 Norky joined #gluster
09:17 rjoseph joined #gluster
09:23 kovshenin joined #gluster
09:24 suliba joined #gluster
09:26 mhulsman1 joined #gluster
09:28 fsimonce joined #gluster
09:29 jrm16020 joined #gluster
09:29 arcolife joined #gluster
09:30 suliba joined #gluster
09:36 Slashman joined #gluster
09:39 dirkcjelli joined #gluster
09:41 adamaN joined #gluster
09:41 dirkcjelli Hi everybody ! Is is possible to copy filesystems (via xfsdump) from a working replica brick to its failed counterpart and re-populating the .glusterfs directory ? We have tried healing the volumes and only get <10MB/s which will take ages
09:45 ctria joined #gluster
09:49 dusmant joined #gluster
09:49 Merlin__ joined #gluster
09:50 skylar1 joined #gluster
09:56 creshal joined #gluster
10:00 mhulsman joined #gluster
10:01 dirkcjelli joined #gluster
10:01 RedW joined #gluster
10:01 haomaiwa_ joined #gluster
10:02 mhulsman2 joined #gluster
10:06 zhangjn joined #gluster
10:07 nishanth joined #gluster
10:08 DRoBeR joined #gluster
10:12 creshal Is there some memory leak in the glusterfs process…? I was copying some 50 GB data onto a replica and the process was eventually kicked by the OOM killer for gobbling up 8+ GiB RAM.
10:14 creshal …wait, not 8 GiB, 8 million pages. So 32 GiB RAM, aka all the machine had.
10:14 creshal That can't be right.
10:15 itisravi dirkcjelli: If the clients are still accessing the volume, it is not advisable.  You can trigger lookups from multiple clients to speed up the heal.
10:17 itisravi creshal: what was the process that got OOM'ed - the mount or the brick process?
10:18 jrm16020 joined #gluster
10:20 creshal itisravi: https://gist.github.com/creshal/ae1df50ac3f101497975 Already hardreset'd the machine, don't have any more data, sorry.
10:20 glusterbot Title: dmesg · GitHub (at gist.github.com)
10:22 mobaer joined #gluster
10:24 aravindavk joined #gluster
10:33 bluenemo joined #gluster
10:34 gorfel joined #gluster
10:35 dirkcjelli itisravi: For some obscure reason, as soon as I start the "find ..... stat{}", everything slows to a crawl; I get <10MB/s heal speed and blocking io from the clients
10:36 gorfel How do you guys deal with glusterfs' inability to mount subfolders of the volume in web server cluster environments? Create a new volume for each instance which need glusterfs' HA functionality?
10:39 csim github--
10:39 glusterbot csim: github's karma is now -1
10:39 csim seriously, as soon as you try to do something differnet than what they propose, that's just annoying
10:41 MrAbaddon joined #gluster
10:43 _shaps_ joined #gluster
10:46 deepakcs joined #gluster
10:50 atalur joined #gluster
10:55 nishanth joined #gluster
10:59 shubhendu atinm, ping
10:59 glusterbot shubhendu: Please don't naked ping. http://blogs.gnome.org/markmc/2014/02/20/naked-pings/
11:01 haomaiwa_ joined #gluster
11:09 dirkcjelli Is there any possibility to copy data to a broken brick and getting it online without  self-heal
11:09 dirkcjelli ?
11:17 dusmant joined #gluster
11:27 kenansulayman joined #gluster
11:27 creshal itisravi: Okay, I can get it reproduced. The process `/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs` is hogging memory like crazy.
11:28 itisravi skoduri:  ^^
11:28 itisravi creshal: what version of gluster is this?
11:29 Guest52346 joined #gluster
11:29 creshal 3.5.2 ?
11:29 mlncn joined #gluster
11:30 itisravi ok, that is a bit dated.
11:30 creshal …Ah, just noticed that I don't have the official repos enabled on that machine. I'll try again with 3.5.4.
11:30 itisravi creshal: 3.7.6 is the latest release.
11:31 creshal Whatever. :D
11:33 RameshN joined #gluster
11:48 Philambdo joined #gluster
11:50 Humble joined #gluster
12:02 kotreshhr left #gluster
12:04 skoduri creshal, yes as itisravi has mentioned please upgrade the gluster binaries and check the setup
12:09 cked350 joined #gluster
12:12 creshal Okay, updated both to 3.7.6 and… nothing works anymore. I don't even know where to start un-fucking this.
12:12 creshal (both nodes, that is.)
12:15 creshal Okay, manually a) rebooting b) restarting glusterfs-server.service and c) restarting each volume on every server *seems* to work for now.
12:22 capri left #gluster
12:26 cabillman joined #gluster
12:26 cked350 left #gluster
12:34 EinstCrazy joined #gluster
12:42 lpabon joined #gluster
12:44 harish joined #gluster
12:45 dirkcjelli joined #gluster
12:45 dirkcjelli [repost b/c of error] Is there any possibility to copy data to a broken brick and getting it online without  self-heal
12:48 _shaps_ left #gluster
12:59 haomaiwang joined #gluster
12:59 zhangjn joined #gluster
13:00 zhangjn joined #gluster
13:01 haomaiwa_ joined #gluster
13:07 Philambdo joined #gluster
13:10 skylar1 joined #gluster
13:24 haomaiwang joined #gluster
13:28 deniszh1 joined #gluster
13:32 mhulsman joined #gluster
13:34 mhulsman1 joined #gluster
13:40 unclemarc joined #gluster
13:49 arcolife joined #gluster
13:51 ahino joined #gluster
13:54 rwheeler joined #gluster
13:55 B21956 joined #gluster
13:55 nishanth joined #gluster
13:59 julim joined #gluster
14:01 nbalacha joined #gluster
14:01 ira joined #gluster
14:01 haomaiwa_ joined #gluster
14:02 kdhananjay joined #gluster
14:03 plarsen joined #gluster
14:07 overclk joined #gluster
14:11 atinm joined #gluster
14:29 creshal Okay, with 3.7.6 the memleak seems to be gone. The nfs glusterfs process still needs quite a bit of RAM, but it's stable now.
14:31 atinm joined #gluster
14:33 mhulsman joined #gluster
14:34 DV__ joined #gluster
14:35 mhulsman1 joined #gluster
14:36 shyam joined #gluster
14:40 dgandhi joined #gluster
14:41 dgandhi joined #gluster
14:42 skylar joined #gluster
14:44 ToMiles joined #gluster
14:45 creshal …apart from a kernel 0pointer when trying to mount gluster nfs volumes.
14:49 dlambrig_ joined #gluster
14:50 Merlin__ joined #gluster
14:50 Philambdo joined #gluster
14:51 bluenemo joined #gluster
14:57 chirino joined #gluster
14:59 creshal https://gist.github.com/creshal/ef993caaf081060f38de This happened when I was just trying to bind-mount something onto a mounted gluster volume. Afterwards, gluster processes (and everything accessing mount data and/or any gluster mount points) just locked up.
14:59 glusterbot Title: gist:ef993caaf081060f38de · GitHub (at gist.github.com)
14:59 spalai joined #gluster
15:00 creshal How the hell am I supposed to deploy something like that in production? I can't reboot nodes every few hours just because something inside gluster hangs up without giving me as much as an error message.
15:00 julim_ joined #gluster
15:01 haomaiwang joined #gluster
15:07 Merlin___ joined #gluster
15:20 dusmant joined #gluster
15:22 Philambdo joined #gluster
15:23 ahino joined #gluster
15:25 Philambdo joined #gluster
15:30 ctria joined #gluster
15:31 mhulsman joined #gluster
15:33 josh__ joined #gluster
15:36 cristov joined #gluster
15:37 skoduri joined #gluster
15:38 dlambrig_ joined #gluster
15:42 6JTACJNJZ joined #gluster
15:42 7YUAAD0V7 joined #gluster
15:53 wushudoin joined #gluster
15:53 hagarth_ joined #gluster
15:55 maserati joined #gluster
16:01 coredump joined #gluster
16:01 haomaiwang joined #gluster
16:06 gem joined #gluster
16:07 JoeJulian creshal: Gluster doesn't run in the kernel. The kernel oops is from a kernel function.
16:07 dblack joined #gluster
16:07 Park joined #gluster
16:08 cholcombe joined #gluster
16:09 Park hi, anyone knows why not able to add brick which is not part of the volume, but it always complain "already part of a volume" ?
16:09 JoeJulian @path or prefix
16:09 glusterbot JoeJulian: http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/
16:10 Peppard joined #gluster
16:10 Park I also tried clear the xattrs:
16:10 Park # setfattr -x trusted.gfid /data/brk07/srv
16:10 Park setfattr: /data/brk07/srv: No such attribute
16:10 Park setfattr -x trusted.glusterfs.vol0 /data/brk07/srv
16:10 Park setfattr: /data/brk07/srv: No such attribute
16:10 JoeJulian That's kind-of random.
16:11 Park # ls /data/brk07/srv/ -a
16:11 Park .  ..
16:11 Park gluster volume info vol0 | grep brk07
16:11 Park #
16:12 JoeJulian Park: read the article
16:12 ToMiles Anyone that can help me figure out why I can't do  a full heal on my volumes since I upgrade my nodes from v3.7.2 to 3.7.6 ? always get commit failed for at least 2 nodes in my 6 node (2repl with dist) setup on ubuntu
16:13 JoeJulian When you attempt the command, look for the error in your glusterd logs. At least one of them has some problem with it.
16:13 JoeJulian logstash++
16:13 glusterbot JoeJulian: logstash's karma is now 1
16:14 Park JoeJulian, thanks, but I actually restarted glusterd on all of the nodes in the cluster, but it doesn't help.  And the most weried thing is the brick actually were not part of any volume previously, it's a completely new cluster.
16:14 Pablo|off| joined #gluster
16:14 ToMiles thats the thing those that work have "starting full sweep on" in the log but those with commit fail have no messages at the time of the command
16:14 JoeJulian Park: That article says nothing about restarting nodes.
16:15 JoeJulian It's probably marked from a failed attempt to create a volume or add a brick.
16:15 Park JoeJulian, yes, I tried to add-brick once, but it failed.
16:15 JoeJulian Park: Oh! I see what you did there.
16:16 JoeJulian volume-id is actually the literal string, "volume-id"
16:16 JoeJulian not the name of your volume.
16:16 Telsin left #gluster
16:17 Telsin joined #gluster
16:17 ToMiles and when I attempt the command from an other node the commit failed nodes are not exactly the same 2, and those that failed before run a full sweep
16:17 Park JoeJulian, nah... it works now,  thanks for pointing out my stupid mistake, never mind. :-)
16:17 JoeJulian btw... if you follow any examples on my blog, I always try to use $something for things you need to replace.
16:17 JoeJulian :)
16:18 EinstCrazy joined #gluster
16:18 Park ok, actually I already read some of your articles,  appreciate for the nice writings..
16:18 JoeJulian Thank you.
16:20 ToMiles its always at least one member of the other 2 replicate pairs other then the one the command is run on that give fail commit
16:21 _feller joined #gluster
16:22 Merlin__ joined #gluster
16:23 JoeJulian So is there an error in a log?
16:24 Merlin__ joined #gluster
16:24 ToMiles no error or any other message at the time of command attempt
16:25 ToMiles expect on the workign nodes that report start and succes of sweep
16:25 JoeJulian Odd
16:25 ToMiles indeed
16:25 cristov joined #gluster
16:25 JoeJulian The error reported from the cli has to originate somewhere.
16:26 ToMiles true indeed my experience with other such problems before, not this time :-)
16:28 ToMiles and its not volume specific because same issue with newly create test volume
16:28 JoeJulian So the pieces involved are the cli, of course, glusterd (all of them) and glustershd (all of them) with logs at $logpath/cli.log on the machine from which you run the command, $logpath/{etc-glusterfs-glusterd.vol.log,glustershd.log}.
16:30 ToMiles scanning thru them now
16:30 JoeJulian My guesses include something not getting restarted after the upgrade, and something to do with unprivileged ports.
16:30 Humble joined #gluster
16:31 spalai left #gluster
16:32 ToMiles Self-heal Daemon is only local right? no tcp port?
16:32 JoeJulian right
16:33 overclk joined #gluster
16:34 ToMiles ah I only get error when I try to run full heal command on the failing commit nodes itself
16:34 ToMiles W [socket.c:588:__socket_rwv] 0-glusterfs: readv on /var/run/gluster/quotad.socket failed (Invalid argument)
16:35 ToMiles warning sorry no error
16:36 ToMiles ah bingo: [glusterfsd-mgmt.c:619:glusterfs_handle_translator_op] 0-glusterfs: Couldn't get xlator xl-0
16:37 ToMiles but only when heal coommand is run on the problem nodes, logs nothign when run from other nodes
16:37 JoeJulian ToMiles: which version is this?
16:37 JoeJulian and which log was that in?
16:37 ToMiles glusterfs 3.7.6 built on Nov  9 2015 15:17:09
16:37 ToMiles glustershd
16:38 plarsen joined #gluster
16:38 JoeJulian try stopping glusterd, killing the glustershd process (pkill -f glustershd), and starting glusterd again on all your servers and see if that solves it.
16:39 JoeJulian If it does not, that looks like a pointer error.
16:39 ToMiles remember trying the restart before without luck,but  I'll double check to be sure
16:49 ToMiles restarted all gluster-server and glustershd no change, "xlator xl-0" error when heal on problem nodes
16:49 skoduri joined #gluster
16:49 ToMiles or should I have restarted brick deamons too?
16:52 timotheus1 joined #gluster
16:54 shubhendu joined #gluster
16:54 JoeJulian ToMiles: no
16:58 JoeJulian hmm, mmkay.. xl-%d is expected. Not sure how that'd dereferenced though.
16:58 ToMiles since the 0 in xl-0 is just the key in a for loop of the glusterfs_handle_translator_op is there a way to tell which translator it is err on?
17:00 zoldar Is that a normal behavior when issuing "gluster volume heal volume-name info" (log excerpt from glfsheal-*.log ) ? http://dpaste.com/03V8DXC
17:00 glusterbot Title: dpaste: 03V8DXC (at dpaste.com)
17:01 6JTACJN9N joined #gluster
17:02 JoeJulian Nothing there looks abnormal if that's what you're asking. None of it has anything to do with that command though.
17:03 jiffin joined #gluster
17:03 zoldar JoeJulian: this is output consistently with every invocation of the command
17:05 zoldar JoeJulian: the reason I'm asking is that every couple of invocations there's one where I see some files undergoing heal when there's no loss of connection or anything like that happening
17:07 ToMiles zoldar: isn't that expected behaviour since the heal is trigger upon accessing a file, unless a full heal is requested?
17:08 zoldar ToMiles: so, even though I issue a query for heal info, the actual heal is force?
17:08 zoldar forced*
17:08 JoeJulian That log is simply the startup of a client.
17:09 JoeJulian Every client started should look like that.
17:10 zoldar JoeJulian: Ok, so maybe a bit different question - does "gluster volume ... info" force self heal at the same time? I'm using it in my nagios monitoring script and it brings up false alarms
17:10 JoeJulian no
17:11 bennyturns joined #gluster
17:11 JoeJulian zoldar: which version are you running?
17:11 zoldar 3.7.6
17:12 gem joined #gluster
17:12 zoldar JoeJulian: with a patch applied for arbiter issues (I'm using a 2+1 setup)
17:13 ir2ivps5 joined #gluster
17:16 zoldar JoeJulian: this is the related bug report from me https://bugzilla.redhat.com/show_bug.cgi?id=1283956
17:17 glusterbot Bug 1283956: unspecified, unspecified, ---, bugs, NEW , Self-heal triggered every couple of seconds and a 3-node 1-arbiter setup
17:18 ToMiles must been what we ran into when test driving the arbiter stuff in our test setup, we decided to stay with regular 2 repl setup
17:18 zoldar oh bummer :(
17:18 turkleton joined #gluster
17:19 turkleton Good morning folks.
17:19 ToMiles that and very slugish performance compared to 2 repl seti[
17:20 ToMiles I might reconsider later when it stablises
17:21 zoldar ToMiles: ok, thanks for confirming, I will hold off with it then too
17:22 zoldar ... probably
17:22 ToMiles was on 3.72 tho so dont know if still valid for 3.7.6
17:22 mlncn joined #gluster
17:26 ToMiles JoeJulian: how would you advise me to proceed with the heal/xlator error? open bug report or perfer me to debug this further here first later today?
17:26 JoeJulian Please do file a bug report.
17:26 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
17:27 ToMiles ok thanks for your assistance
17:28 rjoseph joined #gluster
17:28 JoeJulian I'll still try to see what I can figure out, but it's too damned abstracted.
17:28 ToMiles true xl-0 is not much to go on :-)
17:30 ToMiles have to go for a bit now but I'll post it on bugzilla later on
17:33 dlambrig_ joined #gluster
17:34 ayma joined #gluster
17:35 Rapture joined #gluster
17:52 Gill joined #gluster
17:58 haomaiwang joined #gluster
18:01 MrAbaddon joined #gluster
18:04 ivan_rossi left #gluster
18:10 EinstCrazy joined #gluster
18:14 arcolife joined #gluster
18:15 F2Knight joined #gluster
18:16 Gill_ joined #gluster
18:44 shyam joined #gluster
18:52 ira joined #gluster
18:54 kovshenin joined #gluster
18:58 calavera joined #gluster
19:01 Park joined #gluster
19:07 dblack joined #gluster
19:12 jbrooks joined #gluster
19:17 chirino joined #gluster
19:17 ahino joined #gluster
19:17 Merlin__ joined #gluster
19:21 MrAbaddon joined #gluster
19:46 Dave joined #gluster
19:47 jwd joined #gluster
19:58 rwheeler joined #gluster
19:59 wushudoin joined #gluster
20:02 Merlin__ joined #gluster
20:05 turkleton joined #gluster
20:06 turkleton Hey folks! I'm working on automated recovery of a replica 2 brick Gluster volume, and it was working fine. After a few iterations, it now seems to be failing. http://termbin.com/0g2i
20:06 turkleton I'm using EBS for the storage volume, and I've moved /var/lib/glusterd to the EBS volume and put a symlink from /var/lib/glusterd to the location on the EBS volume
20:07 turkleton the glusterfs-server daemon fails to start, and I'm using 3.6.6
20:07 turkleton Any ideas?
20:07 turkleton I've seen it work several times before
20:09 mlncn joined #gluster
20:11 Merlin__ joined #gluster
20:27 JoeJulian turkleton: "resolve brick failed in restore" suggests that a server hostname is not resolving.
20:27 turkleton ah, hmm
20:27 turkleton Lemme check the other box
20:27 turkleton I'm using Route53 DNS for the two nodes
20:28 turkleton Ah, hrm
20:28 turkleton peer in rejected state
20:28 Merlin__ joined #gluster
20:28 turkleton http://termbin.com/un7e
20:32 hagarth_ joined #gluster
20:33 turkleton http://www.gluster.org/community/documentation/index.php/Resolving_Peer_Rejected - Interesting approach. It seems like the general message is "just keep restarting it til it works"
20:35 mhulsman joined #gluster
20:36 turkleton http://termbin.com/td2u - "bad" node
20:36 turkleton good node http://termbin.com/bb6z
20:36 turkleton good node thinks bad node is good
20:36 turkleton Only one is running the GlusterFS services for the volume
20:37 JoeJulian I really hate the peering output.
20:37 turkleton Yeah :(
20:37 turkleton http://termbin.com/y88x
20:37 turkleton That's the "broken" node
20:39 JoeJulian btw... instead of a symlink, you could just change the location in /etc/glusterfs/glusterd.vol in your image.
20:39 turkleton hahaha
20:39 turkleton *facepalm*
20:39 JoeJulian both servers show the same uuid.
20:41 turkleton oh weird
20:41 turkleton oh goddammit
20:44 turkleton hmm
20:44 turkleton that's interesting
20:44 turkleton I changed it in /var/lib/glusterd/glusterd.info to match the peer UUID on the other node, and then GlusterFS died on the working node
20:44 turkleton changed it back, and now it's running on both
20:45 turkleton The UUIDs are the same on both in that file
20:45 turkleton is that a problem? (assuming it is since they aren't universally unique...)
20:49 turkleton Working node isn't working any more
20:49 turkleton http://termbin.com/ck8p
20:50 turkleton and broken node now: http://termbin.com/k2of
20:57 JoeJulian turkleton: correct, it's a problem for those to be the same on multiple servers.
20:58 turkleton Should I grab the correct UUID from the peers file?
20:58 JoeJulian yes
20:59 turkleton Is the only file I need to modify /var/lib/glusterd/glusterd.info ?
21:04 JoeJulian yes
21:04 turkleton Ah sonuvabitch
21:04 turkleton So, I noticed another issue
21:04 turkleton Route53 DNS entry was wrong
21:04 turkleton for one of the nodes
21:04 JoeJulian That would make it a little more difficult.
21:07 turkleton hahaha
21:07 turkleton yeah
21:07 turkleton just a little bit
21:07 turkleton I love doing all this automated recovery testing though. It's a great way to learn a lot more about the internals of Gluster.
21:10 turkleton The article I posted above about the peer rejected suggested deleting everything but the glusterd.info file under /var/lib/glusterd. Will I need to do anything special to get the old broken node to "see" the volume again?
21:10 turkleton Or will it automatically send that info over?
21:10 turkleton Nvm
21:10 turkleton question answered itself
21:10 turkleton it sent the data over
21:10 turkleton and volume is back to a working state
21:11 DV joined #gluster
21:14 turkleton and I broke it again wheeeeeeee
21:14 turkleton haha
21:15 turkleton and fixed
21:15 turkleton although... I don't know how it broke like this
21:16 turkleton I generate a UUID per side
21:16 josh__ do user serviceable snapshots consume more space than a regular snapshot?  if not, is there any benefit to doing regular snapshots and mounting using fuse to copy/backup the data instead of using the uss feature?
21:16 turkleton I think it's because the UUIDs within GFS got mixed up somehow (/var/lib/glusterd/glusterd.info)
21:16 chirino joined #gluster
21:19 diegows joined #gluster
21:21 mlhamburg1 joined #gluster
21:46 EinstCrazy joined #gluster
21:56 cyberbootje joined #gluster
22:04 josh__ another snapshot question: i have am running 3.7.6-1.el7 and am having problems creating snapshots.  i keep getting "Snapshot is supported only for thin provisioned LV. Ensure that all bricks of joshtest are thinly provisioned LV".  I'll post the brick data and result of lvs.
22:04 josh__ Brick joshtest.midwestlabs.com:/srv/ovirt/local
22:04 josh__ and /srv/ovirt/local is a mounted thinly provisioned lv
22:07 DV__ joined #gluster
22:17 josh__ i lied.  disregard my question altogether.  i must have forgot the -T when i created it.  sorry for the dumb question
22:17 cristov joined #gluster
22:24 JoeJulian No problem josh__. You'd be surprised how often that same monologue has happened in here.
22:34 dlambrig_ joined #gluster
22:49 chirino joined #gluster
23:07 shyam joined #gluster
23:14 Telsin joined #gluster
23:16 ira joined #gluster
23:30 coredump joined #gluster
23:34 Telsin joined #gluster
23:34 EinstCrazy joined #gluster
23:40 skylar joined #gluster
23:55 jrm16020 joined #gluster
23:59 diegows joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary