Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-14

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 kris joined #gluster
00:05 primechuck joined #gluster
00:05 REdOG so close .... other than these loud ass Invalid argument logs I cannot seem to find anything wrong
00:07 jag3773 joined #gluster
00:08 ktosiek_ joined #gluster
00:09 pdrakewe_ joined #gluster
00:12 _dist RedOG what kind of noise is in your brick logs?
00:12 dbruhn joined #gluster
00:12 REdOG [2014-02-14 00:08:15.491744] I [server-rpc-fops.c:1572:server_open_cbk] 0-tartC-server: 244924: OPEN /Tarttrazine.img (de1ad484-bf0b-4c8f-96b2-7c7e4c624f8d) ==> (Invalid argument)
00:12 REdOG [2014-02-14 00:08:15.492336] E [posix.c:1850:posix_open] 0-tartC-posix: open
00:13 REdOG about 500 a second
00:15 REdOG they only happen on the zfs backed host
00:15 REdOG the other one is humming along quite nicely
00:17 REdOG 882286 Invalid argument errors in a little over an hour
00:18 kris joined #gluster
00:18 * REdOG doesn't think that will make it to production
00:21 _dist REdOg: I'm running ZFS and I just searched my brick log I don't seem to have any of those
00:21 REdOG zfs linux or solaris?
00:22 _dist linux
00:22 REdOG hmm
00:22 REdOG 0.6.2 here
00:22 _dist same, from debian ZoL repo
00:23 REdOG im going to dig for a few more hours
00:23 _dist I also don't have a single entry mention posix.c or "Invalid" when I cat/grep it.
00:23 _dist so there's hope yet
00:23 REdOG then ill write the mailing list or something
00:24 _dist I'm raidz3 --> zvol --> xfs --> gluster. My zpool is 4k ashift 12, my volblocksize is 64k, my xfs is 512
00:24 _dist but I gotta head out, thursday night is steak night! I'll be around again tomorrow though
00:25 REdOG gn
00:25 _dist JoeJulian: Thanks for helping me with that test, makes me feel a lot better. I'll let you know when I get to the bottom of it
00:27 rfortier joined #gluster
00:35 diegows joined #gluster
00:46 psyl0n joined #gluster
00:49 vpshastry joined #gluster
01:02 jflilley1 ç
01:06 jflilley1 set
01:06 jflilley1 exit
01:07 jflilley joined #gluster
01:09 jflilley joined #gluster
01:11 jflilley quit
01:11 jflilley joined #gluster
01:11 KyleG left #gluster
01:17 theron joined #gluster
01:18 mohankumar__ joined #gluster
01:25 mohankumar__ joined #gluster
01:30 mohankumar__ joined #gluster
01:57 mohankumar__ joined #gluster
01:57 recidive joined #gluster
01:58 k4nar_ left #gluster
02:08 mohankumar__ joined #gluster
02:17 davinder joined #gluster
02:18 bennyturns joined #gluster
02:19 diegows joined #gluster
02:20 pdrakeweb joined #gluster
02:26 mohankumar__ joined #gluster
02:37 jporterfield joined #gluster
02:44 badone__ joined #gluster
02:57 mohankumar__ joined #gluster
03:22 mohankumar__ joined #gluster
03:36 bharata-rao joined #gluster
03:38 harish joined #gluster
03:41 bennyturns joined #gluster
03:41 xeed0 joined #gluster
03:45 mohankumar__ joined #gluster
03:46 shubhendu_ joined #gluster
03:49 masterzen joined #gluster
03:56 mohankumar__ joined #gluster
04:00 RameshN joined #gluster
04:07 jporterfield joined #gluster
04:09 mohankumar__ joined #gluster
04:14 sahina joined #gluster
04:17 mohankumar__ joined #gluster
04:22 shyam joined #gluster
04:33 hchiramm_ joined #gluster
04:38 kanagaraj joined #gluster
04:39 RameshN joined #gluster
04:43 ndarshan joined #gluster
04:51 johnmark joined #gluster
04:52 jag3773 joined #gluster
04:54 kdhananjay joined #gluster
04:57 spandit joined #gluster
05:00 tokik joined #gluster
05:09 itisravi joined #gluster
05:10 bala joined #gluster
05:15 sputnik13net joined #gluster
05:19 nightwalk joined #gluster
05:22 prasanth joined #gluster
05:24 aravindavk joined #gluster
05:27 davinder joined #gluster
05:28 rastar joined #gluster
05:28 nshaikh joined #gluster
05:30 mohankumar__ joined #gluster
05:34 shylesh joined #gluster
05:44 rjoseph joined #gluster
05:49 lalatenduM joined #gluster
05:50 hagarth joined #gluster
05:52 ajha joined #gluster
05:57 surabhi joined #gluster
05:58 raghu joined #gluster
06:00 bala joined #gluster
06:10 itisravi joined #gluster
06:13 ktosiek joined #gluster
06:17 dusmant joined #gluster
06:22 pixelgremlins_ba joined #gluster
06:25 meghanam joined #gluster
06:25 saurabh joined #gluster
06:44 CheRi joined #gluster
06:50 vimal joined #gluster
07:09 Nev___ joined #gluster
07:11 kdhananjay joined #gluster
07:15 jtux joined #gluster
07:16 shubhendu joined #gluster
07:30 pdrakeweb joined #gluster
07:33 mohankumar__ joined #gluster
07:39 rossi_ joined #gluster
07:41 ekuric joined #gluster
07:57 eseyman joined #gluster
08:01 ctria joined #gluster
08:02 kdhananjay joined #gluster
08:06 mohankumar joined #gluster
08:12 TvL2386 joined #gluster
08:15 TvL2386 hi guys,
08:15 TomKa joined #gluster
08:26 spiekey joined #gluster
08:28 ngoswami joined #gluster
08:32 keytab joined #gluster
08:32 hchiramm_ joined #gluster
08:33 mohankumar joined #gluster
08:42 Elico joined #gluster
08:43 TvL2386 hi guys, unfortunately my replicated volume uses ip addresses for peers and now I need to change the ip address. I will put it in /etc/hosts with an alias from now on, but how can I migrate my replicated volume to new ip addresses?
08:43 TvL2386 documentation online is very thin
08:50 andreask joined #gluster
08:54 simon__ joined #gluster
08:55 dusmant joined #gluster
08:56 TomKa joined #gluster
09:00 glusterbot New news from resolvedglusterbugs: [Bug 1000019] Bogus dates in RPM changelog <https://bugzilla.redhat.com/show_bug.cgi?id=1000019>
09:04 hybrid512 joined #gluster
09:07 ccha2 hello since which version the glusterfs-client option backupvolfile-server work ?
09:09 al_ joined #gluster
09:12 liquidat joined #gluster
09:16 al0 joined #gluster
09:16 ProT-0-TypE joined #gluster
09:18 kanagaraj joined #gluster
09:18 al joined #gluster
09:19 tryggvil joined #gluster
09:20 al joined #gluster
09:26 davinder2 joined #gluster
09:31 pdrakeweb joined #gluster
09:31 ppai joined #gluster
09:31 d-fence joined #gluster
09:33 shubhendu joined #gluster
09:35 qdk_ joined #gluster
09:41 ngoswami joined #gluster
09:43 mohankumar joined #gluster
09:50 RedShift joined #gluster
09:59 ndarshan joined #gluster
10:04 nshaikh joined #gluster
10:05 kanagaraj joined #gluster
10:05 davinder joined #gluster
10:21 cxx joined #gluster
10:25 cxx Hi there, does anyone have a clue why I am getting "/gluster/vol0 or a prefix of it is already part of a volume" after trying to add a freshly installed brick? The IP was not in that cluster, the hostname was not used before, the path is a freshly created subdirectory on a freshly formatted xfs-partition. I don't get it.
10:25 glusterbot cxx: To clear that error, follow the instructions at http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/ or see this bug https://bugzilla.redhat.com/show_bug.cgi?id=877522
10:26 cxx Thanks glusterbot, I've tried that around a thousand times :)
10:29 diegows joined #gluster
10:37 ninkotech joined #gluster
10:37 ninkotech_ joined #gluster
10:41 dusmant joined #gluster
10:42 getup- joined #gluster
10:43 shylesh joined #gluster
10:51 hchiramm_ joined #gluster
10:51 hchiramm_ joined #gluster
10:52 zapotah joined #gluster
10:52 irimi joined #gluster
10:53 TBlaar joined #gluster
10:54 vsa joined #gluster
10:54 irimi Hey I am using Gluster 3.2 on about 100+ servers and now I wanted to test another distribution on a new node that ships with Gluster 3.4; appearently this one doesn't connect without any error message. Can it be, that Gluster 3.4 is not compatible to 3.2 ?
10:55 vsa Hi all . Gluster 3.4.2. How to delete geo-replica ? I can't stop it, and don't know how to delete it. Can anybody help?
10:57 pk1 joined #gluster
11:06 jmarley joined #gluster
11:07 tziOm joined #gluster
11:08 vsa joined #gluster
11:09 vsa Hi all . Gluster 3.4.2. How to delete geo-replica ? I can't stop it, and don't know how to delete it. Can anybody help?
11:15 jporterfield joined #gluster
11:15 ndarshan joined #gluster
11:16 kanagaraj joined #gluster
11:17 dusmant joined #gluster
11:19 recidive joined #gluster
11:22 ngoswami joined #gluster
11:24 Slash_ joined #gluster
11:32 pdrakeweb joined #gluster
11:36 RameshN joined #gluster
11:36 georgeh|workstat joined #gluster
11:39 al joined #gluster
11:44 al joined #gluster
11:45 burn420 joined #gluster
11:48 DV joined #gluster
11:51 pk1 left #gluster
11:51 ira joined #gluster
11:52 lalatenduM @fileabug
11:52 glusterbot lalatenduM: Please file a bug at http://goo.gl/UUuCq
11:56 ccha2 irimi: yes it is not compatible with 3.3 and 3.4
12:10 jporterfield joined #gluster
12:14 rfortier1 joined #gluster
12:25 klaxa joined #gluster
12:28 harish joined #gluster
12:29 surabhi joined #gluster
12:30 vsa Hi all . Gluster 3.4.2. How to delete geo-replica ? I can't stop it, and don't know how to delete it. Can anybody help?
12:37 Philambdo joined #gluster
12:52 pdrakeweb joined #gluster
12:56 zerick joined #gluster
13:02 mbukatov joined #gluster
13:04 Philambdo joined #gluster
13:05 marbu joined #gluster
13:09 vsa Hi all . Gluster 3.4.2. How to delete geo-replica ? I can't stop it, and don't know how to delete it. Can anybody help?
13:11 tdasilva left #gluster
13:14 RameshN joined #gluster
13:16 edward1 joined #gluster
13:16 davinder joined #gluster
13:22 vsa Hi all . Gluster 3.4.2. How to delete geo-replica ? I can't stop it, and don't know how to delete it. Can anybody help?
13:33 sroy joined #gluster
13:39 mbukatov joined #gluster
13:42 Philambdo joined #gluster
13:47 ccha2 what you mean by "delete it" ?
13:47 ccha2 when you stop geo-replica there is no anymore
13:49 cxx joined #gluster
13:51 vsa when i run #gluster volume geo-rep... stop  - i see error gluster command filed.
13:52 ccha2 what about gluster volume geo-replication status
13:53 ccha2 if there is none, there is no geo anymore
13:54 vsa Ohh.. i fixed it by added rsa_id to slave host, and then rerun command (I have some trables with it). Thank's!)
14:00 cxx joined #gluster
14:05 keytab joined #gluster
14:06 prasanth joined #gluster
14:08 failshell joined #gluster
14:08 rwheeler joined #gluster
14:11 japuzzo joined #gluster
14:11 failshel_ joined #gluster
14:12 ihre joined #gluster
14:15 plarsen joined #gluster
14:23 tryggvil joined #gluster
14:30 bennyturns joined #gluster
14:31 B21956 joined #gluster
14:38 theron joined #gluster
14:40 hagarth joined #gluster
14:41 theron joined #gluster
14:42 an_ joined #gluster
14:43 theron joined #gluster
14:43 zapotah joined #gluster
14:43 zapotah joined #gluster
14:43 calum_ joined #gluster
14:48 jmarley joined #gluster
14:48 jmarley joined #gluster
15:04 tdasilva joined #gluster
15:10 cxx left #gluster
15:11 psyl0n joined #gluster
15:14 bugs_ joined #gluster
15:14 sprachgenerator joined #gluster
15:14 theron joined #gluster
15:21 nshaikh joined #gluster
15:30 ProT-O-TypE joined #gluster
15:31 shyam joined #gluster
15:33 aixsyd joined #gluster
15:33 aixsyd Hey gents - anyone alive?
15:33 aixsyd Trying to replace a crashed server and the official docs are very vague on a few steps
15:33 aixsyd http://gluster.org/community/documentation/index.php/Gluster_3.4:_Brick_Restoration_-_Replace_Crashed_Server
15:34 glusterbot Title: Gluster 3.4: Brick Restoration - Replace Crashed Server - GlusterDocumentation (at gluster.org)
15:36 aixsyd JoeJulian: Im reading your blog on the subject, but its just as vague on this critical aspect
15:37 aixsyd "The work-around is to add the volume-id to the new brick" - then you show some code. WHERE does that code go?
15:39 dbruhn joined #gluster
15:42 psyl0n joined #gluster
15:47 plarsen joined #gluster
15:50 theron joined #gluster
15:55 primechuck aixsyd: The cod eat the bottom of the page can just be upt into a terminal.  On the server you're bringing back online.  It just recreates the xattrs for gluster to use the new blank bricks as the old bricks by pulling the information from the volume file.
15:55 primechuck It does require you to add the sever to the cluster again, and do the volume sync first to get the required ID information.
16:01 aixsyd what in the heck is going on - i'm getting a max of 5MB/s read and write out of my cluster O_O
16:01 dbruhn :/
16:02 aixsyd this is pitiful. what is going on?
16:02 jag3773 joined #gluster
16:03 jikz joined #gluster
16:05 aixsyd iperf shows 450mbits to my gluster server.
16:05 plarsen joined #gluster
16:06 REdOG I still cannot locate the source of these invalid argument posix errors
16:06 plarsen joined #gluster
16:07 vpshastry joined #gluster
16:14 kkeithley REdOG: invalid argument errors? Do you have quota enabled?
16:16 zapotah joined #gluster
16:16 zapotah joined #gluster
16:18 daMaestro joined #gluster
16:23 tdasilva joined #gluster
16:24 primechuck aixsyd:  What translators are you using?  How many Bricks do you have?
16:24 aixsyd i fixed it. cabling issue
16:25 aixsyd Proxmox was going through 3 switches to get to Glusterfs. was supposed to be on the same swith
16:25 aixsyd *switch
16:25 aixsyd moving it fixed it
16:34 hchiramm__ joined #gluster
16:34 dberry joined #gluster
16:35 dusmant joined #gluster
16:38 zapotah joined #gluster
16:41 B21956 joined #gluster
16:48 shylesh joined #gluster
16:52 davinder joined #gluster
16:56 semiosis REdOG: posix xlator runs on the brick.  have you checked brick logs?  feel free to pastie.org them
17:00 pdrakeweb joined #gluster
17:08 quique joined #gluster
17:12 REdOG kkeithley: I have quota enabled on the zvols
17:12 REdOG semiosis: the brick logs are where the errors are
17:12 REdOG they are HUGE
17:12 REdOG same errors over and over
17:12 REdOG [2014-02-14 17:12:48.452823] E [posix.c:1850:posix_open] 0-v7-posix: open on /awz0/v7_65G/brick6/v7.img: Invalid argument
17:12 REdOG [2014-02-14 17:12:48.452874] I [server-rpc-fops.c:1572:server_open_cbk] 0-v7-server: 248967: OPEN /v7.img (a2f467c4-2418-4dfb-8658-69fe50d77cd7) ==> (Invalid argument)
17:14 REdOG I have 4 vm's running on the system
17:14 REdOG performing quite well
17:14 ira joined #gluster
17:17 pdrakeweb joined #gluster
17:18 KyleG joined #gluster
17:18 KyleG joined #gluster
17:30 theron joined #gluster
17:30 theron joined #gluster
17:37 Mo_ joined #gluster
17:47 pdrakeweb joined #gluster
17:47 edward1 joined #gluster
18:05 psyl0n joined #gluster
18:10 mohankumar joined #gluster
18:16 pixelgremlins joined #gluster
18:18 Slash_ joined #gluster
18:38 sputnik13 joined #gluster
18:57 NeatBasis joined #gluster
19:04 hchiramm__ joined #gluster
19:08 edward1 joined #gluster
19:14 _dist joined #gluster
19:15 _dist JoeJulian: that issue I had yesterday, totally not gluster. It "appeared" to be, but it was actually a corrupt disk. On a plus note, fresh install of windows server 2012 took less than 10 min
19:15 _dist So sorry for wasting your time chasing it, the log entry looked worse than it was. It looks like libgfapi just complains the whole time a node is down even though it found a new path already
19:17 psyl0n joined #gluster
19:18 qdk joined #gluster
19:21 Matthaeus joined #gluster
19:33 sputnik13net joined #gluster
19:37 rwheeler joined #gluster
19:40 diegows joined #gluster
19:44 StarBeast joined #gluster
19:45 kris joined #gluster
20:00 gmcwhistler joined #gluster
20:32 zapotah joined #gluster
20:32 zapotah joined #gluster
20:37 nikk really silly question.. in the glusterd.vol config there's a management volume but nothing else - i've never touched that file.. do i need to?  i've done all my volume management through the cli.
20:37 nikk i see messages pertaining to that every now and again
20:37 nikk no idea what it is/does
20:44 _dist nikk: sometimes, only for special options. The main one I know people touch it for is insecure setting to allow libgfapi to connect (aparently as non-root)
20:45 nikk hmm
20:49 _dist usually volume set is all you need
20:50 nikk yeah
20:50 nikk just got back to a problem i was having the other day.. i have a single replicated volume (two nodes) - i try to add two more nodes to it and i just get
20:50 nikk [2014-02-14 15:46:36.261799] E [glusterd-op-sm.c:3719:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Add brick', Status : -1
20:50 sputnik13 joined #gluster
20:50 nikk on the node i'm trying to add
20:51 nikk i've tried everything at this point, not a clue what it could be
20:51 nikk /sigh
20:52 nikk i can't put this into production if i can't add nodes to a replicated volume :(
20:54 _dist gluster volume vol add-brick replica +1 newserver:/newbrick ? and it fails?
20:54 _dist (x+1 I should say, where x is current bricks)
20:54 JoeJulian Wait, what?
20:55 JoeJulian nikk: What's a "node" mean to you?
20:55 nikk sorry, four servers (nodes), one volume, replica value = 2
20:55 nikk i should be able to keep that set to 2 as long as the number of nodes is even (which is required for replica i think)
20:55 JoeJulian And you want to add two more bricks
20:55 JoeJulian @glossary
20:55 glusterbot JoeJulian: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
20:56 nikk gluster> volume add-brick gv0 replica 2 rhel3:/gluster/gv0 rhel4:/gluster/gv0
20:56 nikk volume add-brick: failed:
20:56 nikk correct
20:56 JoeJulian The error you posted comes from which log?
20:57 nikk the first thing i pasted comes from etc-glusterfs-glusterd.vol.log on rhel3 (first new brick)
20:58 JoeJulian @pastestatus
20:58 glusterbot JoeJulian: Please paste the output of gluster peer status from more than one server to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:00 nikk http://ur1.ca/gmmqd
21:00 glusterbot Title: #77400 Fedora Project Pastebin (at ur1.ca)
21:01 nikk so originally i created a volume with two bricks (rhel1 and rhel2) and had this problem.
21:01 nikk next i deleted that volume completey and made a new one with four bricks (all four servers0
21:01 nikk that worked fine.. as a test i removed two now i can't add them back (same problem as the first time through)
21:02 nikk so i know this setup works with four bricks however i can't add two more once the volume is created
21:02 JoeJulian Thanks, can I now see the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log from both the server you  executed the add-brick operation on and the two servers being added?
21:02 nikk during the commands i pasted just now?
21:03 JoeJulian Yes
21:03 ^^rcaskey I've got a 1-node gluster and when I added a second node with replica=2 ... my storage speed increased. How is this possible?
21:04 JoeJulian insufficient data to make a conclusion.
21:05 nikk http://ur1.ca/gmmr6
21:05 glusterbot Title: #77406 Fedora Project Pastebin (at ur1.ca)
21:05 JoeJulian That's IT? no other entries at all? I would have expected at least some other info entries... :(
21:06 nikk that's it
21:06 nikk lemme put it into debug mode actually
21:08 nikk actually nothing more with debug on
21:08 nikk if it helps, this is rhel7-beta, i haven't tried a 6.5 host yet to see if it's o/s specific
21:08 nikk was thinking about trying that
21:09 nikk but i can't imagine that it would matter that much due to the nature of gluster itself
21:11 ^^rcaskey It flummoxes me how you can have a slow drive and add a faster drive into the cluster and get better perfomrance when replicas=2
21:11 ^^rcaskey (and nodes=2 as well)
21:14 dbruhn ^^rcaskey, on read or on write?
21:14 ^^rcaskey write
21:14 dbruhn that's interesting, I have an answer for read, lol
21:14 JoeJulian nikk: Is this on the community release, or RHS?
21:14 ^^rcaskey I have an answer for read oto :P
21:15 nikk glusterfs-server.x86_64           3.4.2-1.el7                    @glusterfs-epel
21:15 JoeJulian I'm mostly sure fuse is informed of a completed write when the first replica finishes the write.
21:17 nikk i have glusterfs-epel and glusterfs-noarch-epel enabled, the dev repo is disabled
21:17 JoeJulian nikk: I'm walking the code path trying to find one that has no error messages...
21:17 JoeJulian I can't find one.
21:17 JoeJulian nikk: selinux?
21:18 nikk off
21:18 nikk iptables also off
21:18 JoeJulian figured, but I hate trying to figure stuff out only to have that be the end result... :D
21:18 nikk hehe of course
21:19 JoeJulian on rhel{3,4}, stop glusterd and just run  "glusterd --debug" and try again. There's got to be some other clue...
21:19 nikk k se c
21:22 nikk http://ur1.ca/gmmtu
21:23 glusterbot Title: #77412 Fedora Project Pastebin (at ur1.ca)
21:23 nikk i believe #587 is where i ran the add-brick command
21:23 nikk this is on rhel3
21:23 nikk do you care about rhel1?
21:23 nikk that's not running manually
21:24 JoeJulian I don't think I care about rhel1 yet...
21:24 nikk k
21:30 JoeJulian Ho! I may have found something...
21:30 nikk orly
21:31 jag3773 joined #gluster
21:31 JoeJulian meh, nevermind...
21:31 jag3773 left #gluster
21:32 JoeJulian I thoughtI found a code path that always results in a failure, but I missed an assignment.
21:32 jag3773 joined #gluster
21:32 nikk baw
21:32 nikk yeah i've never ran into anything that didn't return a real error other than this
21:41 JoeJulian So what I'm seeing is that glusterd_op_stage_add_brick succeeded as far as glusterd_new_brick_validate...
21:41 plwab joined #gluster
21:42 nikk is there any other info i can provide?
21:43 spiekey joined #gluster
21:43 plwab are the following steps still the correct way to replace a dead server with a new one in 3.4.2? 1) transfer bricks to new server 2) mount them 3) wipe trusted.glusterfs.volume-id and trusted.gfid from each brick 4) rm -rf .glusterfs on each brick 5) gluster peer attach $newserver 6) gluster volume $volname replace-brick $deadserver:/brickN $newserver:/brickN commit force.
21:47 plwab http://gluster.org/community/documentation/index.php/Gluster_3.4:_Brick_Restoration_-_Replace_Crashed_Server assumes the new server has the hostname (which it does not in my case), and the link to the article that supposedly talks about this use case is a 404
21:47 glusterbot Title: Gluster 3.4: Brick Restoration - Replace Crashed Server - GlusterDocumentation (at gluster.org)
21:47 plwab ...and the admin guide doesn't exist any more
21:48 JoeJulian sure it does... but it wasn't in there...
21:49 JoeJulian plwab: essentially, copy /var/lib/glusterd from a working server to the new one. Replace the uuid in glusterd.info with the uuid for this server as shown in "gluster peer status" from one of the working servers...
21:50 JoeJulian plwab: remove the peer file for the new server from the peers/ directory.
21:50 JoeJulian You should not need to do your step 3 or 4.
21:50 plwab JoeJulian: oh, right.
21:50 JoeJulian nor should you need to replace brick.
21:53 tryggvil joined #gluster
21:53 JoeJulian nikk: find /var/lib/glusterd on each server please and paste
21:55 nikk http://ur1.ca/gmmzf
21:55 glusterbot Title: #77428 Fedora Project Pastebin (at ur1.ca)
21:59 plwab JoeJulian: the hostname's changed though - what about all those $hostname:[...] files under the bricks folder?
21:59 plwab and the hostname in the files under peers/, for that matter
22:00 JoeJulian ooh, that's what I was looking for!
22:01 JoeJulian grep -r hostname /var/lib/glusterd/peers and see if any of them are wrong.
22:01 JoeJulian Also, make sure they can each ping all four servers by hostname.
22:02 JoeJulian I think I'm seeing that rhel3 doesn't recognize itself...
22:02 nikk k sec
22:03 nikk they all contain three entries and everything matches up between them
22:03 plwab JoeJulian: (side question: removing the .glusterfs folder shouldn't have any ill effects, right? It will just get regenerated?)
22:04 nikk all can ping
22:04 JoeJulian plwab: heal ... full and it should, yes.
22:04 JoeJulian Though any hardlinks will be broken.
22:05 JoeJulian ... or may be broken... interesting question... the gfid's would still match so it may repair them... Not sure on that.
22:05 plwab I'm just trying to figure out why I had that in my notes
22:10 jobewan joined #gluster
22:12 JoeJulian plwab: You would do that if you were trying to add the brick to the volume. Since we're essentially trying to cheat and tell the volume, "Look! I'm back! No, really, it's me!" then there's no need.
22:13 theron joined #gluster
22:14 JoeJulian nikk: Please file a bug report. Include a step-by-step I can follow to repro the problem. I'll spin up a few rhel7 vms (assuming I can get my hands on rhel7) and try to duplicate it.
22:14 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
22:15 JoeJulian I'll also try in centos6 since I can do that in minutes.
22:16 nikk i can provide the iso if you can't find it
22:16 nikk how much log-wise do you want from me
22:16 nikk or just the steps to reproduce
22:17 tdasilva left #gluster
22:17 psyl0n joined #gluster
22:19 JoeJulian That debug log you pasted should be sufficient.
22:20 recidive joined #gluster
22:20 JoeJulian For me: ,,(undocumented options)
22:20 glusterbot me: Undocumented options for 3.4: http://www.gluster.org/community/documentation/index.php/Documenting_the_undocumented
22:20 glusterbot Undocumented options for 3.4: http://www.gluster.org/community/documentation/index.php/Documenting_the_undocumented
22:34 Matthaeus1 joined #gluster
22:49 glusterbot New news from newglusterbugs: [Bug 1065551] Unable to add bricks to replicated volume <https://bugzilla.redhat.com/show_bug.cgi?id=1065551>
22:50 nikk JoeJulian: https://bugzilla.redhat.com/show_bug.cgi?id=1065551
22:50 glusterbot Bug 1065551: medium, unspecified, ---, kparthas, NEW , Unable to add bricks to replicated volume
22:50 nikk do you need any more information aside from that?
22:50 primechuck joined #gluster
22:52 JoeJulian nikk: df /gluster/vol1
22:52 JoeJulian Just to satisfy a curiosity
22:53 plwab 6pm on a friday, check. bunch of sticky bit files showing up on my volume all of a sudden, check.
22:54 JoeJulian plwab: Assuming it was the brick you just re-added, shut it back down.
22:54 plwab JoeJulian: unfortunately looks unrelated
22:55 JoeJulian plwab: You can pretty safely (unless you use this for your own uses) find $brickdir -perm 1000 -size 0 -exec /bin/rm {} \;
22:56 JoeJulian If they're needed they'll be recreated.
22:57 plwab I just had a closer look at the first sticky file I could find. It's on 4 bricks instead of 2. Two of them are 0 bytes, the other two are the expected size. All four are 1000, though.
22:57 plwab is that what you would expect?
22:57 JoeJulian plwab: replica 2?
22:57 plwab JoeJulian: yup
22:58 JoeJulian that's what I would expect under normal circumstances.
22:58 JoeJulian The sticky shouldn't show up in a client mount though.
22:58 plwab I'll just try on that one file and see
22:59 nikk JoeJulian: k added
23:00 JoeJulian nikk: Ah, crap! They're on root and you didn't "force".
23:00 JoeJulian Should still throw an error, but I bet that's the problem.
23:00 plwab JoeJulian: eh, still sticky
23:00 nikk i did force
23:00 nikk :)
23:01 JoeJulian plwab: Sorry, I missed the "all four are 1000" bit... Doing too many things at once I guess.
23:02 JoeJulian plwab: The non-zero files should not have been made sticky.
23:02 JoeJulian I wonder if that has to do with healing from the replaced server.
23:04 plwab2 joined #gluster
23:04 plwab2 JoeJulian: my other webchat tab froze up for some reason. In case this didn't get through: the file was never on the server that died in the first place
23:06 JoeJulian plwab2: strange. If I were a betting man, I would wager that the replaced brick is replicated with the one that had the 0 size sticky file (dht link file).
23:07 kris joined #gluster
23:07 plwab2 JoeJulian: nope
23:07 plwab2 the two zero byte stickies were on bricks that mirror each other
23:08 plwab2 shoot, gotta run for now. Thanks for the help so far. I'll have a second look over the weekend.
23:08 JoeJulian o/
23:09 plwab2 \o
23:16 theron_ joined #gluster
23:17 kris joined #gluster
23:32 recidive joined #gluster
23:36 kris joined #gluster
23:37 kris joined #gluster
23:48 sputnik13 joined #gluster
23:51 sputnik13 joined #gluster
23:55 psyl0n joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary