Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-01-07

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 eclectic_ joined #gluster
00:02 Cenbe_ joined #gluster
00:03 bolazzles joined #gluster
00:03 prasanth__ joined #gluster
00:04 daMaestro joined #gluster
00:04 sac`away joined #gluster
00:04 fidevo joined #gluster
00:04 yosafbridge joined #gluster
00:09 johnbot11 joined #gluster
00:20 recidive joined #gluster
00:28 mkzero joined #gluster
00:29 mattappe_ joined #gluster
00:33 plarsen joined #gluster
00:37 mattappe_ joined #gluster
00:47 jbrooks joined #gluster
00:49 mattappe_ joined #gluster
00:52 mattapp__ joined #gluster
00:56 pingitypong joined #gluster
01:00 mattappe_ joined #gluster
01:00 harish joined #gluster
01:04 yinyin joined #gluster
01:11 recidive joined #gluster
01:14 askb joined #gluster
01:16 mattapperson joined #gluster
01:18 mattapperson joined #gluster
01:18 mattappe_ joined #gluster
01:23 pingitypong joined #gluster
01:24 mattappe_ joined #gluster
01:38 askb joined #gluster
01:39 mattapperson joined #gluster
01:42 mattapperson joined #gluster
01:42 harish joined #gluster
01:43 mattapperson joined #gluster
01:46 mattappe_ joined #gluster
01:47 harish joined #gluster
01:49 GLHMarmot joined #gluster
01:52 mattappe_ joined #gluster
01:54 askb joined #gluster
02:05 hjmangalam joined #gluster
02:12 zwu joined #gluster
02:13 r0b joined #gluster
02:16 askb joined #gluster
02:29 harish joined #gluster
02:32 recidive joined #gluster
02:35 askb joined #gluster
02:48 zaitcev joined #gluster
02:53 askb joined #gluster
02:53 nueces joined #gluster
02:55 hjmangalam1 joined #gluster
02:55 askb joined #gluster
02:56 zwu joined #gluster
02:57 askb joined #gluster
02:58 zaitcev joined #gluster
03:06 kshlm joined #gluster
03:09 bharata-rao joined #gluster
03:22 shubhendu joined #gluster
03:30 vpshastry joined #gluster
03:30 vpshastry left #gluster
03:37 mattapperson joined #gluster
03:40 badone__ joined #gluster
03:42 mattappe_ joined #gluster
03:47 itisravi joined #gluster
03:53 RameshN joined #gluster
03:54 dylan_ joined #gluster
03:54 Shdwdrgn left #gluster
04:03 johnbot11 joined #gluster
04:06 bala joined #gluster
04:08 hjmangalam joined #gluster
04:24 mattappe_ joined #gluster
04:29 dalekurt joined #gluster
04:34 ngoswami joined #gluster
04:36 shyam joined #gluster
04:41 fidevo joined #gluster
04:42 dalekurt joined #gluster
04:44 rjoseph joined #gluster
04:44 rjoseph left #gluster
04:45 spandit joined #gluster
04:45 shubhendu joined #gluster
04:45 dalekurt joined #gluster
04:49 dalekurt joined #gluster
04:50 davinder joined #gluster
04:53 overclk joined #gluster
04:56 hagarth joined #gluster
04:57 saurabh joined #gluster
04:57 ndarshan joined #gluster
04:58 satheesh joined #gluster
04:59 mohankumar joined #gluster
04:59 MiteshShah joined #gluster
05:00 ababu joined #gluster
05:01 lalatenduM joined #gluster
05:03 dalekurt joined #gluster
05:03 dalekurt joined #gluster
05:03 dalekurt joined #gluster
05:04 dalekurt joined #gluster
05:04 dalekurt joined #gluster
05:04 dalekurt joined #gluster
05:06 johnbot11 joined #gluster
05:11 dalekurt joined #gluster
05:11 kdhananjay joined #gluster
05:12 ppai joined #gluster
05:13 vpshastry joined #gluster
05:16 dalekurt joined #gluster
05:17 prasanth joined #gluster
05:17 dalekurt hello
05:17 glusterbot dalekurt: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
05:18 dalekurt joined #gluster
05:18 dalekurt joined #gluster
05:18 dalekurt joined #gluster
05:21 CheRi joined #gluster
05:25 yinyin joined #gluster
05:25 askb joined #gluster
05:28 eclectic joined #gluster
05:28 d-fence_ joined #gluster
05:29 mohankumar joined #gluster
05:31 ndarshan joined #gluster
05:31 prasanth__ joined #gluster
05:32 juhaj_ joined #gluster
05:32 nixpanic_ joined #gluster
05:32 nixpanic_ joined #gluster
05:33 Amanda_ joined #gluster
05:34 tru_tru_ joined #gluster
05:35 JonathanS joined #gluster
05:36 aravindavk joined #gluster
05:36 bharata-rao joined #gluster
05:36 lalatenduM joined #gluster
05:36 shapemaker joined #gluster
05:36 harish joined #gluster
05:37 dusmant joined #gluster
05:38 _feller joined #gluster
05:40 prasanth joined #gluster
05:41 inodb_ joined #gluster
05:41 ajha joined #gluster
05:41 shylesh joined #gluster
05:42 klaxa joined #gluster
05:45 haakon_ joined #gluster
05:45 lyang0 joined #gluster
05:48 satheesh joined #gluster
05:49 nixpanic joined #gluster
05:49 eclectic_ joined #gluster
05:50 nixpanic joined #gluster
05:50 tru_tru joined #gluster
05:57 prasanth joined #gluster
05:57 juhaj_ joined #gluster
05:57 askb joined #gluster
05:57 fidevo joined #gluster
06:00 psharma joined #gluster
06:04 raghu joined #gluster
06:07 MiteshShah joined #gluster
06:09 DV joined #gluster
06:09 bala joined #gluster
06:13 rastar joined #gluster
06:18 Philambdo joined #gluster
06:24 glusterbot New news from newglusterbugs: [Bug 1033576] rm: cannot remove Directory not empty on path that should be clean already <https://bugzilla.redhat.co​m/show_bug.cgi?id=1033576>
06:27 tor joined #gluster
06:33 DV joined #gluster
06:44 vimal joined #gluster
06:50 nueces joined #gluster
07:03 satheesh joined #gluster
07:03 social joined #gluster
07:09 meghanam joined #gluster
07:18 Gugge joined #gluster
07:19 micu joined #gluster
07:21 jtux joined #gluster
07:23 anoopcs joined #gluster
07:24 anoopcs How to solve "peer probe: failed: Probe returned with unknown errno 107"?
07:24 ngoswami joined #gluster
07:42 glusterbot New news from resolvedglusterbugs: [Bug 968301] improvement in log message for self-heal failure on file/dir in fuse mount logs <https://bugzilla.redhat.com/show_bug.cgi?id=968301>
07:53 ctria joined #gluster
07:58 ekuric joined #gluster
08:02 eseyman joined #gluster
08:11 hybrid5121 joined #gluster
08:12 keytab joined #gluster
08:13 getup- joined #gluster
08:19 anoopcs joined #gluster
08:20 bala joined #gluster
08:25 anoopcs How to solve "peer probe: failed: Probe returned with unknown errno 107" error on fedora 19?
08:28 harish joined #gluster
08:34 anoopcs left #gluster
08:39 andreask joined #gluster
08:43 qdk joined #gluster
08:51 badone__ joined #gluster
08:55 hagarth joined #gluster
09:06 badone__ joined #gluster
09:08 harish joined #gluster
09:13 yinyin joined #gluster
09:23 shubhendu joined #gluster
09:24 blook joined #gluster
09:27 satheesh joined #gluster
09:29 tryggvil joined #gluster
09:29 tryggvil_ joined #gluster
09:34 hurl joined #gluster
09:37 atrius joined #gluster
09:43 recidive joined #gluster
09:44 mgebbe_ joined #gluster
09:45 badone joined #gluster
09:50 diegows joined #gluster
09:56 Philambdo joined #gluster
09:56 nshaikh joined #gluster
09:58 jclift joined #gluster
10:08 rastar joined #gluster
10:08 satheesh joined #gluster
10:09 vpshastry joined #gluster
10:32 rastar joined #gluster
10:33 bharata-rao left #gluster
10:36 NeatBasis joined #gluster
10:37 jclift left #gluster
10:41 jclift joined #gluster
10:51 psyl0n joined #gluster
10:52 shubhendu joined #gluster
10:56 rastar joined #gluster
10:56 recidive joined #gluster
11:02 kanagaraj joined #gluster
11:06 ngoswami joined #gluster
11:13 psyl0n joined #gluster
11:16 kshlm joined #gluster
11:18 shubhendu joined #gluster
11:19 spandit joined #gluster
11:30 benjamin_______ joined #gluster
11:32 kaushal_ joined #gluster
11:41 kkeithley1 joined #gluster
11:54 shyam joined #gluster
11:55 vpshastry joined #gluster
11:56 getup- joined #gluster
11:57 rastar joined #gluster
12:00 Staples84 joined #gluster
12:06 shubhendu joined #gluster
12:07 spandit joined #gluster
12:07 itisravi joined #gluster
12:08 KaZeR_ joined #gluster
12:09 KaZeR_ hi there. i am trying to replace a dead node. i followed http://europe.gluster.org/community/doc​umentation/index.php/Gluster_3.2:_Brick​_Restoration_-_Replace_Crashed_Server, my volume shows as started, but the datas aren't replicated to the replaced node. can anyone help?
12:19 KaZeR_ joined #gluster
12:22 uebera|| joined #gluster
12:29 satheesh joined #gluster
12:30 ababu joined #gluster
12:40 Peanut KaZeR_: did you look at the fattr? What version of gluster are you using?
12:45 KaZeR_ Peanut: i'm using 3.4.0
12:46 KaZeR_ i did not look at the fattr, i will have a look right now
12:47 KaZeR_ mmm
12:47 KaZeR_ on the new node i only have trusted.gfid=0x00000000000000000000000000000001
12:47 KaZeR_ the id matches the one on the other node
12:48 KaZeR_ but on the new node i do not have the other attributes like afr.cinder* or glusterfs.*
12:49 Peanut What I have in my docs for repairing:
12:49 Peanut Put peer UUID from the old serer in /var/lib/glusterd/gluster.info
12:50 Peanut getfattr -m . -d -e hex /export/brick0/sdb1, on the replacement: setfattr -n trusted.glusterfs.volume-id -v <hex value>
12:50 Peanut And ten restart glusterfs, and it should resync
12:52 Peanut http://joejulian.name/blog/repl​acing-a-brick-on-glusterfs-340/
12:52 glusterbot Title: Replacing a brick on GlusterFS 3.4.0 (at joejulian.name)
12:52 Peanut That's where I got that from
12:53 theron joined #gluster
12:53 * social thinks JoeJulian is god for making usefull docs
12:53 KaZeR_ thanks Peanut
12:54 KaZeR_ when restarting glusterd, i get this in the log :  W [posix.c:4205:init] 0-myvolume-directory: Volume is dangling. Please check the volume file.
12:54 jbrooks joined #gluster
12:55 hagarth joined #gluster
13:02 qdk joined #gluster
13:03 RicardoSSP joined #gluster
13:03 RicardoSSP joined #gluster
13:08 blook joined #gluster
13:10 kaushal_ joined #gluster
13:10 KaZeR_ force restarted the volume, then restarted the glusterd service, and now i have some files being replicated
13:15 Peanut That sounds like progress
13:15 clyons joined #gluster
13:17 dneary joined #gluster
13:17 dusmant joined #gluster
13:19 benjamin_______ joined #gluster
13:20 B21956 joined #gluster
13:20 dylan_ joined #gluster
13:21 KaZeR_ indeed
13:26 B21956 joined #gluster
13:41 tziOm joined #gluster
13:43 hurl joined #gluster
13:45 sroy_ joined #gluster
13:47 tjikkun_work joined #gluster
13:57 getup- joined #gluster
13:58 lalatenduM joined #gluster
13:58 vpshastry joined #gluster
13:59 tjikkun_work joined #gluster
14:00 ira joined #gluster
14:01 ira joined #gluster
14:03 jbrooks joined #gluster
14:06 edward2 joined #gluster
14:06 kanagaraj joined #gluster
14:11 robo joined #gluster
14:12 vpshastry left #gluster
14:13 andreask joined #gluster
14:14 ccha4 I have replica 2 with 2 servers, I want to remove 1 replica, but keep the volume
14:14 ccha4 how can I should do ?
14:14 ccha4 remove-brick ?
14:15 JoeJulian yep, remove-brick replica 1 serverfoo:/bar
14:15 ccha4 oh
14:15 ccha4 replica1 works ?
14:15 JoeJulian yep
14:15 ccha4 but you can create replica 1 right ?
14:15 ccha4 can't
14:16 bennyturns joined #gluster
14:17 JoeJulian You can. It's rather redundant to specify though.
14:18 davinder joined #gluster
14:24 ccha4 remove-bric to replica 1 , change the volume type replicate to distribute
14:24 ccha4 is it normal ?
14:24 JoeJulian That's what replica 1 will do.
14:25 JoeJulian When you change it to replica 1 and remove a brick, you'll have a 1 brick volume. When you add a brick without changing the replica count, you'll have a distributed volume.
14:26 plarsen joined #gluster
14:27 ccha4 cool
14:27 theron joined #gluster
14:28 dbruhn joined #gluster
14:29 dewey joined #gluster
14:32 vimal joined #gluster
14:37 CheRi joined #gluster
14:49 getup- joined #gluster
14:56 tryggvil joined #gluster
14:57 tryggvil_ joined #gluster
14:59 ccha4 hum I can't attach a server 3.4.2 to a cluster 3.3.2,... something wrong ?
14:59 ccha4 Peer 10.10.51.150 does not support required op-version
15:05 ira_ joined #gluster
15:11 kkeithley1 joined #gluster
15:13 ngoswami joined #gluster
15:14 dbruhn ccha4, is there any reason you are trying to use a newer version of the client than the server?
15:15 getup- joined #gluster
15:17 hagarth ccha4: you likely have some new option from 3.4 set in one of your volumes .. since the 3.3 server cannot understand the volume, the peer probe would be failing..
15:17 bigl0af joined #gluster
15:22 ccha4 I want to add the new server and remove 1 old server, and change it
15:26 glusterbot New news from newglusterbugs: [Bug 1049470] Gluster could do with a useful cli utility for updating host definitions <https://bugzilla.redhat.co​m/show_bug.cgi?id=1049470>
15:30 bugs_ joined #gluster
15:30 zerick joined #gluster
15:32 wushudoin joined #gluster
15:34 theron joined #gluster
15:34 psyl0n joined #gluster
15:38 getup- joined #gluster
15:38 vpshastry joined #gluster
15:42 vpshastry left #gluster
15:43 ccha4 but so I need to update 3.3 to 3.4 before to add the new server
15:45 ccha4 I have replica 2 with 3.3.2,, if I update one server to 3.4, the volume would still fine, right ?
15:47 ccha4 Why I can't add replica 3 with new 3.4 to the 3.3 ?
15:50 daMaestro joined #gluster
15:52 dewey joined #gluster
15:56 glusterbot New news from newglusterbugs: [Bug 1049481] Need better GlusterFS log message string when updating host definition <https://bugzilla.redhat.co​m/show_bug.cgi?id=1049481>
15:57 failshell joined #gluster
16:01 gunthaa joined #gluster
16:08 vpshastry joined #gluster
16:08 vpshastry left #gluster
16:11 hagarth joined #gluster
16:14 bala joined #gluster
16:19 johnbot11 joined #gluster
16:20 XATRIX joined #gluster
16:21 XATRIX Hi, anyone use glusterfs on centos ?
16:23 dylan_ joined #gluster
16:26 _feller joined #gluster
16:29 johnmilton joined #gluster
16:33 redbeard joined #gluster
16:35 kanagaraj joined #gluster
16:48 JoeJulian XATRIX: I would almost venture to say that most do.
16:50 aravindavk joined #gluster
16:51 semiosis johnmark: ping
16:51 johnbot1_ joined #gluster
16:53 LoudNoises joined #gluster
16:55 jclift left #gluster
16:57 japuzzo joined #gluster
16:57 kkeithhley_ looking at the apache logs on download.gluster.org, going back to December 15, there have been > 50,000 downloads of the glusterfs-3.X RPMs for el6.
16:59 kkeithley_ And just be be clear, that's just the glusterfs-3.x.x core or base RPM. I.e. I did not count the others, e.g. glusterfs-server, glusterfs-libs, glusterfs-cli, etc.  And that's all versions, 3.3.x, 3.4.x, etc.
17:00 semiosis !!!
17:00 johnbot11 joined #gluster
17:00 hagarth kkeithley_: 50k in 3 weeks? :O
17:01 kkeithley_ That doesn't count Fedora either.
17:01 tryggvil joined #gluster
17:01 tryggvil_ joined #gluster
17:01 kkeithley_ 52841 to be exact
17:02 jclift joined #gluster
17:02 kkeithley_ Doesn't count Debian or Ubuntu either
17:03 semiosis kkeithley_: johnmark has a script that pulls stats from launchpad (for ppa downloads).  i need to get a copy of that from him
17:04 XATRIX JoeJulian: Thanks :)
17:04 hagarth amazing. on this happy note, I will sign off today :).
17:04 kkeithley_ yes, that would be interesting. (Because only ~500 downloads of the glusterfs-common DPKG
17:04 hagarth later everybody
17:04 XATRIX But how can i setup my volume to be mounted automatically while the system starts ?
17:04 kkeithley_ good night
17:05 JoeJulian XATRIX: use the _netdev mount option in fstab
17:05 zaitcev joined #gluster
17:05 XATRIX But in Centos parses the fstab file before the network starts
17:05 XATRIX isn't it ?
17:05 JoeJulian XATRIX: use the _netdev mount option in fstab
17:05 johnbot11 joined #gluster
17:06 JoeJulian @google _netdev
17:06 glusterbot JoeJulian: mount - Linux Command - Unix Command: <http://linux.about.com/od/c​ommands/l/blcmdl8_mount.htm>; Mounting shared devices and using the /etc/fstab file: <http://pic.dhe.ibm.com/infocenter/lnx​info/v3r0m0/topic/liaag/ocfs2/l0wocf0​0_ocfs2_mounting_shared_devices.htm>; network shares - How to Set "<options>" in fstab - Manual Mounting ...:
17:06 glusterbot JoeJulian: <http://superuser.com/questions/226561/how-to-set​-options-in-fstab-manual-mounting-is-successfull>; Confusion about _netdev mount option. - Ars Technica OpenForum: <http://arstechnica.com/civis/vi​ewtopic.php?f=16&amp;t=1224111>; networking - How to mount a CIFS share so it doesnt warn about ...: (2 more messages)
17:06 JoeJulian gah, mean to do @lucky
17:07 vpshastry joined #gluster
17:07 vpshastry left #gluster
17:10 Technicool joined #gluster
17:11 johnmark semiosis: ah, I don't have a script that does that
17:11 johnmark at least, not one that can tabulate by day or month
17:11 johnmark semiosis: I figured out how to get *all* downloads but not broken out by time
17:13 dbruhn Is there going to be a community presence at SCALE x12?
17:13 johnmark dbruhn: yes
17:14 johnmark dbruhn: gah, I forgot to include you on an email - forwarding
17:14 dbruhn haha
17:14 dbruhn john mark, I just booked my trip for SCALE, so if I can help in anyway let me know.
17:14 johnbot11 joined #gluster
17:14 aixsyd joined #gluster
17:15 aixsyd https://bugzilla.redhat.com/show_bug.cgi?id=895830
17:15 glusterbot Bug 895830: unspecified, medium, ---, rhs-bugs, ASSIGNED , [QEMU/KVM-RHS] Creating qcow2 image on fuse mount with preallocation=metadata option fails.
17:15 aixsyd :'( :'( :'( :'( :'(
17:15 aixsyd almost a year later, and no one cares :'(
17:17 johnmark dbruhn: excellent! we will have something going on there, although I rpobably wont' be there
17:17 johnmark dbruhn: sent
17:17 dbruhn got it
17:20 JoeJulian aixsyd: Looks like the ioctl option isn't supported by fuse. That's a deficiency in the kernel.
17:21 aixsyd D:
17:21 zerick joined #gluster
17:21 aixsyd It works fine without that preallocation flag - but Proxmox's GUI has that hard coded
17:22 JoeJulian Did you also file a bug with Proxmox?
17:22 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
17:22 JoeJulian glusterbot: thank you
17:22 glusterbot JoeJulian: you're welcome
17:23 aixsyd JoeJulian: filing one with Proxmox now
17:23 aixsyd glusterbot: http://bugzilla.proxmox.com
17:23 JoeJulian aixsyd: That's not to say that it won't get fixed. I've seen the gluster devs have many patches accepted into the kernel fuse module.
17:24 aixsyd Theres a gent on Proxmox's forums thats looking to write a patch if he can reproduce the results
17:34 JoeJulian Just figured a two-pronged approach might be more efficient.
17:34 aixsyd oh, make a bug report on RH's, too?
17:34 XATRIX Do i have to apply some extra options to the fstab entry for glusterfs ?
17:34 XATRIX I have defaults,_netdev
17:34 XATRIX maybe some noatime,nodirtime, or whatever ?
17:34 Mo_ joined #gluster
17:34 rvignacio joined #gluster
17:34 rvignacio hello! does anybody know where this guide went to? http://community.gluster.org/q/a-replica-no​de-has-failed-completely-and-must-be-replac​ed-with-new-empty-hardware-how-do-i-add-the​-new-hardware-and-bricks-back-into-the-repl​ica-pair-and-begin-the-healing-process/
17:34 rvignacio I'm trying to replace a crashed brick with a new one with different hostname
17:34 semiosis rvignacio: iirc you need to probe the new server into the pool then do a replace-brick commit force
17:35 psyl0n joined #gluster
17:35 semiosis then a heal full would probably be a good idea
17:35 jobewan joined #gluster
17:36 semiosis @seen chirino
17:36 glusterbot semiosis: chirino was last seen in #gluster 18 weeks, 6 days, 23 hours, 30 minutes, and 20 seconds ago: <chirino> semiosis: pong
17:40 rvignacio thanks semiosis! I found some emails suggesting that process but I wasn't sure if it was secure... I need to be sure that gluster will recognize the new brick as empty and copy the contents from the others to it (instead of assuming the new brick as good and emptying the rest :P )
17:41 semiosis just make sure you're replacing the bad brick with the new empty brick
17:41 tomased joined #gluster
17:42 aixsyd JoeJulian: https://bugzilla.proxmox.com/show_bug.cgi?id=485
17:42 glusterbot Bug 485: medium, medium, ---, johnsonm, CLOSED CURRENTRELEASE, Integrate changes from UltraPenguin 1.1.9
17:42 rvignacio ok, I will try it, thanks!
17:44 fyxim_ joined #gluster
17:44 skered- joined #gluster
17:45 johnmwilliams_ joined #gluster
17:45 marcoceppi_ joined #gluster
17:51 delhage joined #gluster
17:51 verdurin joined #gluster
17:51 delhage joined #gluster
17:51 verdurin joined #gluster
17:53 vpshastry joined #gluster
17:53 vpshastry left #gluster
17:56 Peanut semiosis: how difficult/risky is it to downgrade my gluster back to your 3.4.0 packages?
17:56 JoeJulian 3.4.0???
17:56 JoeJulian Why?
17:57 * semiosis waves to JoeJulian
17:57 semiosis Peanut: no idea
17:57 JoeJulian I'd say it's risky. There's some really nice management bugs in 3.4.0.
17:58 * JoeJulian waves back
17:58 kuroneko4891 joined #gluster
17:58 aixsyd JoeJulian: so say I had a raid failure on cluster node #2. I rebuilt the raid, now i wanna rejoin the new RAID brick to an existing cluster. volume heal is unsuccessful. thoughts?
17:59 JoeJulian check log files and find out why
17:59 aixsyd that IS the command, though?
17:59 aixsyd not a reblance, or something
18:00 aixsyd stopping the volume works, starting it again fails
18:00 JoeJulian Assuming your raid brick was offline, as in the glusterfsd process for that brick was not running, and it's still the same server, and the replaced filesystem is mounted, and the brick was started (gluster volume start $volume force) and it succeeded... yes.
18:00 aixsyd sweeet.
18:01 aixsyd force option fails, too
18:01 jclift left #gluster
18:01 JoeJulian It probably fails because your new filesystem isn't tagged as a brick.
18:02 JoeJulian ... a process that SHOULD be handled through the cli, but I don't think anyone's filed a bug report on that yet.
18:03 Peanut JoeJulian: what happened is that I upgraded from 3.4.0 to 3.4.1 yesterday. Now, if I live-migrate a virtual host from one gluster machine to the other, the gluster storage for that host becomes unavailable ('permission denied') not only to libvirt, but to any application trying to read that file on that half of the gluster.
18:03 JoeJulian So... on a good brick: getfattr -m trusted.glusterfs.volume-id -d -e hex $good_brick_path
18:03 JoeJulian aixsyd: ^
18:04 aixsyd done
18:04 aixsyd trusted.glusterfs.volume-id=0xa​8735bce15ee47f09b32de637dfd6794
18:04 JoeJulian aixsyd: With that volume-id in hand (I'll call it $volume_id), on the new brick: setfattr -n trusted.glusterfs.volume-id -v $volume_id
18:04 XATRIX joined #gluster
18:04 aixsyd that command failed
18:04 aixsyd not failed
18:04 aixsyd wrong syntax?
18:05 JoeJulian aixsyd: With that volume-id in hand (I'll call it $volume_id), on the new brick: setfattr -n trusted.glusterfs.volume-id -v $volume_id $new_brick_path
18:05 aixsyd ah
18:05 JoeJulian Sorry, got sidetracked mid-command.
18:05 aixsyd :P
18:05 aixsyd k, done
18:05 JoeJulian Should start now.
18:05 aixsyd it did
18:05 aixsyd hot damn
18:05 johnbot11 joined #gluster
18:05 * JoeJulian should probably blog about that.
18:06 Peanut JoeJulian: you already havea  blog page about it, that's where I learned about getfattr/setfattr.
18:06 aixsyd my wife says i'm getfattr
18:06 aixsyd :(
18:06 JoeJulian lol
18:07 aixsyd so the heal command was successful, but i'm not seeing any io on either node
18:07 JoeJulian If my wife does that I just threaten to s/old wife/new girlfriend/g
18:07 aixsyd xD
18:08 delhage_ joined #gluster
18:08 JoeJulian Then she reminds me that I can't afford a girlfriend...
18:08 flrichar joined #gluster
18:10 verdurin joined #gluster
18:10 verdurin joined #gluster
18:10 JoeJulian Peanut: So after you attempt to live-migrate, any access to a file on the server you attempted to migrate to results in an EPERM through a fuse mount?
18:11 Peanut Yup, until I bring down the virtual host, then I can read the file fine again.
18:11 Peanut Even with dd from teh command line - just no access, permission denied.
18:12 Peanut And with 3.4.0, live migration worked a treat.
18:12 JoeJulian Weird. That sounds like a variation on but 1019053
18:12 JoeJulian bug 1019053
18:12 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1019053 medium, urgent, ---, libvirt-maint, CLOSED NEXTRELEASE, libvirt/Glusterfs ports conflict with qemu live migration
18:14 Peanut That looks quite simliar, at least the symptoms.
18:14 ndevos Peanut: I thought your dmesg pointed to access-denied by AppArmor?
18:14 JoeJulian Ahhh
18:15 JoeJulian See how you are...
18:15 Peanut ndevos: no, on further investigation, it still fails with app-armor switched to complain instead of enforce.
18:15 johnbot11 joined #gluster
18:16 ndevos Peanut: hmm, and reading the image with md5sum (or whatever you used) still fails too?
18:16 Peanut Yes, it fails with 'permission denied' when using md5sum or a simple dd.
18:16 johnbot11 joined #gluster
18:16 Peanut Which also points to this not being an app-armor problem.
18:17 ndevos unless app-armor is broken ;)
18:17 JoeJulian Or it's app-armor on the server.
18:18 JoeJulian What if you don't migrate the VM, but rather start it on the server that's giving you fits?
18:18 JoeJulian Does that work?
18:19 Peanut Yes, works perfectly. I've tried that two ways: live migrate, then destroy/start, or simply undefine it on the one machine, then import the XML on the other side.
18:19 Peanut So the second case is exactly what you suggest. Works reliably.
18:19 johnbot1_ joined #gluster
18:20 Peanut virsh destry; virsh undefine on the one machine, then virsh define; virsh start on the other. That's how I do my migrations now, as live-migration is broken.
18:20 JoeJulian mmkay...
18:20 Peanut Excerpts from glusterfs.log: W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 509: FSYNC() ERR => -1 (Permission denied)
18:20 Peanut I get that on every access to a broken vm image, even from the cmdline.
18:21 Peanut The other possibly relevant entry is W [defaults.c:1291:default_release] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.1/xla​tor/performance/open-behind.so(ob_wake_cbk+0x10c) [0x7fe671b3de1c] (-->/usr/lib/x86_64-linux-gnu/libglu​sterfs.so.0(call_stub_destroy+0xe6) [0x7fe677ae3e66] (-->/usr/lib/x86_64-linux-gnu/li​bglusterfs.so.0(fd_unref+0x143) [0x7fe677ae5143]))) 0-fuse: xlator does not implement release_cbk
18:22 JoeJulian Just to make sure. Just the vm image, or any files on that brick?
18:22 Peanut Only that particular VM, the other ones run fine, and other files are fine too.
18:23 Peanut virsh doesn't even notice the error, but the guest OS suddenly can't write journal entries to its FS anymore and becomes quite unhappy ;-)
18:23 JoeJulian Ah, ok.
18:24 Peanut So it's not the port allocation problem at first glance.
18:24 JoeJulian No
18:24 JoeJulian I'm guessing some sort of lock problem
18:25 Peanut The file stays unavailable as long as the qemu for that vm runs. Stopping it and then starting it releases the file.
18:26 Peanut Must be some kernel level lock though, probably somewhere in fuse?
18:26 Peanut This is on Ubuntu Raring with glusterfs-3.4.1 from the Ubuntu PPA.
18:31 xymox joined #gluster
18:31 JoeJulian I'm not seeing anything in the logs that changed from 3.4.0 to 3.4.1 that looks likely to cause that...
18:32 Peanut What about open-behind.so ?
18:33 JoeJulian Could you trigger that with debug level logs and paste them somewhere?
18:33 Peanut Sure.. how do I get debug level logs?
18:36 aixsyd "Server and Client lk-version numbers are not same"  - wat.
18:36 glusterbot aixsyd: This is normal behavior and can safely be ignored.
18:36 aixsyd oh. well then.
18:36 Peanut Nice bot..
18:36 JoeJulian gluster volume set diagnostics.client-log-level DEBUG
18:36 aixsyd [socket.c:514:__socket_rwv] 0-gv0-client-0: readv failed (No data available)  Wheres your messiah now, glusterbot ?
18:36 Peanut JoeJulian: ah, so I can do that without a reboot/gluster restart? I'll get on it right away.
18:36 JoeJulian When done, gluster volume reset diagnostics.client-log-level
18:38 JoeJulian Actually... I haven't tested that in a while. Last time I tried it didn't actually reset to the default. I had to set it to INFO then reset it again.
18:38 JoeJulian ... forgot to file that bug....
18:38 aixsyd this is so strange. self heal is just reading the good node like crazy, and aboslutely nothing is happening on the new/rebuilt node. waaaaaaat?
18:39 JoeJulian logs or it didn't happen. ;)
18:40 wushudoin joined #gluster
18:41 avati joined #gluster
18:42 aixsyd_ joined #gluster
18:43 Peanut Ok, which logs would you like to see?
18:44 JoeJulian I just want to see neutron logs that show the ipsec connection being established with my cisco.
18:44 JoeJulian You, however, might like to look at the glustershd.log and see if that tells you anything useful.
18:45 JoeJulian bbiab... Daughter needs to eat something...
18:45 johnbot11 joined #gluster
18:47 wgao__ joined #gluster
18:48 SFLimey_ joined #gluster
18:48 johnbot11 joined #gluster
18:49 Peanut Me too. Logfiles: http://loopy.jive.nl/~boven/gluster-logs/
18:49 glusterbot Title: Index of /~boven/gluster-logs (at loopy.jive.nl)
18:53 verdurin_ joined #gluster
18:55 xymox joined #gluster
18:57 psyl0n joined #gluster
18:59 johnbot11 joined #gluster
19:01 a2_ joined #gluster
19:02 JoeJulian Peanut: Can I see the brick logs at 2014-01-07 18:42:17 +- 2 seconds?
19:06 Peanut It's now in the same directory online
19:07 johnbot11 joined #gluster
19:08 johnbot1_ joined #gluster
19:09 aixsyd JoeJulian: heal says its done. node 1 has 118gb used, node two has 8.2gb used. dafuq?
19:10 jdarcy joined #gluster
19:13 aixsyd this makes zero sense.
19:14 JoeJulian aixsyd: heal full
19:14 aixsyd did that. it crawled for a while, reading on good node like a mother, with no io activity on bad node. then after 8.3gb, it stopped
19:15 aixsyd http://fpaste.org/66502/38912215/
19:15 glusterbot Title: #66502 Fedora Project Pastebin (at fpaste.org)
19:18 aixsyd interesting.
19:18 aixsyd the missing GB are in files that havent been used/accessed
19:19 zapotah joined #gluster
19:19 zapotah joined #gluster
19:20 wanye joined #gluster
19:20 dbruhn JoeJulian, have you by chance come across that issue where files and directories will show up twice in the file system since that last time we talked about it?
19:22 JoeJulian Yesterday
19:22 JoeJulian dbruhn: It was because a file existed on multiple distribute subvolumes with differing gfids (split-brain cause).
19:23 johnbot11 joined #gluster
19:24 johnb____ joined #gluster
19:25 dbruhn JoeJulian: I checked that and it's coming back matching on all of the bricks.
19:27 JoeJulian Peanut: Let's try it again with diagnostics.server-log-level DEBUG
19:28 JoeJulian Peanut: The error's clearly coming from the server, but I can't see why.
19:28 JoeJulian dbruhn: fpaste your volume info and the xattrs from that file on each brick.
19:28 Peanut JoeJulian: Ok, coming up
19:29 JoeJulian aixsyd: I suspect you'll see another batch of activity 10 minutes after the first batch started.
19:30 tryggvil joined #gluster
19:30 tryggvil_ joined #gluster
19:30 theron joined #gluster
19:31 dbruhn I have about 1000 directories effected, I am assuming you want just a sample, or do you want the containing directory?
19:31 JoeJulian Just a sample.
19:31 JoeJulian possibly the containing directory also...
19:32 dbruhn kk
19:32 dbruhn sec
19:32 JoeJulian Yours are directories, not files, right?
19:32 dbruhn it seems to be everything under it, I think this all started with a rebalance fix layout that went bad and took the system down
19:32 jdarcy joined #gluster
19:33 Peanut "Did you mean diagnostics.brick-log-level?"
19:33 JoeJulian d'oh
19:33 Peanut That's a yes, then? :-)
19:33 JoeJulian yes
19:34 dbruhn Yeah, I know for sure I have about 3k files I have in split brain that I am working on as well
19:35 Peanut Done - which logfile would you like? Or all of them again?
19:37 JoeJulian dbruhn: I found an easy way to work on split-brain. I'm trying to write a tool for it, but I have to interface with libglusterfs to retrieve the volume definition, which is a lot of re-implementation of header files in python. Maybe I should use cython...
19:37 dbruhn I have built a couple scripts that go through each of my bricks and dump the meta info back out to me, then I choose one and it goes through and cleans it up
19:38 dbruhn they are pretty hack at the moment, but make it go way faster
19:40 johnbot11 joined #gluster
19:40 Peanut JoeJulian: http://loopy.jive.nl/~boven/gluster-server-logs/
19:40 glusterbot Title: Index of /~boven/gluster-server-logs (at loopy.jive.nl)
19:40 zapotah joined #gluster
19:40 zapotah joined #gluster
19:40 JoeJulian Oops, need the brick log again.
19:41 Peanut Sure, just a minute
19:43 Peanut Brick  logfile Is in place too.
19:47 JoeJulian Peanut: Only slightly less helpful. :/ Can you to it with brick log level at TRACE ? I was hoping to see where the EPERM originated.
19:47 JoeJulian All I need to see is the brick log.
19:48 Peanut Sure, anything you ask right now :-) Will be a few minutes
19:50 __feller joined #gluster
19:52 dbruhn JoeJulian: Volume info, and the output of the fs from the mount point showing the duplicates.
19:52 dbruhn http://fpaste.org/66505/38912387/
19:52 glusterbot Title: #66505 Fedora Project Pastebin (at fpaste.org)
19:52 dbruhn the attributes from all of the bricks
19:52 dbruhn http://fpaste.org/66509/12420613/
19:52 glusterbot Title: #66509 Fedora Project Pastebin (at fpaste.org)
19:54 dbruhn odd thing, I copied the files out of the directory, deleted the data, and copied them back in, and now those directories don't have trusted.afr.ENTV04EP-client-xx attributes
19:54 dbruhn those attributes existed before while it was exhibiting the same behavior though
19:55 JoeJulian normal. Those attributes are created if a directory is operated on while a replica brick is down. Otherwise they're not there.
19:55 dbruhn ahh ok
19:56 Peanut JoeJulian: is uploaded, same location as last one
19:58 dbruhn so if you know a directory is not split brain anymore can you delete those attributes? or is it better to set them to 0x00
20:03 JoeJulian dbruhn: I would delete them on directories just to see if the problem goes away. Watch the split-brain log to see if that broke something, but I've never had a problem just removing them.
20:03 JoeJulian Peanut: uname -a please
20:03 Peanut Linux cl0 3.8.0-35-generic #50-Ubuntu SMP Tue Dec 3 01:24:59 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
20:04 JoeJulian nothing interesting there...
20:04 dbruhn which log is the split brain log specifically?
20:05 JoeJulian Do you change limits.conf at all, Peanut?
20:05 Peanut Nope, I don't think they even have that anymore.
20:05 JoeJulian dbruhn: all client logs, and glustershd logs. If you aggregate your logs, that's easy, otherwise check heal info split-brain
20:09 JoeJulian Peanut: You've got me stumped... If I were debugging this myself, I'd strace the glusterfsd (brick) process and look for the open to see the results.
20:09 Peanut Ouch, that bad, eh?
20:09 Peanut So this happened when I installed patches on Monday. I could try to revert to 3.4.0, or revert the kernel (which also got a slight upgrade).
20:09 Peanut So nobody else is running into this? I was rather pleased so far with how well gluster was behaving! :-)
20:09 JoeJulian The message clearly makes it to the server, but I don't see it going through to the posix translator. Seems to only get as far as the io threads.
20:11 JoeJulian But there was only one change to that file between 3.4.0 and 3.4.1, and that change simply moves the log message before a function call.
20:13 Peanut And what aboutn the open-behind.so? Didn't that get introduced in 3.4.1?
20:14 JoeJulian All that's telling you is that after the call fails, the cleanup process doesn't exist for that function.
20:15 JoeJulian If you can, try the strace... strace -f -v -s 256 -e trace=file -p $(pidof glusterfsd)
20:16 Peanut To what extent is this issue unique to libvirt? It's really odd that the file becomes accessible again once I close libvirt.
20:17 Peanut I'll give that a shot after dinner
20:20 JoeJulian Peanut: I'm sure it's some specific set of circumstances that are unique to libvirt. It's got to be a bug, though. I'm hoping we can isolate it so it can get fixed.
20:20 Peanut I'm willing to put effort into that, it's not-quite-production yet so I can muck around with it.
20:21 JoeJulian Or maybe it's already fixed in 3.4.2 and figuring out what the problem is will better allow us to decide if it's fixed, or just a fluke.
20:23 dbruhn JoeJulian, removing the trusted.afr attributes seem to have resolved it
20:24 dbruhn odd that it was duplicating them even though they matched
20:25 gmcwhistler joined #gluster
20:26 JoeJulian Yeah, that is odd.
20:26 gmcwhistler joined #gluster
20:28 kaptk2 joined #gluster
20:29 andreask joined #gluster
20:29 JoeJulian dbruhn: file a bug
20:29 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
20:30 theron joined #gluster
20:31 dbruhn on it
20:35 dbruhn hmm
20:35 dbruhn I guess I'm wrong about that, now instead of showing up duplicate on a couple of the clients it's showing up duplicate on all of the clients
20:36 JoeJulian lol
20:36 dbruhn I was using wc -l on the end of my ls -l :/
20:37 dbruhn uniformity is not fixed I guess
20:37 dbruhn lol
20:37 JoeJulian dbruhn: What about checking .glusterfs/2c/fd/2cfd*46d2 on all the bricks. See if there's some mismatch there.
20:37 JoeJulian They /should/ all be symlinks.
20:38 dbruhn kk
20:39 JoeJulian dbruhn: Any particular reason you for i in $(seq 1 12) instead of 1 24 ?
20:39 dbruhn 12 servers /var/brick* grabs both bricks
20:40 JoeJulian Oh, right.
20:40 JoeJulian I did notice that...
20:40 dbruhn one of my older systems before your suggestion to do /var/brick/aa etc
20:43 dbruhn http://fpaste.org/66522/91274131/
20:43 glusterbot Title: #66522 Fedora Project Pastebin (at fpaste.org)
20:45 JoeJulian well that's thoroughly boring... ;)
20:46 dbruhn lol
20:47 harold[mtv] joined #gluster
20:47 dbruhn I honestly am a little lost as to what I am looking at here
20:47 JoeJulian The symlink for a directory gfid points to it's parent directory.
20:48 JoeJulian its
20:48 dbruhn so follow the chain till I find something wrong?
20:48 dbruhn like maybe this? http://fpaste.org/66527/89127711/
20:48 glusterbot Title: #66527 Fedora Project Pastebin (at fpaste.org)
20:50 JoeJulian Oh! It's a directory instead of a symlink! I bet that's it.
20:50 JoeJulian You should just be able to "rm -rf /var/brick13/.glusterfs/e5/b0/e5b0​1d3f-9606-4589-99c3-02df978e6ab8"
20:51 dbruhn do I want to recreate the symlink?
20:52 JoeJulian It'll recreate itself.
20:52 dbruhn kk
20:52 JoeJulian or it should...
20:53 _zerick_ joined #gluster
20:55 tryggvil joined #gluster
20:57 tryggvil_ joined #gluster
20:59 zapotah joined #gluster
20:59 zapotah joined #gluster
21:00 theron joined #gluster
21:01 theron joined #gluster
21:05 dbruhn I ended up having to make it by hand for it to start showing the directories properly
21:06 JoeJulian good to know
21:08 dbruhn This system is so weird, I have 3 systems and this is the one I have always have issues with. And before these system I had two other ones
21:08 JoeJulian So if you did a "find $brick/.glusterfs/*/*/* -type d" that should show you any others that need fixed.
21:09 dbruhn sweet, that will help a lot
21:13 ira joined #gluster
21:20 dalekurt joined #gluster
21:21 rcaskey joined #gluster
21:21 _Bryan_ joined #gluster
21:22 _zerick_ joined #gluster
21:24 zerick joined #gluster
21:26 dbruhn Joe, have you seen a directory have split brain because of a mismatched gfid, or just a file?
21:30 JoeJulian Yes, directory
21:32 JoeJulian Take, for instance, a replica 2, 2 brick volume. One brick is offline. mkdir foo. bring the other brick up and take down the first brick. Do this before self-heal has a chance to do anything. directory foo will not exist. create it again. bring the first brick back up.
21:32 johnbot11 joined #gluster
21:32 JoeJulian Also works if there's a netsplit.
21:33 JoeJulian I'm not sure how that could happen with twelve servers though.
21:33 JoeJulian Other than a self-heal bug.
21:35 dbruhn I am just trying to recreate the scenario and instead of seeing duplicated, I am getting an I/O error which is more of what I expect
21:35 JoeJulian The bug is that a gfid file should NEVER be a directory.
21:36 JoeJulian If it is, an error should be logged and that gfid file should be ignored.
21:48 ^rcaskey joined #gluster
21:56 tyl0r joined #gluster
21:56 dalekurt joined #gluster
21:58 tyl0r I'm having trouble setting the translate-uid option through the CLI. Does anyone know the correct syntax for v3.2.4 ?
21:58 tyl0r http://www.gluster.org/community/docume​ntation/index.php/Translators/features
21:58 glusterbot Title: Translators/features - GlusterDocumentation (at www.gluster.org)
22:12 ctria joined #gluster
22:16 rcaskey joined #gluster
22:29 dylan_ joined #gluster
23:05 dbruhn joined #gluster
23:08 ira joined #gluster
23:09 dalekurt joined #gluster
23:38 pingitypong joined #gluster
23:54 dalekurt joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary