Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-05-14

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:14 primusinterpares joined #gluster
00:14 Ark joined #gluster
00:24 coredump joined #gluster
00:27 jag3773 joined #gluster
00:33 lyang0 joined #gluster
00:34 primechuck joined #gluster
00:38 yinyin_ joined #gluster
00:45 diegows joined #gluster
00:57 hagarth joined #gluster
01:02 bala joined #gluster
01:34 mjsmith2 joined #gluster
01:45 PLATOSCAVE joined #gluster
02:08 gdubreui joined #gluster
02:12 bharata-rao joined #gluster
02:16 mjsmith2 joined #gluster
02:20 hexaclock joined #gluster
02:25 vpshastry joined #gluster
02:42 edong23 http://fpaste.org/101557/00035357/
02:42 glusterbot Title: #101557 Fedora Project Pastebin (at fpaste.org)
02:43 edong23 hm.... rdma anyone?
02:44 vpshastry joined #gluster
02:46 ceiphas_ joined #gluster
02:48 hexaclock hi all, i have a few questions about general glusterfs setup - would appreciate any input
02:49 hexaclock i understand that gluster has two significant performance limitations - in the area of thousands of small files, and in the area of large files (would need to setup striping for files bigger than what a brick can hold as i understand)
02:50 hexaclock for the first case, i imagine 'ls' would be quite slow, but was wondering if there is some configuration option/workaround to speed that up?
02:51 badone__ joined #gluster
02:51 hexaclock and for the second, suppose striping is not enabled. would gluster just block writes to that file (and other files stored on the same brick)?
02:53 _ndevos joined #gluster
02:53 _ndevos joined #gluster
02:54 hexaclock for the second case, i've read about re-balancing, but does it work based on filesize or # of files that need to be balanced?
02:55 sjm joined #gluster
03:13 kumar joined #gluster
03:26 shubhendu joined #gluster
03:40 RameshN joined #gluster
03:41 kanagaraj joined #gluster
03:52 cogsu joined #gluster
03:56 itisravi joined #gluster
04:15 DV joined #gluster
04:22 Ark joined #gluster
04:25 bala1 joined #gluster
04:29 kdhananjay joined #gluster
04:29 aviksil joined #gluster
04:34 ndarshan joined #gluster
04:37 tryggvil joined #gluster
04:39 ktosiek joined #gluster
04:41 dusmant joined #gluster
04:42 shilpa_ joined #gluster
04:48 sahina joined #gluster
04:52 ramteid joined #gluster
04:53 Ark joined #gluster
04:58 bala2 joined #gluster
05:00 ppai joined #gluster
05:01 jag3773 joined #gluster
05:06 DV joined #gluster
05:08 nshaikh joined #gluster
05:12 davinder joined #gluster
05:14 sputnik1_ joined #gluster
05:19 davinder joined #gluster
05:22 aviksil joined #gluster
05:22 prasanthp joined #gluster
05:23 DV joined #gluster
05:28 sputnik1_ joined #gluster
05:28 davinder joined #gluster
05:30 MugginsM joined #gluster
05:33 mjrosenb morning, all!
05:40 mjrosenb so, it looks like I did someting awkward, and now the permissions on files is wronc and chmod does't affect it.
05:41 mjrosenb how do permissions on linkto stubs work?
05:44 nishanth joined #gluster
05:52 dusmant joined #gluster
05:56 DV joined #gluster
05:58 Bardack joined #gluster
06:04 vpshastry joined #gluster
06:05 ktosiek joined #gluster
06:06 kanagaraj joined #gluster
06:09 vimal joined #gluster
06:13 vpshastry1 joined #gluster
06:15 an joined #gluster
06:18 rahulcs joined #gluster
06:27 api984 joined #gluster
06:27 rahulcs joined #gluster
06:29 sputnik1_ joined #gluster
06:30 psharma joined #gluster
06:31 dusmant joined #gluster
06:33 aviksil joined #gluster
06:39 meghanam joined #gluster
06:39 meghanam_ joined #gluster
06:42 rahulcs joined #gluster
06:43 aravindavk joined #gluster
06:52 glusterbot New news from resolvedglusterbugs: [Bug 1057846] Data loss in replicate self-heal <https://bugzilla.redhat.com/show_bug.cgi?id=1057846>
06:52 shilpa_ joined #gluster
06:53 nishanth joined #gluster
06:55 haomaiwa_ joined #gluster
07:00 mjrosenb ... that is the type of bug title that strikes fear into my heart.
07:05 ctria joined #gluster
07:05 eseyman joined #gluster
07:09 haomaiwa_ joined #gluster
07:20 hagarth joined #gluster
07:23 fsimonce joined #gluster
07:29 sputnik1_ joined #gluster
07:42 nishanth joined #gluster
07:46 davinder2 joined #gluster
07:53 RameshN joined #gluster
07:54 vpshastry1 joined #gluster
07:55 edward2 joined #gluster
07:56 ngoswami joined #gluster
07:59 prasanthp joined #gluster
08:00 dusmant joined #gluster
08:08 liquidat joined #gluster
08:23 Ark joined #gluster
08:27 Norky joined #gluster
08:35 ProT-0-TypE joined #gluster
08:39 GabrieleV joined #gluster
08:39 an joined #gluster
08:40 raghu joined #gluster
08:41 Slashman joined #gluster
08:41 TvL2386 joined #gluster
08:46 saravanakumar joined #gluster
08:50 maduser joined #gluster
08:57 rahulcs joined #gluster
09:01 dcherednik joined #gluster
09:14 bala1 joined #gluster
09:36 aravinda_ joined #gluster
09:36 rahulcs joined #gluster
09:37 jmarley joined #gluster
09:37 jmarley joined #gluster
09:44 prasanthp joined #gluster
09:46 rahulcs joined #gluster
09:55 rahulcs joined #gluster
10:00 rahulcs joined #gluster
10:01 rahulcs joined #gluster
10:12 vpshastry1 joined #gluster
10:19 rahulcs joined #gluster
10:21 kanagaraj joined #gluster
10:24 rahulcs joined #gluster
10:29 rahulcs joined #gluster
10:35 rahulcs joined #gluster
10:47 rahulcs joined #gluster
10:50 rjoseph joined #gluster
11:01 glusterbot New news from newglusterbugs: [Bug 1091677] Issues reported by Cppcheck static analysis tool <https://bugzilla.redhat.com/show_bug.cgi?id=1091677>
11:02 rahulcs joined #gluster
11:05 diegows joined #gluster
11:14 andreask joined #gluster
11:18 rahulcs joined #gluster
11:20 rahulcs joined #gluster
11:29 olisch joined #gluster
11:29 ceiphas i experiment using glusterfs as a root filesystem and have got it working so that the diskless machine starts up, but i have some errors remaining: first i get inpuit/output errors when i try to ls some directories (like /etc), second the machine crashes on halt as the glusterfs process gets killed and then there is no reboot or halt binary accessible
11:31 edward2 joined #gluster
11:33 DV__ joined #gluster
11:46 an joined #gluster
11:47 ceiphas but sometimes i get the first error on /sbin and that means the system refuses to boot
11:49 rahulcs joined #gluster
11:53 rahulcs joined #gluster
11:53 askb joined #gluster
11:55 ceiphas hello?
11:55 glusterbot ceiphas: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
11:55 rahulcs joined #gluster
11:56 ceiphas rahulcs: please check your connection... you are leavin and joining every minute
11:58 rahulcs joined #gluster
12:00 ndarshan joined #gluster
12:03 ndarshan joined #gluster
12:03 Ark joined #gluster
12:11 calum_ joined #gluster
12:11 itisravi joined #gluster
12:13 social ceiphas: huh you are probably first person I heared about running on gluster as /root, you use fuse mount?
12:14 ceiphas social: yes i do
12:15 ceiphas social: i think it is a good idea especially for backing my virtual machines up and for thin clients, too
12:15 social ok and what's the state, it would be nice to see whether you get io errors because of some split-brain or something else, could you provide client and brick logs?
12:15 ceiphas social: the problem is, i can't give you client logs, as the mount happens in the initramfs, and the logs stay there
12:16 ceiphas the server logs show no split brain.
12:16 ceiphas i can access all files, but the ls is borked
12:16 social hmm you might be able to turn on syslog for client logs?
12:16 ceiphas social: for an existing mount? how?
12:18 ceiphas social: i tried to re-mount the rootfs to get the logs into the rootfs, but it didn't work "according to mtab GlusterFS is already mounted on /"
12:18 social on server do gluster volume set help | less
12:18 social ceiphas: look for syslog ,)
12:19 social ceiphas: I have to warn you about acls/xattrs and adding bricks, I havent check for while my old patches so I don't know the real state but last time I remember gluster could screw up acls/xattrs badly with add-brick operations
12:20 ceiphas social: i haven't added any bricks after starting the volume
12:20 ceiphas social: and syslog shows nothing when i do "ls -al /sbin"
12:21 social you get no errors?
12:21 social did you enable the syslog feature?
12:28 ceiphas what syslog feature?
12:28 vpshastry joined #gluster
12:30 cyberj left #gluster
12:30 B21956 joined #gluster
12:31 fsimonce joined #gluster
12:37 dusmant joined #gluster
12:38 ceiphas social: i enabled syslog logging on the brick, but how do i enable it for the client?
12:39 japuzzo joined #gluster
12:41 sjm joined #gluster
12:43 chirino joined #gluster
12:44 sroy_ joined #gluster
12:45 sroy joined #gluster
12:50 bennyturns joined #gluster
12:51 mjsmith2 joined #gluster
12:54 sjm left #gluster
12:55 sjm joined #gluster
12:55 shilpa_ joined #gluster
12:59 ndevos ceiphas: those directories for which you get an error, do they have more than ca. 20 files/entries?
12:59 ndevos ceiphas: and, are you mixing 32-bit and  64-bit architectures?
13:00 ceiphas ndevos: i compiled it with your patch
13:00 ceiphas ndevos: some dirs with many entries work, others not
13:01 ceiphas ndevos: client is 64bit, bricks are 32 and 64
13:01 ndevos ceiphas: oh wow, you have a mix of architectures for the bricks?
13:01 chirino joined #gluster
13:01 * ndevos never tried that
13:01 ceiphas ndevos: why not?
13:02 ceiphas ndevos: your patch works for 32bit clients, but in this specific situation (32/64bit bricks, 64bit client) i still get input/output errors
13:04 ceiphas ndevos: if i mount with "--enable-ino32" the folder /sbin has this error, without i get the error on /etc. th other is working fine in both constellation
13:05 ndevos ceiphas: the patch works for my 32-bit servers and 64-bit clients too, but there may be some other issues related to 32/64-bit bricks, I think someone mentioned that somewhere...
13:06 theron joined #gluster
13:07 cvdyoung Hi, in our environment, I have 2 gluster servers, each with 2 bricks for a volume called homegfs.  Each brick is part of a volume group with a single logical volume of about 60TB in size.  Would it be better to have multiple logical volumes instead of the larger 60TB?  Thank you!
13:08 tryggvil joined #gluster
13:08 ceiphas cvdyoung: best is not more than one brick per hard drive
13:08 edward2 joined #gluster
13:09 jobewan joined #gluster
13:09 jmarley joined #gluster
13:09 jmarley joined #gluster
13:10 plarsen joined #gluster
13:11 ceiphas ndevos: do you think it would work better if i remove the 32bit brick?
13:12 ndevos ceiphas: that is possible, I think the advise is to have only one architecture per volume
13:14 ceiphas ndevos: i removed the 32bit brick (now i just have replica 1
13:15 ceiphas now i don't get the i/o errors
13:15 ndevos hmm, interesting
13:15 ceiphas you should repair the gluster protocol to support different frame sizes, as this is really annoying (and a bit embarrasing)
13:16 primechuck joined #gluster
13:16 ndevos it's not the protocol itself, I think, but how the list of entries is received and merged on the client side
13:16 ceiphas ndevos: like a handshake, client sends range of supportet framesizes to server, server sens ack with a matching frame size and all conversation now uses this frame size
13:17 ndevos could you file a bug about it? and maybe you could check if the problem happens when you mount a 23/64-bit volume over nfs?
13:17 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
13:18 ndevos if it happens on nfs too, its one of the lower xlators, if it does not happen on nfs, it'll likely be a problem in the fuse-interface
13:20 rahulcs joined #gluster
13:23 ceiphas ndevos: cannot check it with nfs as my bricks must use kernel nfs
13:25 ndevos ceiphas: okay, maybe something libgfapi based?
13:27 rahulcs joined #gluster
13:35 edong23 joined #gluster
13:35 stickyboy joined #gluster
13:37 ceiphas ndarshan: if you explain what that means, maybe
13:37 ceiphas ndevos: ndarshan: sorry, wrong nick
13:38 ceiphas will /ignore block the join/part messages for a user? rahulcs is getting on my nerves
13:39 ndevos ceiphas: libgfapi is a library that can be used for accessing a volume without mounting it, you can write a c program for it, or use the python/ruby/java bindings
13:40 ceiphas ndevos: no time to write software for it, i already use too much time to explore the possibilities of gluster regarding virtualization
13:50 ndevos ceiphas: sure, no problem, anyone working on the bug will test with nfs/libgfapi to narrow down the place that needs a correction
13:52 vincent_vdk ceiphas: are you using it to store vm images?
13:52 zaitcev joined #gluster
13:52 coreping joined #gluster
13:53 vincent_vdk i'm interested in that myself
13:53 ceiphas vincent_vdk: no, i wand to store the vms
13:53 ceiphas i use the volume as the vm's root fs
13:55 gmcwhistler joined #gluster
13:56 vincent_vdk i see
13:56 vincent_vdk i would like to use gluster to store VM disk images
13:57 vincent_vdk so i'm searching around for as much info to get this perfomant and stable
13:58 rahulcs joined #gluster
13:58 davinder joined #gluster
13:59 mjsmith2 joined #gluster
14:05 ceiphas vincent_vdk: storing images seems rather easy for me
14:05 ceiphas vincent_vdk: create volume, create image in volume, use image in volume
14:05 sahina joined #gluster
14:06 kaptk2 joined #gluster
14:07 ceiphas hah, got it working
14:09 nishanth joined #gluster
14:13 vincent_vdk ceiphas: getting the setup right is a bit trickier imho
14:14 dusmant joined #gluster
14:16 shubhendu joined #gluster
14:17 wushudoin joined #gluster
14:18 kanagaraj joined #gluster
14:20 RameshN joined #gluster
14:22 micu joined #gluster
14:22 dcherednik joined #gluster
14:23 rahulcs joined #gluster
14:24 kkeithley Gluster Community Meeting in ~30 minutes in #gluster-meeting
14:24 api984 kkeithley: wow a meeting….
14:24 thornton joined #gluster
14:26 kmai007 joined #gluster
14:29 hchiramm_ joined #gluster
14:34 ceiphas how do i check the file attributes again? getfattr and which param?
14:35 thornton Try ‘attr -l <path>’
14:38 sprachgenerator joined #gluster
14:40 kmai007 ceiphas: getfattr -h -d -m trusted.gfid -e hex  <brick_path>
14:41 ceiphas thx
14:49 rahulcs joined #gluster
14:50 scuttle_ joined #gluster
14:52 vpshastry joined #gluster
14:55 ceiphas is there a syntax to allow exactly two clients to access a volume?
15:00 JustinClift
15:00 JustinClift *** Gluster Community Meeting time ***
15:00 JustinClift (in #gluster-meeting)
15:02 kmai007 ceiphas: please explain
15:02 kmai007 the clients just need to mount the volume
15:02 shelby742 joined #gluster
15:02 ceiphas kmai007: yes
15:03 shelby742 Hello
15:03 glusterbot shelby742: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
15:03 ceiphas kmai007: and i want only these clients to be able to access the volume
15:03 shelby742 Is there paid support option from Gluster for the enterprise?
15:03 kmai007 shelby742: redhat storage
15:03 an joined #gluster
15:04 kmai007 is the supported product
15:04 kmai007 ceiphas: only root can mount it up to the clients, so are there other people that have root that can mount it up?
15:04 shelby742 Thank you
15:04 kmai007 you can restrict it by NFS-clients
15:05 kmai007 there is a host allow feature
15:05 kkeithley http://www.redhat.com/products/storage-server/
15:05 glusterbot Title: Red Hat | storage-server (at www.redhat.com)
15:05 vpshastry left #gluster
15:05 kmai007 gluster volume set help
15:05 kmai007 will give you a description of what feature you want
15:05 ceiphas kmai007: i know, but i can just add a ip with wild cards
15:06 ceiphas and i want to add two ips without wild card, abt i don't know what the syntax for that is
15:06 ceiphas because it isn't in the manual that i already read
15:18 vpshastry joined #gluster
15:20 zerick joined #gluster
15:20 Intensity joined #gluster
15:21 sputnik1_ joined #gluster
15:23 Ark joined #gluster
15:25 systemonkey2 joined #gluster
15:26 John_HPC joined #gluster
15:26 Rydekull joined #gluster
15:28 hchiramm_ joined #gluster
15:29 daMaestro joined #gluster
15:29 LoudNoises joined #gluster
15:30 sputnik1_ joined #gluster
15:31 John_HPC I am trying to use the splitmount utility; but its not mounting any drives. Does it require a specific glusterfs-package I may not have installed?
15:32 [o__o] joined #gluster
15:33 sputnik1_ joined #gluster
15:39 aravinda_ joined #gluster
15:52 an joined #gluster
15:53 jbd1 joined #gluster
16:03 kmai007 John_HPC: i think it requires python of some sorts
16:04 John_HPC kmai007: I have python installed, everything seems to install/run correctly. I get no errors. It says its mounted under  /tmp/sbfix/r0 /tmp/sbfix/r1 but nothing is there
16:04 chirino joined #gluster
16:06 kmai007 interesting, i've not used splitmount yet
16:07 kmai007 message JoeJulian
16:16 sks joined #gluster
16:21 kmai007 John_HPC: it works for me
16:21 kmai007 i have my mounts
16:22 kmai007 ./splitmount.py omhq1826 zoobar /tmp/./splitmount.py omhq1826 zoobar /tmp/zoobar
16:22 kmai007 sorry
16:22 kmai007 ./splitmount.py omhq1826 zoobar /tmp/zoobar
16:22 kmai007 and it gives me /tmp/zoobar/r1  /tmp/zoobar/r2  /tmp/zoobar/r3 /tmp/zoobar/r4
16:23 kmai007 https://github.com/joejulian/glusterfs-splitbrain
16:23 glusterbot Title: joejulian/glusterfs-splitbrain · GitHub (at github.com)
16:24 John_HPC I think I'm missing something, just trying to figure out what
16:25 John_HPC what version of python are you using?
16:26 kmai007 python-2.6.6-52.el6.x86_64
16:26 John_HPC my version is older. I tried 3.4 that was to new
16:27 John_HPC going to recompile 2.7's latest and trythat
16:29 John_HPC ll
16:29 John_HPC doh wrong window
16:36 John_HPC /root/.local/bin/python /root/.local/bin/splitmount gluster05 glustervol01 /tmp/sbfix
16:36 John_HPC Your split replicas are mounted under /tmp/sbfix, in directories r1 through r0
16:36 John_HPC and nothing
16:38 John_HPC glusterfsd and glusterd are running
16:40 ramteid joined #gluster
16:43 John_HPC maybe I don't have any split-brain files?
16:43 John_HPC the files its reporting at split-brain seem to match with the checksum
16:44 GabrieleV joined #gluster
16:44 Mo___ joined #gluster
16:46 _dist joined #gluster
16:47 sputnik1_ joined #gluster
16:50 qdk joined #gluster
16:51 kmai007 u don't need split-files to have the mount work
16:51 kmai007 do you have a /tmp filesystem?
16:52 kmai007 how many replicas does your volume gluster05 have?
16:53 kmai007 sorry
16:53 kmai007 glustervol01 i mean
16:53 John_HPC Its a 18x2
16:53 John_HPC I have 6 physical servers, each with 6 bricks
16:54 John_HPC df
16:54 John_HPC as for your previous question. I do have /tmp, but its a normal folder off of root. it's not its own mountpoint.
16:56 John_HPC from my understanding of the 18x2 setup. gluster01 and gluster02 are mirrors.... 3/4 and 5/6
16:58 sjusthome joined #gluster
16:59 John_HPC Here is what my split-brain looks like: http://paste.ubuntu.com/7463702/
16:59 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
16:59 lmickh joined #gluster
17:05 davinder joined #gluster
17:06 kmai007 have you tried to correct it the manual way?
17:07 kmai007 @splitbrain
17:07 glusterbot kmai007: I do not know about 'splitbrain', but I do know about these similar topics: 'split-brain'
17:07 John_HPC Not yet.
17:08 John_HPC I could just delete one and see what happens. But the wierd thing is, both seem to have the same sha1sum; so they appear *not* to be split-brain?
17:09 kmai007 i would focus on 1 brick
17:09 kmai007 and work your way down, looks like its complaining about the same file
17:09 John_HPC I know gluster05 is good data. I had a "punctured block" from a raid rebuild on gluster06
17:11 firemanxbr joined #gluster
17:13 glusterware joined #gluster
17:14 zerick joined #gluster
17:25 ktosiek joined #gluster
17:30 dusmant joined #gluster
17:35 systemonkey joined #gluster
17:54 diegows joined #gluster
17:58 _dist good afternoon, I was wondering if anyone has tried gluster 3.5 for virtualization storage, and if so does it still have the issue where volume heal info shows healthy files as healing?
17:58 _dist (this is in a replica volume)
17:59 andreask joined #gluster
18:03 bennyturns joined #gluster
18:07 sroy__ joined #gluster
18:10 John_HPC joined #gluster
18:14 dbruhn joined #gluster
18:16 bennyturns joined #gluster
18:32 pdrakewe_ joined #gluster
18:36 qdk joined #gluster
18:36 chirino joined #gluster
18:48 ktosiek joined #gluster
18:50 pdrakeweb joined #gluster
18:50 bennyturns joined #gluster
18:53 failshell joined #gluster
18:55 JoeJulian John_HPC: Did it create the tempfiles?
19:05 an joined #gluster
19:07 jeffrin joined #gluster
19:08 John_HPC JoeJulian: It did not.
19:09 JoeJulian John_HPC: selinux?
19:09 John_HPC selinux is disabled
19:10 JoeJulian Hmm... it can't get to the "Your split replicas..." text without succesfully creating those temp files.
19:11 John_HPC Your split replicas are mounted under /dev/shm/sbfix, in directories r1 through r0
19:11 John_HPC [42361 refs]
19:11 John_HPC I know!
19:11 John_HPC I think*
19:11 JoeJulian /dev/shm?
19:12 John_HPC I was trying to write to different locations. that was the tmpfs
19:12 John_HPC does it just do a mount or some type of copy? My system's drive is 100GB, but some fo the files are nearly 200GB?
19:13 JoeJulian It creates the tempfiles which are hacked up vol files. Then mounts the halves of the volume from those vol files.
19:16 kmai007 has anybody seen this log message?
19:16 rahulcs joined #gluster
19:16 kmai007 0-devstatic-dht: subvol: devstatic-replicate-0; inode layout - 0 - 0; disk layout - 2147483647 - 4294967295
19:21 John_HPC JoeJulian: Then it shouldn't be a file size issue...
19:26 jag3773 joined #gluster
19:26 glusterbot New news from resolvedglusterbugs: [Bug 1073442] large NFS writes to Gluster slow down then stop <https://bugzilla.redhat.com/show_bug.cgi?id=1073442>
19:28 JoeJulian John_HPC: And, again, an exception would happen if it was unable to create those temp files. It never deletes them so the fact that they're not there should say something.
19:30 John_HPC JoeJulian: That's what I was thinking; maybe there are no splitbrain files; but the gluster volume heal glustervol01 info split-brain returns a few entries; then again, maybe I am misunderstanding that output.
19:31 JoeJulian Maybe. Did you paste it? (I'm feeling too lazy to scroll back)
19:31 John_HPC ya 1 sec
19:32 John_HPC http://paste.ubuntu.com/7463702/
19:32 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
19:32 John_HPC The setup is a replicated 18x2
19:36 JoeJulian Ok... you can do a "heal glustervol01 full" to resolve the gfid to a filename, or you can use the other tool... let's see if it's ,,(resolver)
19:36 glusterbot I do not know about 'resolver', but I do know about these similar topics: 'gfid resolver'
19:36 JoeJulian @gfid resolver
19:36 glusterbot JoeJulian: https://gist.github.com/4392640
19:37 jeffrin left #gluster
19:37 chirino joined #gluster
19:44 John_HPC Ok. Thanks
19:45 ctria joined #gluster
19:46 JoeJulian Maybe you can do an strace of splitmount and paste it somewhere: strace -e trace=file $python $splitmount $server $volume $path
19:49 rahulcs joined #gluster
19:51 John_HPC will do
19:53 John_HPC http://paste.ubuntu.com/7464498/
19:53 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
19:56 John_HPC just an fyi, those gfids are the same file, the disk dump of eab1e81b-079a-11e0-beba-001e680f689d.vhd
19:56 failshell left #gluster
19:59 JoeJulian Alrighty then. Looks like it's legitimate split-brain to me, so far. You can look at the glustershd logs to find that gfid and see what criteria qualify it as such.
20:05 dbruhn joined #gluster
20:10 cvdyoung joined #gluster
20:14 John_HPC [2014-05-14 20:07:31.439459] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-glustervol01-replicate-14: Unable to self-heal contents of '<gfid:53dc39e2-e15e-48f5-915e-5193456e189f>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 1 ] [ 16777216 0 ] ]
20:15 John_HPC this was caused by a bad raid rebuild; had 3 hdds go down; replaced one at a time and on the 2nd one. got some stuff.
20:15 John_HPC I know gluster05 copy is good
20:15 John_HPC so I'll just delete the one from gluster06
20:16 John_HPC JoJulian: Thanks for your help./
20:17 andreask joined #gluster
20:22 rahulcs joined #gluster
20:24 John_HPC sob, this maybe the issue I'm seeing. the drive rebuild (I started on Friday morning) hasn't finished. May have to wait to finish before trying to heal
20:27 jiffe98 gluster really needs to fix the healing, adding a new replica kills everything
20:29 atrius joined #gluster
20:33 John_HPC didn't add a new replica, just replaced a failed drive on one of the replicas
20:33 dcherednik joined #gluster
20:36 jiffe98 John_HPC: I just added a new replica
20:36 jiffe98 and everything is dead now
20:36 John_HPC oh, thought you were refering to me
20:36 John_HPC eep
20:47 _dist jifee98: what kind of replica did you add? I'm considering adding one myself in the near future
20:52 badone__ joined #gluster
21:01 jiffe98 _dist: I updated a volume from no replicas to having 1 replica
21:03 dcherednik joined #gluster
21:08 basso joined #gluster
21:09 chirino joined #gluster
21:10 theron joined #gluster
21:11 dcherednik_ joined #gluster
21:14 tryggvil joined #gluster
21:15 dcherednik__ joined #gluster
21:17 jiffe98 luckily this is just a dev environment so no huge loss if its down
21:17 jiffe98 we have a production web environment that went down due to self healing
21:18 sroy_ joined #gluster
21:19 dcherednik_ joined #gluster
21:27 JoeJulian define "dead" and "went down"
21:29 jiffe98 inaccessible through the fuse client
21:30 JoeJulian Ok, so no crashes, just deadlock.
21:30 MugginsM joined #gluster
21:30 JoeJulian Which would make sense if you exceed the background-self-heal count.
21:34 dcherednik_ joined #gluster
21:35 kmai007 JoeJulian: have you used nagios for gluster monitoring?
21:37 JoeJulian kmai007: Not in any way that I am entirely happy with.
21:44 jiffe98 JoeJulian: I'm not sure I understand that value
21:45 jiffe98 so if I add a new replica to a volume with lots of files, its going to try to heal all those files but cluster.background-self-heal-count needs to be higher than the file count?
21:53 dcherednik_ joined #gluster
22:05 JoeJulian jiffe98: No, just higher than the number of files any one client is going to try to access before the previous background-self-heal count has completed healing.
22:05 JoeJulian Once the bsh queue is full, the next heal triggered will be in the foreground.
22:06 JoeJulian More clients, or a longer queue, or a limit on the number of files the client is trying to simultaneously access.
22:07 JoeJulian Of course the more clients idea will only work on highly distributed volumes or else you're going to saturate the network.
22:10 wushudoin left #gluster
22:11 jiffe98 how do I find out what that value is set to right now?
22:12 jiffe98 in this setup I have 2 servers and 2 clients and the servers and clients share the same machine
22:13 jiffe98 very little activity
22:14 JoeJulian gluster volume set help
22:16 primechuck joined #gluster
22:16 MeatMuppet joined #gluster
22:25 diegows joined #gluster
22:30 sjm joined #gluster
22:34 james__ joined #gluster
22:44 siel joined #gluster
22:45 primechuck joined #gluster
22:50 MeatMuppet left #gluster
22:51 sjm left #gluster
22:51 fidevo joined #gluster
23:07 Sunghost joined #gluster
23:09 Sunghost hello i have a problem after one brick failes - i cant move nor delete files and folders - any idea?
23:09 MugginsO joined #gluster
23:10 Sunghost i got socket error on glusterfs nativ mount
23:15 Sunghost i can create a dir but after that i cant delete it
23:18 JoeJulian Distribute volume without replication?
23:19 Sunghost distribute only
23:19 JoeJulian yep, that's expected.
23:19 Sunghost i must say that i talkt about that last week with lalatenduM - a supporter from redhat
23:20 Sunghost we talked about the possibilities and he adviced me to do so
23:20 JoeJulian directories exist on all bricks. Modifying those directories with a brick offline will fail.
23:20 Sunghost dam
23:20 Sunghost so what could i do?
23:20 JoeJulian You can remove the brick.
23:20 JoeJulian "gluster volume remove-brick..."
23:20 Sunghost whats happend with the data
23:20 Sunghost on both
23:21 Sunghost brick1 is only accessible via live cd
23:21 Sunghost and the vol1 with brick2 and brick3 starts without problems
23:21 Sunghost i catually try to copy the files from brick1 to vol1
23:21 JoeJulian Well files are stored whole, so if you can get to your filesystem that was a brick, you should be able to recover any files.
23:22 Sunghost thats what i expected
23:22 Sunghost but if i copied from brick1 to an usb-disk and from there to the vol i got socket error and cant move over existing files
23:22 JoeJulian The question in my mind is, why can you access the filesystem with a live cd but not with the installed os?
23:23 Sunghost ok the os is on an raid and this has crashed i cant boot from it
23:23 JoeJulian Right, like I said, you'll have to remove-brick brick1
23:23 Sunghost what happends with the files, i think nothing or?
23:23 JoeJulian Nothing. It just changes the volume layout.
23:24 Sunghost as i understand it will only remove brick1 and after that the permissions to write to the vol whould work
23:25 Sunghost right?
23:26 Sunghost gluster volume info says for brick1 :Brick1: 192.168.0.41:/media/node01/vol1
23:26 Sunghost so must i enter gluster volume remove-brick vol1 192.168.0.41 or the full path?
23:29 JoeJulian full path
23:30 Sunghost ok thanks worked - will now try to move files
23:31 Sunghost yes he copied files ;) many thx for simple solution but i didnt find any info in the net ;) saved my night thx JoeJulian
23:32 Sunghost should i now remove the brick1 from storage pool or later ?
23:33 JoeJulian You should be able to do it now. Make sure you committed your change.
23:35 Ark joined #gluster
23:36 Sunghost how do you meen "committed your change" ?
23:41 JoeJulian gluster volume remove-brick 192.168.0.41:/media/node01/vol1 commit
23:43 Sunghost twice?
23:44 Sunghost one without and second with commit or only one time with commit ;)? sorry for asking stupid
23:44 JoeJulian Oh, if you already commited then no.
23:44 sjm joined #gluster
23:44 Sunghost ah ok but i dindt use option commit
23:44 JoeJulian With commit once would have been fine. But for sure you need at least once with commit.
23:44 Sunghost ok so i enter it a second with commit, right
23:45 JoeJulian Without commit it tries to migrate all the files off the brick being removed. Since it can't do that...
23:45 JoeJulian yes, once with commit.
23:45 Sunghost volume remove-brick commit: failed: Incorrect brick 192.168.0.41:/media/node01/vol1 for volume vol1
23:45 Sunghost sounds it has happend before at the first time
23:46 JoeJulian Or it's an older version. You should be good to remove the server if you want.
23:46 Sunghost thats the plan and after backup all files i will reinstall brick1 / the server
23:47 Sunghost ah ok you mean by using now detach command
23:47 JoeJulian right
23:47 Sunghost simple gluster peer detach 192.168.0.41 ?
23:48 Sunghost is there a controll command before and after running this?
23:49 JoeJulian that simple, yes
23:55 Sunghost its about removing the brick1 from trusted pool -right? how could i see which bricks inside the pool?
23:56 gdubreui joined #gluster
23:57 Sunghost ok found it gluster peer status
23:58 Sunghost ok i think its now all ok and i can go further to backup my data -thx again JoeJulian

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary