Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-18

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 hagarth joined #gluster
00:05 yinyin joined #gluster
00:11 badone joined #gluster
00:19 davidbitton joined #gluster
00:26 hagarth joined #gluster
00:31 davidbitton i noticed in the source code that there is an selinux option to both glusterd and glusterfsd. i added the selinux option in my glusterd init script, however the glusterfsd instance launch w.o the option. how can I manage the launch options for glusterfsd when it's glusterd that's doing the launching
00:43 jdarcy joined #gluster
00:46 hagarth joined #gluster
01:06 hagarth joined #gluster
01:22 jules_ joined #gluster
01:26 yinyin joined #gluster
01:45 kevein joined #gluster
01:53 jules_ joined #gluster
02:10 ultrabizweb joined #gluster
02:21 hagarth joined #gluster
02:48 hagarth joined #gluster
02:49 bala joined #gluster
03:04 DaveQB joined #gluster
03:06 davidbitton joined #gluster
03:07 bharata joined #gluster
03:12 saurabh joined #gluster
03:14 DaveQB I have noticed since moving to 3.3 that doing a peer probe on host A sees host B have host A listed by IP. As a result, all commands refering to host A needs to use that IP now. It won't respect the hostname given in the peer probe command.
03:18 yinyin joined #gluster
03:20 raghug joined #gluster
03:41 shylesh joined #gluster
03:44 mynameisdeleted joined #gluster
03:44 mynameisdeleted so.... any way I can change 4 nodes with 4x1 replication to 2x2?
03:44 mynameisdeleted without destroying a volume?
03:49 sgowda joined #gluster
04:15 m0zes mynameisdeleted: I think you can do something like "gluster volume remove-brick <volname> replica 2 server:/brick/to/remove server:/brick/to/remove" when that is finished, mkfs the bricks you removed and add-brick.
04:16 bala joined #gluster
04:17 yinyin joined #gluster
04:21 Kins joined #gluster
04:25 anmol joined #gluster
04:27 vshankar joined #gluster
04:28 vpshastry joined #gluster
04:29 sahina joined #gluster
04:32 raghug joined #gluster
04:34 Kins joined #gluster
04:38 _pol joined #gluster
04:49 sripathi joined #gluster
05:00 lalatenduM joined #gluster
05:01 aravindavk joined #gluster
05:03 yinyin joined #gluster
05:04 bulde joined #gluster
05:05 deepakcs joined #gluster
05:18 yinyin joined #gluster
05:24 mohankumar joined #gluster
05:28 test joined #gluster
05:36 rastar joined #gluster
05:50 satheesh joined #gluster
05:51 rastar joined #gluster
05:51 sgowda joined #gluster
05:52 hagarth joined #gluster
05:53 vshankar joined #gluster
06:02 Humble joined #gluster
06:07 zhashuyu joined #gluster
06:09 bulde1 joined #gluster
06:09 bulde joined #gluster
06:10 raghu joined #gluster
06:11 puebele joined #gluster
06:13 rotbeard joined #gluster
06:15 DaveQB I have noticed since moving to 3.3 that doing a peer probe on host A sees host B have host A listed by IP. As a result, all commands refering to host A needs to use that IP now. It won't respect the hostname given in the peer probe command.
06:20 hagarth joined #gluster
06:20 sgowda joined #gluster
06:29 raghu joined #gluster
06:29 ngoswami joined #gluster
06:35 yinyin joined #gluster
06:46 guigui joined #gluster
06:51 hagarth joined #gluster
06:52 mistigi joined #gluster
06:54 shireesh joined #gluster
06:54 displaynone joined #gluster
06:55 bulde joined #gluster
06:57 jcaputo joined #gluster
07:02 Nevan joined #gluster
07:03 jcaputo I"m really  hoping somebody can help me out here
07:03 jcaputo on one of our gluster servers, there's an issue:
07:03 jcaputo http://dpaste.com/1026358/
07:03 glusterbot Title: dpaste: #1026358: no such file or directory error, by joe (at dpaste.com)
07:03 jcaputo and that has been happening for the past week
07:04 jcaputo can't seem to self heal that file
07:04 jcaputo i've also tried to fix it by triggering a self heal manually
07:04 jcaputo but no luck
07:05 jcaputo and it's only one file that needs fixing :(
07:07 jcaputo any idea as to how to remove/fix this one file?
07:08 jcaputo when i run the command to check for split-brain, I can confirm there are no files listed on any brick
07:08 jcaputo when i run the command to search for files that have failed to heal, that one file/directory does indeed show up
07:09 jcaputo gfid:b40ba46e-8779-4c85-aade-0467fb5cc675
07:11 jcaputo any ideas fellow glusterfs users? :)
07:12 rgustafs joined #gluster
07:16 jtux joined #gluster
07:18 vimal joined #gluster
07:25 rgustafs joined #gluster
07:26 m0zes_ joined #gluster
07:31 hagarth joined #gluster
07:33 jcaputo_ joined #gluster
07:35 jcaputo joined #gluster
07:35 jcaputo hmmm, everybody must be in bed :)
07:35 rotbeard not really, but some people aren't able to help you maybe ;)
07:36 rotbeard (including myself)
07:36 jcaputo :)
07:36 jcaputo that's too bad - was really hoping somebody has seen that before
07:49 vpshastry joined #gluster
07:57 ctria joined #gluster
08:07 Nagilum jcaputo: replicated set?
08:08 bala joined #gluster
08:09 jcaputo hi back
08:09 jcaputo sorry
08:10 jcaputo it is a distributed replicate
08:10 jcaputo thanks for looking @Nagilum
08:12 Nagilum hmm, the way I understand it is that these gfid's identify inodes, if there would be a way to get the inode out of that gfid..
08:13 jcaputo oh, i've seen errors about inodes as well
08:13 jcaputo when i run the status of the heal
08:13 jcaputo i get an inode error
08:13 jcaputo one second
08:13 jcaputo let me find :)
08:15 jcaputo here is an example:
08:15 jcaputo http://dpaste.com/1026406/
08:15 glusterbot Title: dpaste: #1026406: inode error, by joe (at dpaste.com)
08:15 eiki joined #gluster
08:15 Nagilum jcaputo: which filesystem underneath?
08:16 jcaputo xfs
08:16 jcaputo so those errors appear when i run this for example:  gluster volume heal VOLNAME info
08:18 jcaputo have you seen this before @Nagilum?
08:18 Nagilum no
08:18 jcaputo :|
08:18 jcaputo that's no good :P
08:19 Nagilum yeah, looks like glusterfs still has some maturing to do
08:19 jcaputo yeah, maybe you are correct
08:19 jcaputo but in the meantime, what should I do I wonder
08:19 jcaputo it seems it is in a bad state
08:20 jcaputo i don't think it's just spam in the log.  I'm affraid if one other node goes down for one reason or another, then the volume will be  unavailable :(
08:20 jcaputo because this is self-healing still, does it mean that it's not ready  to be part of the volume yet?
08:21 jcaputo i'm a bit concerned for sure
08:21 jcaputo worried is more like it
08:21 Nagilum afaik sealf healing run online, but I've been using it barely a week :>
08:22 jcaputo yes, you are correct I believe
08:22 jcaputo but if it can't self heal, that's an issue I think :)
08:22 Nagilum I agree
08:22 jcaputo wish mr. @JoeJulian was around - he's awesome at figuring this stuff out.  One of the best
08:23 vpshastry joined #gluster
08:23 jcaputo well thanks for looking buddy
08:23 jcaputo really appreciate it
08:23 jcaputo hopefully somebody else can share some insight.
08:24 Nagilum documentation also needs a bit of work
08:24 jcaputo it's very cool technology
08:24 jcaputo :)
08:24 Nagilum doesn't even say how to replace a failed server
08:24 Nagilum or how to dissolve a gluster
08:25 Nagilum so that the servers can become members of a new gluster
08:25 andreask joined #gluster
08:26 jcaputo yeah...
08:29 Staples84 joined #gluster
08:30 tjikkun_work joined #gluster
08:31 spai joined #gluster
08:35 mohankumar joined #gluster
08:37 sripathi joined #gluster
08:37 saurabh joined #gluster
08:46 sahina joined #gluster
08:47 raghug joined #gluster
08:47 samppah is there any known issues with 3.4.0.alpha2 georeplication?
08:51 bulde samppah: we are trying to work on multiple things related to geo-replication starting this month :-)
08:52 bulde so mostly lot of 'performance' enhancements will get into geo-rep by 3.5.0 (or if there is a popular demand, to 3.4.x)
08:53 bulde as of now, the most serious known issue with geo-replication is the performance, when the number of files inside volume increases, or even number of bricks in the volume increases
08:53 samppah ah okay, but basicly there isn't currently any known show stoppers on 3.4?
09:01 yinyin joined #gluster
09:04 sripathi joined #gluster
09:07 kevein joined #gluster
09:10 maxiepax joined #gluster
09:10 inodb joined #gluster
09:14 spai joined #gluster
09:18 vpshastry joined #gluster
09:18 bulde joined #gluster
09:20 sahina joined #gluster
09:23 sgowda joined #gluster
09:23 mooperd joined #gluster
09:25 shireesh joined #gluster
09:28 dobber_ joined #gluster
09:30 hagarth joined #gluster
09:30 glusterbot New news from newglusterbugs: [Bug 922432] Upstream generated spec file references non-existing patches <http://goo.gl/ThpfV>
09:34 test__ joined #gluster
09:37 tryggvil__ joined #gluster
09:42 joeto joined #gluster
09:57 displaynone joined #gluster
10:00 glusterbot New news from newglusterbugs: [Bug 921024] Build process not aware of --prefix directory <http://goo.gl/uUplT>
10:01 hagarth joined #gluster
10:02 deepakcs joined #gluster
10:02 joeto1 joined #gluster
10:09 pmuller_ joined #gluster
10:09 jclift joined #gluster
10:11 shireesh joined #gluster
10:12 jcaputo joined #gluster
10:12 jcaputo_ joined #gluster
10:17 mohankumar joined #gluster
10:18 jcaputo_ anybody online see my last gpaste of the issue I'm having?
10:18 jcaputo_ http://dpaste.com/1026358/
10:18 glusterbot Title: dpaste: #1026358: no such file or directory error, by joe (at dpaste.com)
10:20 jcaputo_ our team is going live with a feature utilizing glusterfs and I'm afraid making it live with the state it's in...:|
10:20 raghug joined #gluster
10:24 sahina joined #gluster
10:26 bharata joined #gluster
10:33 vpshastry joined #gluster
10:42 Nagilum joined #gluster
10:45 Nagilum jcaputo_: you can try a "find /gfs -type f >/dev/null" and hope for the best?
10:46 jdarcy joined #gluster
10:50 jdarcy_ joined #gluster
10:52 jcaputo_ i'm really not sure exactly what that will do
10:52 yinyin joined #gluster
10:54 ndevos jcaputo_: I *think* a directory on the gv_savedata-client-5 brick is missing, on an other brick it would have the gfid b40ba46e-8779-4c85-aade-0467fb5cc675
10:56 jcaputo_ so what am I to do about this @ndevos?
10:57 jcaputo_ is this solvable?
10:57 ndevos jcaputo_: you can check the .glusterfs/<1st-gfid-part>/<​2nd-gfid-part>/<whole-gfid> on a brick different from gv_savedata-client-5 and check where the gfid (it's a symlink) is pointing to
10:58 ndevos jcaputo_: after that, you can check if the directory is indeed missing on the gv_savedata-client-5 brick
10:58 jcaputo_ and if it is?
10:59 ndevos you create it, and set the same xattrs as the other replicated brick has
10:59 jcaputo_ ok, i'll need to find out how to do that last part you mention about the xattrs
10:59 ndevos try to heal the volume and, see if the self-heal daemon still logs the errors every 10 minutes
11:00 ndevos it's like 'getfattr -m . -e hex /path/to/brick/to/subdir' and some setfattr commands
11:01 ndevos mainly the trusted.* xattrs are important
11:02 jcaputo_ ok
11:02 jcaputo_ i'll try and give this a shot
11:02 jcaputo_ thanks very much for your help
11:02 jcaputo_ oh, sorry, do you know off hand where the .glusterfs dir is located?
11:03 ndevos it's on the bricks, well, if you have glusterfs-3.3
11:03 jcaputo_ ah right
11:03 jcaputo_ ok
11:07 jcaputo_ so ndevos, you're saying the dir i'm looking for should be in the .glusterfs/b4/0b/blahahahaha?
11:08 ndevos jcaputo_: correct, where blahahahahah is b40ba46e-8779-4c85-aade-0467fb5cc675
11:09 ndevos jcaputo_: I expect the .glusterfs/b4/0b/b40ba46e-8​779-4c85-aade-0467fb5cc675 to be a symlink to the directory on the brick
11:09 jcaputo_ hmm
11:09 jcaputo_ seems as though it does not exist
11:09 ndevos jcaputo_: right, on note of the bricks that make up the replicated volume?
11:10 ndevos s/note/none/
11:10 glusterbot What ndevos meant to say was: jcaputo_: right, on none of the bricks that make up the replicated volume?
11:10 jcaputo_ yes, i'm on a different server
11:10 jcaputo_ that has a brick
11:10 jcaputo_ that makes up the volume
11:13 jcaputo_ http://dpaste.com/1026534/
11:13 glusterbot Title: dpaste: #1026534: dir listing, by joe (at dpaste.com)
11:15 yinyin_ joined #gluster
11:15 jcaputo_ no such file or dir...
11:15 jcaputo_ hmmm
11:18 jcaputo_ oh
11:18 jcaputo_ ndevos
11:18 jcaputo_ it exists on the bad brick though
11:18 jcaputo_ on the brick it's complaining on
11:18 jcaputo_ so it exists on the bad one, but not on the 5 others
11:19 ndevos jcaputo_: hmm, and does 'getfattr -m . -e hex $path' show the trusted.gfid of the directory?
11:19 tryggvil__ joined #gluster
11:20 jcaputo_ $path being?
11:20 ndevos jcaputo_: $path being the location the gfid-symlink points to
11:20 jcaputo_ oh, i haven't got to that yet
11:20 jcaputo_ i'm not sure where it points to
11:22 ndevos jcaputo_: doesnt 'ls -l' or 'readlink' show that?
11:22 jcaputo_ i did an ls -la
11:22 jcaputo_ but i don't see that
11:22 jcaputo_ it shows like the output i posted on dpaste
11:24 ndevos jcaputo_: ah, okay, so the affected gfid isn't a directory, but a file
11:25 jcaputo_ looks that way
11:25 ndevos jcaputo_: can you dpaste the xattrs from that gfid?
11:25 jcaputo_ one second...
11:27 jcaputo_ http://dpaste.com/1026546/
11:28 jcaputo_ there you go sir
11:28 glusterbot Title: dpaste: #1026546: getfattr, by joe (at dpaste.com)
11:28 ndevos ah, sorry, missed the -d option to getfattr, it would be good to have the values of the xattrs :)
11:29 jcaputo_ oh lol
11:29 jcaputo_ k
11:29 jcaputo_ sec
11:29 jcaputo_ so this command yes?
11:29 jcaputo_ getfattr -m -d . -e hex /mnt/brick3/.glusterfs/b4/0b/b40b​a46e-8779-4c85-aade-0467fb5cc675
11:29 ndevos almost, getfattr -m . -d -e hex .....
11:30 ndevos the '.' is a parameter to the -m option
11:30 jcaputo_ ah, ok
11:30 jcaputo_ sorry about that :)
11:31 rwheeler joined #gluster
11:31 jcaputo_ http://dpaste.com/1026549/
11:31 glusterbot Title: dpaste: #1026549: with -d, by joe (at dpaste.com)
11:32 jcaputo_ so does this mean that another brick does have this file?
11:32 jcaputo_ client-4?
11:32 jcaputo_ btw, just so you know this is a 6 node distributed replicate
11:33 jcaputo_ 3 x 2
11:33 ndevos jcaputo_: so, trusted.afr.gv_savedata-client​-5=0x000000020000000000000000 means that the file on gv_savedata-client-5 is missing two changes to its contents
11:33 jcaputo_ ok, and how is this resolved?
11:33 jcaputo_ why can't it get the changes?
11:34 ndevos jcaputo_: well, its a ,,(split-brain) and gluster does not seem to know the state of the file on gv_savedata-client-4
11:34 glusterbot jcaputo_: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
11:35 jcaputo_ i've read about split-brain
11:35 jcaputo_ but when i ran a command to see if there was split-brains, it looked like there were none
11:35 jcaputo_ all the bricks had 0
11:35 ndevos jcaputo_: okay, so, you'll need to find the filein the normal directory structure, as you pointed out that the gfid-link does not exist
11:36 jcaputo_ ok...
11:36 ndevos jcaputo_: the entries in .glusterfs are hardlinks, so they have the same inode as their companion
11:37 ndevos which means that you canuse 'ls -i $gfidfile' and 'find $brick -inum $inode' to locate it
11:37 jcaputo_ this is a bit over my head - i'm fairly new to linux sorry, :(
11:38 jcaputo_ $gfidfile is what?
11:38 jcaputo_ that same id we used previously?
11:38 ndevos jcaputo_: thats the /mnt/brick3/.glusterfs/b4/0b/b40b​a46e-8779-4c85-aade-0467fb5cc675 file
11:38 jcaputo_ ok, cool, that's what I thought.  bear with me :)
11:38 Staples84 joined #gluster
11:39 ndevos the inode is a number that identifies an entry on the filesystem
11:39 jcaputo_ 1143534524 /mnt/brick3/.glusterfs/b4/0b/b40b​a46e-8779-4c85-aade-0467fb5cc675
11:40 ndevos where 'entry' is the contents of the file - it has two names, one /mnt/brick3/.glusterfs/b4/0b/b40b​a46e-8779-4c85-aade-0467fb5cc675 and an other one
11:40 flrichar joined #gluster
11:40 ndevos so, with 'find /mnt/brick3/ -inum 1143534524' you can search for all files that have that inode
11:41 jcaputo_ so now: find /mnt/brick3 -inum 1143534524
11:41 jcaputo_ ?
11:41 jcaputo_ oh, k
11:41 jcaputo_ sorry we wrote that at the same time. :)
11:41 ndevos yeah, but that will cause quite some i/o :)
11:41 jcaputo_ hmmm
11:41 jcaputo_ that could be bad then
11:41 jcaputo_ as it's in prod :(
11:42 jcaputo_ but do i have a choice?
11:42 jcaputo_ and is there a way to stop it if i need to?
11:42 ndevos well, "some i/o" is relative...
11:42 ndevos sure, press CTRL+c to kill the find
11:42 jcaputo_ i can stop it if i need to k
11:43 jcaputo_ and when i find the file?
11:43 jcaputo_ or files?
11:43 ndevos jcaputo_: verify them, and check on the other brick(s) of the replica of the files are there too
11:44 jcaputo_ ok, found
11:44 ndevos jcaputo_: delete the wrong files and do the healing for the split-brain
11:44 jcaputo_ /mnt/brick3/6/5/5/5c0556.bin
11:44 jcaputo_ so one second
11:45 jcaputo_ on each server that has a brick for this volume, i want to search for this file
11:45 jcaputo_ and if it exists, delete it from each server?
11:45 ndevos yes, and now you know the real filename, you can check that easier
11:46 jcaputo_ i thought you were not supposed to do any file modifications on the bricks themselves?
11:46 ndevos it can be that one brick contains a different (over written?) file, causing the two files on different bricks have a different gfid
11:46 ndevos jcaputo_: you really should not, but in order to resolve a split-brain, you sometimes have to
11:46 jcaputo_ ok, i'm going to go and delete this file from each server
11:47 bulde joined #gluster
11:47 jcaputo_ i guess i could just do a rm /mnt/brick3/6/5/5/5c0556.bin
11:47 jcaputo_ on each server
11:47 ndevos jcaputo_: well, you want to not delete all of them, only the bad ones
11:47 jcaputo_ correct?
11:48 jcaputo_ im a bit confused - so the bad one would be the one that had the errors in the log in the first place no?
11:48 ndevos gluster does not know which file is bad, otherwise it would be able to resolve the split-brain by itself
11:48 jcaputo_ so how do i know which file is bad?
11:49 ndevos you inspect the file? maybe the date is was last modified?
11:49 ndevos maybe you have a checksum in a db somewhere?
11:49 jcaputo_ hmmm, ok
11:50 jcaputo_ im not sure how that file is generated
11:50 jcaputo_ i could ask one of the devs tomorrow
11:50 jcaputo_ it might be that it isn't even important
11:50 ndevos also, sometimes the contents of the files are actually the same, so you can checksum them on the bricks too
11:51 ndevos yeah, a dev should be able to tell you how you can verify which copy of the file is the correct one
11:51 andreask joined #gluster
11:52 jcaputo_ i'm just going through all of them quickly
11:53 jcaputo_ thanks so much ndevos
11:53 jcaputo_ if you could just bear with me for a few more min
11:53 jcaputo_ i really appreciate this
11:54 ndevos you're welcome! just check the contents, xattrs and other details you can think of
11:54 * ndevos will be in a meeting shortly, and not able to respond much more
11:55 jcaputo_ it doesn't exist yet so far on any of them i've checked
11:56 jcaputo_ does not exist on any of them
11:56 jcaputo_ except the bad one
11:57 jcaputo_ which means I could possibly delete the file from the bad brick?
11:59 jcaputo_ and does this also mean it is not a split brain?
11:59 jcaputo_ i'm a bit confused on this
12:00 rubbs joined #gluster
12:07 jdarcy joined #gluster
12:10 hagarth joined #gluster
12:10 jcaputo_ also, does this also mean that if this output does not return any entries, there may still possibly be a split-brain scenario?  http://dpaste.com/1026575/
12:10 glusterbot Title: dpaste: #1026575: split brain?, by joe (at dpaste.com)
12:10 bala joined #gluster
12:11 saurabh joined #gluster
12:14 jcaputo_ i think what I'd like to do at this point for this one file that exists only on the one server is log into the server that has this file and run:
12:15 jcaputo_ rm- f /mnt/brick3/6/5/5/5c0556.bin
12:15 jcaputo_ rm -f /mnt/brick3/.glusterfs/b4/0b/b40b​a46e-8779-4c85-aade-0467fb5cc675
12:16 jcaputo_ i've confirmed it does not exist on any other server...
12:17 Nagilum maan, these error messages are just crap: "Operation failed"
12:19 Nagilum even the log borderlining: [2013-03-18 12:16:35.191291] I [glusterd-brick-ops.c:380:​glusterd_handle_add_brick] 0-glusterd: Received add brick req
12:20 Nagilum [2013-03-18 12:16:35.191404] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: b1dc4e1b-aeea-4507-b8d5-90a39e3a720c, lock held by: b1dc4e1b-aeea-4507-b8d5-90a39e3a720c
12:20 Nagilum [2013-03-18 12:16:35.191428] E [glusterd-handler.c:458:glusterd_op_txn_begin] 0-management: Unable to acquire local lock, ret: -1
12:23 jcaputo_ lonce i get the go from you @ndevos, I'm going to go ahead and delete those files from the bad server
12:23 jcaputo_ :)
12:23 jcaputo_ that sucks Nagilum
12:24 jcaputo_ sorry I can't help out (yet), just getting into this stuff myself
12:25 Nagilum it's ok, I'm guessing it's because there is a brick-replace running, but I would expect some kind of indication in the error message
12:31 H__ what gluster version is this ?
12:31 Nagilum 3.3.1
12:32 aliguori joined #gluster
12:34 Nagilum ah great, now it says migration complete..after having migrated about 10%
12:34 Nagilum when I do a commit it hangs
12:35 robos joined #gluster
12:36 Nagilum ah great, glusterd can't be killed on the server where I issued the commit
12:48 bennyturns joined #gluster
12:49 awheeler joined #gluster
12:51 jcaputo_ @ndevos, please say hello when you're back.  Before I make this change, I want to get your opinion first...
12:51 jcaputo_ don't want to break anything :)
13:01 mynameisbruce joined #gluster
13:01 mynameisbruce_ joined #gluster
13:01 glusterbot New news from newglusterbugs: [Bug 922760] Volume type changes don't end with add-brick/remove-brick commands <http://goo.gl/4LYzx>
13:01 mynameisbruce_ left #gluster
13:02 edward1 joined #gluster
13:05 dustint joined #gluster
13:08 yinyin joined #gluster
13:15 tryggvil__ joined #gluster
13:22 bulde joined #gluster
13:23 manik joined #gluster
13:26 hagarth joined #gluster
13:29 jtux joined #gluster
13:29 stat1x joined #gluster
13:31 glusterbot New news from newglusterbugs: [Bug 922765] build: package version string should show 3.4 <http://goo.gl/qHua3> || [Bug 918917] 3.4 Beta1 Tracker <http://goo.gl/xL9yF>
13:34 coredumb joined #gluster
13:35 coredumb Hello, is there some numbers to show of guest VM running on KVM native implementation over glusterfs ?
13:42 displaynone joined #gluster
13:48 brunoleon_ joined #gluster
13:50 sjoeboo joined #gluster
13:50 plarsen joined #gluster
13:55 rwheeler joined #gluster
13:56 alex88 joined #gluster
13:56 alex88 hi guys
13:57 alex88 does gluster support ipv6?
13:57 alex88 I've tried to connect to an ipv6 host and I get "DNS resolution failed on host storage.site.com"
13:57 alex88 but using host storage.site.com I get the ip
13:57 alex88 it seems to use ipv6 since it calls gf_resolve_ip6
13:58 alex88 any idea?
13:59 alex88 I'm using 3.3.1
14:00 alex88 maybe because the dns server is using ipv4? (but still returns ipv6)
14:03 ndevos jcaputo_: so, that file should have been removed but is still available on one brick?
14:06 awheeler Is it just me, or does the 3.4 alpha2 release break UFO/Swift?
14:08 ndevos jcaputo_: those commands would completely remove the file from your volume, without being able to access it again - so, only do that when the file indeed is not needed anymore
14:10 statix_ joined #gluster
14:16 jclift That's weird.  Some strange messages in the older online docs: "Error creating thumbnail: Unable to save thumbnail to destination"
14:16 jclift http://gluster.org/community/documen​tation/index.php/Gluster_3.1:_Explor​ing_the_Gluster_Management_Console
14:16 glusterbot <http://goo.gl/zUzOZ> (at gluster.org)
14:16 jclift Tried editing the page source, and it's not there.  Seems to be a problem in the page generation.
14:17 alex88 seems that using host storage.site.com works fine, using nslookup on the same address which uses same server isn't working.. any idea?
14:21 alex88 ok, nm, adding set q=any works on nslookup, so why gluster is saying unable to resolve address?
14:21 guigui joined #gluster
14:23 jclift Maybe it's forcibly looking for ipv6 only entries.  (no real idea tho)
14:23 * jclift isn't that deep into the gluster code yet
14:26 alex88 jclift: it's very strange
14:27 alex88 need a tool to sniff dns queries
14:27 alex88 to see what's different
14:27 jclift wireshark I guess?
14:28 jclift alex88: The other option that might work for DNS debugging if you don't like wireshark, is to install dnsmasq and tell it to generate a log entry for every dns query that hits it
14:28 jclift alex88: I've used that technique before, and it works ok
14:28 alex88 jclift: I'm already using dnsmasq for dns/dhcp
14:29 jclift alex88: Aha, might be the go then.
14:29 jclift alex88: When I'm using it for that, I install it on the host having problems, point /etc/resolv.conf to 127.0.0.1, and watch the dnsmasq.log file.
14:29 alex88 jclift: I've just one host since it's an initial setup
14:30 alex88 not that much garbage will be generated ;)
14:30 jclift Cool, give that a shot maybe? :)
14:30 alex88 already doing that :) changing config
14:30 jclift :)
14:32 sahina joined #gluster
14:32 alex88 jclift: http://pastie.org/private/qkbc1lz6xodra8n7nhjuhw it queries just A
14:32 glusterbot <http://goo.gl/X8I2J> (at pastie.org)
14:33 lh joined #gluster
14:33 lh joined #gluster
14:33 alex88 instead of host that queries everything
14:34 jclift Ahhh.
14:34 jclift alex88: You don't have storage.site.com in your /etc/hosts ?
14:35 alex88 jclift: nope, since it's public
14:35 alex88 it's resolvable
14:35 * jclift just tried storage.site.com and got told "no such entry"
14:35 vpshastry joined #gluster
14:35 plarsen joined #gluster
14:35 alex88 jclift: site is just a placeholder
14:36 alex88 as you see from the dns server it resolves fine
14:36 sgowda joined #gluster
14:36 * jclift looks properly
14:39 alex88 anyone else guys?
14:40 deepakcs joined #gluster
14:40 jclift alex88: Definitely worth filing a BZ request for that.
14:40 alex88 BZ?
14:40 jclift A bug report. :)
14:40 jclift bugzilla.redhat.com
14:40 jclift Sorry, "BZ" is the short name everyone uses, as typing Bugzilla a lot is a pita. :D
14:41 alex88 oh :) sure sorry :)
14:41 jclift alex88:  sure the Gluster core coders will be interested
14:41 jclift Gah
14:42 alex88 meh, maybe it's just my fault
14:42 alex88 somewhere
14:42 jclift alex88: Reasonably sure the Gluster core coders will be interested in Gluster not working IPv6 only name resolution.
14:43 jclift alex88: If you file a BZ :) then it should get looked at by someone more clueful than me with this kind of things.
14:43 alex88 sure, I'll do this evening maybe
14:43 alex88 too lazy to create user and/or signup on mailing list
14:44 semiosis file a bug
14:44 glusterbot http://goo.gl/UUuCq
14:44 semiosis ^^^
14:47 jclift alex88: I'll file it then.  Might as well get it in the system. :D
14:47 jclift alex88: Which version of Gluster are you running, so I can put in the right details?
14:48 alex88 jclift: glusterfs 3.3.1 built on Feb 21 2013 03:24:40
14:48 alex88 got by the ubuntu ppa
14:48 jclift thx
14:48 alex88 jclift: thanks you man!
14:49 ProT-0-TypE joined #gluster
14:49 alex88 so basically, mount -t glusterfs storage.site.com:/site-development /mnt/site-development/
14:49 alex88 makes just an A query that doesn't resolv
14:52 yinyin joined #gluster
14:54 stat1x joined #gluster
14:56 manik joined #gluster
14:56 jclift alex88: https://bugzilla.redhat.com/show_bug.cgi?id=922801
14:56 glusterbot <http://goo.gl/79JLl> (at bugzilla.redhat.com)
14:56 glusterbot Bug 922801: unspecified, unspecified, ---, amarts, NEW , Gluster not resolving hosts with IPv6 only lookups
14:57 alex88 thanks jclift!
14:57 jclift :)
14:57 bugs_ joined #gluster
15:01 glusterbot New news from newglusterbugs: [Bug 922801] Gluster not resolving hosts with IPv6 only lookups <http://goo.gl/79JLl>
15:04 daMaestro joined #gluster
15:05 jbrooks joined #gluster
15:05 lge joined #gluster
15:13 _pol joined #gluster
15:14 tryggvil joined #gluster
15:28 nirik joined #gluster
15:29 nirik hey folks. Quick query... I have a 2 system replicate setup... 'gluster volume info name' shows the bricks on host01 and host02, but 'gluster volume status name' shows only one brick, and indeed it's only using the one. ;( How can I get it to start using the other brick again?
15:34 yinyin joined #gluster
15:39 zaitcev joined #gluster
15:45 tjstansell nirik: you can check if the glusterfsd processes are running on both hosts ... and maybe try re-starting your volume with force option to force it to bring up the brick if it's not up
15:45 nirik yeah. :) It looks like it wasn't... and there was a dns/resolution error preventing it.
15:45 nirik got the brick back now.
15:45 tjstansell cool
15:45 jdarcy joined #gluster
15:51 jdarcy joined #gluster
15:54 lh joined #gluster
15:54 lh joined #gluster
15:57 yinyin joined #gluster
15:59 nirik left #gluster
16:01 go2k joined #gluster
16:03 go2k Guys, do you know how to monitor glusterfs auto-replication, on account of in what status it is? Say if I had an outage of a node for 5 hours how can I know that the node which was offline got updated?
16:03 go2k I apologize, self-healing it's called.
16:06 nueces joined #gluster
16:07 mooperd_ joined #gluster
16:15 Norky go2k: gluster volume heal home info
16:15 Norky go2k: gluster volume heal VOLNAME info
16:16 Norky "gluster volume heal VOLNAME info healed" will show you what has been healed, I think
16:16 Chiku|dc joined #gluster
16:16 Chiku|dc hi
16:16 glusterbot Chiku|dc: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:17 Chiku|dc I created a volume and deleted and want to reuse it
16:17 Chiku|dc but I got this error "...or a prefix of it is already part of a volume"
16:17 glusterbot Chiku|dc: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
16:17 ndevos glusterbot strikes again \o/
16:18 Chiku|dc I try to follow this http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/ but doesn't work
16:18 glusterbot <http://goo.gl/YUzrh> (at joejulian.name)
16:21 wushudoin joined #gluster
16:21 bstansell left #gluster
16:22 sgowda joined #gluster
16:26 cw joined #gluster
16:27 H__ Chiku|dc: you did all that and still get 'already part' ?!
16:28 vimal joined #gluster
16:29 bala joined #gluster
16:31 hagarth joined #gluster
16:32 Norky Chiku|dc, double check you did exactly what that guide says
16:33 Chiku|dc ah maybe I forget the last part... restart glusterd
16:34 Norky copy and paste exactly what you did to http://fpaste.org or similar
16:34 glusterbot Title: Fedora Pastebin (at fpaste.org)
16:41 bulde joined #gluster
16:45 Mo_ joined #gluster
16:49 duerF joined #gluster
16:50 shylesh joined #gluster
16:51 _pol joined #gluster
16:56 _pol joined #gluster
16:57 dustint joined #gluster
17:05 _pol joined #gluster
17:23 nirik joined #gluster
17:24 nirik ok, new issue. ;) When trying to heal over to a new readded brick, it heals some things fine, but its not syncing some top level dirs to the new brick, and giving self heal errors for all the stuff under that.
17:31 wushudoin left #gluster
17:35 Humble joined #gluster
17:36 displaynone joined #gluster
17:37 nirik "background  meta-data data entry missing-entry gfid self-heal failed on" :(
17:39 ProT-0-TypE does an active active nfs export works with gluster + ctdb + rrdns?
17:39 cw joined #gluster
17:43 alex88 hi guys, I've a volume that I'm unable to delete, it says it is unsuccesfull, is there a way to force delete it?
17:47 alex88 found it :)
17:48 y4m4 joined #gluster
17:48 shylesh joined #gluster
17:50 go2k joined #gluster
17:55 _pol_ joined #gluster
17:59 nirik so, my problem seems to be now: "meta-data data entry missing-entry gfid self-heal failed" and indeed the two bricks have different gfid on those files. How best to fix it?
18:00 * nirik guesses he can bring one down and try removing those files so they replicate again perhaps.
18:00 semiosis nirik: glusterfs version?
18:01 nirik 3.3.0
18:03 semiosis do the files on the bricks have the same content?
18:03 semiosis same metadata (perms/owners & times)?
18:03 semiosis any other differences besides gfid?
18:05 nirik so, here's the story:
18:05 Ryan_Lane joined #gluster
18:05 nirik 2 nodes, 2 bricks, replicated.
18:06 nirik we noticed today that one brick was not being used. (glusterfsd wasn't running on the node and it had a dns/hostip issue).
18:06 nirik so, I fixed that and restarted it, but it's not really healing very happily. lots of errors.
18:07 nirik so, perhaps I should remove the old brick entirely and add a new one?
18:10 nirik "reason: lookup detected pending operations" makes me think the errors are just that it has other pending things first, but then why is it an error. ;(
18:17 jbrooks joined #gluster
18:21 morse joined #gluster
18:23 nirik ok, I think it's back on track now. Not fully sure why it was doing that... but it's replicating data ok now.
18:30 _pol joined #gluster
18:32 wrale joined #gluster
18:36 glusterbot New news from resolvedglusterbugs: [Bug 764890] Keep code more readable and clean <http://goo.gl/p7bDp>
18:46 disarone joined #gluster
18:55 lpabon joined #gluster
19:15 edong23 joined #gluster
19:41 robos joined #gluster
19:44 jcaputo_ k
19:51 nirik left #gluster
19:55 disarone joined #gluster
20:04 jcaputo joined #gluster
20:06 jcaputo_ joined #gluster
20:11 jdarcy joined #gluster
20:16 jcaputo_ hmmm
20:16 jcaputo_ it seems after I deleted those files, i'm still receiving the error
20:19 _pol joined #gluster
20:19 jcaputo_ http://dpaste.com/1027023/
20:19 glusterbot Title: dpaste: #1027023: errors still, by joe (at dpaste.com)
20:19 jcaputo_ so that ouput is after I deleted the bad files
20:20 jcaputo_ and after running gluster volume heal gv_savedat
20:20 _pol joined #gluster
20:20 JoeJulian did you also delete the .glusterfs counterparts?
20:20 jcaputo_ is @ndevos still around? :)
20:21 jcaputo_ hi Joe!
20:21 jcaputo_ well, i deleted
20:21 jcaputo_ rm -f /mnt/brick3/.glusterfs/b4/0b/b40b​a46e-8779-4c85-aade-0467fb5cc675
20:22 jcaputo_ so i basically did this Joe:
20:22 jcaputo_ http://dpaste.com/1027030/
20:22 glusterbot Title: dpaste: #1027030: command ran, by joe (at dpaste.com)
20:27 JoeJulian Is that the 6th brick in the volume info?
20:28 jcaputo_ actually Joe, it might be good now, i just ran gluster volume heal gv_savedata info
20:28 jcaputo_ and all bricks return 0 now
20:28 jcaputo_ so i may be in the clear...
20:29 jcaputo_ i think
20:29 jcaputo_ let me check the volume info quickly
20:29 JoeJulian I think it sometimes tries to heal metadata before the gfid file, causing it to fail because the file doesn't exist.
20:29 JoeJulian That's just a guess though. I haven't actually traced it out.
20:30 jcaputo_ ok
20:30 JoeJulian Later when the gfid hardlink is created, it doesn't need to heal the metadata because the file already had it.
20:30 jcaputo_ yes it does show in the volume info
20:31 jcaputo_ so after the deletion of those two files, and running 'gluster volume heal gv_savedata', is that all i need to do?
20:31 JoeJulian should be
20:31 jcaputo_ kk
20:32 jcaputo_ i'm going to run 'gluster volume heal gv_savedata info'
20:32 jcaputo_ i was getting inode errors before...
20:33 jcaputo_ would you able to guess why that file was left over and couldnt' recover itself?
20:35 JoeJulian no clue
20:37 jcaputo_ so you have not seen that before eh
20:37 jcaputo_ darn
20:37 jcaputo_ yeah, didn't exist on any nodes except that one
20:49 jcaputo_ Joe - thanks for the help
20:49 jcaputo_ yet again :)
20:50 rwheeler joined #gluster
20:51 jcaputo_ when you decide you want to check out Australia, let me k now and I'll make sure you're taken care of ;)
20:51 jcaputo_ you can stay with us
21:21 tryggvil joined #gluster
21:23 lh joined #gluster
21:23 lh joined #gluster
21:27 wenzi_ joined #gluster
21:28 _pol joined #gluster
21:35 edward1 joined #gluster
21:35 go2k joined #gluster
21:36 go2k Norky: I use gluster 3.2.7 and there is no heal option in the gluster "submenu" of commands. Where would I find it? In 3.3.0 ?
21:36 H__ yes
21:37 H__ in 3.2 branch heal is triggered by stat of the files
21:37 go2k so someone has basically do "ls -la" from time to time, right?
21:37 go2k :D
21:41 JoeJulian yes
21:41 semiosis ~targeted self heal | go2k
21:41 glusterbot go2k: http://goo.gl/E3b2r
21:41 semiosis that
21:41 JoeJulian Theoretically you should know if a server was offline.
21:41 semiosis poor man's self-heal daemon
21:45 go2k yeah, I will know it... right. And thanks for the link. Basically, then, the procedure is to run the command and wait until its completion.
21:45 go2k Nevertheless I think I'll update to gluster 3.3.*
21:47 _pol joined #gluster
21:52 lh joined #gluster
21:52 jcaputo joined #gluster
21:52 jcaputo_ joined #gluster
21:54 samppah any idea what gid-timeout mount option does actually? mentioned here http://shell.gluster.com/~vijay/GlusterFS-3.4.pdf and https://bugzilla.redhat.com/show_bug.cgi?id=902350
21:54 glusterbot <http://goo.gl/zv3Ef> (at shell.gluster.com)
21:54 glusterbot Bug 902350: unspecified, medium, ---, divya, CLOSED NOTABUG, The mount option gid-timeout should be documented
22:05 duffrecords joined #gluster
22:06 duffrecords I'm starting to see lots of "no active sinks for performing self-heal on file" messages in glustershd.log and several of the VMs that are stored on this volume have become unresponsive.  what does this message mean?
22:10 _pol joined #gluster
22:11 _pol joined #gluster
22:25 manik joined #gluster
22:35 semiosis duffrecords: sounds like maybe a brick is down?
22:35 semiosis gluster volume status
22:40 duffrecords all bricks are online
22:40 duffrecords turns out the VMs being unresponsive was because of an LDAP hiccup but the "no active sinks" thing has me concerned
22:44 JoeJulian Get you get a "getfattr -m . -d -e hex" and a stat of the file that there are no active sinks for?
22:45 vex Is there any chance gluster isn't doing synchronous writes between two nodes? i.e. I write a file to a mount (mounted with gluster client) and there can be up to 60seconds delay till the file is accessable by clients requesting it
22:46 JoeJulian My guess, especially since that's an INFO message, is that glustershd checked that file because the xattrs showed pending. They were probably both pending the same update simultaneously (not unexpected during a write) so neither were "sinks" for the file because both were pending.
22:51 JoeJulian There is a chance, depending on cache and posix options, yes.
22:51 JoeJulian If you need a file flushed, flush the file or include the appropriate options on open.
22:53 vex JoeJulian: how do you mean?
22:56 duffrecords JoeJulian: I ran both the commands you mentioned.  what am I looking for?
23:05 JoeJulian vex: google O_SYNC, fsync, fadvise. man sync.
23:05 vex thx ;)
23:06 JoeJulian duffrecords: Conflicting non-zero afr entries, differing sizes, times, owners, etc.
23:06 jdarcy joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary