Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-05-08

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:16 chirino_m joined #gluster
00:19 gmcwhist_ joined #gluster
00:21 bala joined #gluster
00:28 jcsp joined #gluster
00:43 bala joined #gluster
00:46 theron joined #gluster
00:52 ceiphas joined #gluster
00:55 yinyin joined #gluster
00:59 gdubreui joined #gluster
01:05 theron_ joined #gluster
01:12 badone joined #gluster
01:15 jmarley joined #gluster
01:15 jmarley joined #gluster
01:21 Honghui joined #gluster
01:22 DV joined #gluster
01:30 gdubreui joined #gluster
01:55 glusterbot New news from newglusterbugs: [Bug 1095525] GlusterFS MacOSX FUSE loop on symlink <https://bugzilla.redhat.co​m/show_bug.cgi?id=1095525>
02:02 harish joined #gluster
02:09 DV__ joined #gluster
02:12 sputnik1_ joined #gluster
02:16 gdubreui joined #gluster
02:21 askb joined #gluster
02:33 aviksil joined #gluster
02:36 Licenser joined #gluster
02:39 sprachgenerator joined #gluster
02:41 yinyin_ joined #gluster
02:53 ceiphas_ joined #gluster
03:13 gmcwhistler joined #gluster
03:15 NCommander joined #gluster
03:18 Honghui joined #gluster
03:29 kanagaraj joined #gluster
03:31 RameshN joined #gluster
03:38 itisravi joined #gluster
03:39 Dasberger joined #gluster
03:42 bharata-rao joined #gluster
03:46 badone joined #gluster
03:53 kumar joined #gluster
03:57 shubhendu joined #gluster
04:18 ndarshan joined #gluster
04:19 nthomas joined #gluster
04:19 nishanth joined #gluster
04:29 dusmant joined #gluster
04:30 hagarth joined #gluster
04:40 Honghui joined #gluster
04:44 kdhananjay joined #gluster
04:45 atinmu joined #gluster
04:47 saurabh joined #gluster
04:47 sputnik1_ joined #gluster
04:47 sputnik1_ joined #gluster
04:50 aviksil joined #gluster
04:51 kanagaraj joined #gluster
04:52 haomaiwang joined #gluster
04:52 ktosiek joined #gluster
04:54 DV joined #gluster
04:57 bala joined #gluster
05:06 ppai joined #gluster
05:06 vpshastry joined #gluster
05:09 badone joined #gluster
05:14 prasanthp joined #gluster
05:22 raghu joined #gluster
05:33 nshaikh joined #gluster
05:33 kanagaraj_ joined #gluster
05:36 sputnik1_ joined #gluster
05:37 surabhi joined #gluster
05:37 rjoseph joined #gluster
05:38 lalatenduM joined #gluster
05:42 dusmant joined #gluster
05:44 aravindavk joined #gluster
05:50 davinder joined #gluster
05:53 hagarth joined #gluster
05:54 chirino joined #gluster
05:55 ricky-ti1 joined #gluster
05:56 jbrooks joined #gluster
05:58 ricky-ticky1 joined #gluster
06:02 abelur_ joined #gluster
06:05 chirino joined #gluster
06:10 glusterbot New news from resolvedglusterbugs: [Bug 824233] Stick to IANA standard while allocating brick ports <https://bugzilla.redhat.com/show_bug.cgi?id=824233>
06:22 DV joined #gluster
06:24 JoeJulian @later tell MeatMuppet https://bugzilla.redhat.co​m/show_bug.cgi?id=1089758
06:24 glusterbot JoeJulian: The operation succeeded.
06:24 glusterbot Bug 1089758: high, unspecified, ---, pkarampu, ASSIGNED , KVM+Qemu + libgfapi: problem dealing with failover of replica bricks causing disk corruption and vm failure.
06:24 sputnik1_ joined #gluster
06:26 glusterbot New news from newglusterbugs: [Bug 1095594] Stick to IANA standard while allocating brick ports <https://bugzilla.redhat.co​m/show_bug.cgi?id=1095594> || [Bug 1095595] Stick to IANA standard while allocating brick ports <https://bugzilla.redhat.co​m/show_bug.cgi?id=1095595> || [Bug 1095596] Stick to IANA standard while allocating brick ports <https://bugzilla.redhat.co​m/show_bug.cgi?id=1095596>
06:26 rjoseph joined #gluster
06:26 JoeJulian @later tell MeatMuppet No, there is no way to do a rebalance that only creates files on the newly added bricks. Shouldn't be too difficult to add that feature though. I can think of a couple of ways to make that work off the top of my head.
06:26 glusterbot JoeJulian: The operation succeeded.
06:27 dusmant joined #gluster
06:30 kanagaraj joined #gluster
06:30 davinder2 joined #gluster
06:30 JoeJulian ktosiek: When healing many files, if you lookup more files than the background-self-heal queue, the next one cannot be backgrounded and will heal in the foreground. This will cause the pid that triggered the lookup to hang until the self-heal is complete.
06:31 nishanth joined #gluster
06:31 ktosiek JoeJulian: thanks
06:32 hagarth joined #gluster
06:32 ktosiek took me a minute to remember the context for that message tough :-)
06:32 JoeJulian I hope that answered your question. I didn't scroll all the way back.
06:32 ktosiek can I check if that's what is happening?
06:32 JoeJulian Should be logged in the client log.
06:33 ktosiek ok, I'll check that at the next hang :-)
06:33 JoeJulian Sorry to just point you in the general direction and run, but I've driven a couple hundred miles on top of a full work day and I'm heading to bed.
06:33 nthomas joined #gluster
06:34 ktosiek JoeJulian: no problem, I don't feel entitled to stop you :-)
06:34 ktosiek good night
06:34 JoeJulian :)
06:36 nishanth joined #gluster
06:38 nishanth joined #gluster
06:38 nthomas joined #gluster
06:39 davinder joined #gluster
06:39 nishanth joined #gluster
06:40 ktosiek hmm, new problems: after running out of inodes, changing tha quota, and restarting the container with glusterfs server, I'm getting "Failed to get extended attribute trusted.glusterfs.volume-id for brick dir [...] Reason : No data available" when starting one of my volumes (only one of them, and even not the one that caused the inode exhaustion)
06:42 badone joined #gluster
06:51 aviksil joined #gluster
06:53 pdrakeweb joined #gluster
06:54 ghenry joined #gluster
06:54 ghenry joined #gluster
06:56 MrAbaddon joined #gluster
06:56 glusterbot New news from newglusterbugs: [Bug 1078847] DHT- mkdir fails and gives error, 'invalid argument' if hashed sub-volume is down <https://bugzilla.redhat.co​m/show_bug.cgi?id=1078847>
07:00 ekuric joined #gluster
07:01 ctria joined #gluster
07:07 warci joined #gluster
07:07 vcauw_ joined #gluster
07:12 aravindavk joined #gluster
07:12 vpshastry joined #gluster
07:23 liquidat joined #gluster
07:33 fsimonce joined #gluster
07:36 rahulcs joined #gluster
07:37 psharma joined #gluster
07:43 aravindavk joined #gluster
07:48 rahulcs_ joined #gluster
07:52 ghghz joined #gluster
07:52 ghghz Hey, what can be the problem? I can create directory, but can't remove it?
07:52 ghghz Transport endpoint is not connected
08:06 andreask joined #gluster
08:07 ngoswami joined #gluster
08:36 MrAbaddon joined #gluster
08:41 davinder joined #gluster
08:52 rastar joined #gluster
09:02 aviksil joined #gluster
09:02 Askon joined #gluster
09:03 Askon Hi there, is anybody can halp me with qemu+glusterfs related question?
09:05 DV joined #gluster
09:06 Askon My problem is mapped share mode for qemu guests....it has very slow performace in mapped mode
09:06 Askon I can't use passthrought, because qemu run as unpriveleged user, qemu controls by libvirt
09:07 tryggvil joined #gluster
09:08 Askon According to qemu-devel - "With mapped security mode we use "user.virtfs" namespace is used to store the virtFs related attributes. So hide it from user." but this behavor give me alot of errors in briks log file -   getxattr failed on ...: user.virtfs.gid (No data available)
09:09 Askon I have set diagnostics.brick-log-level: CRITICAL, it reduced log file size but performance of gluster share on qemu guest is very poor
09:10 Askon dd if=/dev/zero of=./test bs=1M count=1000 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 8.95105 s, 117 MB/s
09:10 Askon dd if=./test of=/dev/null 2048000+0 records in 2048000+0 records out 1048576000 bytes (1.0 GB) copied, 188.211 s, 5.6 MB/s
09:14 lalatenduM Askon, are u using fuse mount of gluster ?
09:15 lalatenduM fuse mount is same as native glusterfs mount
09:17 Askon Yes, I'm using fuse
09:17 Askon hereis my fstab on host - /etc/glusterfs/portage.vol /usr/portage glusterfs rw 0 0
09:18 bazzles joined #gluster
09:19 Askon when I switch to passthrough mode from mapped - no issue with performance, but can't create any new dir or file...
09:20 Askon looks like something works incorrect in chain virtfs<-->glusterfs
09:22 vpshastry joined #gluster
09:30 Askon forgot to notice, I'm using gluster with ZFS (ZoL), xattr=sa
09:34 qdk joined #gluster
09:34 kumar joined #gluster
09:42 lalatenduM Askon, I have never used ZFS
09:55 Askon this is not ZFS problem, because in qemu passthrough no user.virtfs related errors at all
09:57 aravindavk joined #gluster
10:05 tryggvil joined #gluster
10:10 ira joined #gluster
10:14 tryggvil joined #gluster
10:16 bazzles is there any difference in setting up georep in 3.5 compared to 3.4 ?
10:22 hagarth bazzles: yes, there is. Referring to the 3.5 documentation on geo-replication should help - https://github.com/gluster/glusterfs/b​lob/release-3.5/doc/admin-guide/en-US/​markdown/admin_distributed_geo_rep.md
10:22 glusterbot Title: glusterfs/doc/admin-guide/en-US/mar​kdown/admin_distributed_geo_rep.md at release-3.5 · gluster/glusterfs · GitHub (at github.com)
10:26 aviksil joined #gluster
10:27 chirino_m joined #gluster
10:44 Sunghost joined #gluster
10:45 aravindavk joined #gluster
10:50 twx_ joined #gluster
10:50 Amanda_ joined #gluster
10:51 mjrosenb1 joined #gluster
10:51 sputnik1_ joined #gluster
10:51 JustinCl1ft joined #gluster
10:51 Sunghost Hello - sorry this is my first time in irc - hope i did nothing wrong.
10:51 JonathanS joined #gluster
10:53 rahulcs joined #gluster
10:54 Moe-sama joined #gluster
10:54 crashmag_ joined #gluster
10:54 RobertLaptop_ joined #gluster
10:54 xiu_ joined #gluster
10:55 james__ joined #gluster
10:55 ctria joined #gluster
10:55 hagarth @channelstats
10:55 glusterbot hagarth: On #gluster there have been 300211 messages, containing 12056800 characters, 1993764 words, 7405 smileys, and 1013 frowns; 1556 of those messages were ACTIONs. There have been 128293 joins, 3699 parts, 124707 quits, 24 kicks, 289 mode changes, and 7 topic changes. There are currently 220 users and the channel has peaked at 239 users.
10:55 codex joined #gluster
10:57 Sunghost Need some help for replacing a brick who has die
11:02 rahulcs joined #gluster
11:03 shubhendu_ joined #gluster
11:04 ghghz_ joined #gluster
11:04 Sunghost #help
11:05 Sunghost joined #gluster
11:05 Honghui joined #gluster
11:10 ceiphas_ what could be the cause for following problem: Server A and Server B form a volume with replication. Client C mounts this volume over FUSE. When i delete a file on C it reappears after about 5 minutes. All log files are empty, gluster volume heal <volume> info shows nothing. What should i do to get more information for debugging this behavior?
11:17 Sunghost Hello i have a question about rescue a brick with failed boot on raid on distributed. I can only boot over live-cd and see the raw files of the volume. the question is can i simply backup the data reinstall the brick, join the volume and copy the files back or what whould be the best way?
11:27 harish joined #gluster
11:28 gmcwhistler joined #gluster
11:28 jcsp joined #gluster
11:30 crashmag joined #gluster
11:33 ctria joined #gluster
11:33 haomaiw__ joined #gluster
11:38 Honghui joined #gluster
11:40 vpshastry joined #gluster
11:42 atinmu joined #gluster
11:43 ceiphas what could be the cause for following problem: Server A and Server B form a volume with replication. Client C mounts this volume over FUSE. When i delete a file on C it reappears after about 5 minutes. All log files are empty, gluster volume heal <volume> info shows nothing. What should i do to get more information for debugging this behavior?
11:43 lalatenduM Sunghost, what type of volume u r using?
11:44 Sunghost hi, distributed
11:44 Sunghost data spread over 3 bricks
11:44 Sunghost had kernel panic on brick1
11:44 Sunghost after that raid6 = md0 for root was damage
11:44 Sunghost i could recreate it but after the reboot grub reports error and shows grub rescue mode
11:45 lalatenduM Sunghost, backup the data reinstall the brick, join the volume and copy the files back wil not work
11:45 Sunghost ok thats what i thought but never done before
11:46 Sunghost for me to reunderstand - i whould mount the md2 on a live-cd and copy all files in my folder vol1 to a backup disk, right
11:46 Sunghost after that i can kill brick one and reinstall him?
11:46 Sunghost after that install gluster, and join the volume
11:46 lalatenduM Sunghost, I mean you can't just copy back data to brick partition and expect gluster recognize the files
11:47 lalatenduM Sunghost, 1st beackup the data from failed brick
11:47 burn420 joined #gluster
11:47 lalatenduM 2nd, do a remove-brick for the failed brick (use force option)
11:47 Sunghost ok that would be the data in my vol dir
11:47 lalatenduM 3rd, do a peer detach for the failed hots
11:47 lalatenduM s/hots/host/
11:47 glusterbot What lalatenduM meant to say was: 3rd, do a peer detach for the failed host
11:48 primechuck joined #gluster
11:48 lalatenduM 4, then re install the host, fix hard ware issue
11:48 lalatenduM 5. do a peer probe for the reinstalled hots
11:49 lalatenduM 6. do add-brick for the new brick
11:49 lalatenduM 7. Mount the gluster volume somewhere, and copy back the files from the backup
11:49 lalatenduM <end>
11:50 * ceiphas applaudes
11:50 lalatenduM Sunghost, that's the reason you should use replicate distribute volume when you care for data, so that these kind of issues will be automatically handled
11:51 ceiphas what could be the cause for following problem: Server A and Server B form a volume with replication. Client C mounts this volume over FUSE. When i delete a file (doesn't matter on which machine, A, B or C) it reappears after about 5 minutes. All log files are empty, gluster volume heal <volume> info shows nothing. What should i do to get more information for debugging this behavior?
11:51 lalatenduM ceiphas, yeah looking in to ur issue
11:51 Sunghost yes you are right i know this, but in this case the money was important sorry was not my decision
11:52 Sunghost but i am a bit confused above you wrote "you cant just copy back data to brick"
11:52 Sunghost but as i understand i will do so
11:52 Sunghost short for recapitulation: 1. backup volume data <- direct folder from mounted brick
11:52 lalatenduM ceiphas, try a self-heal force, i ca't remember the exact command, but let force gluster to do forceful selfheal
11:53 lalatenduM s/let/lets/
11:53 glusterbot What lalatenduM meant to say was: ceiphas, try a self-heal force, i ca't remember the exact command, but lets force gluster to do forceful selfheal
11:54 lalatenduM ceiphas, so that self-heal will crawl across all files and see if something needs to self-healed
11:54 Sunghost 2. remove brick 3. reinstall host 4 add host to volume 5. mount the volume and copy the files back
11:54 lalatenduM Sunghost, you need to copy back data to the mount point, not to the brick directly
11:55 Sunghost ok understand -> the files on brick1 are in plain so i simply can copy that and after the new brick is installed i mount the same old volume = vol1 but copy the files over the mountpoint onto the vol1 NOT DIRECT ON BRICK1 <- right
11:56 Sunghost lalatenduM ;) ok - there is no metadata or something like that which would prevent this - ok its a simple plain copy of files onto the volume not the brick itself
11:56 lalatenduM Sunghost, yes, Because when you copy on to the mount point, gluster will be aware of these files But if you copy files to backend i.e. brick directly , gluster will never come to know abt them
11:57 Sunghost ok that is what i understand so far too from glusterfs, but i was unsure how to handle it in this case, while i thougt there could be a kind of metafile or something like that which would not recognised this files or has the old one still allive
11:57 lalatenduM Sunghost, yes :) thats the beauty of Glusterfs implementation , the meta data is stored with file itself as extended metadata. When you copy back the files on to the mount point metadata wil be recreated according to the current  layout
11:58 lalatenduM s/extended metadata/extended attributes/
11:58 glusterbot What lalatenduM meant to say was: Sunghost, yes :) thats the beauty of Glusterfs implementation , the meta data is stored with file itself as extended attributes. When you copy back the files on to the mount point metadata wil be recreated according to the current  layout
11:59 Sunghost ah ok and i wasted over 4 days and nights to recreate the raid and partitiontable - if it that simple i had done it earlier
12:01 ceiphas lalatenduM: i tried "gluster volume heal dbtemp full" and the "find . | xargs stat" method to start a full heal, didn't work the files keep reappearing
12:01 Sunghost perhaps a short hint, i use on all my glusterfs ext4 and i offen read that you have to choose xfs for glusterfs, while performance and some other facts, what do you think? should i install the brick with xfs or still use ext4?
12:02 itisravi_ joined #gluster
12:03 lalatenduM Sunghost, xfs is definitely better, but I will suggest same fs across all bricks in a setup
12:03 ceiphas lalatenduM: i think i deleted the files about 50 times now
12:03 ndarshan joined #gluster
12:03 ceiphas is there any debugging option to check if the files get deleted on a brick?
12:04 lalatenduM Sunghost, usually with ext4 will brought issues like:  no space for extended attributes or inodes  later when your setup grows very large
12:05 lalatenduM ceiphas, you can check the bricks ,
12:05 lalatenduM ceiphas, just go to the partition and check
12:05 Sunghost thanks so much for this information - this is my first time in irc and after minutes of wating i thought i get no help, but now i think this is a good think and your are so helpfull - thanks
12:05 ceiphas the files are gone from the brick dirs on both machines
12:05 lalatenduM s/partition/brick partition/
12:05 glusterbot What lalatenduM meant to say was: ceiphas, just go to the brick partition and check
12:05 andreask joined #gluster
12:06 ceiphas lalatenduM: aaaand the files are back again
12:06 lalatenduM Sunghost, np.
12:06 Sunghost lalatenduM - your right that was my initial problem with ext4 and limit to 16tb in old version of efs2progs, newer version good to more than 16tb
12:07 lalatenduM ceiphas, may be the volume has gone mad, try restarting glusterd and the volume too
12:07 lalatenduM on all nodes
12:08 Sunghost ok -thanks alot - i will try xfs and rebuild step by step all bricks - @ceiphas excuse for interrupting - bye to all
12:09 lalatenduM Sunghost, do you have link to the admin guide for gluster
12:10 lalatenduM Sunghost, wait for 5 mins, I trying similar steps on a test setup :)
12:10 ceiphas lalatenduM: so unmount the volume from from all clients, stop it, stop glusterd and then all back up?
12:11 abelur_ joined #gluster
12:11 Sunghost yes i know the guide from redhat and the gluster online documentation
12:12 lalatenduM ceiphas, if you stop glusterd on all nodes IO will fail on clients, so I dont think we need to unmount gluster volume
12:12 Sunghost but it is like all things if it went good and you have no problems the guides and knowledge are enough but if things go wrong you need experience or a good documention of the same situation with steps to the solution
12:12 Sunghost lalatendum <- thanks but are you unsure about that? ;)
12:13 lalatenduM Sunghost, I am sure about it, but doing again just to be more sure
12:13 sauce joined #gluster
12:13 Sunghost i deleted my test vm for 1min ;) after the final answer from you i think it sounds good ;)
12:15 ceiphas lalatenduM: i stopped glusterd on all machines, but still can access all the files...
12:15 lalatenduM ceiphas, may be they are cached in the client
12:15 ceiphas lalatenduM: i think the gluster "server" is still running
12:16 ceiphas lalatenduM: glusterfsd and glusterfs are still up
12:16 ceiphas lalatenduM: should i kill them?
12:16 lalatenduM Sunghost, we are good :) it worked on my setup
12:17 Sunghost no you are good - fast and good in technic ;)
12:17 lalatenduM ceiphas, use pkill glusterd, pkill glusterfsd, pkill gluster
12:18 lalatenduM ceiphas, these things will kill all process related to gluster
12:18 basso joined #gluster
12:18 uebera|| joined #gluster
12:18 tjikkun_work joined #gluster
12:18 lalatenduM then in all nodes start glusterd "service start glusterd"
12:18 ceiphas now i get socket not connected
12:19 lalatenduM ceiphas, where?
12:19 Sunghost are you the first supporter of glusterfs and have many prepaired testsystems or why you are so fast with testing ;)
12:19 ceiphas on all clients, even after starting glusterd
12:20 lalatenduM Sunghost, I work for Glusterfs :) (Red Hat pays me for that)
12:20 ceiphas Sunghost: i have three test vms, too. kvm makes testing easy
12:21 ppai joined #gluster
12:21 Sunghost Good Job +1, i whould do it in your case exactly the same way - virtualisation and kvm too is a good thing - nice job bye
12:21 lalatenduM ceiphas, you are running any application on the mount points ?
12:22 ceiphas lalatenduM: maybe
12:22 ceiphas lalatenduM: i remounted the mount points
12:22 lalatenduM ceiphas,  have you started the volume?
12:22 ceiphas i didn't stop it
12:23 ceiphas you said i should just restart glusterd
12:23 lalatenduM ceiphas, ok np, lets restart the volume now
12:23 ceiphas the volume works fine now...
12:23 ceiphas lets see how the deleting works
12:24 lalatenduM ceiphas, my orig plan was to restart glusterd and the volume too
12:24 ceiphas oh, ok
12:24 ceiphas should i restart again?
12:25 lalatenduM ceiphas, if it is working for you, lets not disturb it :)
12:25 ceiphas let's see if the files reappear
12:25 lalatenduM ceiphas, which gluster version you are using?
12:25 ceiphas 3.4.2
12:25 lalatenduM ok
12:25 ceiphas the files are back
12:26 lalatenduM :/
12:26 ceiphas i got this one for a short time:
12:27 ceiphas a file named .201-f8h4.dat.T55Lr7 existed for about 1 sec and was then 201-f8h4.dat
12:27 shubhendu_ joined #gluster
12:27 ceiphas is that a self healing thing?
12:28 lalatenduM ceiphas, you are deleting the file from the mount poinbt ?
12:28 lalatenduM s/poinbt/point/
12:28 ceiphas yes, fuse mounted
12:28 glusterbot What lalatenduM meant to say was: ceiphas, you are deleting the file from the mount point ?
12:28 ceiphas after deletion the files are gone from the mount point and the bricks but after about five minutes they are back as if they haunt me
12:29 Askon Guys, anyone - I have tons of this in my logs: E [posix.c:2668:posix_getxattr] 0-portage-posix: getxattr failed on /CLOUD/portage/: user.virtfs.rdev (No data available)
12:29 ceiphas my boss gets really angry... i persuaded him to use a clustered fs for this, and now it fails in deleting files
12:30 lalatenduM ceiphas, can you pastebin "gluster v <vol> status", also "gluster v heal <volname> info"
12:30 Ark joined #gluster
12:30 Askon every ls -l fills logs with this strings, log file grow ~5M in 30 sec.
12:31 mjsmith2 joined #gluster
12:31 lalatenduM Askon, this ur qemu-kvm setup right?
12:32 Askon lalatenduM, yes
12:32 bala1 joined #gluster
12:32 lalatenduM Askon, I have less knowledge abut qemu kvm, Humble might know
12:33 lalatenduM ceiphas, not sure if something wrong with the setup, I never heard of this issue for 3.4.2
12:33 ndarshan joined #gluster
12:33 Askon lalatenduM, thx! Will pm him
12:33 ceiphas lalatenduM: http://pastebin.com/vpQdFySk http://pastebin.com/KRXAat4f
12:34 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
12:34 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
12:34 ceiphas i know, but i use pastebin, thank you
12:34 lalatenduM ceiphas, if you want , you can file a bug for it, I wil try to get the bug to some dev in gluster ..http://www.gluster.org/community/document​ation/index.php/Bug_reporting_guidelines
12:34 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
12:37 lalatenduM ceiphas, you are manually deleting the files right? I hope no other client creating the file you are deleting
12:38 ceiphas lalatenduM: this is the last thing i have to check, the script that generates these files runs untouched since 1996 but still it might behave different now
12:38 lalatenduM ceiphas, check /var/log/glusterfs/glustershd.log, also pastebin the log
12:44 bala1 joined #gluster
12:45 rahulcs joined #gluster
12:46 ndk joined #gluster
12:49 lalatenduM ceiphas, it very unlikely that glusterfs is bringing back the files as if the files are deleted from brick (after deleting from the mount point) , glusterfs does have any info on these fiels
12:49 lalatenduM files*
12:49 ceiphas lalatenduM: /var/log/glusterfs/glustershd.log > http://paste.ubuntu.com/7415811/
12:49 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
12:50 ceiphas lalatenduM: i just created a file on the volume and it got deleted about 10sec later
12:51 d-fence joined #gluster
12:51 lalatenduM ceiphas, ;( bad,
12:51 lalatenduM ceiphas, I can see "E [afr-self-heald.c:1067:afr_find_child_position] 0-dbtemp-replicate-0: getxattr failed on dbtemp-client-0 - (Transport endpoint is not connected)" in the logs
12:51 ceiphas lalatenduM: and nothing in the logs
12:52 ceiphas lalatenduM: i killed gluster, you remember?
12:52 lalatenduM ceiphas, ohh the old log, cool
12:52 lalatenduM ceiphas, check output of gluster v heal <volume> info split-brain
12:53 ceiphas o for both bricks
12:53 ceiphas 0
12:53 ceiphas why are two keys with similar letters next to each other
12:53 lalatenduM ceiphas, you mean if you do "touch a1" on the mount point, a1 disappears after some time
12:53 ceiphas lalatenduM: yes
12:54 lalatenduM ceiphas, which keys
12:54 lalatenduM ?
12:54 ceiphas o0
12:54 ceiphas on your keyboard
12:55 ceiphas at least on a german keyboard, don't know for others
12:55 sjm joined #gluster
12:56 lalatenduM ceiphas, lets restart the volume now
12:56 ceiphas i will now stop the volume stop glusterd and kill all remaining processes and bring everything up again, let's see what happens
12:56 ceiphas you won
12:56 lalatenduM ceiphas, yeah lets try that
13:00 chirino joined #gluster
13:01 ceiphas lalatenduM: f*ck there are still processes accessing the volume... means i cannot stop now
13:01 bennyturns joined #gluster
13:01 lalatenduM ceiphas, how many clients u have?
13:04 ceiphas lalatenduM: about 300 and some machines that use the data on the volume to produce goods
13:05 ceiphas windows to be prcise
13:05 ceiphas ceiphas: i cannot kill samba and dovecot just to check this
13:05 ceiphas why do i talk to myself
13:06 lalatenduM ceiphas, just restart the volume
13:06 lalatenduM ceiphas, "gluster v start <volume> force"
13:08 harish joined #gluster
13:08 lalatenduM ceiphas, with start force the volume will be unavailable for very less time, there is high chance that they will not notice the restart
13:09 lalatenduM ceiphas, sorry just came to know that  "gluster v start <volume> force" does not stop and start, it only start brick process if they are not running
13:12 tdasilva joined #gluster
13:12 B21956 joined #gluster
13:17 rahulcs joined #gluster
13:18 ceiphas lalatenduM: this is the worst case for me... if i just had a log which said "you dumbass, you did xy wrong" but nothing, gluster pretends that everything is fine
13:18 ceiphas if i just could debug the moment in which the file gets re-created
13:19 dusmant joined #gluster
13:20 jag3773 joined #gluster
13:21 lalatenduM ceiphas, sorry cant help you with ur issue, as a lastthing I wan to see glusterd logs /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
13:22 lalatenduM ndevos, are u back from vacation ?
13:22 kaptk2 joined #gluster
13:22 ndevos lalatenduM: I'm not on vacation, just in a different timezone and with very bad internet
13:23 lalatenduM ndevos, ohh
13:23 jobewan joined #gluster
13:24 ceiphas hi ndevos
13:24 * ndevos _o/ ceiphas
13:24 ceiphas i have a new bug, maybe you can help there, too
13:25 ceiphas lalatenduM: /var/log/glusterfs/etc-glusterfs-glusterd.vol.log has lots of [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) i grep them off to get to the point
13:25 ndevos ceiphas: what kind of problem is that?
13:26 lalatenduM ndevos, ceiphas is seeing a weird issue. it would good to get ur views on that
13:27 ceiphas ndevos: i have a volume with your patched gluster mounted
13:27 ceiphas if i delete files from the volume they reappear some minutes later
13:27 ceiphas if i create files they disappear some minutes later
13:27 ceiphas logs are clean
13:28 ndevos ceiphas: uh, hmm, sounds like something in the replication/healing processes
13:28 tdasilva joined #gluster
13:28 ceiphas ht/var/log/glusterfs/etc-g​lusterfs-glusterd.vol.log > tp://paste.ubuntu.com/7416029/
13:28 ceiphas /var/log/glusterfs/etc-glusterfs-glusterd.vol.log > http://paste.ubuntu.com/7416029/
13:28 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:29 ndarshan joined #gluster
13:29 ndevos ceiphas: I think checking the glustershd.log (self-heal-daemon) would give mode info
13:30 hchiramm_ joined #gluster
13:30 ceiphas /var/log/glusterfs/glustershd.log > http://paste.ubuntu.com/7415811/
13:30 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:30 ceiphas ndevos: i killed gluster once, that's the cause for the connection errors
13:31 ira joined #gluster
13:32 plarsen joined #gluster
13:32 Guest72892 joined #gluster
13:32 dbruhn joined #gluster
13:33 ndevos ceiphas: can you verify that you only have one self-heal-daemon running for that volume?
13:33 japuzzo joined #gluster
13:34 ceiphas ndevos: volume status > http://paste.ubuntu.com/7416057/
13:34 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:36 ceiphas i checked and there are only these two processes with glustershd in the command line
13:36 ceiphas one on each peer
13:38 ndevos ceiphas: you only should have one glustershd per volume
13:38 ceiphas why do i have two?
13:39 rahulcs joined #gluster
13:39 ndevos ceiphas: I suspect that these processes are competing somehow, one creates the missing files, the other thinks those files should have been deleted
13:39 ndevos I have no idea how you an get two of those...
13:39 ceiphas look at my status, you see both of them: http://paste.ubuntu.com/7416057/
13:39 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:40 ceiphas i thought, you have one per brick
13:41 Humble Askon, pm ?
13:42 ndevos ceiphas: hmm, yeah, I have the same thing, seems that it runs on all servers
13:42 lmickh joined #gluster
13:42 ceiphas how would it heal another brick if it didn't run there?
13:43 sjoeboo joined #gluster
13:43 ndevos ceiphas: the daemon contacts too all the bricks in the volume
13:44 ndevos ceiphas: you can check if the self-healing is at fault, just disable it for a while
13:44 ceiphas but if the host witth the shd goes boom, there needs to be another shd
13:44 ceiphas how do i disable shd on this volume?
13:45 ndevos ceiphas: 'gluster volume set $VOL cluster.self-heal-daemon off'
13:46 davinder joined #gluster
13:47 ceiphas ndevos: disabled self heal, let's see if the files reappear
13:48 ndevos ceiphas: you should probably also check if the directory is not in a ,,(split brain) situation
13:48 glusterbot ceiphas: I do not know about 'split brain', but I do know about these similar topics: 'split-brain'
13:48 ndevos bleh, glusterbot!
13:49 ndevos @split-brain
13:49 glusterbot ndevos: (#1) To heal split-brain, use splitmount. http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/, or (#2) For additional information, see this older article http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/
13:49 ceiphas ndevos: i checked volume heal info split-brain but it showed nothing
13:49 sputnik13 joined #gluster
13:49 ndevos ceiphas: check and compare the xattrs for the directory: getfattr -d -m. -ehex /path/to/brick/to/dir
13:51 ceiphas ndevos: look identic to me
13:52 mjsmith2 joined #gluster
13:54 ndevos ceiphas: hmm, okay... lets see if the shd is the issue here, and start them later again
13:54 Askon Just for googlers with same question as mine - accessmode='squash' in libvirt solved my issue with glusterfs to qemu guest with 9p. Thank you, guys!
13:55 ceiphas ndevos: the files reappeared
13:55 ceiphas is there any possibility which process created the file?
13:56 Humble Askon, \o/
13:56 ndevos ceiphas: it can be happening on a LOOKUP/stat() call, that triggers the self-heal too
13:56 ndevos ceiphas: can you check+compar the xattrs for those files too?
13:57 ceiphas ndevos: also identic
13:57 ceiphas ndevos: if i delete the files they are also gone from the brick dir
13:57 glusterbot New news from newglusterbugs: [Bug 1095775] Add support in libgfapi to fetch volume info from glusterd. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1095775>
13:57 ndevos ceiphas: very strange...
13:58 ceiphas there are no processes that could possibly re-create these files all the time
13:58 ndevos ceiphas: it sounds as if the gfid link (under /path/to/brick/.glusterfs/...) is not being deleted
13:59 ceiphas ndevos: how can i check this, or find the link?
13:59 ndevos ceiphas: could it be that this file is a hardlink and is referenced multiple times?
14:00 ndevos ceiphas: the truster.gluster.gfid xattr is the value of the gfid, /path/to/brick/.glusterfs/<1st-2-​bytes>/<2nd-2-bytes>/<full-gfid>
14:00 ceiphas ndevos: the file has a 1 in the second column of ls -al
14:00 ndevos ceiphas: ok
14:02 hagarth joined #gluster
14:02 ceiphas ndevos: the gfid file is gone, too
14:06 ndevos ceiphas: gone on both bricks?
14:06 ceiphas didn't check, wait
14:07 ndevos ceiphas: without the gfid-file, self-heal will not be able to heal anything....
14:08 sroy_ joined #gluster
14:08 ceiphas ndevos: it's gone on both, but the file reappeared with a new gfid
14:08 cvdyoung Hi, is there a 32 group limit with gluster volumes?  And is there a way to work around that?  Thanks!
14:09 ndevos ceiphas: that sounds as if something else is creating those files
14:10 dusmant joined #gluster
14:10 ndevos cvdyoung: the limit is around 93 groups, see http://thread.gmane.org/gmane.com​p.file-systems.gluster.devel/6180 for some details
14:10 glusterbot Title: Gmane Loom (at thread.gmane.org)
14:11 tdasilva left #gluster
14:11 keytab joined #gluster
14:11 ceiphas ndevos: lalatenduM: OMG i'm sooo stupid
14:12 ndevos ceiphas: that sounds as if you found the cause :)
14:12 ceiphas i have the volume mounted to /mount/dbtemp and then a bind mount to /dbtemp. before i had the bind mount i synced /dbtemp to /mount/dbtemp every five minutes
14:13 ceiphas i forgot to delete the cron job
14:13 ndevos lol
14:13 ceiphas strange thing is, that in rsync of a bind mount to the root mount creates files
14:15 ira__ joined #gluster
14:19 jbrooks joined #gluster
14:20 ndevos ceiphas: sorry, what do you mean?
14:21 ceiphas ndevos: /dbtemp is a bind mount to /mount/dbtemp ; i did a rsync from dbtemp to /mount/dbtemp and it re-created deleted files
14:22 ndevos ceiphas: depending on how big those files are, that might be expected - rsync creates a .<original-filename>.<somethingrandom> while doing a transfer
14:23 scuttle_ joined #gluster
14:24 ceiphas ndevos: i experienced these files
14:25 ceiphas ndevos: a file named .201-f8h4.dat.T55Lr7 existed for about 1 sec and was then 201-f8h4.dat
14:25 bala1 joined #gluster
14:25 ndevos ceiphas: righ, that is typical rsync
14:26 ceiphas but the files are not that big, just about 2m
14:26 ndevos ceiphas: rsync does a full directory tree scan first, and then starts transferring the files
14:28 ceiphas aaand i found out that it wasn't a bind mount, as my boss removed the bind mount and a synced an old state every five minutes
14:30 ceiphas sorry for disturbing the cannel for about two days with my stupidity
14:31 mkzero joined #gluster
14:32 tziOm joined #gluster
14:37 LoudNois_ joined #gluster
14:44 rahulcs joined #gluster
14:46 tdasilva joined #gluster
14:51 rwheeler joined #gluster
14:52 lalatenduM ceiphas, no prob, glad that we found the root cause \o/
14:54 haomaiwang joined #gluster
14:54 zaitcev joined #gluster
15:07 jag3773 joined #gluster
15:17 sprachgenerator joined #gluster
15:21 sputnik13 joined #gluster
15:25 GabrieleV joined #gluster
15:26 pdrakeweb joined #gluster
15:30 daMaestro joined #gluster
15:36 hchiramm__ joined #gluster
15:48 plarsen joined #gluster
15:51 jbd1 joined #gluster
15:52 vpshastry joined #gluster
15:58 primechuck joined #gluster
16:02 chirino_m joined #gluster
16:07 davinder joined #gluster
16:08 coredump joined #gluster
16:12 jag3773 joined #gluster
16:13 ProT-0-TypE joined #gluster
16:17 Mo_ joined #gluster
16:19 hagarth joined #gluster
16:19 vpshastry joined #gluster
16:25 theron joined #gluster
16:26 arya joined #gluster
16:31 LoudNoises joined #gluster
16:31 ndk joined #gluster
16:38 naveed joined #gluster
16:40 John_HPC joined #gluster
16:44 ndk joined #gluster
16:46 vpshastry left #gluster
16:47 kanagaraj joined #gluster
16:56 cvdyoung On my volume, I can only have 32 groups be recognized.  Is there a way to increase that?
16:58 aafa joined #gluster
16:59 aafa /msg nickserv register nitro7600
17:00 * aafa wave hello
17:03 aafa First gluster installed today!
17:04 aafa created two node replicated with geo-replication all seems to work
17:06 JoeJulian ghghz: Check your client log to see what the error is.
17:09 hagarth aafa: welcome and good to hear that!
17:12 scuttle_ joined #gluster
17:14 aafa Does gluster allow for simultaneous file updates? Did not see any locking service
17:15 JoeJulian Yes, standard posix locking.
17:16 aafa JoeJulian: Thanks!
17:22 kanagaraj joined #gluster
17:25 GabrieleV joined #gluster
17:27 Ark joined #gluster
17:29 Scott6 joined #gluster
17:36 GabrieleV joined #gluster
17:37 daMaestro joined #gluster
17:44 sjusthome joined #gluster
17:57 aafa left #gluster
17:57 ira joined #gluster
18:01 Ark joined #gluster
18:03 MeatMuppet joined #gluster
18:07 wushudoin| joined #gluster
18:08 wushudoin| joined #gluster
18:09 sputnik13 joined #gluster
18:09 sputnik13 joined #gluster
18:13 ThatGraemeGuy joined #gluster
18:17 JonathanS joined #gluster
18:17 tziOm joined #gluster
18:18 DV joined #gluster
18:19 rahulcs joined #gluster
18:26 dusmant joined #gluster
18:28 glusterbot New news from newglusterbugs: [Bug 1095888] dist-geo-rep: while doing rm -rf on master mount, slave glusterfs crashed in io-cache. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1095888>
18:40 rahulcs joined #gluster
18:41 arya joined #gluster
18:44 arya joined #gluster
18:48 rahulcs joined #gluster
18:56 Ark joined #gluster
19:01 nullck joined #gluster
19:05 rahulcs joined #gluster
19:06 Uguu joined #gluster
19:12 tdasilva joined #gluster
19:15 rahulcs_ joined #gluster
19:22 ktosiek joined #gluster
19:26 rotbeard joined #gluster
19:30 systemonkey If one of my head OS goes belly up and I had to do a new install, would the partition distributed brick lose it's data with other cluster? Would heal fix and reattach to the cluster?
19:31 MrAbaddon joined #gluster
19:31 rahulcs joined #gluster
19:47 Ark joined #gluster
19:49 jbd1 systemonkey: distributed or distributed-replicated?
19:50 systemonkey jbd1: just distributed
19:55 ghghz left #gluster
19:59 failshell joined #gluster
20:07 chirino joined #gluster
20:13 tdasilva joined #gluster
20:21 theron joined #gluster
20:21 wushudoin joined #gluster
20:36 rahulcs joined #gluster
20:37 chirino_m joined #gluster
20:40 lmickh joined #gluster
20:45 rahulcs joined #gluster
20:47 rahulcs joined #gluster
20:49 badone joined #gluster
20:57 naveed joined #gluster
21:05 MrAbaddon joined #gluster
21:19 andreask joined #gluster
21:30 zerick joined #gluster
21:33 gdubreui joined #gluster
21:45 jbd1 joined #gluster
21:45 Ark joined #gluster
21:46 ernetas joined #gluster
21:46 ernetas Hey guys
21:47 ernetas How do I mount glusterfs so that the root folder would remain owned by a specific user?
21:47 ernetas After a reboot it always falls back to root, even though I chowned it before mounting as well...
21:53 ktosiek joined #gluster
22:00 qdk joined #gluster
22:01 failshel_ joined #gluster
22:04 jbd1 joined #gluster
22:09 ernetas Note: this happens only when rebooting. Otherwise, the chown stays...
22:10 chirino joined #gluster
22:12 MrAbaddon joined #gluster
22:14 cvdyoung I am having the same problem, my volume is exported and mounted by a client running 3.5 and the group ownership is changing after a gluster server reboots and comes back up.
22:25 MacWinne_ joined #gluster
22:40 VerboEse joined #gluster
22:47 VerboEse joined #gluster
22:47 sjm joined #gluster
22:57 anotheral left #gluster
23:20 tryggvil joined #gluster
23:22 ira joined #gluster
23:28 [o__o] joined #gluster
23:33 badone joined #gluster
23:43 chirino_m joined #gluster
23:49 JoeJulian ernetas, cvdyoung: what version and what mount options? I have my vmimages volume with the root of the volume owned by nova.nova and I've never had a problem.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary