Camelia, the Perl 6 bug

IRC log for #gluster, 2013-02-12

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:13 mkultras i had this earlier when i had one of the gluster nodes off :  "Skipping entry self-heal because of gfid absence"
00:13 mkultras thats normal if one of the nodes is down right
00:24 glusterbot New news from newglusterbugs: [Bug 910188] swift checks for mount points which are alphanumeric and gluster uses ones with dashes <http://goo.gl/fjvSp>
00:33 johndescs_ joined #gluster
00:36 sashko_ joined #gluster
01:01 mkultras when i setup a new volume, i have to copy the files into the volume through the gluster client , or can i have prexisting files on the mount and serve them?
01:02 mkultras i think thats why i have the error above now i see i cant do what i had done
01:05 kkeithley joined #gluster
01:07 bfoster joined #gluster
01:12 sashko joined #gluster
01:16 raven-np joined #gluster
01:24 JoeJulian I suspect cw's problem was that he'd run out of ports <= 1024 on the server he was having trouble getting the cli to connect to.
01:28 JoeJulian mkultras: On 3.0.2, even I don't remember what's normal for log entries anymore. It's been a long time.
01:29 plarsen joined #gluster
01:38 semiosis JoeJulian: want to @later that for cw?
01:39 JoeJulian Nah, just thought I'd throw it out there generally.
01:39 semiosis ok
01:39 semiosis afk
01:40 stopbit joined #gluster
02:05 dustint joined #gluster
02:33 * m0zes is annoyed that the glusterfs nfs server sometimes fails to register with rpc. not sure what/how, I just know that when I (rarely) try to use it, I can't until I kill it and restart glusterd
02:33 m0zes I just wrote a nagios check to keep an eye on it for me, though.
02:35 m0zes I'm not sure if rpc is just dropping it occasionally, or if glusterd fires up nfs before something else is ready.
02:40 m0zes what services outside of glusterfs are needed for the nfs-server? rpcbind, rpc.statd(?), anything else?
02:53 vigia joined #gluster
03:00 overclk joined #gluster
03:02 raven-np joined #gluster
03:08 bharata joined #gluster
03:09 hagarth joined #gluster
03:14 aravindavk joined #gluster
03:15 JoeJulian m0zes: For EL based distros, rpcbind.
03:17 m0zes JoeJulian: forgot to mention Gentoo (whoops). it seems the Gentoo init script doesn't depend on rpcbind. It could be that rpcbind wasn't ready/started before glusterd started and that is why it never registered.
03:19 JoeJulian Gentoo probably uses portmapper then?
03:19 JoeJulian debian/ubuntu use portmapper, I think.
03:20 m0zes no, it uses rpcbind. whoever wrote the init script for Gentoo didn't get the deps right is all.
03:20 m0zes I had rpcbind set to start, it just didn't start until after glusterd.
03:23 m0zes and the maintainer for the Gentoo ebuild for the package doesn't like changing *anything* other than version numbers for the ebuild. I've provided a patch to fix another ugly bug in the init script, and there has been no response (for ~10 months) https://bugs.gentoo.org/show_bug.cgi?id=413417
03:23 glusterbot Bug 413417: was not found.
03:25 ajm m0zes: fortunately very very easy to roll your own portage tree
03:26 m0zes ajm: I've been doing that for a long long time. I just thought others might like a working init script ;)
03:27 ajm publish it somewhere, i'll use it :)
03:27 ajm m0zes: too lazy to look at the moment, but is the maintainer generally idle? you can apply to takeover the package.
03:31 m0zes he's not terribly active. I don't have the time to maintain any public trees at the moment. I'm one of 2 sysadmins for a 2000+ core cluster, and I am still training the other ;)
03:46 plarsen joined #gluster
03:48 VSpike joined #gluster
03:52 bulde joined #gluster
04:06 rastar joined #gluster
04:29 sahina joined #gluster
04:31 vpshastry joined #gluster
04:33 sripathi joined #gluster
04:46 * cicero see portage
04:46 * cicero runs for the hills
04:47 * cicero learns how to properly type
04:48 rastar1 joined #gluster
04:49 ajm idk, i've used a lot of package managers through the years
04:49 ajm portage still my favorite
04:51 sgowda joined #gluster
04:55 bala joined #gluster
04:55 m0zes I understand many people don't like source based distros, but for an hpc/image based environment, it makes a *lot* of sense. I definitely prefer portage over yum/apt/rpm/etc.
04:57 mkultras what os is portage from , freebsd?
04:57 ajm your thinking ports
04:57 ajm which is a nightmare in its own right.
04:57 mkultras ya
04:57 ajm i think people see "source compile" and think you have to compile everything all the time, which you definitely don't
04:58 ajm portage has great binary package support
04:58 mkultras is it a bsd though?
04:58 shylesh joined #gluster
04:58 ajm mkultras: no
04:59 ajm portage is native to gentoo
04:59 Ryan_Lane joined #gluster
05:00 mkultras I really liked my experience with gentoo I wondered why it wasnt more popular
05:01 ajm not so user-friendly community, IMO
05:01 ajm lots of people who will tell you to RTFM, not so kindly
05:04 m0zes my first linux experience gentoo in 2004, over dial-up. I learned so much about my hardware. it took three attempts to get a good install, though.
05:04 m0zes s/experience/experience was with/
05:04 glusterbot What m0zes meant to say was: my first linux experience was with gentoo in 2004, over dial-up. I learned so much about my hardware. it took three attempts to get a good install, though.
05:05 sripathi joined #gluster
05:06 ajm hah, i like glusterbot
05:09 mkultras i'd like to setup a bot sometime to plugin to my nagios and jira and stuff when i have time, give it some quirks though for personality.
05:10 ajm i'm sure someone did that for jira
05:10 ajm I have my nagios message me
05:11 ngoswami joined #gluster
05:11 mkultras i was thinking maybe have it google and scrape song lyrics or sometihng for random things to say once and awhile
05:11 m0zes I have nagios tweet some things. mostly downtime notifications and issues with the fileservers.
05:12 DWSR joined #gluster
05:17 mkultras my nagios keeps telling me i need to add a brick to this 3.0.2 gluster server because it has 19% left only. I cant believe I never saved the admin guide or documentation for that version anywhere that says how to go about adding another brick. theres no gluster command I can see on there so i guess its editing the .vol file or using gluster-volgen to edit it
05:19 m0zes yeah, 3.2 was the first with the gluster command iirc. <=3.1 needed manual vol file editing.
05:21 morimoto joined #gluster
05:21 semiosis m0zes: 3.1 introduced the cli
05:22 m0zes fair enough. I started with 3.2, so I've never been without it.
05:23 semiosis i've never been without it either, having started with 3.1
05:24 cicero mkultras: a hacky stop-gap... if /mnt/vol1 is your volume, add another volume at /mnt/vol2, rsync -av /mnt/vol1/foobar/ /mnt/vol2/foobar/ and ln -s /mnt/vol2/foobar/ /mnt/vol1/foobar/
05:25 cicero mkultras: that is, create a brand new volume, copy the data and symlink ... just ensure /mnt/vol2 is available on all clients on which /mnt/vol1 is available
05:25 cicero mkultras: that's a "solution" completely outside of gluster.
05:25 semiosis mkultras: afaik you would need to edit more than one volfile, at least one per brick you want to add, then also the client volfile which would need to be updated to know about the new brick(s) and distributed to all clients
05:25 ngoswami joined #gluster
05:25 semiosis thats basically what the gluster cli does for you
05:26 semiosis ok, just popped into cause some trouble before bed, now i'm off
05:26 semiosis catch you all tmrw
05:28 cicero night
05:28 raghu joined #gluster
05:41 mkultras night
05:44 lala joined #gluster
05:49 ramkrsna joined #gluster
05:49 ramkrsna joined #gluster
05:51 mkultras i've been taking care of a 3 month old pit bull for the last month, i work from home so i'm almost always with the thing, i left it alone today for the afternoon and it ate my headset and some monster hdmi cables, trashed the place.
05:51 mkultras he acts up when hes alone super bad
05:52 mkultras have to get him used to being alone, i bet if there was another dog to play with theyd be fine
05:55 glusterbot New news from newglusterbugs: [Bug 910239] open behind issue <http://goo.gl/i9ZRQ>
05:57 mohankumar joined #gluster
05:58 kanagaraj joined #gluster
06:02 samppah mkultras: ouch
06:14 ajm mkultras: at 3 months he probably should be crated when alone, that's too much space for them
06:15 ajm he may feel responsible for protecting it all since your gone, its a big burden for a little guy :)
06:21 mkultras ajm: thats a good way to put it, makes sense
06:22 ajm << amateur dog psychologist
06:33 mkultras cicero: that rsync hack i believe will suit my purposes just fine as i'm supposed to replace the old gluster 3.0.2 with a 3.3 version couple of servers in a couple days anyways, thanks.
06:34 mkultras im going to go take puppy for a walk and try and tire him out while the rsync goes
06:35 satheesh joined #gluster
06:36 sripathi joined #gluster
06:43 vimal joined #gluster
06:46 Ryan_Lane joined #gluster
06:47 hagarth joined #gluster
06:52 sashko joined #gluster
07:10 rgustafs joined #gluster
07:16 hagarth joined #gluster
07:52 guigui1 joined #gluster
07:52 Nevan joined #gluster
08:04 andreask joined #gluster
08:05 ctria joined #gluster
08:09 hybrid512 joined #gluster
08:09 ctria joined #gluster
08:14 sripathi joined #gluster
08:24 dobber joined #gluster
08:33 gbrand_ joined #gluster
08:34 tjikkun_work joined #gluster
08:35 gbrand_ joined #gluster
08:39 sripathi joined #gluster
08:40 Humble joined #gluster
08:43 sripathi1 joined #gluster
08:43 WildPikachu joined #gluster
08:50 tryggvil joined #gluster
08:52 VSpike I'm creating a new cluster and I peer probe'd host1 from host2 and vice versa. Now trying to create volume from host1 and gluster volume create gv0 replica 2 host1:/foo host2:/foo says "Host host1 not a friend"
08:52 VSpike I get similar trying from host2
08:53 VSpike I notice as well that gluster treats hostname.domain as different from hostname when it comes to peer names - I guess this is expected?
08:54 JoeJulian Yes. If you want to use fqdn for brick definition, you need to probe with fqdn.
08:55 VSpike Right - in this case, I'm happy just to use a hostname
08:55 VSpike But the hosts don't seem to know themselves
08:56 JoeJulian Did you probe the first host from any 1 other?
08:56 VSpike I probed each from the other
08:57 VSpike So peer status on each shows the other only, by name
08:57 JoeJulian Ok, and you're using those same names as the hostname component of the brick definition.
08:57 VSpike Yes
08:58 JoeJulian fpaste the peer status from 2 servers
08:59 VSpike http://fpaste.org/AoX1/
08:59 glusterbot Title: Viewing Peer status by VSpike (at fpaste.org)
09:03 VSpike JoeJulian: Was the process I went through essentially correct (as I have described it at least)?
09:04 VSpike If so, I'm happy to purge Gluster and all its settings and try again. I did clone the machines after installing Gluster, but before adding any peers or volumes
09:04 VSpike I did delete /var/lib/glusterd/glusterd.info
09:05 Staples84 joined #gluster
09:09 vpshastry1 joined #gluster
09:17 vpshastry joined #gluster
09:17 JoeJulian Sorry, getting ready for bed between things. The peer status looks good. Try restarting glusterd on both servers.
09:18 VSpike Hmm.. no change
09:22 VSpike http://fpaste.org/newD/
09:22 glusterbot Title: Viewing Volume Create by VSpike (at fpaste.org)
09:22 JoeJulian I would stop glusterd on both. Wipe /var/lib/glusterd and start over. I'm not sure what's wrong and I'm too tired to do a good job of figuring it out.
09:23 Norky joined #gluster
09:24 VSpike :)
09:25 VSpike Thanks - at least you've confirmed that as far as you can tell I'm not doing something actually really wrong
09:25 JoeJulian Make sure your hostnames resolve correctly to themselves as well.
09:25 VSpike That's a real help in itself!
09:25 VSpike I will... they do, but I'll double check
09:29 sripathi joined #gluster
09:35 bauruine joined #gluster
09:53 sripathi joined #gluster
10:04 H__ JoeJulian: might you be awake ?
10:05 H__ JoeJulian: My 3.2.5 glusterfsd always dies with a sig11 (within 90 seconds) on a replace-brick command. The brick log: http://dpaste.org/hDSoF/ I can restart the process by manually starting a new glusterfsd and doing a replace-brick pause and start but then it dies again within 90 seconds.
10:05 glusterbot Title: dpaste.de: Snippet #218994 (at dpaste.org)
10:05 H__ I've tried to find whether 3.2.6 and 3.2.7 have any relevant fixes but could not find the release notes anymore on gluster.org.
10:05 H__ can you recommend a minimum but > 3.2.5 version for replace-brick capable of dealing with lots of directories ?
10:22 hagarth joined #gluster
10:22 sahina joined #gluster
10:30 mooperd joined #gluster
10:31 guigui4 joined #gluster
10:31 sripathi1 joined #gluster
10:33 cw joined #gluster
10:37 mooperd_ joined #gluster
10:38 Nevan joined #gluster
10:40 lala joined #gluster
10:42 sripathi joined #gluster
10:45 sripathi joined #gluster
10:50 mtanner_ joined #gluster
11:04 lala joined #gluster
11:09 joeto1 joined #gluster
11:21 gbrand__ joined #gluster
11:24 gbrand___ joined #gluster
11:29 hagarth joined #gluster
11:37 duerF joined #gluster
11:43 lala_ joined #gluster
11:44 sripathi joined #gluster
11:50 Zenginee1 joined #gluster
11:50 ctria joined #gluster
11:56 kkeithley1 joined #gluster
12:01 VSpike I deleted /var/lib/glusterd on both machines, but still the same.
12:01 VSpike This is utterly baffling :) http://fpaste.org/99wX/
12:01 glusterbot Title: Viewing Gluster Problems by VSpike (at fpaste.org)
12:04 NuxRo VSpike: if you `peer probe $IP` instead, does it make a difference?
12:04 VSpike NuxRo: I was wondering that. Might wipe /var/lib/glusterd and try again with IPs
12:05 NuxRo if that works, then you have a DNS/naming problem in your setup
12:06 NuxRo and you only need to do the peer probe command on one host
12:06 VSpike I thought if you're using names you needed to do it the other way too, to set the name on the first one?
12:07 NuxRo no, once you do it the other node automatically knows about it
12:07 VSpike Did that change recently?
12:07 NuxRo possibly, i'm a new glusterfs user, started with 3.3
12:07 VSpike The quick start says "Note: When using hostnames, the first server needs to be probed from one other server to set it's hostname."
12:08 NuxRo and in 3.3 you only need to run the command on one server
12:08 VSpike I'm sure I observed that at some point - then again I started with an older version, 3.1 I think
12:08 NuxRo never read that.. :D
12:08 VSpike Maybe that's my problem then
12:08 VSpike Perhaps doing it from both is harmful
12:09 NuxRo are you using 3.3?
12:10 szopa joined #gluster
12:11 VSpike Version: 3.3.1-ubuntu1~precise5
12:11 VSpike No, I get the same result if I only do the probe from one. The second server then shows the first server as "Hostname: 192.168.102.60" so it doesn't know the name of it
12:12 VSpike Let me try again with IPs
12:13 VSpike Yes, works with IPs
12:13 NuxRo i think replacing the IPs with hostnames directly in the vol files undr /var/lib/glusterfs is doable
12:13 NuxRo yep, so you have a naming issue somewhere
12:14 NuxRo make sure you use full fqdn and you have proper RDNS set up (not sure if it matters)
12:14 NuxRo alternatively you can maintain a set of /etc/hosts files with your servers across your setup
12:14 VSpike AFAICT DNS and /etc/hosts and /etc/hostname are all configured correctly
12:14 NuxRo that's odd
12:16 NuxRo now try to replace the IPs in the vol files with hostnames, restart gluster, look for any abnormalities
12:22 pipopopo joined #gluster
12:25 pipopopo Anyone using gluster with proxmox ?
12:27 NuxRo pipopopo: not me, but if you have a problem just ask and wait for someone to reply
12:27 H__ NuxRo: i've seen advice here to peer proble from both sides, sometimes one side only sees the IP address initially which gets solved by the peer probe from both sides (assuming 2 machines)
12:28 pipopopo NuxRo: No problems, just thinking about implementing it :-)
12:30 NuxRo H__: right, good to know, cheers
12:30 NuxRo pipopopo: good luck with it :-)
12:31 NuxRo v3.4 should be _very_ nice for KVM btw
12:32 pipopopo NuxRo: Thx for the info
12:32 NuxRo http://www.gluster.org/2013/02/​new-release-glusterfs-3-4alpha/
12:32 glusterbot <http://goo.gl/GuQav> (at www.gluster.org)
12:32 pipopopo NuxRo: For what are you using gluster at the moment ?
12:34 edward1 joined #gluster
12:36 vpshastry joined #gluster
12:37 mooperd joined #gluster
12:40 NuxRo pipopopo: just bulk storage for stuff atm, but plan to use it as backups/snapshots storage for the "cloud" and will probably set up some striped volumes for r1soft/idera backups (sadly they don't split their stuff in smaller parts..)
12:41 gbrand__ joined #gluster
12:43 manik joined #gluster
12:46 JFK Hi all. Is there any way to lower cpu usage of glusterfsd process?
12:47 JFK i have three node replica setup and almost 50% of 24 cores machine is used by glusterfsd
12:48 JFK it seems too much
12:51 bauruine joined #gluster
12:53 NuxRo JFK: might not be too much, depending on version and especially your specific work load
12:54 NuxRo if you do believe something is off, you should open a bug on bugzilla.redhat.com and/or send an email to the mailing list
12:54 JFK NuxRo: web service - lots of reads
12:55 JFK i dont think this is bug in soft, more probable is that config is not as tuned up as it should be
12:56 JFK that's why i googled a lot and asking here if somenone had similar problem and solved it.
12:58 plarsen joined #gluster
12:58 NuxRo JFK: joejulian wrote this article, maybe it helps http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/
12:58 glusterbot <http://goo.gl/uDFgg> (at joejulian.name)
12:59 JFK seen that already :-)
12:59 NuxRo I'd try to cache the hell out of it, especially if you do a lot of reads
12:59 NuxRo as in, local cache, of course
13:01 JFK we do cache as much as possible- varnish, memcache,nginx with own memory cache etc.
13:01 NuxRo hm
13:01 hybrid512 joined #gluster
13:02 JFK waht is suprising for me that glusterfsd seems to use same amount of cpu regardless of how much do we read from gluster mount point
13:02 ekuric joined #gluster
13:02 NuxRo right, well, you could take it to the mailing lists, should give you more coverage with the devs or more skilled ppl
13:02 NuxRo sounds like a bug then
13:03 NuxRo don't forget to also mention platform and glusterfs version used
13:04 JFK sure thing
13:05 tjikkun_work joined #gluster
13:11 jclift_ joined #gluster
13:22 tjikkun_work joined #gluster
13:38 bauruine joined #gluster
13:41 bauruine joined #gluster
13:42 manik joined #gluster
13:43 dustint joined #gluster
13:52 vimal joined #gluster
13:54 melanor9 joined #gluster
13:55 melanor9 left #gluster
13:58 hagarth joined #gluster
14:00 samppah hmm.. does libvirt have support for glusterfs storage pool?
14:04 _ilbot joined #gluster
14:04 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
14:06 johnmark samppah: yes, as of December
14:06 johnmark the latest release
14:06 ctria joined #gluster
14:07 samppah johnmark: do you know if there's any documentation available? i can find some examples of disk xml but no pool xml
14:07 johnmark that I can't help you with. sorry :(
14:07 * johnmark adds to his todo list a bunch of docs needed for the new features
14:07 samppah oh well, i'll keep searching.. thanks you :)
14:08 samppah i'd like to test qemu gluster with foreman but i think that foreman requires that storage pool is configured into libvirt
14:08 rastar1 joined #gluster
14:10 _ilbot joined #gluster
14:10 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
14:14 H__ My 3.2.5 glusterfsd always dies with a sig11 (within 90 seconds) on a replace-brick command. The brick log: http://dpaste.org/hDSoF/ I can restart the process by manually starting a new glusterfsd and doing a replace-brick pause and start but then it dies again within 90 seconds.
14:14 glusterbot Title: dpaste.de: Snippet #218994 (at dpaste.org)
14:15 H__ I've tried to find whether 3.2.6 and 3.2.7 have any relevant fixes but could not find the release notes anymore on gluster.org. Can anyone recommend a minimum but > 3.2.5 version for replace-brick capable of dealing with lots of directories ?
14:18 vpshastry joined #gluster
14:21 ngoswami joined #gluster
14:22 venkat joined #gluster
14:22 flrichar joined #gluster
14:22 jclift_ johnmark samppah: Hmmm, it would be good to have the libvirt storage info page(s) updated to include Gluster info.  i.e. http://libvirt.org/storage.html
14:22 glusterbot Title: libvirt: Storage Management (at libvirt.org)
14:23 jclift_ johnmark: If there's someone around who knows enough to write it, then the libvirt guys are very open to patches for updating website content.
14:24 cw Will the Self-heal Daemon shut down on its own, once a heal is done?
14:24 cw or is there some other way to see if healing is in progress?
14:24 jclift_ johnmark: The libvirt website content is part of the libvirt tarball, and it's reasonably straightforward.
14:24 jclift_ Just saying. :)
14:25 mkultras how do you tell a self heal is going, there is a daemon for it?
14:25 cw mkultras: my current guess is that gluster volume status shows healing daemons running if it's in progress
14:27 mkultras im about to startup 2 nodes one has a full volume and the other directory is empty, i'm goin to make a volume replica 2 of them
14:27 mkultras so one should heal the other full i imagine
14:27 mkultras so ill chekc
14:30 cw isn't gluster volume heal $VOL_NAME info  supposed to list all files missing heal? their real paths
14:30 cw it looks like this to me; https://gist.github.com/Jippi/486aa0a08ee02d0b302c  - not sure what those <gfid:c26c9b3c-24b9-46b7-9db4-3fc19a7c3d6d> means
14:30 glusterbot <http://goo.gl/cjJpM> (at gist.github.com)
14:32 cw according to the documentation, it should be file names, not these gfid ?
14:35 _ilbot joined #gluster
14:35 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
14:35 cw I have a split-brain by the look of it - https://gist.github.com/Jippi/a88584ac01640477cdcb
14:35 glusterbot <http://goo.gl/N6AsI> (at gist.github.com)
14:36 _ilbot joined #gluster
14:36 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
14:42 raven-np joined #gluster
14:42 cw any suggestion on how to fix the split-brain ? I've found http://www.joejulian.name/blog/fix​ing-split-brain-with-glusterfs-33/ - but seems impractical on 1500+ files
14:42 glusterbot <http://goo.gl/FzjC6> (at www.joejulian.name)
14:42 cw and for some reason, having <fgid> in the file output doesn't seem like a good thing to me
14:55 lala joined #gluster
14:56 sjoeboo joined #gluster
14:58 stopbit joined #gluster
15:00 mkultras i dont have the heal command
15:00 mkultras oh its v 3.2.5
15:04 sjoeboo joined #gluster
15:06 aliguori joined #gluster
15:12 Humble joined #gluster
15:16 ctria joined #gluster
15:18 vpshastry joined #gluster
15:20 GLHMarmot joined #gluster
15:26 morse joined #gluster
15:31 mkultras ok so if i want to setup a replica 2 volume, on v3.2.5 now, do i have to setup an empty partition and fill the files via the gluster client to get tham all assigned a fgid? i had the files on one node only and thought they would self heal over to the other but they dont appear to be doing that, maybe if i just rsync them up first then create the volume
15:31 mkultras or maybe i need to setup the volume on the server, and load the files via the gluster client
15:31 mkultras that seems the most reliable way
15:31 semiosis loading through the client is the "normal" way
15:32 semiosis however you should be able to preload one replica (in a pure replicate volume) then do a ,,(repair) via the client
15:32 mkultras ok
15:32 glusterbot http://goo.gl/uA812
15:32 semiosis before 3.3 there was no self heal daemon, so healing only happened on stat of a file
15:32 mkultras ah ok
15:32 mkultras thanks semiosis !
15:32 semiosis yw
15:33 balunasj joined #gluster
15:38 wushudoin joined #gluster
15:44 samppah jclift_, johnmark: this seems to do the trick http://pastie.org/6132864.. yet i don't know if it uses qemu or fuse with vm's
15:48 jclift_ samppah: It might also be worth jumping in the #virt channel on OFTC if you're not there already, as that's where the libvirt guys hang out.
15:48 jclift_ Just saying. :D
15:49 samppah good point :)
15:53 jskinner_ joined #gluster
15:58 johnmark samppah: nice find
15:59 johnmark but I'm betting it simply points to a FUSE mount
16:00 Ryan_Lane joined #gluster
16:01 daMaestro joined #gluster
16:01 bluefoxxx joined #gluster
16:01 aliguori joined #gluster
16:01 sjoeboo joined #gluster
16:02 mohankumar joined #gluster
16:02 bluefoxxx hi
16:02 glusterbot bluefoxxx: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:02 bluefoxxx wow shotgun.
16:02 bdperkin joined #gluster
16:03 bluefoxxx Anyway.  I"m having what appear to be a quorum issue
16:03 bluefoxxx If I mount gluster-server-a:/nfs and then reboot gluster-server-b, the NFS volume hangs until gluster-server-b comes back
16:03 bluefoxxx Relevant page in the documentation?
16:04 bdperkin joined #gluster
16:08 jskinner_ joined #gluster
16:09 bugs_ joined #gluster
16:10 Norky bluefoxxx, what kind of volume?
16:10 bluefoxxx Norky, replication, 2 nodes.
16:10 Norky hmm, ought to work AFAIK
16:11 jskinner joined #gluster
16:12 bluefoxxx doesn't seem to
16:12 bluefoxxx this is on Gluster 3.3
16:12 Norky what happens if you use a gluster client, i.e. no NFS?
16:12 bluefoxxx is that still recommended against?
16:12 Norky eh?
16:12 bluefoxxx I'm using nfs because I'm dealing with reading lots and lots of small files (also, I'm using cachefilesd with nfs)
16:13 wushudoin left #gluster
16:13 Norky I believe that performance of the 'native' gluster FUSE client for lots of small files is worse than NFS
16:14 bluefoxxx nod
16:14 jskinner_ joined #gluster
16:14 Norky but it is worth testing with the gluster FUSE client to isolate the problem
16:16 bluefoxxx bah, glusterfuse won't listen
16:17 Norky "won't listen"?
16:17 jskinner joined #gluster
16:18 bluefoxxx it fails.  Fixed my /etc/hosts, apparently that's needed.  At least the logs indicated that.
16:18 ramkrsna joined #gluster
16:19 bluefoxxx yeah
16:19 bluefoxxx it still halts when I reboot
16:20 bluefoxxx and continues when it comes back up.
16:20 Norky the gluster-fuse client hangs?
16:20 bluefoxxx yes
16:20 bluefoxxx well, I have a for loop that keeps updating a file every 0.2 seconds and printing an output on the client
16:20 bluefoxxx THAT hangs, so the fuse client hangs yes.
16:21 semiosis bluefoxxx: check your client log file... i'd guess you're never really connected to both bricks in the first place
16:22 bluefoxxx http://pastebin.com/cngXmvBV
16:22 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
16:23 bluefoxxx semiosis, hmm
16:23 bluefoxxx Connected to 63.88.62.170:24009, attached to remote volume '/mnt/s
16:23 bluefoxxx ilo0'.
16:23 bluefoxxx Connected to 63.88.62.171:24009, attached to remote volume '/mnt/silo0'.
16:23 semiosis unless the client mount is on host hq-ext-store-2 that pastie doesnt help
16:23 bluefoxxx nod, that's just volume info in both states
16:23 bluefoxxx but yeah, the logs on the client indicated I connected to both
16:23 semiosis best thing would be to look at the actual client mount log file, that will tell you exactly what is going wrong
16:24 semiosis also, is this a graceful reboot or a pull-the-plug reboot?
16:24 bluefoxxx graceful shutdown -r
16:28 cw joined #gluster
16:29 bluefoxxx gathering logs now
16:29 andrei joined #gluster
16:31 bluefoxxx http://fpaste.org/qjJf/
16:31 glusterbot Title: Viewing Paste #276694 (at fpaste.org)
16:31 bluefoxxx semiosis, rebooting either server hangs the FS.
16:33 portante joined #gluster
16:33 bluefoxxx http://community.gluster.org/q/2-node-rep​lication-gluster-hangs-when-node-reboots/
16:33 glusterbot <http://goo.gl/Da297> (at community.gluster.org)
16:34 bluefoxxx ^^^ the answer given here seems to indicate that glusterfs is not high-availability and replication is not meant to continue to supply service in the event of a downed node :|
16:34 bitsweat joined #gluster
16:35 bluefoxxx http://gluster.org/pipermail/glust​er-users/2011-November/009124.html same issue, no resolution given
16:35 glusterbot <http://goo.gl/dpHF2> (at gluster.org)
16:41 VSpike I'm trying to set up geo-replication but my command "gluster volume geo-replication gv0 drgvgs1::gv0 start" just gives me "geo-replication command failed"
16:41 VSpike I don't get anything in the logs at all
16:42 VSpike Ignore me..
16:42 VSpike Why is only when you paste your command to IRC that you spot the obvious typo? :)
16:42 sjoeboo joined #gluster
16:43 cicero it's a variation of duck programming
16:43 cicero :)
16:45 mtanner_ joined #gluster
16:45 bluefoxxx ok, with a hard shutdown, it comes back in about 60 seconds with gluster-fuse.  With NFS... need some advanced mounting options.  -o vers=3,tcp,fsc,soft,inter and then it also comes back in 60 seconds.
16:50 VSpike Referring to section 8.2.5.2 of the admin guide, should the option mountbroker-* lines be added to the glusterd.vol on the master or the slave?
16:51 VSpike I'm assuming slave, but I'm not 100% sure
16:52 VSpike In which case, I'm unsure how the part in step 6 is done... "Setup master to access Slave as geoaccount@Slave" ... I can't see where to set the username used for ssh access on the master
16:54 aliguori_ joined #gluster
16:59 sashko joined #gluster
17:01 bulde joined #gluster
17:02 piotrektt joined #gluster
17:04 piotrektt hey. i've set the tutorial example gluster on 2 vmachines. everything seems to work. nodes see each other, volume is woriking, status reports everything is alright - nevertheless its replicated volume creating something on one node does nothing on the other
17:04 piotrektt i am lost here. no error no nothing
17:05 piotrektt it was working before i set samba on the nodes but ive removed samba to test it and it still does not work.
17:06 flrichar joined #gluster
17:07 piotrektt and when i try to deatach the second node that is not responding - it says "Brick(s) with the peer ADDRESS exist in cluster"
17:18 disarone joined #gluster
17:19 Norky piotrektt, VMs you say? How did you create them?
17:20 piotrektt i ve set up 2 debian systems with shell only...
17:20 VSpike Aha, figured it out. Simple when you know how.
17:20 piotrektt the worst thing was they were working fine
17:20 piotrektt and i dont know at wich moment they broke because i started to set up samba to connect to windows
17:21 VSpike piotrektt: did you start samba directly on the bricks?
17:21 piotrektt yes
17:21 Norky that wont work
17:21 piotrektt i mean on nodes
17:21 piotrektt i have two debians witch one brick each
17:21 Norky Samba shoudl be configured to share out the gluster mount
17:22 piotrektt it was
17:22 piotrektt but its not the point
17:22 piotrektt i purged samba
17:22 piotrektt and gluster is not working any more
17:24 piotrektt ok i will rephrase it. ive set up gluster on 2VM and everything was working fine... set up samba... and wanted to test it stopped... removed sama... it still does not work
17:24 piotrektt but... gluster works, nodes see each other
17:25 VSpike Wondering if you have split brain, but I'm not sure how you can detect that
17:25 piotrektt volume works
17:25 VSpike What count did you give for replica when you created?
17:26 piotrektt 2
17:26 piotrektt ok.. only thing that comes to my mind may be that ive created some directory in .gluster on one of those nodes and delated it
17:26 piotrektt can this be an issue?
17:26 VSpike A question - if you have geo-replication running from a master volume to a slave volume, and a client modifies something on the slave volume, what will happen?
17:27 VSpike Will replication stop, or will it overwrite the changes? Or some other outcome?
17:28 piotrektt ive only tested it from windows side - creating something only creates thing on master volume
17:28 piotrektt when i swith off master volume
17:28 piotrektt it sees slave and writes on slave
17:28 piotrektt what does not work is replication
17:28 piotrektt :)
17:30 piotrektt oh sorry - i dont have geo-replication started
17:30 VSpike I'm just setting it up for the first time
17:32 piotrektt dang
17:32 piotrektt i wonder what i did wrong, because everyghing seems to work but the replication
17:34 Norky possibly split brain? do a ls -lR on a gluster-fuse mount (NOT on the brick directly)
17:34 Norky and see if that triggers a self-heal
17:34 Ryan_Lane joined #gluster
17:34 piotrektt gluster-fuse? is that f.e linux with gluster client?
17:35 Norky see my earlier question
17:35 Norky or rather, VSpike's question
17:37 Norky are you mounting the gluster volume on the Samba server, then re-sharing that?
17:37 Norky becuase if you sharing the bricks directly that till NOT work
17:38 piotrektt the thing is i've removed samba completely - as the first thing that might have casused trouble
17:38 VSpike Are you running the gluster client somewhere?
17:38 Norky yes, I know
17:38 piotrektt yes i have third debian i can run now with client
17:39 VSpike Yep, then it's on the mounted FS on there that you need to do the ls -lR
17:39 Norky but if you samba-shared the bricks directly, you've probably broken gluster's idea of what's going on
17:39 piotrektt what do you mean directly?
17:39 piotrektt so i cant set up samba on a computer that has a gluster brick?
17:40 VSpike Not exactly...
17:40 VSpike more that you shouldn't write directly to the bricks on the nodes by any means
17:40 VSpike Only through the client
17:40 VSpike Doesn't matter if you do it through samba or any other method - it will still confuse gluster
17:41 Norky you can run samba on a gluster server. The server must mount the glsuter volume as a client, then samba must share that
17:41 mooperd joined #gluster
17:42 piotrektt ok
17:42 Norky e.g. "mount -t glusterfs localhost:/volname /volname" and samba shares /volname
17:43 edong23 joined #gluster
17:43 VSpike Normally if it sees a difference between the files on the two nodes it will be able to figure out from metadata how to heal it... for example, if one server goes offline and you change something (through client), then when that server comes back on gluster will know how to get it back into step
17:44 VSpike But if you change something directly on the underlying FS on the bricks, it may view that as a situation that it can't recover from and stop replicating correctly
17:44 Norky indeed
17:44 piotrektt wait. so what you want to tell me is that to samba work alright i need a GLUSTER SERVER with installed GLUSTER CLIENT and then i can set up samba?
17:44 Norky do NOT directly change bricks
17:44 Norky yes
17:44 VSpike That would be one way to do it, yep
17:45 piotrektt wouldnt that kill performance?
17:45 VSpike Or run the gluster client + samba on a 3rd server
17:45 piotrektt ok, what about if i have one node with lets say 3 bricks... do i still need client?
17:45 VSpike it's not specific to samba. All access to the distributed FS must go through the client (gluster, or NFS)
17:46 trapni joined #gluster
17:46 VSpike What would be the point of that?
17:46 VSpike If you only have one node, it's not a distributed FS - just skip gluster and use the FS
17:46 piotrektt it will be the environment i will work with... but the volume wont be replicated
17:47 piotrektt we need a lot of storage
17:47 piotrektt so couple bricks with huge amount of data on them
17:47 piotrektt shared to windows clients cia CIFS
17:47 VSpike So you only want one node?
17:48 Mo_____ joined #gluster
17:48 piotrektt not exactly, not on the long run... but we want to avoid redistribution
17:49 VSpike So the single node setup is just for testing?
17:49 piotrektt yes
17:49 VSpike In that case, you still need to use the client. Otherwise, it's not testing is it?
17:50 VSpike Anyway, even with a single node, if you don't use the client then gluster will be broken
17:50 VSpike So you'll never be able to switch to multiple nodes
17:53 piotrektt i will be - with gluster 3.5
17:53 piotrektt :P
18:06 mooperd joined #gluster
18:08 samu60 joined #gluster
18:09 samu60 hi all
18:09 samu60 using 3.2.2 in a distributed replicated environment
18:10 samu60 we've start seeing [2013-02-12 19:08:59.51819] I [server3_1-fops.c:1240:server_writev_cbk] 0-cloud-server: 2400420735: WRITEV 7 (37067) ==> -1 (Cannot allocate memory)
18:10 samu60 the server has memory enough
18:10 samu60 we're could the problem come from?
18:15 luckybambu joined #gluster
18:15 luckybambu joined #gluster
18:16 gbrand_ joined #gluster
18:16 jiffe98 with 3.31 is None still the default for cluster.quorum-type ?
18:17 jiffe98 and is there a way for me to find the value of a setting if it isn't explicitely set besides a documented default ?
18:20 andreask joined #gluster
18:20 semiosis jiffe98: try gluster volume set help, that will probably tell you what the defaults are
18:21 piotrektt___ joined #gluster
18:22 jiffe98 hmm, cluster.quorum-type isn't listed in there
18:23 samu60 hi there
18:24 samu60 i've got an issue with a 2-replicated distributed environment
18:24 samu60 [2013-02-12 19:24:14.459743] I [server3_1-fops.c:1240:server_writev_cbk] 0-cloud-server: 2400485947: WRITEV 7 (37067) ==> -1 (Cannot allocate memory)
18:24 samu60 lots of above message appearing
18:24 samu60 could it be a memory issue?
18:55 bitsweat left #gluster
18:56 Ryan_Lane joined #gluster
19:08 mooperd joined #gluster
19:10 edong23 joined #gluster
19:14 mooperd joined #gluster
19:15 bauruine joined #gluster
19:16 Ryan_Lane1 joined #gluster
19:33 bcipriano joined #gluster
19:34 nueces joined #gluster
19:34 bcipriano Hello - does anyone have any advice for upgrading an Ubuntu system from 3.3.0 to 3.3.1? 3.3.0 was installed using a .deb package, 3.3.1 is via PPA. I haven't been able to upgrade yet without losing my volume information, the two versions seem to store their data in different locations.
19:35 andrei joined #gluster
19:55 andrei__ joined #gluster
20:00 gbrand_ joined #gluster
20:01 semiosis bcipriano: ,,(3.3 upgrade notes) may be helpful
20:01 glusterbot bcipriano: http://goo.gl/qOiO7
20:02 luckybambu_ joined #gluster
20:02 semiosis although 3.3.0 and 3.3.1 should both store their configs in /var/lib/glusterd
20:04 luckybambu joined #gluster
20:05 Ryan_Lane joined #gluster
20:11 hybrid5122 joined #gluster
20:12 bcipriano Great, thank you. One more question - if data exists on a brick when you add it, will gluster keep it around? Is this supported behavior? It appears the gluster sees the existing data and doesn't touch it, builds metadata for it, etc. We're considering upgrading by setting up new Gluster servers, then just moving our existing bricks over.
20:13 semiosis that should work
20:13 drockna1 joined #gluster
20:13 bcipriano Great, thank you.
20:13 semiosis yw
20:13 semiosis just be sure you create the new volume with the same layout & move the bricks over to their same place
20:14 semiosis actually glusterfs will tell you the brick is already part of a volume
20:14 semiosis actually glusterfs will tell you the path or prefix is already part of a volume
20:14 semiosis hrm
20:15 semiosis actually glusterfs will tell you the path or a prefix of it is already part of a volume
20:15 glusterbot semiosis: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
20:15 semiosis :)
20:15 bcipriano Ah, excellent. Thanks again.
20:16 semiosis yw
20:16 drockna1 is there anything else i need to do to set up gluster to use ipv6 other then 'transport.address-family inet6' in my config file.
20:16 semiosis uh, i wasnt aware that anything was required
20:16 semiosis also generally speaking editing config/vol files is not recommended
20:16 semiosis i was under the impression that ipv6 just worked
20:16 jskinner_ joined #gluster
20:16 semiosis though never tried it
20:18 drockna1 not for us.
20:18 drockna1 semiosis: it appears that without any configuration some part of gluster (client or server) is trying to force a 'get_addr_info'  over ipv4 when we do a peer probe.
20:24 semiosis drockna1: what version of glusterfs?
20:27 drockna1 ive tried 3.2.x 3.3.1 and 3.4qa8
20:28 drockna1 semiosis: to be clear i have also tried it on centos 6 and ubuntu 12.04
20:28 drockna1 always get the same data
20:28 drockna1 https://gist.github.com/lyondhill/c91​2814de3c8ac16ac11#file-gluster-log-L8
20:28 glusterbot <http://goo.gl/Ccx8I> (at gist.github.com)
20:29 semiosis hrm, sorry idk much about glusterfs on ipv6, stick around i'm sure someone else will chime in sooner or later
20:30 drockna1 semiosis: well hopefully.
20:41 lkoranda_ joined #gluster
20:51 aliguori_ joined #gluster
20:56 jskinner_ joined #gluster
20:58 nightwalk joined #gluster
20:58 drockna1 left #gluster
21:01 georgeh|workstat question: have a distributed-replicated volume that I'm trying to setup geo-replication with, when starting geo-replication it fails with Errno 95, xtime not supported, any ideas what I'm doing wrong?
21:12 jag3773 joined #gluster
21:17 andreask joined #gluster
21:28 flrichar joined #gluster
21:29 dopry_ joined #gluster
21:32 aliguori joined #gluster
21:32 Ryan_Lane joined #gluster
21:34 bcipriano joined #gluster
21:38 sashko joined #gluster
21:38 y4m4 joined #gluster
21:39 y4m4 joined #gluster
21:40 RicardoSSP joined #gluster
21:41 Mo_ joined #gluster
21:41 y4m4_ joined #gluster
22:11 hattenator joined #gluster
22:17 polenta joined #gluster
22:19 luckybambu joined #gluster
22:55 JoeJulian georgeh|workstat: afaict, it means that you don't have the marker xlator in your volume definition. What version?
22:55 xavih_ joined #gluster
22:55 piotrektt____ joined #gluster
22:56 georgeh|workstat JoeJulian, using version 3.3.1-1
22:56 piotrektt_______ joined #gluster
22:58 JoeJulian Did you upgrade from a previous version?
22:58 georgeh|workstat no, started from scratch
22:58 JoeJulian hrm...
22:59 JoeJulian fpaste.org the log that shows the error
22:59 georgeh|workstat actually, I believe it is some kind of permission issue, I tried doing the geo-replication with root and it works, but I can't do that in a production environment
23:01 xavih_ left #gluster
23:02 georgeh|workstat http://fpaste.org/fGK1/
23:02 glusterbot Title: Viewing Paste #276831 (at fpaste.org)
23:04 georgeh|workstat http://fpaste.org/oTna/
23:04 glusterbot Title: Viewing Paste #276832 (at fpaste.org)
23:07 JoeJulian Oh, right. It has to run as root or it won't be able to read or write the xattrs.
23:09 georgeh|workstat okay, well now I feel a bit stupid, what is the purpose of the mountbroker?
23:10 georgeh|workstat I thought you could run it as an unprivileged user
23:10 JoeJulian You're one up on me on that one... looking now.
23:12 JoeJulian Hmm, maybe it's bug 811672 regarding selinux?
23:12 glusterbot Bug http://goo.gl/bmyrd urgent, urgent, ---, vshankar, ASSIGNED , mountbroker initiated umounts fail with EACCES on RHEL systems.
23:13 JoeJulian Mmm, maybe not. That seems to be more umount related.
23:13 * JoeJulian hasn't ever looked at mountbroker before.
23:14 VSpike I just set up georeplication using it, and it works as an unpriv user
23:14 georgeh|workstat did you use mountbroker?
23:14 VSpike yep
23:14 georgeh|workstat as an aside, I have selinux disabled for testing purposes
23:17 VSpike If you have geo-replication running from a master volume to a slave volume, and a client modifies something on the slave volume, what will happen?
23:17 VSpike Will replication stop, or will it overwrite the changes? Or some other outcome?
23:22 VSpike Since AIUI it's one-way replication using rsync, I'd expect it to overwrite but I'm not sure
23:24 kminooie joined #gluster
23:24 kshlm|AFK joined #gluster
23:24 kshlm|AFK joined #gluster
23:24 bdperkin joined #gluster
23:24 dblack joined #gluster
23:32 kminooie hi everyone
23:33 bcipriano left #gluster
23:34 kminooie I had an unusual experience. the gluster daemon on 3 of my 6 nodes cluster shut it self down and I don't know why. after I restarted the service everything  is working fine. and all i found in the log file is this:
23:35 kminooie [glusterfsd.c:727:cleanup_and_exit] (-->/lib/libc.so.6(clone+0x6d) [0x7f369049e92d] (-->/lib/libpthread.so.0(+0x68ca) [0x7f36907378ca] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x40536d]))) 0-: received signum (15), shutting down
23:35 kminooie does anyone have any idea how this could have happen?
23:36 elyograg 15 is a TERM signal.  usually that means something told it to shut down.
23:37 kminooie ? but could have possibly done that?
23:37 kminooie but what could....
23:37 elyograg could be the OOM killer, or someone typing 'kill <pid>'
23:37 elyograg or someone ran the init script with a 'stop' param.
23:38 kminooie i know that there was no human interaction with the cluster. I don't know what OOM is thou?
23:39 elyograg out of memory.
23:39 elyograg when memory gets super low the kernel will decide which processes to kill based on an algorithm it's got internally.
23:40 elyograg you'd see that in the kernel log - dmesg shows you the last bits of that for the current boot, it might be stored in /var/log somewhere too.
23:40 kminooie naa  i know kernel can do that but I am sure that hasn't been the case here
23:40 kminooie how about network ?
23:41 kminooie I am seeing line like this:  E [name.c:253:af_inet_client_get_remote_sockaddr] 0-glusterfsStorage1-client-0: DNS resolution failed on host d1r2n2
23:41 kminooie [2012-10-23 13:46:30.597660] W [socket.c:1494:__socket_proto_state_machine] 0-2locosStoragei-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.0.128:24009)
23:41 glusterbot kminooie: That's just a spurious message which can be safely ignored.
23:41 elyograg i wonder which line he triggered on.
23:41 kminooie could a network failure results in the daemon to shut itself down?
23:42 semiosis reading from socket failed. Error (Transport endpoint is not connected)
23:42 kminooie :) the second one --> [2012-10-23 13:46:30.597660] W [socket.....
23:43 elyograg I don't know if a problem with dns could maybe cause glusterd to terminate glusterfsd processes.  don't know enough about it.
23:44 kminooie :) welcome to the party
23:44 kminooie left #gluster
23:45 elyograg do any of the other gluster logs (including those on other servers) have anything at the same time? (hopefully your servers re time synced)
23:46 elyograg time to catch a train.  i'll still be logged in here.
23:47 edong23 joined #gluster
23:54 bauruine joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary