Camelia, the Perl 6 bug

IRC log for #gluster, 2013-07-12

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 krink joined #gluster
00:27 _pol_ joined #gluster
00:37 badone joined #gluster
00:56 glusterbot New news from newglusterbugs: [Bug 961892] Compilation chain isn't honouring CFLAGS environment variable <http://goo.gl/xy5LX>
01:09 shapemaker joined #gluster
01:13 bala joined #gluster
01:17 krink joined #gluster
01:24 krink 'gluster peer probe ip.address' fails when trying to add peer on a dedicated backend storage network, But works on the front end network.  gluster 3.3.1-ubuntu.  This setup has worked for me before on older deployments.  What can I do to get the peer probe to work on this network?
01:31 harish joined #gluster
01:31 kevein joined #gluster
01:32 krink_ joined #gluster
01:33 krink_ 'gluster peer probe ip.address' fails when trying to add peer on a dedicated backend storage network, But works on the front end network.  gluster 3.3.1-ubuntu.  This setup has worked for me before on older deployments.  What can I do to get the peer probe to work on this network?
01:34 krink_ left #gluster
01:43 _pol joined #gluster
01:58 coredumb https://access.redhat.com/site/solutions/126183 < someone has the answer? i cannot seem to mount a gluster volume through NFS ...
01:58 glusterbot Title: It is not possible to mount a GlusterFS share over NFS on a Gluster server - Red Hat Customer PortalRed Hat Customer Portal (at access.redhat.com)
01:58 raghug joined #gluster
02:10 harish joined #gluster
02:31 semiosis krink: sounds like a routing issue.  is there any NAT in play here?
02:47 badone_ joined #gluster
02:49 harryxiyou Has gluster suppoted API for python?
02:51 harryxiyou semiosis: Has gluster supported API for python?
02:51 semiosis @glupy
02:52 harryxiyou semiosis: Thanks, let me have a deep look ;-)
02:52 semiosis jdarcy has done some work providing a python interface: http://hekafs.org/index.php/2012/08/glupy​-writing-glusterfs-translators-in-python/
02:52 glusterbot <http://goo.gl/D48zo> (at hekafs.org)
02:52 semiosis https://github.com/jdarcy/glupy
02:52 glusterbot Title: jdarcy/glupy · GitHub (at github.com)
02:53 semiosis i dont know much about it though
02:56 harryxiyou semiosis: Yeah, I have found this one ;-)
03:00 _pol joined #gluster
03:04 sprachgenerator joined #gluster
03:04 bulde joined #gluster
03:23 edong23 joined #gluster
03:29 mohankumar joined #gluster
03:34 bulde joined #gluster
03:36 CheRi joined #gluster
03:42 itisravi joined #gluster
03:43 vpshastry joined #gluster
03:45 sprachgenerator joined #gluster
03:48 JoeJulian jag3773: Nope, there was no bug report for a leak against 3.2.7.
03:49 shylesh joined #gluster
03:52 JoeJulian krink: insufficient information to form a hypothesis
03:53 JoeJulian ~nfs | coredumb
03:53 glusterbot coredumb: To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
03:57 krink semiosis: no nat, single backend switch attaching the two nodes.  i can ping each other, and ssh into each other.  I've even added host names to the /etc/hosts file.
03:58 krink the backend switch and interface is 10GbE (netgear xs712t), and I set the mtu to 8192.
04:00 JoeJulian jag3773: Upon further inspection, there was a patch http://review.gluster.com/#/c/3641/ that was merged into the release-3.2 branch after the last 3.2 version was released.
04:00 glusterbot Title: Gerrit Code Review (at review.gluster.com)
04:01 JoeJulian krink: define "fails"
04:01 semiosis krink: iptables?
04:01 JoeJulian semiosis: midnight?
04:04 semiosis yep
04:05 krink no iptables.  fails as in just hangs for a long time and then finally returns with no peer.  I can telnet into the peer port 24007 just fine.
04:05 raghug joined #gluster
04:06 semiosis krink: check the glusterd log files on both ends, /var/log/glusterfs/etc-glusterfs-gluster.log or something like that
04:06 krink [2013-07-11 21:06:11.658995] I [glusterd-handler.c:685:glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.168.100.253 24007
04:06 krink [2013-07-11 21:06:11.659694] I [glusterd-handler.c:428:glusterd_friend_find] 0-glusterd: Unable to find hostname: 10.168.100.253
04:07 krink [2013-07-11 21:06:11.659728] I [glusterd-handler.c:2245:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.168.100.253 (24007)
04:07 krink [2013-07-11 21:06:11.660089] I [rpc-clnt.c:968:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
04:07 krink [2013-07-11 21:06:11.660729] I [glusterd-handler.c:2227:glusterd_friend_add] 0-management: connect returned 0
04:07 semiosis pastie.org please
04:10 krink http://pastie.org/8133508
04:10 glusterbot Title: #8133508 - Pastie (at pastie.org)
04:13 bet_ joined #gluster
04:16 krink http://pastie.org/8133525
04:16 glusterbot Title: #8133525 - Pastie (at pastie.org)
04:17 krink can I bind the gluster daemon to a specific interface?
04:22 anand joined #gluster
04:29 badone_ joined #gluster
04:42 hagarth joined #gluster
04:53 kshlm joined #gluster
04:53 rjoseph joined #gluster
04:58 JoeJulian hmm... I was just asleep... I'm going to head to bed. No, krink, the daemons listen on 0.0.0.0.
04:59 bulde joined #gluster
04:59 krink thanks: JoeJulian, I'm going to go through the networking side of things again, and start from there.
04:59 JoeJulian krink: I think if you use hostnames you'd be fine.
05:01 JoeJulian or maybe not... let me read this again...
05:03 JoeJulian krink: Neither server has a peer in "gluster peer status", right?
05:03 krink correct.
05:05 JoeJulian can I see /etc/hosts
05:06 vpshastry joined #gluster
05:07 sgowda joined #gluster
05:08 krink JoeJulian: http://pastie.org/8133608
05:09 glusterbot Title: #8133608 - Pastie (at pastie.org)
05:10 krink the backend network (the 10. net) does not have default gw, the connected nodes are on an isolated vlan with no other uplink or connections.
05:13 JoeJulian and you /can/ telnet to 24007 on each other...
05:14 krink yes. se@vsrv01:~$ telnet  10.168.100.253 24007
05:14 krink Trying 10.168.100.253...
05:14 krink Connected to 10.168.100.253.
05:14 krink Escape character is '^]'.
05:15 JoeJulian which version is this?
05:16 krink se@vsrv01:~$ dpkg --list |grep gluster
05:16 krink ii  glusterfs-client                   3.3.1-ubuntu1~raring9            amd64        clustered file-system (client package)
05:16 krink ii  glusterfs-common                   3.3.1-ubuntu1~raring9            amd64        GlusterFS common libraries and translator modules
05:16 krink ii  glusterfs-server                   3.3.1-ubuntu1~raring9            amd64        clustered file-system (server package)
05:17 krink semiosis ppa
05:17 lalatenduM joined #gluster
05:20 bala joined #gluster
05:21 JoeJulian try restarting both glusterd.
05:22 JoeJulian I'm reading through source code, and I can't find anything that would explain that behavior...
05:24 JoeJulian "connect returned 0" is actually a success...
05:25 JoeJulian but then it times out... Did you look in the target's glusterd.vol.log?
05:26 samppah semiosis: are you around?
05:26 JoeJulian He should be asleep... :)
05:26 samppah or JoeJulian, are you using logstash to store glusterfs logs by any chnce? :)
05:27 JoeJulian I am.
05:27 krink I have /var/log/gluster/etc-glusterfs-glusterd.vol.log  is that a different log?
05:28 JoeJulian krink: That's the one. I was just curious if there was any different error in that log on .253
05:29 krink iptables -vnL is clear too ( no firewall )
05:30 krink i guess i'm going to turn down all interfaces, except the 10.net.  bounce the box's, and see if i can peer probe.
05:32 JoeJulian I doubt it's a network issue.
05:33 krink I thought maybe the 8192 mtu, but I have deployed this in the past; same mtu on a dedicated isolated backend switch.  the only diff now is this is 10GbE.
05:33 JoeJulian You could use wireshark and trace out the handshake if you think it is, though.
05:33 anand joined #gluster
05:34 JoeJulian But if each end can establish a tcp connection to 24007 on the other, then I think the network's ok. If it wasn't, even syn packets would get lost.
05:34 krink its just odd, I can ssh between the hosts, and even telnet into the 24007 on each port.  I'm just getting perplexed…  bluster 3.3.1 version?  perhaps I should down grade Or upgrade to newer QA version?
05:35 samppah JoeJulian: GlusterFS 3.4 is using UTC time in logs. I'm still bit new with Logstash but it seems like it converts timestamps to UTC. Servers have been configured to use EET (UTC +3) and my best guess is that somehow Logstash removes 3 hours from timestamp. This causes that because time in GlusterFS logs is already in UTC time they are actually converted to UTC-3
05:35 samppah sorry for the poor explanation... I hope you undestand what I'm trying to say :)
05:36 JoeJulian I've been using and supporting 3.3.1 since it came out. It should be working. That's why I wanted you to try restarting both glusterd. It's as if something else is causing a hang.
05:36 bala joined #gluster
05:37 JoeJulian samppah: are you using semiosis' grok patterns?
05:37 samppah JoeJulian: yes
05:38 samppah @logstash
05:38 glusterbot samppah: semiosis' logstash parser for glusterfs logs: https://gist.github.com/1499710
05:38 JoeJulian Cool! I didn't know that factoid was set.
05:38 psharma joined #gluster
05:38 lalatenduM joined #gluster
06:04 raghu joined #gluster
06:05 kevein joined #gluster
06:06 satheesh joined #gluster
06:13 JoeJulian samppah: I haven't forgotten you. I'm still trying to figure this out. What should really happen is that we should file a bug report against 3.4. The timestamp should really be in ISO8601.
06:13 glusterbot http://goo.gl/UUuCq
06:13 ricky-ticky joined #gluster
06:13 harryxiyou semiosis: What are the differences between 'pool' and 'trusted pool'?
06:14 JoeJulian nothing...
06:14 JoeJulian Unless you work for your community rec center....
06:15 harryxiyou I cannot understand your words...
06:16 JoeJulian It's called a trusted pool because once a peer trust is established, no other nodes can add themselves to it. Only the trusted peers can invite (probe) another node to be a server.
06:17 jtux joined #gluster
06:17 JoeJulian (My son is a swimming instructor in a community recreation center. A trusted pool, to him, is a (swimming) pool that hasn't had any children in it yet.)
06:17 samppah lol :)
06:20 JoeJulian samppah: Try https://gist.github.com/jo​ejulian/5982126/revisions
06:20 glusterbot <http://goo.gl/Un63D> (at gist.github.com)
06:20 JoeJulian I doubt it will work, but....
06:22 samppah JoeJulian: thanks, i will test it
06:24 harryxiyou Thanks all, l see ;-)
06:26 satheesh joined #gluster
06:29 andreask joined #gluster
06:30 dobber_ joined #gluster
06:32 JoeJulian samppah: Nope, but this /should/ be closer: https://gist.github.com/joejulian/5982126
06:32 glusterbot Title: gist:5982126 (at gist.github.com)
06:33 JoeJulian The mutate filter I think should append the +0000 to the timestamp so it won't be misinterpreted.
06:40 8OWAAJA0F joined #gluster
06:44 samppah JoeJulian: close but no cigar..
06:51 ekuric joined #gluster
06:52 samppah JoeJulian: thanks for your efforts! i think you got me to right direction :) i have to focus on some $dayjob stuff right now and i'll look at this later
06:54 harryxiyou Where is glusterfs latest git tree?
06:57 puebele joined #gluster
06:58 vshankar joined #gluster
06:58 harryxiyou semiosis: Where is the latest glusterFS git tree?
07:02 ramkrsna joined #gluster
07:04 saurabh joined #gluster
07:04 satheesh joined #gluster
07:04 satheesh1 joined #gluster
07:06 jtux joined #gluster
07:12 ngoswami joined #gluster
07:14 dpaul joined #gluster
07:18 satheesh joined #gluster
07:27 atrius joined #gluster
07:29 harryxiyou semiosis: Where is the latest glusterFS git tree?
07:31 harryxiyou I cannot find openssl-devel package on Debian Wheezy. Any commnets?
07:36 dobber___ joined #gluster
07:36 satheesh joined #gluster
07:39 theron joined #gluster
07:40 stigchristian joined #gluster
07:40 SteveCooling joined #gluster
07:40 NeonLicht joined #gluster
07:42 deepakcs joined #gluster
07:44 JoeJulian joined #gluster
07:44 yosafbridge joined #gluster
07:44 RobertLaptop joined #gluster
07:44 penglish joined #gluster
07:45 nightwalk joined #gluster
07:56 mooperd joined #gluster
07:59 dobber_ joined #gluster
08:06 ricky-ticky joined #gluster
08:08 xavih joined #gluster
08:45 johnmark joined #gluster
08:45 coredumb JoeJulian: thx, but all that is ok
08:46 coredumb i get this refusal in messages logs:  refused mount request from 127.0.0.1 for /vol0 (/vol0): unmatched host
08:46 coredumb i have "nfs.rpc-auth-allow: 127.0.0.1" in my volume config
08:50 _Bryan_ joined #gluster
08:51 rastar joined #gluster
08:58 vimal joined #gluster
09:04 pkoro joined #gluster
09:08 GabrieleV joined #gluster
09:12 harish joined #gluster
09:16 raghug joined #gluster
09:18 itisravi_ joined #gluster
09:22 _pol joined #gluster
09:28 rastar joined #gluster
09:39 GabrieleV joined #gluster
09:40 raghug joined #gluster
09:41 guigui1 joined #gluster
10:03 skyw joined #gluster
10:07 skyw joined #gluster
10:09 _harryxiyou joined #gluster
10:09 _harryxiyou left #gluster
10:18 yosafbridge joined #gluster
10:19 portante joined #gluster
10:19 cyberbootje joined #gluster
10:20 vpshastry joined #gluster
10:21 eryc joined #gluster
10:21 eryc joined #gluster
10:21 mjrosenb joined #gluster
10:33 eryc joined #gluster
10:33 mjrosenb joined #gluster
10:43 andreask joined #gluster
10:48 ctria joined #gluster
10:51 GabrieleV joined #gluster
10:55 spider_fingers joined #gluster
11:00 bulde1 joined #gluster
11:00 ingard_ oldbug 723
11:00 glusterbot Bug http://goo.gl/cs146 medium, high, 3.0.5, raghavendra, CLOSED CURRENTRELEASE, enhancements to quick read
11:01 lalatenduM joined #gluster
11:05 lala_ joined #gluster
11:17 harish joined #gluster
11:21 bulde1 vpshastry: ping
11:22 bulde1 vpshastry: did we actually solve the 'colon' in the path issue in quota?
11:26 skyw joined #gluster
11:34 rcheleguini joined #gluster
11:42 vpshastry bulde1: no..
11:43 edward1 joined #gluster
11:50 balunasj joined #gluster
12:01 rwheeler joined #gluster
12:13 raghug joined #gluster
12:22 saurabh joined #gluster
12:24 andreask1 joined #gluster
12:25 aliguori joined #gluster
12:31 mkollaro joined #gluster
12:32 glusterbot New news from newglusterbugs: [Bug 983975] Not all tests call 'cleanup' in the end, causing difficulties with single test runs <http://goo.gl/IZZj1>
12:35 bet_ joined #gluster
12:42 robos joined #gluster
12:43 skyw joined #gluster
12:45 NeatBasis joined #gluster
13:21 ndevos kkeithley: what do you think about packaging the tests subdirectory in a glusterfs-test subpackage?
13:22 ndevos I'm not sure yet where to put the test suite in the installed system though...
13:23 vpshastry joined #gluster
13:23 vpshastry left #gluster
13:23 rickytato joined #gluster
13:26 kkeithley yes, I think that would be good, although I wonder how many developers out there who don't already have it from `git clone`. Are we getting people asking for it as an RPM?
13:31 kkeithley s/how many developers out there/how many developers there are out there/
13:31 glusterbot What kkeithley meant to say was: yes, I think that would be good, although I wonder how many developers there are out there who don't already have it from `git clone`. Are we getting people asking for it as an RPM?
13:32 ndevos well, I don't know, but it would make it easier to run the tests during any of the test days
13:33 kkeithley agreed
13:33 chirino joined #gluster
13:34 mkollaro joined #gluster
13:35 sprachgenerator joined #gluster
13:37 pkoro joined #gluster
13:53 failshell joined #gluster
13:53 plarsen joined #gluster
13:57 bugs_ joined #gluster
14:04 jthorne joined #gluster
14:06 semiosis i'm happy to see activity on bug 852578
14:06 glusterbot Bug http://goo.gl/p1DgN is not accessible.
14:06 semiosis wat?
14:07 semiosis maybe bug 832609
14:07 glusterbot Bug http://goo.gl/WLzCC urgent, high, ---, rabhat, ASSIGNED , Glusterfsd hangs if brick filesystem becomes unresponsive, causing all clients to lock up
14:07 semiosis yeah thats the ticket
14:19 jdarcy joined #gluster
14:21 Guest98373 joined #gluster
14:21 Debolaz joined #gluster
14:25 ndevos semiosis: thats in the master branch already, you're free to test it ;)
14:36 kkeithley joined #gluster
14:37 kedmison joined #gluster
14:45 chirino joined #gluster
14:50 spider_fingers joined #gluster
14:50 jebba joined #gluster
14:50 rcheleguini joined #gluster
14:52 chirino joined #gluster
14:53 spider_fingers left #gluster
14:54 lubko joined #gluster
14:54 lubko hi
14:54 glusterbot lubko: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:54 lubko what's the purpose of features/index translator, please?
14:58 coredumb Hi folks, appart from ovirt, anyone of you is using a "cloud" application and runs VMs on top of glusterfs using native implementation to have real performances ?
15:00 JoeJulian lubko: the features/index translator is used for geo-replication, iirc.
15:01 JoeJulian coredumb: "real performance" is an end result. Don't design that way. Determine your workload and design the system that executes that workload for the best return on investment.
15:04 lubko JoeJulian: found out that it keeps track of changed files that failed to replicate for the needs of self-healing
15:06 JoeJulian lubko: Ah, right... it's marker that's used for georep
15:06 redbeard joined #gluster
15:07 coredumb JoeJulian: i love glusterfs for it's simplicity, but without the native implementation running VMs on top of it is quite painful... It's a big plus of 3.4, qemu and libvirt support that without a hitch but it doesn't seem that cloud solutions developpers are interested in
15:10 JoeJulian coredumb: Most closed-source cloud solution developers are interested in selling whatever product they're pushing. In the open-source arena, they will accept whatever patches you're willing to offer.
15:10 JoeJulian Have you put in feature requests for the solution you would like to use?
15:11 coredumb i think it's already in dev for next version of openstack
15:18 lpabon joined #gluster
15:19 daMaestro joined #gluster
15:20 joelwallis joined #gluster
15:21 kkeithley joined #gluster
15:22 cleeming[foxx] joined #gluster
15:24 raghug joined #gluster
15:24 cleeming[foxx] hello. Can anyone here thnk of any reason why reading files direct from the gluster brick dir would be a bad thing? In my situation, we have a lot of read throughput on millions of small files, and we're getting poor perf from it.. A colleague suggested reading files direct from the brick dir, whilst writing them to the glusterfs mount.. Is this a bad idea?
15:27 semiosis cleeming[foxx]: use noatime on your bricks if you're going to do that, otherwise (by default) even reads will modify the filesystem which could cause trouble for glusterfs
15:29 cleeming[foxx] semiosis: ah good idea, thank you!
15:29 semiosis also helps performance in general
15:30 puebele1 joined #gluster
15:31 roo9 joined #gluster
15:37 JoeJulian cleeming[foxx]: Also for every replica you add, you divide your available write bandwidth and increase the self-heal check latency and the potential for split-brain. So that option doesn't scale well. If you don't plan on your needs growing it's probably not a big deal.
15:38 JoeJulian If you do, then the other option would be to use geo-replication. Your shared write volume would then sync out to your fast-read media.
15:41 cleeming[foxx] got it - thanks for the advice guys, much appreciated!
15:49 bsaggy joined #gluster
15:50 krink_ joined #gluster
15:52 chirino joined #gluster
15:59 krink joined #gluster
16:00 shylesh joined #gluster
16:04 guigui1 left #gluster
16:04 zaitcev joined #gluster
16:04 _pol joined #gluster
16:12 bsaggy_ joined #gluster
16:13 bsaggy__ joined #gluster
16:19 dewey joined #gluster
16:25 cleeming[foxx] question.. if glusterfs crashes (signal received: 11), are the crash details logged anywhere? cant seem to find anything in dmesg or /var/log
16:26 JoeJulian /var/log/glusterfs/*
16:26 JoeJulian Which * depends on which ,,(process)
16:26 glusterbot I do not know about 'process', but I do know about these similar topics: 'processes'
16:26 JoeJulian @processes
16:26 glusterbot JoeJulian: The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/F6jqx for more information.
16:26 cleeming[foxx] JoeJulian: yeah i had a look in there too.. it showed that it crashed with signal 11, but didnt tell me why
16:26 cleeming[foxx] the file was; /var/log/glusterfs/mnt-brick1.log
16:27 cleeming[foxx] http://pastebin.com/xs5tQ21Q
16:27 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
16:27 _pol_ joined #gluster
16:27 semiosis @paste
16:27 glusterbot semiosis: For RPM based distros you can yum install fpaste, for debian and ubuntu it's dpaste. Then you can easily pipe command output to [fd] paste and it'll give you an url.
16:27 semiosis @forget paste
16:27 glusterbot semiosis: The operation succeeded.
16:28 cleeming[foxx] my bad on the paste.
16:28 JoeJulian Why forget?
16:28 semiosis @learn paste as For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [fd] paste [binit] and it'll give you a URL.
16:28 glusterbot semiosis: The operation succeeded.
16:28 cleeming[foxx] http://fpaste.org/24934/37364650/
16:28 glusterbot Title: #24934 Fedora Project Pastebin (at fpaste.org)
16:28 semiosis it's pastebinit on ubuntu, not dpaste.  afaik no such thing as dpaste
16:28 JoeJulian huh
16:28 cleeming[foxx] seems that it failed due to "0-brick1-replicate-0: size differs for /bonnie/Bonnie.8783"
16:29 cleeming[foxx] so im guessing its now stuck with not being able to heal itsef
16:29 semiosis @forget paste
16:29 glusterbot semiosis: The operation succeeded.
16:29 semiosis @learn paste as For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
16:29 cleeming[foxx] tho `gluster volume info` shows the volume is started
16:29 glusterbot semiosis: The operation succeeded.
16:31 mooperd joined #gluster
16:32 jag3773 JoeJulian, thanks for mentioning that NFS bug (http://review.gluster.com/#/c/3641/) -- Seems like that is what I am running into has it is the nfs process that is causing problems
16:32 glusterbot Title: Gerrit Code Review (at review.gluster.com)
16:32 JoeJulian Usual advice.... upgrade... ;)
16:33 jag3773 i've got that planned for sun night ;)
16:37 puebele1 left #gluster
16:38 raghug joined #gluster
16:41 dewey joined #gluster
16:42 _pol_ joined #gluster
16:43 sprachgenerator_ joined #gluster
16:46 chlunde joined #gluster
16:46 vincent_1dk joined #gluster
16:46 GLHMarmot joined #gluster
16:46 ingard__ joined #gluster
16:47 GabrieleV_ joined #gluster
16:53 krink JoeJulian:  Success!  peer probe works now.  It was the switch and its settings; netgear xs712t had Auto Denial of Service on.  I turned that off, and whala,  peer in cluster connected.  There are 17 DoS config options for this switch, I'm going to walk through each one and verify which is killing the service.
16:53 masterzen joined #gluster
16:56 semiosis krink: wow thats pretty interesting
16:56 semiosis please do let us know which feature is killing it
16:56 semiosis s/feature/"feature"/
16:56 glusterbot semiosis: Error: u's/feature/ feature"/" please do let us know which feature is killing it' is not a valid regular expression.
16:56 semiosis ha
16:57 krink semiosis: Certainly will let you know, I've been ripping my hair out on this!
16:58 cyberbootje joined #gluster
16:58 semiosis i went through something similar trying to get my voip phones working... in the end turned out to be some obscure & hidden sip proxy feature of our router
16:58 semiosis it was excruciating
16:58 daMaestro joined #gluster
17:00 vimal joined #gluster
17:03 JoeJulian krink: Hah! That's awesome.
17:03 mooperd joined #gluster
17:03 JoeJulian sip proxies usually suck.
17:04 vincent_vdk joined #gluster
17:04 ultrabizweb joined #gluster
17:06 RobertLaptop_ joined #gluster
17:07 raghug joined #gluster
17:14 daMaestro joined #gluster
17:20 Deformative joined #gluster
17:20 Deformative Hi.
17:20 glusterbot Deformative: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
17:20 Deformative Ugh, I always forget that the glusterbot doesn't like that.
17:20 Deformative Anyway, power was cut to my cluster and now all the files that had locks on them when the power went out are not released.
17:20 Deformative How do I release these locks?
17:24 JoeJulian In 3.3 you can use the "clear-locks" command.
17:30 Deformative What do I use for path?
17:30 Deformative I can't figure out how to make it work, I do "gluster colume clear-locks home path kind all
17:31 Deformative What do I put for path?
17:36 Deformative I think I also have split-brain files.
17:37 rcheleguini joined #gluster
17:40 Deformative My log has a lot of these: [2013-07-12 13:37:12.303294] E [afr-self-heal-common.c:2160:​afr_self_heal_completion_cbk] 0-home-replicate-7: background  meta-data data self-heal failed on /joemp/runs/40.error
17:40 Deformative Any ideas how to fix this?
17:40 jthorne joined #gluster
17:46 JoeJulian @split-brain
17:46 glusterbot JoeJulian: (#1) To heal split-brain in 3.3, see http://goo.gl/FPFUX ., or (#2) learn how to cause split-brain here: http://goo.gl/Oi3AA
17:46 raghug joined #gluster
17:47 JoeJulian @forget split-brain 2
17:47 glusterbot JoeJulian: The operation succeeded.
17:48 JoeJulian Deformative: It's been a year since I last used the clear-locks command, but iirc, the path is the path on the volume, so if the volume was mounted on /mnt/foo and the file that's locked was /mnt/foo/bar/baz the path is /bar/baz
17:50 Deformative JoeJulian, Hm.  I see.
17:51 Deformative JoeJulian, I get a bunch of "Number of entries: 0" when I run "gluster volume home info split-brain"
17:51 Deformative But those lines are still showing up in my log.
17:52 Deformative So I don't understand.
17:53 skyw joined #gluster
17:54 Deformative Could I just turn off replication?
17:54 Deformative Seems like an eaiser solution.
18:01 Deformative joined #gluster
18:08 JoeJulian Deformative: Sorry for the delays, feeding my daughter... It's French Toast day.
18:09 Deformative Hah, it is ok.
18:09 JoeJulian What that error says is that the metadata self-heal failed. That doesn't necessarily mean split-brain. Is that a file or directory?
18:10 Deformative Thanks for helping.
18:10 Deformative It is a file.
18:11 JoeJulian Ok, I suspect the entries leading up to that error probably paint the picture of why the heal is failing.
18:11 Deformative There are like 100s of similar errors.
18:11 JoeJulian And that error would show up in "volume heal $vol info heal-failed"
18:11 Deformative I run an HPC cluster.
18:11 Deformative And I use gluster as the results folder.
18:11 Deformative So a bunch of jobs try to write data into that folder.
18:12 Deformative Then I had a power outage and half hte nodes went down.
18:12 JoeJulian If you don't want to have to walk through clearing locks and can afford downtime, stop and start the volume.
18:13 Deformative I can.
18:13 Deformative Do I need to manually remount it on all the nodes?
18:13 Deformative Also, would these errors be less likely to occur if my volume had no replication?
18:13 JoeJulian Shouldn't.
18:13 JoeJulian Yes, there's no need for self-heal if there's no replication.
18:14 Deformative So what would happen in a power outage?
18:14 Deformative The data would just be a little messed up?
18:14 Deformative (This is reasonable for my stuff since the experimetns could just be re-run.)
18:15 JoeJulian self-heal is about repairing consistency between replicas. A power outage would do the same thing it does with a local filesystem. Whatever was in mid-write is corrupt.
18:17 Deformative Ok.
18:17 Deformative That's totally fine.
18:17 chirino joined #gluster
18:17 Deformative I suppose disk failure becomes a bigger problem.
18:17 Deformative What whatever.
18:17 Deformative The users could live with it.
18:17 Deformative It isn't supposed to be their personal storage anyway.
18:17 Deformative Just temporary results.
18:18 Deformative I will remake the volume without replication.
18:18 JoeJulian As long as your political heirarchy is supportive of that decision. :)
18:19 Deformative Yeah, I am in charge of its managment.
18:19 Deformative Well there are two of us.
18:20 Deformative Whatever we say, goes.
18:20 JoeJulian Nice
18:20 Deformative They are supposed to be fsyncing the data to their local raided disks.
18:20 Deformative They are jsut too lazy.
18:20 JoeJulian Heh, that's a common story.
18:21 Deformative The big issue we are having is that we occasionally get power outages, then when we try to re-run the experiments that were running at the time of the power outage the jobs hang trying to modify the files.
18:21 Deformative I suspect it is because of this replication problem.
18:21 Deformative But I am not certain.
18:22 Deformative So I am removing replication to see if it gets fixed.
18:22 JoeJulian *** Is anyone else here going to the Gluster event at OSCON? I'll be presenting on managing Gluster with Puppet.
18:22 JoeJulian Deformative: As long as you don't need replication, it's more common in HPC environments not to have it.
18:23 Deformative Is there anything else than can cause a hang like that?
18:24 Deformative Like sometimes I try to do rm results.txt or whatever and it just sits there.
18:24 Deformative Doing nothing.
18:24 Deformative Then I have a bunch of errors in the log.
18:24 JoeJulian Most things would be in the client log, unless it's a brick filesystem or disk problem.
18:27 kedmison joined #gluster
18:27 raghug joined #gluster
18:41 skyw joined #gluster
18:43 jthorne joined #gluster
19:05 gmcwhistler joined #gluster
19:12 _pol joined #gluster
19:39 mooperd left #gluster
19:45 ThatGraemeGuy joined #gluster
19:45 krink JoeJulian:  switch netgear XS712T DoS setting "Denial of Service TCP SYN" causes the issue I had.  All other default DoS setting for this switch are enabled and peer probe is fine.  Just this one causes peer probe to timeout.  Here is what the NetGear readme for the setting says:  Denial of Service TCP SYN - Enabling TCP SYN DoS prevention causes the switch to drop packets that have TCP Flags SYN set.
19:46 krink Gotta gras lunch.
19:46 JoeJulian Netgear being stupid again...
20:01 _pol joined #gluster
20:06 kaptk2 joined #gluster
20:07 brosner joined #gluster
20:11 Deformative Is there a way to get a list of all machines which have a volume mounted?
20:12 nightwalk joined #gluster
20:16 semiosis netstat?
20:19 JoeJulian gluster volume $vol status client
20:20 JoeJulian er... bad syntax, but you get the idea
20:20 semiosis ooh
20:20 JoeJulian gluster volume status $vol clients
20:21 JoeJulian You can't do that, semiosis.
20:21 JoeJulian Or did you sneak an upgrade in somewhere?
20:21 semiosis NEVAR!
20:21 JoeJulian hehe
20:21 semiosis but i *know* where my clients are
20:21 semiosis puppet makes them
20:22 semiosis we are about to lose power here
20:22 semiosis apparently there's a storm coming in
20:22 JoeJulian Uh-oh... stormy huh?
20:23 Deformative Hmm
20:24 Deformative I am trying to find a way to just get a list of machines that failed to mount on my cluster.
20:25 semiosis so a list of machines that dont have the volume mounted?
20:25 semiosis long list ;)
20:25 Deformative Well, I have an exhaustive list of all machines.
20:25 chirino joined #gluster
20:25 * JoeJulian looks at icinga and says, "None of mine are in error states." and looks away innocently.
20:25 Deformative So the subtraction of hte set that are mounted.
20:26 semiosis Deformative: ah yes so not as long as i feared
20:28 Deformative JoeJulian, Icinga looks interesting.
20:28 JoeJulian It's nagios, but with a better interface.
20:29 Deformative Never used either of those.
20:29 Deformative I wonder how easy it would be for me to install.
20:30 Deformative I already have mediawiki as my apache index, so I suspect I cannot just apt get it.
20:30 JoeJulian That's what virtualhost names are for.
20:31 * JoeJulian sidetracks another one!!!
20:36 JoeJulian gluster volume status home clients |cut -d: -f1 | sort -u | grep '^[0-9]\{1,3\}\.[0-9]\{1,3\}\​.[0-9]\{1,3\}\.[0-9]\{1,3\}'| cat - extensive_list_of_client_ips.txt | uniq -u
20:37 JoeJulian s/home/$vol/
20:37 glusterbot What JoeJulian meant to say was: gluster volume status $vol clients |cut -d: -f1 | sort -u | grep '^[0-9]\{1,3\}\.[0-9]\{1,3\}\​.[0-9]\{1,3\}\.[0-9]\{1,3\}'| cat - extensive_list_of_client_ips.txt | uniq -u
20:44 _pol joined #gluster
20:47 chirino joined #gluster
20:50 jebba joined #gluster
20:52 failshell vs
20:57 Deformative I got icinga installed.
20:57 Deformative Trying to figure it out now.
20:58 Deformative JoeJulian, That gives me ip addresses, not hostnames.
20:58 Deformative I suppose I could do a dns lookup somehow.
20:59 JoeJulian It's just some simple bash-foo I thought of when you presented your problem.
21:01 _pol_ joined #gluster
21:03 Deformative JoeJulian, Yeah, I like it.
21:03 Deformative I just need to figure out how to translate it.
21:09 _pol joined #gluster
21:10 krink the new distributed-striped-replicated volume.  can I create on a two node cluster, or should I reserve this for 4 plus nodes?
21:12 cleeming[foxx] hmm - strange problem. running fio/bonnie against a glusterfs volume seems to cause it to crash with signal 11... no extra info is given as to why. any ideas?
21:15 gmcwhistler joined #gluster
21:17 cleeming[foxx] Program received signal SIGSEGV, Segmentation fault.
21:17 cleeming[foxx] 0x00007fbda1d52ae9 in ?? () from /usr/lib/glusterfs/3.2.7/xlator/mount/fuse.so
21:17 JoeJulian krink: I don't really have an answer other than ,,(stripe)
21:17 chirino joined #gluster
21:17 glusterbot krink: Please see http://goo.gl/5ohqd about stripe volumes.
21:17 cleeming[foxx] something to do with fuse.. nfi what tho
21:18 JoeJulian cleeming[foxx]: This isn't in production?
21:18 cleeming[foxx] not yet :)
21:18 cleeming[foxx] after doing an strace on the original program, the call being made isl;
21:19 cleeming[foxx] [pid 12227] [    7f37abfda689] fallocate(3, 0, 0, 8589934592 <unfinished ...>
21:19 cleeming[foxx] then gdb catches the seg
21:19 cleeming[foxx] (the seg on gluster obv)
21:19 JoeJulian Then you should be using at least 3.3.1 and I'd probably recommend doing your test environment in the 3.4 beta.
21:19 cleeming[foxx] i havent got debugging symbols enabled tho
21:20 cleeming[foxx] oh... there isnt a stable deb package for anything beyond 3.2.7 tho?
21:20 JoeJulian @ppa
21:20 glusterbot JoeJulian: The official glusterfs 3.3 packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.3 QA: http://goo.gl/5fnXN -- and 3.4 QA: http://goo.gl/u33hy
21:20 cleeming[foxx] ahhhh
21:20 cleeming[foxx] didnt know about that, ttyv
21:20 cleeming[foxx] ttvm*
21:20 semiosis did somebody say stable?
21:20 JoeJulian Hold your horses!
21:21 semiosis ha
21:27 cleeming[foxx] lol well upgrading broke my volume.. is that to be expected?
21:27 cleeming[foxx] (it was only test data - but still)
21:30 JoeJulian The old clients can't connect to the new server. Upgrade everything and it should work.
21:31 cleeming[foxx] got it
21:31 brosner joined #gluster
21:31 krink yep, gonna do it = distributed-striped-replicated volume.  gluster volume will house vm images.  Thanks JoeJulian for the url write up on it.
21:39 cleeming[foxx] hmm - 3.3 doesnt seem to crash under fio/bonnie tests :)
21:39 cleeming[foxx] can you think of anything that might have changed in the last year that would have fixed that? would be nice to know what the actual problem wa
21:39 cleeming[foxx] s
21:40 JoeJulian Way too many things to count. :D
21:40 joelwallis joined #gluster
21:40 cleeming[foxx] lmao ok - ttvm :)
21:40 tstclair joined #gluster
21:42 tstclair eightyeight, JoeJulian - from IRC logs in Dec (http://irclog.perlgeek.de/gluster/2012-12-26) I appear to have stumbled across the same issue on F18.  I'm wondering if there has been a bug filed upstream or if I should file.
21:42 glusterbot Title: IRC log for #gluster, 2012-12-26 (at irclog.perlgeek.de)
21:45 _br_ joined #gluster
21:50 JoeJulian _tstclair: not really sure what the actual cause was. eightyeight found a solution that made him happy using shortnames.
21:51 _tstclair Stumbled across the same issue.  Full Names caused an issue, which honestly shouldn't
21:51 JoeJulian What if your full names ended with a .?
21:52 _tstclair running 3.3.1-15.fc18
21:53 _tstclair they did have multiple .'s though.. my.foo.bar.domain.com
21:53 StarBeast joined #gluster
21:53 _tstclair dns was just fine.
21:55 JoeJulian I was wondering if "my.foo.bar.domain.com." would work when "my.foo.bar.domain.com" doesn't.
21:56 krink some docs are showing mkfs.xfs -i size=512.  my isize=256 for the default xfs.  should I re-mkfs.xfs the file system, and what is special about size=512?
21:57 JoeJulian 512 byte inode sizes allow enough space for all the extended attributes without additional inode lookups. A recent email, however, suggested that it didn't make any measurable performance difference.
22:00 krink good to know.  I'm at a point where if I wanted to re-do the file system, I could right now.  the partition and filesystem was created during the ubuntu installer.  I guess I'll keep it default.
22:02 JoeJulian krink: http://comments.gmane.org/gmane.co​mp.file-systems.gluster.user/12050
22:02 glusterbot <http://goo.gl/V3XyC> (at comments.gmane.org)
22:08 _pol joined #gluster
22:08 krink JoeJulian: another good url and read.  Thanks again!
22:16 _tstclair joined #gluster
22:21 joelwallis joined #gluster
22:21 jebba joined #gluster
22:22 krink …added the inode64 to ftab…
22:28 krink …and re-created the filesystem with -i size=512…  why not?  I'm here right now :-)
22:29 JoeJulian :)
22:31 sprachgenerator joined #gluster
22:42 _ilbot joined #gluster
22:42 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
22:45 brosner joined #gluster
23:00 krink any mount option recommendations for client mounting??  I've used this in the past:  127.0.0.1:/gvol02 /srv/gvol02 glusterfs _netdev,direct-io-mode=enable,noatime 0 0
23:07 JoeJulian noatime doesn't do anything useful for a fuse mount. Would be useful for a brick though. If you're resharing via samba, attribute-timeout=0 is helpful.
23:12 nightwalk joined #gluster
23:22 krink getting odd output testing with the dd command.  http://pastie.org/8135992
23:22 glusterbot Title: #8135992 - Pastie (at pastie.org)
23:23 roo9 left #gluster
23:24 krink it created an empty file, I re-ran command and it created the file.  I can reproduce this...
23:29 sprachgenerator joined #gluster
23:30 krink I'm getting tired…  Gonna turn in for the day, and will do more sanity testing another day.  I think I'll try the distributed-replicated volume instead.
23:33 JoeJulian +1
23:33 krink left #gluster
23:43 _tstclair joined #gluster
23:51 StarBeast joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary