Camelia, the Perl 6 bug

IRC log for #gluster, 2013-04-08

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
05:21 glusterbot New news from newglusterbugs: [Bug 924481] gluster nfs sets unrequested bits in ACCESS reply <http://goo.gl/g6zWA> || [Bug 927648] volume status command not providing host names for NFS and SHD <http://goo.gl/AoK0w> || [Bug 928575] Error Entry in the log when gluster volume heal on newly created volumes <http://goo.gl/KXsmD>
05:52 glusterbot New news from newglusterbugs: [Bug 919916] glusterd compilation failure on OSX due to AT_SYMLINK_NOFOLLOW <http://goo.gl/uOfY6> || [Bug 921215] Cannot create volumes with a . in the name <http://goo.gl/adxIy> || [Bug 916372] NFS3 stable writes are very slow <http://goo.gl/Z0gaJ> || [Bug 916390] NLM acquires lock before confirming callback path to client <http://goo.gl/5eoJv> || [Bug 916406] NLM failure against Solaris
06:22 glusterbot New news from newglusterbugs: [Bug 949406] Rebalance fails on all the nodes when glusterd is down on one of the nodes in the cluster <http://goo.gl/Q8dyW>
10:07 rotbeard joined #gluster
10:10 ingard__ joined #gluster
10:11 Staples84 joined #gluster
10:11 ingard__ Hi. It seems I've run out of client ports.
10:11 ingard__ [2013-04-08 12:11:44] E [addr.c:133:gf_auth] auth/addr: client is bound to port 33224 which is not privileged
10:11 ingard__ [2013-04-08 12:11:44] E [authenticate.c:234:gf_authenticate] auth: no authentication module is interested in accepting remote-client 10.0.20.100:33224
10:11 ingard__ [2013-04-08 12:11:44] E [server-protocol.c:5862:mop_setvolume] server-tcp: Cannot authenticate client from 10.0.20.100:33224
10:11 ingard__ [2013-04-08 12:11:44] N [server-protocol.c:6788:notify] server-tcp: 10.0.20.100:33224 disconnected
10:11 ingard__ keep getting entries like this in logs all over the place all of a sudden
10:12 ingard__ (after adding 48 new bricks)
10:12 ingard__ what would be the correct way to fix it?
10:13 ingard__ my googling turns up CLIENT_PORT_CEILING setting as a suggestion.
10:13 ingard__ are there other ways ?
10:13 bulde1 joined #gluster
10:18 zykure|uni hey, im trying to set up a striped volume across 2 nodes using infiniband (IBoIP), but i can't get the peers to work
10:18 rastar joined #gluster
10:19 zykure|uni interestingly it works in one direction - "Peer in Cluster (Connected)"
10:19 zykure|uni but on the other (main) host it shows "Disconnected"
10:21 zykure|uni hm .. come to think of it, why not just use the working node to create the volume ;)
10:21 JoeJulian ingard__: Sounds like your client is trying to mount as a non-privileged user?
10:22 JoeJulian zykure|uni: probably won't work if one shows disconnected...
10:22 JoeJulian Make sure your hostnames resolve on each server.
10:22 JoeJulian also: ,,(stripe)
10:22 glusterbot also: Please see http://goo.gl/5ohqd about stripe volumes.
10:23 jclift_ joined #gluster
10:23 JoeJulian Why am I still awake?
10:26 ingard__ JoeJulian: why would you say that?
10:27 ingard__ it is running as root
10:27 ingard__ this problem started occuring recently after adding another gluster mountpoint
10:28 ingard__ we now have 19 glustermounts loads of bricks
10:29 zykure|uni thanks JoeJulian, i will look into it
10:29 zykure|uni but it's a little bit weird that it works on one of the two nodes
10:30 JoeJulian Then I can only suggest disabling the privileged port check by setting allow-insecure on for your volumes.
10:30 JoeJulian ingard__: ^
10:30 ingard__ JoeJulian: right. I'll look into it. Thanks :)
10:31 JoeJulian 3:30am... I'm going to bed.
10:31 ingard__ for some reason i couldnt find anything on google pointing in that direction for the errormessages i'm seeing
10:31 JoeJulian I've not seen it come up before.
10:31 ingard__ the client_port_ceiling patch was the only thing i found that made sense
10:31 JoeJulian http://joejulian.name/blog/mounting-a-gl​usterfs-volume-as-an-unprivileged-user/ is about as close as I expect you'll find.
10:31 glusterbot <http://goo.gl/3kpJW> (at joejulian.name)
10:32 JoeJulian But since I don't reference the error message in that article, you wouldn't find it that way.
10:32 ingard__ exactly :)
10:33 ingard__ would you happen to know when that insecure option was introduced?
10:33 JoeJulian Wow... that was a really old patch you found.
10:33 JoeJulian Probably 3.1
10:33 ingard__ those posts were the only ones referencing my error messages
10:33 JoeJulian Before that it was a translator option.
10:33 ingard__ if it is 3.1 i'm going to have a problem
10:34 JoeJulian You're running an older version than 3.1?
10:34 ingard__ yes
10:34 JoeJulian Ouch
10:34 ingard__ 3.0.5
10:35 JoeJulian You run that over Arcnet?
10:35 JoeJulian jk... :P
10:35 ingard__ hehe
10:36 ingard__ its been perfectly fine until now
10:36 ingard__ but what version we're on is something i've inherited
10:36 ingard__ unfortunately
10:36 JoeJulian I'm way too tired to check out a 3.0 version and try to find which option for which translator that would be.
10:37 ingard__ can you point me in a useful direction?
10:37 RobertLaptop_ joined #gluster
10:37 * ingard__ hands JoeJulian a cup of coffee
10:37 ingard__ ;)
10:37 JoeJulian What I would be doing is cloning the git tree (actually I already have it cloned) check out v3.0.5, then "git grep privileged" and see if something pops out.
10:38 arusso| joined #gluster
10:38 Kins_ joined #gluster
10:38 arusso joined #gluster
10:38 JoeJulian It's probably the server translator that's performing the check, so that'll be xlators/cluster/server
10:39 JoeJulian If you want to make it really easy, upgrade. :P
10:40 ingard__ well
10:41 ingard__ that would be 80 servers
10:41 ingard__ with i dunno 50 clients
10:41 ingard__ across ubuntu and gentoo
10:41 ingard__ and 3 different version of ubuntu
10:41 ingard__ :s
10:41 JoeJulian puppet
10:42 JoeJulian Ok, I really do have to get up in a few hours, so I'm out. Good luck.
10:42 ingard__ cheers
10:42 ingard__ good night
10:45 mtanner_w joined #gluster
10:51 andreask1 joined #gluster
10:54 H__ gluster volume status ... "Brick stor1:/gluster/c 24010  Y 18572" ... gluster volume replace-brick vol01 stor1:/gluster/c stor3-idc1-lga:/gluster/b start ... "brick: stor1:/gluster/c does not exist in volume: vol01" ?!
11:11 piotrektt joined #gluster
11:11 hagarth joined #gluster
11:14 piotrektt joined #gluster
11:16 zykure|uni wtf glusterfs .. "Status: Started" -> stop volume gives "Volume not in the started state"
11:23 ricky-ticky joined #gluster
11:29 ChikuLinu__ joined #gluster
11:29 msmith__ joined #gluster
11:29 jboster joined #gluster
11:29 red_solar joined #gluster
11:30 jboster If I mount a drive at /mnt/d1, is it possible to make two blocks /mnt/d1/block1 /mnt/d1/block2 ? The filesystem is already xfs I dont want to reformat.
11:30 vpshastry joined #gluster
11:33 H__ I see a gluster 3.3.1 replace-brick completely lock up all gluster nodes
11:35 balunasj joined #gluster
11:36 rwheeler joined #gluster
11:36 Chiku|dc joined #gluster
11:37 H__ The target node //sbin/glusterfs -f/var/lib/glusterd/vols/vol01/rb_dst_brick.vol eats up 100% CPU and shows no output on strace, and 0 iops on the target brick
11:46 manik joined #gluster
11:46 andreask joined #gluster
11:46 andreask1 joined #gluster
11:47 andreask joined #gluster
11:51 Nevan joined #gluster
11:53 glusterbot New news from newglusterbugs: [Bug 947226] CLI option --xml doesn't have any effect on debian/ubuntu <http://goo.gl/jEc9n>
11:56 zykure|uni anyone knows what peer status "Accepted peer request" means?
11:59 Nevan joined #gluster
12:03 Nr18 joined #gluster
12:08 H__ zykure|uni: sorry I do not. Maybe it's halfway a peer probe sequence ?
12:15 rcheleguini joined #gluster
12:27 balunasj joined #gluster
12:27 zykure|uni damn, seems like iptables messed up my peer connections
12:27 zykure|uni i'll report once i get it to work :)
12:28 zykure|uni H__: i thought the same, but i'm not sure .. maybe it means it's waiting for the reply from the probed peer
12:34 Nr18 joined #gluster
12:34 yinyin joined #gluster
12:38 Nr18 left #gluster
12:40 bennyturns joined #gluster
12:42 robos joined #gluster
12:44 ekuric1 joined #gluster
12:46 ekuric joined #gluster
12:46 hagarth joined #gluster
12:48 dobber__ joined #gluster
12:59 bulde joined #gluster
13:06 aliguori joined #gluster
13:09 dustint joined #gluster
13:14 joeto joined #gluster
13:18 ujjain joined #gluster
13:21 ollivera joined #gluster
13:23 glusterbot New news from newglusterbugs: [Bug 949096] Inconsistent read on volume configured with cluster.quorum-type auto <http://goo.gl/dGVOc>
13:37 ramkrsna joined #gluster
13:37 ramkrsna joined #gluster
13:43 ndevos joined #gluster
13:43 aliguori joined #gluster
13:49 eyalm joined #gluster
13:49 rwheeler joined #gluster
13:50 manik left #gluster
13:51 andrewjs1edge joined #gluster
13:52 andrewjs1edge hi all
13:53 andrewjs1edge having an issue creating a volume: getting "Host xxxxx is not a friend", though it's the same box.
13:53 andrewjs1edge any thoughts?
13:54 andrewjs1edge Also, added itself to /etc/hosts
13:54 andrewjs1edge s/itself/it
13:57 eyalm hello all, do you think using gluster 3.3 on top of xfs a good idea for storing kvm images (qcow2)? we have a 4 nodes in replicated config.. some vm have high writing and the performance is not as expected.
14:00 zykure|uni wheee, finally it works - GlusterFS with 2 distributed volumes, using InfiniBand via IPoIB
14:00 ndevos eyalm: 3.4 has quite some improvements for that, but heavy random writing is mostly not a suitable use-case (not directly on a volume, nor in an image on a volume)
14:01 eyalm ndevos: thanks! I'll wait till 3.4 is stable and available via the repo..
14:06 andrewjs1edge ping
14:17 andrewjs1edge joined #gluster
14:17 andrewjs1edge ping
14:18 dbruhn__ joined #gluster
14:18 andrewjs1edge left #gluster
14:24 lalatenduM joined #gluster
14:30 17WABEHEM joined #gluster
14:33 jbrooks joined #gluster
14:40 rastar joined #gluster
14:46 lh joined #gluster
14:46 lh joined #gluster
14:50 daMaestro joined #gluster
14:53 rwheeler joined #gluster
14:57 neofob joined #gluster
15:12 Supermathie joined #gluster
15:12 chirino joined #gluster
15:12 dbruhn__ So had a weird one happen this weekend, tried to add four bricks to a volume, got a weird message that one of the bricks couldn't be added and then the entire volume stopped working, and now the glistered service wont start and stay running.
15:13 dbruhn__ luckily this wasn't a production system, but any ideas?
15:13 dbruhn__ 3.3.1 el6
15:14 Supermathie avati_: Were you able to take a look at the NFS captures I sent?
15:17 nueces joined #gluster
15:19 jag3773 joined #gluster
15:20 bugs_ joined #gluster
15:24 saurabh joined #gluster
15:26 copec left #gluster
15:29 vpshastry joined #gluster
15:31 H__ dbruhn__: I hope you find errors in the log files
15:44 bulde joined #gluster
15:47 jboster when starting the hadoop jobtracker It looks like the gluster plugin is trying to mount the filesystem, but because the default scripts run as mapred it fails, I have already mounted gluster at /mnt/glusterfs does it not try to detect if its mounted first?
15:54 johnmark jboster: hrm... that's a question for jvyas, if he's around. you might want to send a message to gluster-users
15:57 bet_ joined #gluster
16:01 andrewjs1edge joined #gluster
16:23 lh joined #gluster
16:24 glusterbot New news from newglusterbugs: [Bug 949625] Peer rejected after upgrading <http://goo.gl/HtYU5>
16:25 vpshastry joined #gluster
16:31 Mo___ joined #gluster
16:39 nhm_ joined #gluster
16:48 Supermathie Anybody around who's familiar with the NFSv3 internals of gluster?
16:51 JoeJulian jboster: Yes, but your df will be wrong as it will count the figures from that filesystem twice.
16:51 JoeJulian H__: ext4?
16:52 JoeJulian andrewjs1edge: Make sure you have a localhost entry in /etc/hosts
16:53 JoeJulian dbruhn__: Check the logs. I suspect something about checksum mismatch or something like that. I usually cure that by removing /var/lib/glusterd/vols/* from the server that's failing to start glusterd.
16:53 andrewjs1edge JoeJulian: Thanks, though I figured it out.
16:55 zaitcev joined #gluster
17:02 ingard__ JoeJulian: could you confirm one thing for me pretty please. Is there anything what will let me override the privileged port limit we talked about this morning
17:02 ingard__ (last night for you)
17:03 ingard__ JoeJulian: at least from my limited C skills I concluded there was nothing that would let me override it
17:03 ingard__ https://github.com/gluster/glusterfs/​blob/release-3.0/auth/addr/src/addr.c
17:03 glusterbot <http://goo.gl/pd7uY> (at github.com)
17:03 ingard__ so options are to patch and recompile or switch to user/pass
17:16 dustint joined #gluster
17:21 Supermathie Any way to crank up the debug level of the gluster RPC/NFS code?
17:23 dbruhn__ JoeJulian, all of the servers threw up after I tried to add the bricks. Will I need to recreate the volume after deleting the /var/lib/glusterd/vols/* from those nodes?
17:28 jboster JoeJulian: I ended up modifying the glusterfs client so that it wouldn't try to remount every time the filesystem started up
17:29 jboster Now I'm having trouble launching a hadoop job, says it can't find job.xml in the .staging directory, but it exists. Maybe a race condition between nodes?
17:29 JoeJulian dbruhn__: If you remove all those subdirectories, it will delete the volume definitions. Perhaps just deleting the one failed one would be sufficient.
17:31 JoeJulian jboster: I'm lost. I haven't used hadoop and I don't understand your modification. Let's start with the latter.
17:32 dbruhn__ JoeJulian: That's part of the issue it took down the process on all of the servers, so I have no idea which one is messed up, or if it propagated a bad change through the whole system.
17:33 jboster JoeJulian: Inside GlusterFileSystem.java there is an undocumented parameter fs.glusterfs.automount which looks like it does what I want (not trying to rerun fuse mount every time the FileSystem class is loaded in hadoop). It looks llike that parameter was being ignored, I just modified GlusterFileSystem.java so it wasn't ignored and set it to false.
17:33 JoeJulian dbruhn__: And none of the glusterd will start?
17:34 JoeJulian ah, i see.
17:34 JoeJulian Hadoop's another damned java app, huh? <sigh>
17:34 dbruhn__ JoeJulian: Correct, when I run /etc/init.d/glusterd status on any of the brick servers I get "glusterd dead but subsys locked" as feedback even after a fresh reboot
17:35 JoeJulian dbruhn__: Did you check the logs to make sure we're even going down the right track?
17:35 jboster JoeJulian: Now when a hadoop job is started, an xml file is loaded into the gluster fs, and tasks from all the machines in the cluster try to read it. I'm getting an error that this file does not exist but when I manually look it is there, this makes me think that there is some delay in metadata and all of the nodes don't see this file instantly
17:37 dbruhn__ JoeJulian: Which log would I even start looking at, I am having a hard time seeing anything in the obvious places while trying to start the service.
17:37 JoeJulian jboster: Well, it wouldn't be metadata - at least not in the sense that I'm familiar with. Of you set that flag back to true does it work?
17:38 JoeJulian dbruhn__: /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
17:38 JoeJulian s/Of you/If you/
17:39 glusterbot JoeJulian: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
17:39 * JoeJulian whacks glusterbot
17:41 johnmark jboster: I PM'd you the email of a guy who can help. please CC gluster-users so that we can add to our institutional knowledge
17:44 dbruhn__ JoeJulian: here is my log output http://fpaste.org/6pMv/
17:44 glusterbot Title: Viewing Paste #290411 (at fpaste.org)
17:45 dbruhn__ Looks like the vol file is not happy
17:47 JoeJulian Well that's not helpful... It doesn't even say why it failed...
17:47 dbruhn__ It looks like it might not be happy with the keys?
17:48 JoeJulian That's normal. It's an invalid error message.
17:48 dbruhn__ ahh ok
17:48 JoeJulian Was the last time you tried to start it really Saturday?
17:49 dbruhn__ yep, want me to do it again, luckily this system wasn't in production yet. I had just received two new servers to throw in the mix so I was adding the 4 bricks from them to the volume.
17:50 dbruhn__ I knew I could just go recreate the volume, but I figured it would be more helpful if there was really a bug to bring it to the channel
17:52 dustint joined #gluster
17:57 dbruhn__ also, if I start the process it is not writing anything to the current log file, it's just failing
17:57 JoeJulian Try just running it directly, "glusterd -d"
17:59 dbruhn__ bad option -d
18:05 JoeJulian bah, --debug
18:06 chirino so.. in replicated volume.  Who does the replicating, is it done the client side?  Or does the server replicate writes to other servers?
18:06 JoeJulian It's done client side.
18:08 rwheeler joined #gluster
18:10 chirino JoeJulian: thx.
18:10 chirino so in https://access.redhat.com/site/document​ation/en-US/Red_Hat_Storage/2.0/html-si​ngle/Administration_Guide/index.html#se​ct-User_Guide-Managing_Volumes-Quorum
18:10 glusterbot <http://goo.gl/7yDZ5> (at access.redhat.com)
18:11 chirino in the 3rd paragraph, 'small set of nodes'.. do they mean a client nodes?
18:11 chirino server nodes don't directly communicate do they?
18:17 Supermathie Just sent an email to gluster-devel with details on an rpc decoding error I'm getting on an FSINFO call - I have a debugger attached to the gluster-nfs daemon and a breakpoint set at the decoding function - ready to go! :)
18:17 Supermathie chirino: The server nodes communicate among themselves to replicate & exchange state, etc.
18:21 H__ Are there known replace-brick issues ? I found none in my search yet see replace-brick hang in two setups, both on 3.3.1 (see also todays mail to gluster users)
18:22 andreask joined #gluster
18:26 Supermathie My problem: http://lists.gnu.org/archive/html/g​luster-devel/2013-04/msg00030.html
18:26 glusterbot <http://goo.gl/rvpyQ> (at lists.gnu.org)
18:52 ramkrsna joined #gluster
18:52 ramkrsna joined #gluster
18:53 ramkrsna_ joined #gluster
18:59 dbruhn__ JoeJulian: Sorry here is the output of that debug. http://fpaste.org/1BSs/
18:59 glusterbot Title: Viewing Paste #290437 (at fpaste.org)
19:01 premera joined #gluster
19:02 dbruhn__ I was wrong about all of the servers not connecting, here is the out put of the peer command on what would be peer #8, peers 9/10 are both the new ones I tried adding. http://fpaste.org/8vGE/
19:02 glusterbot Title: Viewing Paste #290438 (at fpaste.org)
19:06 andrewbogott joined #gluster
19:14 andrewbogott Is anyone around who knows about/cares about gluster log rotation?  I want to file a bug but I'm a bit lost in bugzilla looking for duplicates...
19:14 glusterbot http://goo.gl/UUuCq
19:24 semiosis glusterbot wants to help you "file a bug"
19:24 semiosis ahem, file a bug
19:24 glusterbot http://goo.gl/UUuCq
19:24 ramkrsna left #gluster
19:25 semiosis andrewbogott: just ask your question, if someone is around who can answer your question, they eventually will
19:25 themadcanudist joined #gluster
19:25 semiosis hi
19:25 glusterbot semiosis: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
19:25 semiosis ^^^ that
19:25 themadcanudist left #gluster
19:27 andrewbogott Jah, I just filed it.  I'll let the bug wrangler worry about whether or not it's a duplicate.
19:30 semiosis thats reasonable :)
19:50 dustint_ joined #gluster
19:55 glusterbot New news from newglusterbugs: [Bug 949706] Log-rotate needs to hup <http://goo.gl/TwekZ>
20:13 portante joined #gluster
20:54 jbrooks joined #gluster
21:15 bdperkin_gone joined #gluster
21:15 bdperkin joined #gluster
21:19 lh joined #gluster
21:19 lh joined #gluster
21:20 san_ joined #gluster
21:21 san_ any one has idea about 3.4 alpha 2 ubuntu 12 amd64 binaries ?
21:25 glusterbot New news from newglusterbugs: [Bug 949298] Crash in xml output of volume info <http://goo.gl/VVtkv>
21:32 pib1942 joined #gluster
21:34 san_ any one has idea about 3.4 alpha 2 ubuntu 12 amd64 binaries ?
21:42 jthorne joined #gluster
21:43 dbruhn__ when trying to create a volume I am getting this response /var/brick01 or a prefix of it is already part of a volume
21:43 glusterbot dbruhn__: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
21:53 san_ Does gluster 3.4 has rdma support ?
21:59 johnmark san_: yes. that should be merged in the 3.4 branch
21:59 johnmark san_: although we've fixed a few things since alpha2
22:00 san_ johnmark: great, when will it available to test ? Also, if you can help to get ubuntu binaries. Thanks @
22:00 san_ johnmark: great, when will it available to test ? Also, if you can help to get ubuntu binaries. Thanks !
22:01 redsolar_office joined #gluster
22:09 andreask joined #gluster
22:22 JoeJulian Hmm... wasn't it ubuntu 12 that was missing some required library?
22:25 JoeJulian Eh, guess not...
22:25 JoeJulian san_: https://launchpad.net/~semiosis/+archive/ubu​ntu-glusterfs-3.4/+builds?build_state=built
22:25 glusterbot <http://goo.gl/DnU1A> (at launchpad.net)
22:33 san_ JoeJulian: Thanks!
22:34 robo joined #gluster
22:46 andreask joined #gluster
22:57 mtanner joined #gluster
23:01 joehoyle joined #gluster
23:12 tc00per joined #gluster
23:18 tc00per Review of irclog leaves me confused as to whether root-squash is available for gnfs in 3.3.1. Can somebody clue me in?
23:19 tc00per Also, is this documentation valid? http://www.gluster.org/community/documentation/i​ndex.php/Gluster_3.1:_Adding_the_NFSv3_Protocol
23:19 glusterbot <http://goo.gl/sLe9E> (at www.gluster.org)
23:22 johnmark tc00per: it's not in 3.3
23:22 johnmark at least I don't think...
23:25 JoeJulian tc00per: I'm 90% sure that root squash is not available.
23:26 johnmark tc00per: I know there's a patch in the master branch, but it's not in 3.3
23:32 tc00per Thanks johnmark and JoeJulian. I don't see it listed for 3.4 either. Should it be?
23:53 zyk|off joined #gluster
23:53 JoeJulian tc00per: Are you looking through the commit log?
23:54 tc00per No... Features3.4

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary