Camelia, the Perl 6 bug

IRC log for #gluster, 2012-12-13

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 y4m4 semiosis: oh cool thanks
00:00 semiosis yw
00:00 semiosis how've you been?
00:04 peterlin JoeJulian: the pid is nowhere to be found in the statedump file :/
00:10 nueces joined #gluster
00:36 kwevers joined #gluster
00:46 chacken1 joined #gluster
01:00 nightwalk joined #gluster
01:08 yinyin joined #gluster
01:21 _ilbot joined #gluster
01:21 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
01:24 nightwalk joined #gluster
01:53 kevein joined #gluster
01:58 yinyin joined #gluster
02:03 glusterbot New news from newglusterbugs: [Bug 886730] File object is opened and closed when not requested <http://goo.gl/R6xq0>
02:12 hagarth joined #gluster
02:27 mohankumar joined #gluster
02:40 bharata joined #gluster
03:30 mdarade joined #gluster
03:32 mdarade left #gluster
03:33 pelox joined #gluster
03:41 sgowda joined #gluster
03:50 yinyin_ joined #gluster
03:57 xavih joined #gluster
04:01 pelox left #gluster
04:17 sripathi joined #gluster
04:20 mdarade joined #gluster
04:21 yinyin joined #gluster
04:27 deepakcs joined #gluster
04:30 mdarade left #gluster
04:42 nightwalk joined #gluster
04:46 rastar joined #gluster
04:54 vpshastry joined #gluster
05:01 rastar left #gluster
05:07 FyreFoX joined #gluster
05:11 shylesh joined #gluster
05:14 yinyin joined #gluster
05:20 hagarth joined #gluster
05:21 vpshastry joined #gluster
05:23 shylesh joined #gluster
05:29 bulde joined #gluster
05:30 nightwalk joined #gluster
05:30 sripathi joined #gluster
05:33 berend joined #gluster
05:37 rastar joined #gluster
05:41 dbruhn joined #gluster
05:43 raghu joined #gluster
05:45 Humble joined #gluster
05:51 raghu joined #gluster
05:52 shylesh joined #gluster
06:18 bala joined #gluster
06:20 vimal joined #gluster
06:21 rastar1 joined #gluster
06:29 ngoswami joined #gluster
06:40 ramkrsna joined #gluster
06:41 sripathi1 joined #gluster
06:48 rastar joined #gluster
06:52 shireesh joined #gluster
06:56 sripathi joined #gluster
06:58 rwheeler joined #gluster
07:08 puebele joined #gluster
07:09 yinyin joined #gluster
07:33 rastar joined #gluster
07:33 dobber joined #gluster
07:38 peterlin JoeJulian: Once everything was properly self-healed, libvirtd started to work again. It would hang if there was a single self-heal still in progress.
07:39 JoeJulian Glad it's solved, but that's a pretty crappy solution. I wish we could compare with 3.3.1.
07:40 mohankumar joined #gluster
07:48 passie joined #gluster
07:53 guigui3 joined #gluster
07:53 passie left #gluster
07:54 passie joined #gluster
07:57 peterlin JoeJulian: Yes, but at least now I'm aware of it. That will help me plan things out. It would be nice to have the self-heal be less aggressive. I could live with a longer self-heal time if performance for VM's was higher during self-heal.
07:58 JoeJulian You can decrease the number of simultaneous self-heals
07:58 peterlin Oh really? Even ones triggered by file access?
07:58 peterlin Not just self-heald?
07:59 JoeJulian That's the way I'm reading it. They're all background queued, and it looks like if it's hit the limit, they just stay in the queue.
08:00 sripathi joined #gluster
08:01 zwu joined #gluster
08:01 peterlin All right, so how would I do that? There is no setting for it that I can see in the docs. And what would the default value be?
08:04 rudimeyer joined #gluster
08:06 peterlin JoeJulian: I have to go, I'll be back later :)
08:07 xavih left #gluster
08:07 toruonu joined #gluster
08:10 Nevan joined #gluster
08:15 JoeJulian @later tell peterlin cluster.background-self-heal-count default is 16
08:15 glusterbot JoeJulian: The operation succeeded.
08:15 yinyin joined #gluster
08:20 theron_ joined #gluster
08:30 andreask joined #gluster
08:34 mdarade1 joined #gluster
08:38 nissim good morning
08:39 nissim can anyone tell me how can I obtain the rdam patch from gluster git http://review.gluster.com/#change,149
08:39 glusterbot Title: Gerrit Code Review (at review.gluster.com)
08:39 JoeJulian @hack
08:39 glusterbot JoeJulian: The Development Work Flow is at http://goo.gl/ynw7f
08:40 JoeJulian That'll tell you how to get on there so you can use git to download the patch.
08:40 nissim I am running gluster version 3.3.1-3 and I need that patch to resolve my RDMA connectivity issue
08:40 FyreFoX hi im running gluster 3.2.5 on ubuntu 12.04 64bit. its a simple 2 servers which also have the client on and mount the volume with glusterfs. everything works fine except when I do this: cp -RPp /home/deploy/web/tmp/ /home/deploy/web/createdir/ && echo 2ce066d00a4b9ce889fd23eb4a4996ddf20633dc > /home/deploy/web/createdir/REVISION
08:40 FyreFoX -su: /home/deploy/web/createdir/REVISION: Software caused connection abort
08:40 nissim checking
08:41 FyreFoX the mount craps out and i have to umount and remount it
08:41 ndevos nissim: when you go to that review, there is a box where you can click "patch" and "Anonymous HTTP", copy that command and execute it
08:41 FyreFoX can someone clue me in on why this causes a crash
08:41 JoeJulian ndevos: Did they fix the anonymous http?
08:41 FyreFoX if I put a sleep 1 before the echo gluster does not crash
08:41 ndevos JoeJulian: I think I tried it yesterday...
08:42 nissim I am not a git user,therefor I believe installing a git client on linux and running the following patch command will not work
08:42 nissim just tried it
08:42 JoeJulian FyreFoX: known issue.
08:42 FyreFoX JoeJulian: oh! ?
08:42 ndevos nissim: ah, right, that makes things more complicated
08:42 JoeJulian FyreFoX: The workaround with that version and the way you installed it is to put that sleep in there.
08:43 nissim reading the link you sent me now
08:43 FyreFoX JoeJulian: oic. are there any other workarounds or fixes?
08:43 JoeJulian @ppa
08:43 glusterbot JoeJulian: The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY
08:44 ndevos JoeJulian: adding a 'git init nissim && cd nissim && <git-command-from-gerrit-using-http>' works for me
08:44 ndevos well, at least its downloading stuff
08:44 JoeJulian cool
08:45 nissim checking it now
08:45 FyreFoX JoeJulian: awesome thanks. I'll change over to those! any chance you have the link to the original bug ?
08:46 JoeJulian I don't know if there was ever one filed. semiosis just took control and made it happen.
08:46 ndevos nissim: http://fpaste.org/wHLA/ contains your patch
08:46 glusterbot Title: Viewing http://review.gluster.com/#change,149 (at fpaste.org)
08:47 FyreFoX JoeJulian: ahh. ok np. Thanks for your help its been driving me insane all day :)
08:47 nissim running: git fetch git://review.gluster.org/base/glusterfs refs/changes/49/149/11 && git format-patch -1 --stdout FETCH_HEAD
08:47 nissim got the following output: fatal: The remote end hung up unexpectedly
08:48 ndevos nissim: try the http variant, not the git protocol
08:48 nissim this is after running:git init gluster; cd gluster
08:48 JoeJulian Which is what would happen if you didn't go through the registration process using that method.
08:49 nissim JoeJulian, was that for me?
08:49 JoeJulian yep
08:49 nissim working ...
08:49 nissim getting something
08:50 nissim will let you know as soon as its finished
08:50 nissim in the mean time, I have another stupid question :)
08:51 JoeJulian Yes, it's okay to date nuns, as long as you don't get in the habit.
08:51 nissim tried gluster over IPoIB in 3 nodes stripe mode, creating a 10GB file locally took me 8 seconds (1.3GB/s), through gluster 400MB/s
08:52 nissim does this mkae any sence? I would expect peformance to be much higher in stripe mode
08:52 nissim I was just kidding regarding the stupid one ...
08:52 JoeJulian ... oh, not that stupid question...
08:52 JoeJulian oops
08:52 JoeJulian @stripe
08:52 glusterbot JoeJulian: Please see http://goo.gl/5ohqd about stripe volumes.
08:53 nissim so it was a stupid one after all, you say, checking link
08:53 mdarade1 left #gluster
08:54 gbrand_ joined #gluster
08:55 duerF joined #gluster
08:56 blubberdi Hi, is there a way to clear the profile data for a volume?
08:57 JoeJulian blubberdi: Just read it.
08:57 ekuric joined #gluster
08:58 JoeJulian That is the profile data you mean, right?
08:59 blubberdi JoeJulian: I don't know if I understand you. If I watch the data with the top command (gluster volume top www read brick ....) The top entrys are all from files I don't have on the glusterfs anymore. I want to start the profiling new from a "fresh" state.
09:01 JoeJulian gluster volume profile $vol info
09:02 JoeJulian iirc, that resets everything.
09:02 nightwalk joined #gluster
09:02 sgowda joined #gluster
09:04 blubberdi No, I still have the old entrys if I use the top command. I think I just increase the list-cnt, pipe it to a file and diff it to see what changes. Thanks anyway.
09:06 DaveS joined #gluster
09:12 DaveS___ joined #gluster
09:17 FyreFoX JoeJulian: I've added the repo and updated, when I upgrade glusterfs-server seems to get held back the others update ok.. suggestions ?
09:18 JoeJulian I'm not a .deb user, so I would need more information to guess on.
09:18 mohankumar joined #gluster
09:18 FyreFoX ah
09:22 nightwalk joined #gluster
09:33 manik joined #gluster
09:47 peterlin joined #gluster
09:48 manik joined #gluster
09:49 peterlin Anyone know how I can limit the number of concurrent self-heals going on? I'm looking to reduce load during this process. If it takes longer overall then that is ok.
09:51 mdarade joined #gluster
09:53 mdarade left #gluster
09:53 andreask1 joined #gluster
09:55 Norky joined #gluster
09:59 passie Since i'm quite new to gluster and couldn't find the answer: I'm currently testing (read learning) gluster in a virtual environment. On top of the gluster I would like to run a ftp daemon which connects to the shared storage. What is the best approach for this ?  is HA (read pacemaker) the best way to go forward?
09:59 hagarth1 joined #gluster
10:00 passie Within 6 months we are going to build a gluster environment with 50 bricks, which would mean I have to install 50 ftp daemons
10:01 nissim joined #gluster
10:03 Norky I'm no expert (but going to offer my opinion anyway)
10:04 Norky while possible, ftp is arguably missing the point of gluster
10:09 x4rlos passie: Well, i have noticed (i am new too) that ha's are maybe not really required.
10:10 x4rlos As the client that mounts the bricks sends out the writes to all of the nodes, rather than to a single node/ip that will propogate.
10:10 puebele joined #gluster
10:10 x4rlos though as i say, im new, and dont quite understand your requirement :-)
10:11 Norky why so many bricks? I guess it's a capacity requirement, more than speed
10:11 shireesh joined #gluster
10:12 the-me semiosis: ping
10:12 passie Norky capacity indeed. 1.5 PB in June
10:13 Norky your FTP daemon would have to write to a mounted glusterfs. You could have a smaller numebr of FTP servers each acting as FUSE glsuterfs clients, or you coudl mount the glsuterfs on each server, and run FTP servers together
10:13 Norky s/together/on each glsuter server
10:14 xymox joined #gluster
10:14 xymox joined #gluster
10:14 Norky if you wanted the FTP clients to be able to access the same IP address (or pool of addresses) each time, then some kind of HA or load-balancing in front of the FTP service would be needed
10:14 Norky as x4rlos said, HA is not necessary for gluster itself
10:15 passie So I have to mount the storage locally on each server and run a ftp server on top of that and put a HA of load balancer in front of that ?
10:15 passie s/of/or
10:15 Norky that is one possible approach, yes
10:16 passie hmmm would be nice to have some sort of virtual service running on top of gluster. (dreaming out loud)
10:16 Norky I'm assuming you'd have some degree of replication, because with 50 separate servers, the chances of one being offline at any given time are high (or at least, higher than a single server)
10:16 x4rlos passie: (sounds like) go for a couple HA ftp clients, each having glusterfs connections to a bunch of gluster peer groups maybe.
10:17 Norky x4rlos, you mean HA FTP *servers* surely?
10:17 x4rlos Norky: i do. I need more coffee :-)
10:17 passie Each brick will be replicated to it's neighbor  so net. 26 servers
10:18 Norky they would be clients in the sense that they are clients of gluster
10:18 Norky passie, the glsuter server includes FTP server code, so I suppose you are free to add your own FTP server module to that ;)
10:19 Norky passie, the glsuter server includes NFS server code, so I suppose you are free to add your own FTP server module to that ;)
10:19 Norky who needs more coffee? ;)
10:19 the-me <-
10:19 passie <--
10:19 Norky I think we all do
10:19 passie Norky i'll have a look at that
10:20 peterlin I found: "cluster.background-self-heal-count". Does this affect self-heals both going on as a result of a stat, and glustershd?
10:21 Norky passie, note also that glsuter has hooks for starting your own arbitrary scripts/programs. In the case of Red HAt Storage (a 'productised' version of RHEL + Gluster), when you do a "gluster volume start VOLNAME", it automatically mounts the glusterfs as a client on each server, adds an entry to /etc/samba/smb.conf and starts Samba
10:22 Norky the net effect being that Gluster volumes are available throughs CIFS as well
10:23 passie We will go with Redhat Storage (don't ask me why *blames manager*) Cifs would be great though the problem is that the client can speak ftp only :(
10:23 Norky it would probably be easier (quicker to implement, not necessarily cleaner) to do a similar thing for FTP, rather than extending the cluster code
10:23 * passie is getting coffee
10:24 Norky I'm not suggesting CIFS as an alternative - rather I'm suggesting you do for FTP waht RH have done for CIFS
10:25 Norky I'm possibly speaking for the Gluster project when I shouldn't, but I believe that the NFS and especially CIFS support are provided as 'legacy' options (consider that the NFS is v3 only - no v4) until and unless you can use the native gluster protocol
10:26 passie How would a failure of a brick handle the FTP sessions in your suggestion ?
10:27 Norky I think know what the answer will be (no control over it, no choice), but what is the end client application, what does it do and why FTP?
10:27 Norky err, assuming the FTP server is accessing a glsuter volume, and you have replication, the glsuter client will (should) carry on without issue
10:28 Norky if you're running an FTP server on a brick and asking how does FTP handle the loss of an FTP server, well, that's not related to Glsuter :)
10:28 x4rlos Im curious, if both ftp servers have the mount, and then the file: FILE.txt is written by server1, and then the same file is also asked to be written by server2 - what happens in the event of a collision?
10:29 passie the client is a streaming server which can only do ftp (blames the manufactures)
10:30 Norky x4rlos, I think the question is the same irrespective of FTP - i.e. what happens when any process writing to a mounted glusterfs happens to write a file that another client is writing :)
10:32 Norky there is file locking, same as NFS, or indeed any local filesystem
10:32 Norky http://gluster.org/community/documentation/index.p​hp/GlusterFS_Technical_FAQ#How_is_locking_handled.3F
10:32 glusterbot <http://goo.gl/m7JnI> (at gluster.org)
10:33 x4rlos I will take a look. :-)
10:33 x4rlos thanksyou
10:33 Norky there's not much in that link
10:33 x4rlos hehe, i read :-)
10:33 x4rlos Well, I will certainly have a play and see what happens. there's always the likes of noclubber and such so will see what the results are :-)
10:34 x4rlos Im just creating lvm layer on my two test gluster servers now so will see soon.
10:39 Norky x4rlos, I *think* you can replace "FTP server" in your question with "any process" and "gluster mount" with "any shared filesystem" :)
10:39 x4rlos Norky: Yes, i dont disagree. :-)
10:39 x4rlos It just made me think :-)
10:48 xymox joined #gluster
10:53 harshpb joined #gluster
10:55 manik joined #gluster
10:59 passie joined #gluster
11:00 x4rlos Can gluster create volumes over IP instead of dns? Just curious :-)
11:04 flin joined #gluster
11:08 harshpb joined #gluster
11:12 Norky not sure - try it
11:12 Norky you'd probably have to attach to the peers by IP address first
11:17 blubberdi Hi, I have a question for the gluster profile again. After I reboot both server I didn't get any entrys with `gluster volume top www read list-cnt 10`. But profile ist started for this volume and there are reads on this volume. Can someone please help find out what's wrong?
11:19 edward1 joined #gluster
11:19 harshpb joined #gluster
11:20 manik joined #gluster
11:20 bfoster_ joined #gluster
11:21 jdarcy_ joined #gluster
11:21 kkeithley1 joined #gluster
11:23 harshpb joined #gluster
11:24 manik joined #gluster
11:25 bfoster joined #gluster
11:27 FyreFoX JoeJulian: hrm. upgraded to 3.3.1 and problem still exists :(
11:30 harshpb joined #gluster
11:30 toruonu hmm… I'm seeing a stuck volume… all nodes that have the volume mounted through NFS (from different NFS servers) are stuck
11:31 toruonu gluster volume status on a node with the bricks gives status of all up
11:31 toruonu gluster volume heal home0 info doesn't return
11:31 toruonu ok, trying again I now get an error:
11:31 toruonu [root@se3 ~]# gluster volume heal home0 info
11:31 toruonu operation failed
11:31 toruonu [root@se3 ~]#
11:32 toruonu wtf is going on ...
11:32 mohankumar joined #gluster
11:33 Nr18 joined #gluster
11:39 harshpb joined #gluster
11:42 harshpb joined #gluster
11:48 harshpb joined #gluster
11:48 shireesh joined #gluster
11:50 hagarth joined #gluster
11:51 Jippi joined #gluster
11:55 toruonu I don't seem to be able to even stop the volume
11:55 toruonu [root@se1 glusterfs]# gluster volume stop home0
11:55 toruonu Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
11:55 toruonu operation failed
11:55 * toruonu waves his hands in desperation to grab someones attention
11:59 puebele joined #gluster
12:00 x4rlos toruonu: Wish i could help you man :-(
12:00 toruonu well I finally got it to stop the volume, but then couldn't see list or status. Restarted glusterd on all nodes and got gluster volume info, but heal info is still not coming through
12:01 toruonu and that's a tick scary
12:01 toruonu how do I debug: [root@se1 ~]# gluster volume heal home0
12:01 toruonu operation failed
12:01 toruonu [root@se1 ~]#
12:02 toruonu the odd problem seems to be:
12:02 toruonu ==> etc-glusterfs-glusterd.vol.log <==
12:02 toruonu [2012-12-13 14:02:12.134092] I [glusterd-volume-ops.c:492:glu​sterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume home0
12:02 toruonu [2012-12-13 14:02:12.134214] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by: c3ce6b9c-6297-4e77-924c-b44e2c13e58f
12:02 toruonu [2012-12-13 14:02:12.134234] E [glusterd-handler.c:458:glusterd_op_txn_begin] 0-management: Unable to acquire local lock, ret: -1
12:08 chirino joined #gluster
12:12 DaveS_ joined #gluster
12:13 toruonu it seems I'm hitting something similar to this:
12:13 toruonu http://comments.gmane.org/gmane.co​mp.file-systems.gluster.user/9792
12:13 glusterbot <http://goo.gl/JqaJb> (at comments.gmane.org)
12:13 toruonu how do I get gluster to release the lock?
12:15 toruonu hmm… reading this https://bugzilla.redhat.com/show_bug.cgi?id=843003 seems to say that the locking timeout is 30 minutes
12:15 glusterbot <http://goo.gl/kYcf2> (at bugzilla.redhat.com)
12:15 glusterbot Bug 843003: high, urgent, ---, kaushal, MODIFIED , call_bail of a frame in glusterd might lead to stale locks in the cluster
12:15 toruonu so I'll have to wait that :(
12:15 toruonu or can I somehow get that lock killed?
12:20 nightwalk joined #gluster
12:22 nissim JoeJulian, Can I ask another question please?
12:23 nissim To overcome rdma issue I decided to get gluster latest sources from head because there was a bug fix for the following side effect:[2012-12-13 12:21:46.437190] W [rdma.c:4518:gf_rdma_handshake_pollerr] (-->/opt/gluster/sbin/glusterfs(main+0x3ae) [0x40497e] (-->/opt/gluster/lib/libglusterfs.so.0(+0x5a5da) [0x7f63038575da] (-->/opt/gluster/lib/glusterfs/3g​it/rpc-transport/rdma.so(+0x51f6)
12:23 nissim [0x7f62f62a21f6]))) 0-rpc-transport/rdma: vol1-client-0: peer () disconnected, cleaning up
12:23 nissim [2012-12-13 12:21:49.443501] W [common-utils.c:2296:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
12:23 nissim [2012-12-13 12:21:49.443684] E [rdma.c:4601:tcp_connect_finish] 0-vol1-client-0: tcp connect to  failed (Connection refused)
12:23 nissim [2012-12-13 12:21:49.443762] W [rdma.c:4184:gf_rdma_disconnect] (-->/opt/gluster/sbin/glusterfs(main+0x3ae) [0x40497e] (-->/opt/gluster/lib/libglusterfs.so.0(+0x5a5da) [0x7f63038575da] (-->/opt/gluster/lib/glusterfs/3g​it/rpc-transport/rdma.so(+0x5739) [0x7f62f62a2739])^C
12:23 hagarth joined #gluster
12:24 nissim the problem is after compiling everything and running gluster (rdma mount works) I still get these errors
12:24 nissim I am not sure if patch is included in gluster HEAD and how can patch the source
12:24 nissim FIX URL: http://review.gluster.com/#change,149
12:24 glusterbot Title: Gerrit Code Review (at review.gluster.com)
12:25 nissim when I run the command: patch -p1 < patch.diff , I get some errors
12:25 nissim can anyone shade a light here on this issue?
12:26 nissim I am interested to know if patch in already commited to HEAD, and if not how can I patch HEAD to resolve these errors?
12:28 kkeithley joined #gluster
12:28 xymox joined #gluster
12:31 nissim anyone ??
12:39 mdarade joined #gluster
12:40 mdarade left #gluster
12:40 66MAACXPK joined #gluster
12:41 x4rlos FYI : http://gluster.org/community/docume​ntation/index.php/GlusterFS_Technic​al_FAQ#How_to_make_GlusterFS_secure.3F (links to a page thats not there :-l)
12:41 glusterbot <http://goo.gl/Jpif6> (at gluster.org)
12:51 nissim can anyone explain why when I use a single node volume over infiniband (tcp|rdma) I get a dramatic degradation in performance? when I write to local disk I get 1.3GB/s and when I do it on gluster mount point I get 450~550MB/s ???
12:56 Norky nissim, 550MB/s sounds pretty good to me
12:57 toruonu sometimes I hate the fact that most stuff is developed by people in US and I'm in the wrong timezone to get debug help :P
12:57 Norky I'm currently seeing 60MB/s (from 1200MB/s local disk access) over rdma
12:58 balunasj joined #gluster
12:59 toruonu I'd right now be happy already to have a working gluster volume
13:00 toruonu but it seems even 30 minutes didn't expire the lock on my volume
13:00 toruonu [2012-12-13 14:59:49.713194] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by: c3ce6b9c-6297-4e77-924c-b44e2c13e58f
13:00 toruonu it's been at least an hours
13:00 toruonu s/hours/hour/
13:00 glusterbot What toruonu meant to say was: it's been at least an hour
13:01 toruonu and I'm getting a little pissed off now
13:01 Norky that glsuterbot feature is fairly redundant
13:01 Norky s/glsuterbot/glusterbot/
13:01 glusterbot What Norky meant to say was: that glusterbot feature is fairly redundant
13:01 Norky oh the irony
13:02 kkeithley lol
13:03 kkeithley fyi, most of the gluster devs are in India
13:04 kkeithley At last count there are five in the US
13:05 toruonu well India should be online :)
13:05 toruonu I guess I need to write to the list and get a blade and shield to fend off angry users who cannot work
13:06 guest2012 joined #gluster
13:06 * x4rlos just upgraded to 3.3-1
13:07 sunus joined #gluster
13:08 x4rlos ... and now a gluster volume info command returns: "No volumes present"
13:09 x4rlos hmm. Well gluster, your still sharing the volumes.
13:12 kkeithley what about `gluster volume status`
13:12 kkeithley India should be on-line, but it's getting late in the day there.
13:13 x4rlos same :-)
13:13 x4rlos It only for testing atm.
13:13 toruonu well for me volume status shows ok results
13:13 toruonu it's heal that fails to work
13:16 toruonu whoa, another interesting log entry … I just tried to start profiling:
13:16 toruonu [2012-12-13 15:16:25.696308] E [cli-rpc-ops.c:3897:gf_cli3_1_profile_volume_cbk] 0-: Unable to allocate memory
13:17 toruonu but no command works thanks to the lock
13:17 toruonu always: [2012-12-13 15:16:58.310889] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by: c3ce6b9c-6297-4e77-924c-b44e2c13e58f
13:18 x4rlos I feel updating from 2.7 to 3.3 has broken gluster :-)
13:18 kkeithley c3...e58f can't reacquire its own lock!
13:20 toruonu kkeithley: indeed
13:20 toruonu however I think I just got rid of the lock
13:21 toruonu gluster volume statedump home0 actually did a statedump (nothing to display), but the log showed that it got local lock and released it afterwards
13:21 toruonu and now commands work
13:21 harshpb joined #gluster
13:21 toruonu at least I hope it does… heal home0 info still hangs
13:22 toruonu profiling at least worked
13:25 toruonu aaand heal command returned with 0 result
13:26 toruonu aand we have the lock again
13:26 toruonu and now statedump doesn't release the lock either
13:27 toruonu :(
13:29 andreask joined #gluster
13:36 x4rlos toruonu: is clearly experiencing more important problems than i - but i have upgraded to 3.3.1 from 3.2.7 and now gluster tells me this: "/mnt/gluster or a prefix of it is already part of a volume".
13:36 glusterbot x4rlos: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
13:36 x4rlos When i run volume {info,status} i get: No volumes present
13:36 x4rlos I will follow you glusterbot :-)
13:37 toruonu I only wish glusterbot would have something to say about the locking issue too :)
13:37 toruonu @glusterbot know anything about locks?
13:37 toruonu :P
13:37 ndevos @locks
13:37 glusterbot ndevos: I do not know about 'locks', but I do know about these similar topics: 'rhel6 deadlocks', 'centos6 deadlocks'
13:38 toruonu nop…
13:39 toruonu I've googled my ass off for this: [2012-12-13 15:38:51.619071] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by: c3ce6b9c-6297-4e77-924c-b44e2c13e58f
13:39 toruonu and so far 0 resolutions, loads of seeing this
13:39 toruonu it seems that in this case the lock is taken up by the heal command
13:40 toruonu stopping volume doesn't work for the same reason
13:41 toruonu no replies from list either for now...
13:42 guest2012 toruonu, been there, seen the same
13:42 * toruonu gets excited
13:42 toruonu how'd you fix it?
13:43 guest2012 don't know the proper way, unfortunately
13:43 toruonu and the improper one? :)
13:43 guest2012 I think I restarted the glusterfs servers around until it disappeared
13:43 toruonu hmm… I kind of did it once, but no help
13:44 toruonu but I can do it again
13:44 toruonu am attempting to remove all mounts as well
13:45 guest2012 while on rampage I kill -9 all client processes using the mounted volume (from lsof)
13:45 guest2012 then umount -f the volume, repeatedly
13:46 guest2012 it says in can't, oh it really can't umount (but in the end it does)
13:47 vpshastry left #gluster
13:49 guest2012 then, go to servers and stop the daemon. The init.d script shipped with 3.3.0 ubuntu packages (not semiosis ppa) seemed to leave processes around
13:49 toruonu alright
13:50 toruonu indeed removing all clients, restarting glusterd and glusterfsd and it's now working again
13:50 toruonu however… THIS SUCKS :)
13:50 toruonu I'd rather not have to go to full site downtime for this …
13:50 guest2012 I'm pretty sure this could be a nice "don'ts" list in gluster management, but then I'd like to see the "dos" list
13:52 guest2012 toruonu, if I'm right the fix for that issue has been reducing the timeout
13:53 toruonu well I initially waited 1h
13:53 toruonu so it's not really the 30 min timeout
13:53 toruonu and as I said I got rid of the lock so perf monitoring worked
13:53 toruonu i.e. profiling
13:53 toruonu but starting heal locked stuff up again
13:56 guest2012 https://bugzilla.redhat.com/show_bug.cgi?id=843003
13:56 glusterbot <http://goo.gl/kYcf2> (at bugzilla.redhat.com)
13:56 glusterbot Bug 843003: high, urgent, ---, kaushal, MODIFIED , call_bail of a frame in glusterd might lead to stale locks in the cluster
14:01 passie Where can i download the Management console for centos ?
14:01 Norky I think it's part of the oVirt suite of tools
14:02 Norky http://www.ovirt.org/Features/Gluster_Support
14:02 glusterbot Title: Features/Gluster Support (at www.ovirt.org)
14:05 Norky Red Hat make the part of oVirt that manages gluster available as Red Hat Storage Console
14:07 passie reading it now, thanks
14:08 Norky I don't know that CentOS redistributes rhsc, possibly not
14:08 Norky so you might be obliged to install the full oVirt stack (not that I really know if it's that big a task)
14:10 Norky http://middleswarth.net/content/install​ing-ovirt-31-and-glusterfs-using-either​-nfs-or-posix-native-file-system-engine looks more practical, passie
14:10 glusterbot <http://goo.gl/z6J65> (at middleswarth.net)
14:11 manik joined #gluster
14:28 mdarade joined #gluster
14:28 mdarade left #gluster
14:29 kevein joined #gluster
14:53 kkeithley The fix for 843003 is in 3.3.1.
14:53 stopbit joined #gluster
14:59 puebele joined #gluster
15:03 toruonu kkeithley: then it couldn't have been my issue
15:03 toruonu because I run 3.3.1
15:03 dbruhn .
15:06 nightwalk joined #gluster
15:09 guest2012 toruonu, anything in your logs around the line you already posted?
15:11 toruonu I sent the more detailed info at the time to gluster-users list as well
15:11 guest2012 ah, okl
15:11 guest2012 *ok
15:12 toruonu but basically no:
15:12 toruonu ==> etc-glusterfs-glusterd.vol.log <==
15:12 toruonu [2012-12-13 15:10:00.841789] I [glusterd-volume-ops.c:492:glu​sterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume home0
15:12 toruonu [2012-12-13 15:10:00.841910] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3ce6b9c-6297-4e77-924c-b44e2c13e58f, lock held by: c3ce6b9c-6297-4e77-924c-b44e2c13e58f
15:12 toruonu [2012-12-13 15:10:00.841926] E [glusterd-handler.c:458:glusterd_op_txn_begin] 0-management: Unable to acquire local lock, ret: -1
15:12 toruonu I was running a tail -f in the background and that was all there was
15:12 toruonu right now things seem to work
15:13 stopbit joined #gluster
15:14 x4rlos glusterbot: Transport endpoint is not connected
15:16 nullck joined #gluster
15:16 toruonu btw … what to make of gluster volume heal home0 info output where all of the entries are gfid:
15:16 toruonu http://fpaste.org/dtAD/
15:16 glusterbot Title: Viewing Gathering Heal info on volume home0 ... 2dab> (at fpaste.org)
15:16 toruonu no files
15:17 puebele joined #gluster
15:17 toruonu it doesn't claim split-brain on any of them and am not sure they are relevant, but I'd love of course to see 0 files in need of healing anywhere :)
15:18 guest2012 toruonu, oh I've got a comparable amount of entries there. Got used to see them.
15:19 toruonu yeah, but I'd still like to know dafuck they do there :)
15:19 nullck joined #gluster
15:21 * guest2012 glad to see it's not the only one to have trashy entries, and that replica = 3 is not a warranty against those
15:22 kevein joined #gluster
15:22 guest2012 about why they are all gfids and not paths, uhm I remember that was a matter of cache
15:23 guest2012 mine for example are mixed paths and gfids
15:24 guest2012 toruonu, in the mailing list message I think it's missing OS and gluster version
15:25 harshpb joined #gluster
15:25 x4rlos Okay. Gluster over cifs. So i am following http://www.gluster.org/wp-content/up​loads/2012/05/Gluster_File_System-3.​3.0-Administration_Guide-en-US.pdf (Section 6.3.1). I have set up samba shares, but the changes dont seem to be replicating.
15:25 glusterbot <http://goo.gl/bzF5B> (at www.gluster.org)
15:25 x4rlos They write of course to gluster-server1 but not to gluster-server2.
15:25 x4rlos I have also a glusterfs mount here, and that works fine.
15:26 harshpb joined #gluster
15:28 wushudoin joined #gluster
15:29 x4rlos Its just mounting cifs like any other cifs environment :-( Not using gluster. I have clearly missed something.
15:31 theron joined #gluster
15:32 ron-slc joined #gluster
15:33 Norky x4rlos, what path is samba exporting?
15:35 harshpb joined #gluster
15:36 x4rlos ==> /mnt/user-shares
15:36 x4rlos should this read just user-shares?
15:36 Norky /mnt/user-shares is a gluster-fs mount, correct?
15:37 x4rlos yes.
15:37 x4rlos :q!
15:37 x4rlos wrong term :-)
15:37 x4rlos Brick1: client1:/mnt/user-shares
15:37 Norky that is, not a brick, but a gluster client mount
15:37 Norky that's a brick
15:37 Norky dont' share bricks
15:37 x4rlos aaahhh.
15:38 Norky the Samba server must be a client of Gluster
15:38 Norky it can all run on the same machine
15:39 Norky brick0 + brick0 ..brickN make up your volume. You volume is mounted using the glusterfs FUSE driver on /some/path. Samba shares /some/path.
15:39 Norky directly sharing a brick will have results that are... undetermined
15:40 guest2012 Norky: maybe you meant "explosive"?
15:40 x4rlos yes. Yes they were 'undetermined/explosive' :-) Okay.
15:40 x4rlos So i can mount the share on iteself for exposing.
15:41 x4rlos assumably - we do the same for nfs mounts too?
15:41 x4rlos was that clear int he documentation? I maybe just being dumb. Caffiene levels dropping again.
15:41 guest2012 anyway, it seems to me that CIFS with Gluster is nothing more than re-exporting a volume from a client
15:41 Norky exactly so, guest2012
15:42 guest2012 x4rlos, there's a NFS server "embedded" in gluster, so no need to self-mount and re-export
15:42 guest2012 uh, talking 3.3.x here
15:42 Norky I think I said earlier - CIFS, and to a lesser degree, NFS are provided as 'legacy' protocols, while the main body of work goes into the glusterfs protocol itself
15:43 Norky x4rlos, as guest2012 said, NFS (v3 only, no v4) is part of the Gluster server, it shoudl be available automatically
15:43 Norky make sure the 'standard' NFS service is off, and do a "showmount -e" on your glsuter server
15:44 harshpb joined #gluster
15:44 Norky or just look at "gluster volume status" - it will show NFS service
15:44 jbrooks joined #gluster
15:47 Norky or just look at "gluster volume status" - it will show NFS service online
15:47 Norky err, pardon the repeated line
15:49 nueces joined #gluster
15:50 harshpb joined #gluster
15:50 mdarade1 joined #gluster
15:51 nightwalk joined #gluster
15:54 x4rlos Cool, thanks guys, will give that a go.
15:57 manik joined #gluster
15:59 harshpb joined #gluster
15:59 mdarade1 joined #gluster
16:00 mdarade1 left #gluster
16:01 aliguori joined #gluster
16:05 harshpb joined #gluster
16:05 x4rlos a good old: "mount -t glusterfs localhost:user-shares ./samba-link-to-user-shares" seems to have worked a treat. Thanks :-)
16:08 daMaestro joined #gluster
16:09 harshpb joined #gluster
16:10 ekuric1 joined #gluster
16:10 harshpb joined #gluster
16:12 nightwalk joined #gluster
16:18 harshpb joined #gluster
16:21 ekuric1 left #gluster
16:23 theron_ joined #gluster
16:26 mdarade joined #gluster
16:33 nightwalk joined #gluster
16:38 VSpike joined #gluster
16:43 theron_ joined #gluster
16:44 dstywho joined #gluster
16:54 theron joined #gluster
17:01 arusso joined #gluster
17:01 raghu joined #gluster
17:06 x4rlos I have a script that writes files to the folder i have glustered (replica) against server1 and server2. While its running (writing a file ever second) and i restarted server1, upon it booting back up and starting it has 'caught up'.
17:06 x4rlos However, i have done the same now with server 2, and its not catching up. hmmmm.
17:06 x4rlos oh - as i write this, it has just caught up.
17:06 x4rlos hmmmm.
17:07 mdarade left #gluster
17:14 x4rlos Can anyone tells me what happends when instead of adding and catching up i delete files instead? So....
17:14 nightwalk joined #gluster
17:14 x4rlos I restart server1 and while its restarting i delete files on the gluster which takes from server2. When server1 comes back online, it still has the files.
17:15 x4rlos What can we do in this split-brain scenario?
17:16 x4rlos wierdly, the client still sees the files from server1. So i guess i can go ahead and delete them again.
17:17 x4rlos hmmm. And now its caught up.
17:24 x4rlos Is there a cleanup process that runs every 60 seconds or something that checks the states?
17:24 hagarth joined #gluster
17:26 m0zes x4rlos: if you are running 3.3.x there is a self-heal daemon that does what you are describing.
17:28 neofob left #gluster
17:28 x4rlos m0zes: That will be it. I like gluster. Its very clever :-)
17:31 Mo___ joined #gluster
17:39 dbruhn how can you tell which node a file is stored on?
17:41 andreask joined #gluster
17:44 x4rlos dbruhn: I cannot see a way other than jumping on the servers directly :-/
17:44 x4rlos but i am n00b
17:56 x4rlos anyone else thing the set option "nfs.disable" default off is a double-negative? :-)
18:04 flakrat joined #gluster
18:04 flakrat joined #gluster
18:16 dbruhn What are some of the split brain improvements coming down the pipe?
18:34 DaveS joined #gluster
18:38 DaveS____ joined #gluster
18:47 robinr joined #gluster
18:58 Technicool joined #gluster
19:13 y4m4 joined #gluster
19:25 jermudge_ joined #gluster
19:28 nightwalk joined #gluster
19:40 y4m4 joined #gluster
19:42 ValentinVoigt joined #gluster
19:44 ValentinVoigt hi folks! is is correct that if i have a client which connects to one node of a 'replicated storage' network, that this node tells the client which hostnames it can connect to?
19:46 semiosis ~mount server | ValentinVoigt
19:46 glusterbot ValentinVoigt: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
19:48 SteveCooling guys, i just set ut a gluster using avahi and nss-mdns for name service. seems to work fine. any reason i should not do it that way?
19:49 SteveCooling s/ ut / up /
19:49 glusterbot What SteveCooling meant to say was: guys, i just set up a gluster using avahi and nss-mdns for name service. seems to work fine. any reason i should not do it that way?
19:49 SteveCooling thanks, glusterbot :)
19:53 jermudge_ So I have a failed node with a single brick on it. Running 3.3.1 on the working node, I did replace-brick start, and now any other replace-brick commands fail. Can't abort. Apparently I should have done start and then an immediate commit?
19:58 tru_tru joined #gluster
20:09 andreask joined #gluster
20:09 neofob joined #gluster
20:10 redsolar_office joined #gluster
20:20 redsolar joined #gluster
20:21 redsolar joined #gluster
20:25 gluslog_ joined #gluster
20:26 robinr_ joined #gluster
20:28 redsolar joined #gluster
20:30 wica__ joined #gluster
20:30 mjrosenb_ joined #gluster
20:30 DRMacIve1 joined #gluster
20:30 raghaven1rabhat joined #gluster
20:31 JoeJulian SteveCooling: Interesting. I can't think of any reason why not. That'd probably make a good blog post too.
20:32 JoeJulian jermudge_: If the brick failed and there's no data to migrate, do "... replace-brick ... commit force"
20:33 net_list_5 joined #gluster
20:38 tru_tru joined #gluster
20:40 kkeithley left #gluster
20:42 rosco___ joined #gluster
20:42 Zengineer joined #gluster
20:42 sjoeboo_ joined #gluster
20:42 meshugga_ joined #gluster
20:45 smellis_ joined #gluster
20:46 jermudge_ JoeJulian: will try
20:47 social__ joined #gluster
20:47 jermudge_ OK, brick replaced!
20:47 Nuxr0 joined #gluster
20:50 badone joined #gluster
20:51 ron-slc joined #gluster
20:56 badone joined #gluster
21:03 y4m4 joined #gluster
21:34 andreask1 joined #gluster
21:40 gbrand_ joined #gluster
21:43 jiffe1 joined #gluster
21:49 andreask joined #gluster
22:04 redsolar joined #gluster
22:07 glusterbot New news from newglusterbugs: [Bug 884328] quorum needs cli help text <http://goo.gl/33ipV>
22:09 Kizano_werk joined #gluster
22:10 Kizano_werk Good evening all :)
22:14 Kizano_werk I'm trying to enable uid translation in gluster 3.3.1.
22:14 Kizano_werk or at least I think that's what i need
22:14 redsolar joined #gluster
22:14 Kizano_werk the end goal here is to be able to mount files under server1 as user1, then mount the same files under server2, and have user2, user3, and user4 be able to access the files.
22:15 Kizano_werk ** the same files created by user1 on server1
22:15 Kizano_werk Is there a server/client mount option I'm missing to make this happen?
22:15 redsolar joined #gluster
22:16 JoeJulian Seems like group permissions or ACL lists would do that.
22:17 JoeJulian I'm pretty sure the filter translator is currently broken (but I'm not positive).
22:17 Kizano_werk yeah, I think I got it, just a matter of convincing my manager :)
22:17 a2 "New news"
22:17 Kizano_werk JoeJulian:  :/ ouch
22:17 JoeJulian new news?
22:18 JoeJulian Well it hasn't been used since the 2.0 days and it can't be enabled from the cli, so it's been back-burnered.
22:21 redsolar joined #gluster
22:21 swkide_ joined #gluster
22:23 swkide_ left #gluster
22:28 redsolar joined #gluster
22:30 Mo___ joined #gluster
22:32 redsolar joined #gluster
22:37 aliguori joined #gluster
22:41 theron joined #gluster
22:57 noob2 joined #gluster
23:03 redsolar joined #gluster
23:08 red_solar joined #gluster
23:09 FyreFoX JoeJulian: hrm. upgraded to 3.3.1 and problem still exists :(
23:12 JoeJulian I've forgotten which problem this one is...
23:17 FyreFoX -su: /home/deploy/web/createdir/REVISION: Software caused connection abort
23:17 FyreFoX damn
23:17 FyreFoX the one where I copy a file && write to file
23:17 FyreFoX need to put a sleep 1 in there or the mount crashes
23:18 FyreFoX was running ubuntu 12.04.1 with 3.2.5, upgraded to 3.3.1 but problem still exists :(
23:20 redsolar joined #gluster
23:21 a2 FyreFoX, what does the log say?
23:22 FyreFoX how much can I paste here?
23:23 elyograg FyreFoX: use a pastebin website.  the ones recommended here are fpaste.org and dpaste.org.
23:29 FyreFoX http://fpaste.org/DDiM/
23:29 glusterbot Title: Viewing gluster crash (at fpaste.org)
23:30 FyreFoX brb
23:32 gbrand_ joined #gluster
23:41 redsolar joined #gluster
23:42 FyreFoX back
23:42 FyreFoX any useful info in ther?
23:44 zaitcev joined #gluster
23:47 jisk joined #gluster
23:47 jisk Hi there
23:47 jisk I need some help if possible
23:48 jisk I am trying to replace a brick, and after I ran the replace-brick command that source brick is offline, and the volume has issues
23:48 jisk When I try to query the status of the replace-brick operation, I am informed "Source brick ss2:/exports/exp1 is not online."
23:49 jisk Volume is mounted fine at a filesystem level
23:49 jisk Can anyone assist - how do I get this brick back online?
23:50 hattenator joined #gluster
23:53 jermudge_ jisk, is there more detail in a log? (use fpaste.org)
23:54 jisk jermudge_: Seeing a lot of this: http://fpaste.org/fGY2/
23:55 glusterbot Title: Viewing Paste #260255 by jisk (at fpaste.org)
23:55 red_solar joined #gluster
23:56 jisk /backups isn't a directory directly on disk, it's within the gluster volume (storage1)
23:56 jermudge_ got it
23:56 jermudge_ is ss2 resolvable by dns?
23:56 jisk Yep
23:56 jisk Only one node with two bricks
23:56 jisk One of those is being replaced
23:57 jermudge_ probably dumb question, but I just did this exact thing myself (in fact it's auto healing right now). Did you commit the replace-brick?
23:58 jisk I got as far as "start" before this happened

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary