Camelia, the Perl 6 bug

IRC log for #gluster, 2013-07-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 johnlocke Hello! let me try to explain, I have 3 servers, each one have 3 raid arrays of 10+2 (raid6). So we have 36x2TB by server, we have 3 servers, each raid card formatted has 20TB. we have gluster stripe, 9 bricks, 3 static ip's, ctdb,round-robin,samba, because they are used to serve windows machines. We had a problem with one machine and have to change the backplane (it was degrading same disk).
00:05 johnlocke Now it's OK, when I try to start the volume say: It's already started, and when checked status, say: It's not in started mode. Try stopping, starting and gives the wrong message. Start volume volume_name
00:15 y4m4 joined #gluster
00:18 Technicool joined #gluster
00:22 y4m4 semiosis: JoeJulian i just wrote up http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented
00:22 glusterbot <http://goo.gl/Lkekw> (at www.gluster.org)
00:22 y4m4 semiosis: JoeJulian feel free to edit and circulate
00:22 y4m4 :-)
00:32 yinyin joined #gluster
00:45 bala joined #gluster
01:14 chirino joined #gluster
01:18 plarsen joined #gluster
01:38 MACscr joined #gluster
01:42 kevein joined #gluster
01:49 harish joined #gluster
02:00 hagarth joined #gluster
02:06 theron joined #gluster
02:09 raghug joined #gluster
02:24 jag3773 joined #gluster
02:26 phox joined #gluster
02:28 Oneiroi joined #gluster
02:31 harish joined #gluster
02:54 saurabh joined #gluster
03:01 kshlm joined #gluster
03:03 kkeithley joined #gluster
03:15 bulde joined #gluster
03:18 bulde1 joined #gluster
03:22 Technicool joined #gluster
03:26 lalatenduM joined #gluster
03:31 hagarth joined #gluster
03:43 sgowda joined #gluster
03:50 jag3773 Hello, i just upgraded a 3 brick replica from 3.3 to 3.4 and I'm seeing that non of the self-heal daemons are running?
03:51 jag3773 any thoughts on that?
03:52 samppah did you restart glusterd after upgrade?
03:52 jag3773 yes, of course
03:52 jag3773 i followed the guide
03:52 jag3773 rolling upgrade
03:54 satheesh joined #gluster
03:54 jag3773 @paste
03:54 glusterbot jag3773: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
03:55 raghug joined #gluster
03:56 samppah oh ok
03:56 jag3773 the last few lines of the self heal log: http://paste.fedoraproject.org/27136/51759137/
03:56 glusterbot Title: #27136 Fedora Project Pastebin (at paste.fedoraproject.org)
03:57 jag3773 no selinux, no firewall, no dns... just to rule out those suggestions ;)
03:59 kshlm joined #gluster
04:00 bala joined #gluster
04:17 _pol joined #gluster
04:25 bharata joined #gluster
04:30 raghug joined #gluster
04:33 jag3773 well, i'll be around tomorrow if anyone has any ideas on my problem ^ ;)
04:35 satheesh joined #gluster
04:38 ngoswami joined #gluster
04:44 SynchroM joined #gluster
04:47 rjoseph joined #gluster
04:47 raghug joined #gluster
04:48 SynchroM I've been following Vijay's 3.4 upgrade guide here https://vbellur.wordpress.com/2013​/07/15/upgrading-to-glusterfs-3-4, upgrading from a 3.3.0 ubuntu install, and I'm getting this error: "glusterd: undefined symbol: glusterfs_ctx_new", and gluster won't run. This seems to be an old bug, or is related to old versions, but I don't know what I should be looking for. Any ideas?
04:48 glusterbot <http://goo.gl/20AhQQ> (at vbellur.wordpress.com)
04:53 kedmison joined #gluster
04:54 bulde joined #gluster
04:58 CheRi joined #gluster
05:06 vpshastry joined #gluster
05:08 kaushal_ joined #gluster
05:12 18WAD6RRI joined #gluster
05:27 shylesh joined #gluster
05:41 lalatenduM joined #gluster
05:48 ProT-0-TypE joined #gluster
05:52 ngoswami joined #gluster
05:56 psharma joined #gluster
05:57 skyw joined #gluster
06:05 satheesh joined #gluster
06:05 vimal joined #gluster
06:22 skyw joined #gluster
06:24 raghu joined #gluster
06:28 Recruiter joined #gluster
06:35 ProT-0-TypE joined #gluster
06:41 ProT-0-TypE joined #gluster
06:44 rgustafs joined #gluster
06:45 ekuric joined #gluster
06:47 vpshastry1 joined #gluster
06:53 ctria joined #gluster
06:53 ProT-0-TypE joined #gluster
06:54 vshankar joined #gluster
06:56 ricky-ticky joined #gluster
07:00 shireesh joined #gluster
07:06 kaushal_ joined #gluster
07:10 bala joined #gluster
07:11 ujjain joined #gluster
07:17 shireesh joined #gluster
07:19 31NAAANCQ joined #gluster
07:21 skyw joined #gluster
07:21 chirino joined #gluster
07:23 45PAA6XAV joined #gluster
07:25 ProT-0-TypE joined #gluster
07:40 bulde joined #gluster
07:47 ngoswami joined #gluster
07:55 m0zes joined #gluster
08:00 piotrektt joined #gluster
08:12 raghug joined #gluster
08:21 StarBeast joined #gluster
08:27 satheesh1 joined #gluster
08:28 Norky joined #gluster
08:30 vpshastry joined #gluster
08:31 SynchroM In a clean install, any idea why I might get "missing 'option transport-type'. defaulting to "socket"", when transport-type is right there in the default volfile?
08:31 SynchroM (3.4 source install)
08:35 StarBeast joined #gluster
08:36 atrius joined #gluster
08:54 StarBeast joined #gluster
09:22 hybrid5123 joined #gluster
09:34 hateya_ joined #gluster
09:36 psharma joined #gluster
09:39 odyssey4me joined #gluster
09:39 odyssey4me left #gluster
09:44 psharma joined #gluster
09:46 skyw joined #gluster
09:53 manik joined #gluster
09:55 psharma joined #gluster
10:01 spider_fingers joined #gluster
10:01 piotrektt joined #gluster
10:01 piotrektt joined #gluster
10:04 skyw joined #gluster
10:06 psharma joined #gluster
10:30 satheesh joined #gluster
10:30 edward1 joined #gluster
10:31 duerF joined #gluster
10:37 samppah @upgrade
10:37 glusterbot samppah: I do not know about 'upgrade', but I do know about these similar topics: '3.3 upgrade notes', '3.4 upgrade notes'
10:40 samppah @3.4 upgrade notes
10:40 glusterbot samppah: http://goo.gl/SXX7P
10:59 bivak joined #gluster
10:59 rgustafs joined #gluster
10:59 gluslog joined #gluster
10:59 mtanner_ joined #gluster
10:59 jones_d joined #gluster
11:00 tjikkun_work joined #gluster
11:00 JoeJulian joined #gluster
11:00 penglish joined #gluster
11:00 masterzen joined #gluster
11:01 sonne joined #gluster
11:01 duerF joined #gluster
11:01 glusterbot joined #gluster
11:11 satheesh joined #gluster
11:12 ekuric1 joined #gluster
11:20 vpshastry joined #gluster
11:20 CheRi joined #gluster
11:22 lalatenduM joined #gluster
11:40 partner_ hello
11:40 glusterbot partner_: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
11:40 partner_ don't start with me bot.. :)
11:40 portante hello
11:40 glusterbot portante: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
11:41 * portante couldn't resist
11:41 partner_ anyways, i need to check who hinted me about the problem with cluster.min-free-disk but it seems i either confused the bits/bytes or it does not kick in with 3.3.1...
11:41 partner_ how do YOU read this:
11:41 partner_ Options Reconfigured:
11:41 partner_ cluster.min-free-disk: 615726511554
11:42 bulde1 joined #gluster
11:43 portante 615,726,511,554 bytes of minimum free disk space
11:43 skyw joined #gluster
11:43 portante or about 600GB (I think you can also use that syntax)
11:43 partner which translates to 573 GB
11:43 partner i was hinted not to use that syntax due to some issue
11:44 partner so hence i set it in bytes
11:44 portante is this working for you?
11:44 partner i just returned from holiday to find out old bricks are down to under 300 GB each..
11:44 partner so no its not working as i expected
11:45 portante that is a bummer
11:45 partner the newest brick (in distributed setup) still has terabytes free..
11:45 vpshastry1 joined #gluster
11:45 portante what does gluster volume info report?
11:45 partner that number above plus the proper bricks, nothing special
11:46 partner three brick setup
11:46 portante replication?
11:46 partner i just said in distributed setup 4 lines above :)
11:47 partner i could start rebalance again but that just probably will break the volume due to known bug with filehandles..
11:47 portante sorry, partner
11:47 partner or at least i need to be manually starting/stopping/monitoring it constantly
11:48 portante did you add the newest brick after setting the parameter to 573GB?
11:48 partner no, the parameter was set after i already had three bricks and after failed rebalance operations which caused some of the bricks to be in 100% disk utilization for a week..
11:49 portante ugh, sorry
11:49 partner it was set so that old bricks would NOT get new data in during my vacation (to prevent filling up)
11:49 portante do any of the files grow?
11:49 portante or is it just new files that are added?
11:49 partner no, everything is written once and only new files are added
11:50 portante avg size?
11:50 portante lots of small, large?
11:51 partner i had some stats somewhere, i can't find them right now..
11:51 portante k
11:52 portante I would file a bz to give yourself a place to put all this together so that folks can review it and comment
11:54 partner i rather first investigate a bit to find MY error rather than bug more wider audience with my crap
11:55 partner ie. ask around and talk with people
11:55 ricky-ticky joined #gluster
11:58 partner https://bugzilla.redhat.com/show_bug.cgi?id=874554
11:58 glusterbot <http://goo.gl/xbQQC> (at bugzilla.redhat.com)
11:58 glusterbot Bug 874554: unspecified, medium, ---, rtalur, ON_QA , cluster.min-free-disk not having an effect on new files
12:00 ekuric joined #gluster
12:00 partner i think the idea of setting it in bytes was to circumvent the bug
12:00 ekuric joined #gluster
12:02 partner maybe it just doesn't work and only option i have is to try to manage with rebalance somehow. or was i supposed to do some fix-layout after setting that?
12:17 ctria joined #gluster
12:18 manik joined #gluster
12:20 jmeeuwen if you don't mind me repeating yesterday's question - i've configured a 2x2 for testing storing an imap spool on glusterfs (i.e. many small files) - when i shut down the two replicas (on purpose) to check out the automatic self-healing on recovery, it seems that the replicas do not actually automatically start healing themselves - is there anything else i need to do / take in to account, like configure something?
12:21 pkoro joined #gluster
12:26 MACscr joined #gluster
12:31 bulde joined #gluster
12:32 CheRi joined #gluster
12:58 xdexter joined #gluster
12:58 xdexter if I have two bricks replicated between two servers and one of them is off the air for several hours, when that server unavailable to return, it will synchronize data with another server automatically?
12:59 dobber joined #gluster
13:00 mmalesa joined #gluster
13:03 ekuric joined #gluster
13:05 toad JoeJulian, nice the bug report you created fixed also: https://bugzilla.redhat.com/show_bug.cgi?id=987126
13:05 glusterbot <http://goo.gl/Kha58a> (at bugzilla.redhat.com)
13:05 glusterbot Bug 987126: urgent, urgent, ---, kaushal, MODIFIED , core: trying to create 64K dirs, all brick go down with crash
13:10 fleducquede joined #gluster
13:13 satheesh joined #gluster
13:13 jcsp joined #gluster
13:13 failshell joined #gluster
13:15 hybrid5123 joined #gluster
13:20 failshell joined #gluster
13:21 plarsen joined #gluster
13:22 pkoro hi everyone, I have a short question. We are about to upgrade our gluster ring from 3.3.1 to 3.4.0. We have seen the procedure on http://vbellur.wordpress.com/2013/​07/15/upgrading-to-glusterfs-3-4/ .
13:22 glusterbot <http://goo.gl/SXX7P> (at vbellur.wordpress.com)
13:22 pkoro The question is
13:23 pkoro is it preferable to stop the volumes prior to starting the procedure?
13:23 pkoro because this it not explicitly mentioned in the link I mentioned above
13:27 kedmison joined #gluster
13:28 shylesh joined #gluster
13:33 manik joined #gluster
13:40 hybrid5122 joined #gluster
13:49 raghug joined #gluster
13:51 xdexter if I have two bricks replicated between two servers and one of them is off the air for several hours, when that server unavailable to return, it will synchronize data with another server automatically?
13:53 X3NQ joined #gluster
13:57 aliguori joined #gluster
14:00 zaitcev joined #gluster
14:01 plarsen joined #gluster
14:03 ctria joined #gluster
14:04 bugs_ joined #gluster
14:05 hybrid5123 joined #gluster
14:07 jskinner_ joined #gluster
14:09 dbruhn joined #gluster
14:12 yongtaof joined #gluster
14:12 yongtaof Hi
14:12 glusterbot yongtaof: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:13 yongtaof I have encountered an issue related to glusterfs quota.
14:13 yongtaof volume quota limit-usage crashes glusterd
14:13 yongtaof (gdb) bt
14:13 yongtaof #0  0x0000003ef50328a5 in raise () from /lib64/libc.so.6
14:13 yongtaof #1  0x0000003ef5034085 in abort () from /lib64/libc.so.6
14:13 yongtaof #2  0x0000003ef506fa37 in __libc_message () from /lib64/libc.so.6
14:13 yongtaof #3  0x0000003ef5075366 in malloc_printerr () from /lib64/libc.so.6
14:13 yongtaof #4  0x0000003182214365 in data_destroy (data=0x7f26dd329b24) at dict.c:135
14:13 yongtaof #5  0x0000003182214ff3 in _dict_set (this=0x7f26dd50b80c,
14:13 yongtaof key=<value optimized out>, value=0x7f26dd34d7c0) at dict.c:248
14:13 yongtaof #6  dict_set (this=0x7f26dd50b80c, key=<value optimized out>,
14:13 yongtaof value=0x7f26dd34d7c0) at dict.c:302
14:14 yongtaof #7  0x00007f26db2e0c9e in glusterd_quota_limit_usage (volinfo=0x32e9170,
14:14 yongtaof dict=0x7f26dd517074, op_errstr=0x7fffa6fa8d78) at glusterd-quota.c:593
14:14 yongtaof #8  0x00007f26db2e1195 in glusterd_op_quota (dict=0x7f26dd517074,
14:14 yongtaof op_errstr=0x7fffa6fa8d78) at glusterd-quota.c:724
14:14 yongtaof #9  0x00007f26db2acb48 in glusterd_op_commit_perform (op=<value optimized out>,
14:14 yongtaof dict=0x7f26dd517074, op_errstr=<value optimized out>,
14:14 yongtaof rsp_dict=0x7f26dd517020) at glusterd-op-sm.c:3429
14:14 yongtaof #10 0x00007f26db2adf4d in glusterd_op_ac_commit_op (event=<value optimized out>,
14:14 yongtaof ctx=0x33017b0) at glusterd-op-sm.c:3195
14:14 yongtaof #11 0x00007f26db2abbd6 in glusterd_op_sm () at glusterd-op-sm.c:5014
14:14 yongtaof #12 0x00007f26db292f5b in glusterd_handle_commit_op (req=<value optimized out>)
14:14 yongtaof at glusterd-handler.c:669
14:14 yongtaof #13 0x0000003182a0a2e3 in rpcsvc_handle_rpc_call (svc=0x2e69d20,
14:14 yongtaof trans=<value optimized out>, msg=<value optimized out>) at rpcsvc.c:513
14:14 yongtaof ---Type <return> to continue, or q <return> to quit---
14:14 yongtaof #14 0x0000003182a0a453 in rpcsvc_notify (trans=0x3318310,
14:14 yongtaof mydata=<value optimized out>, event=<value optimized out>,
14:14 yongtaof data=<value optimized out>) at rpcsvc.c:612
14:14 yongtaof #15 0x0000003182a0aeb8 in rpc_transport_notify (this=<value optimized out>,
14:14 yongtaof event=<value optimized out>, data=<value optimized out>)
14:14 yongtaof at rpc-transport.c:489
14:14 yongtaof #16 0x00007f26db006784 in socket_event_poll_in (this=0x3318310) at socket.c:1677
14:14 yongtaof #17 0x00007f26db006867 in socket_event_handler (fd=<value optimized out>,
14:14 yongtaof idx=41, data=0x3318310, poll_in=1, poll_out=0,
14:14 yongtaof poll_err=<value optimized out>) at socket.c:1792
14:14 yongtaof #18 0x000000318223e4e4 in event_dispatch_epoll_handler (event_pool=0x2e64e50)
14:14 yongtaof at event.c:785
14:14 yongtaof #19 event_dispatch_epoll (event_pool=0x2e64e50) at event.c:847
14:14 yongtaof #20 0x0000000000407420 in main ()
14:14 yongtaof (gdb)
14:15 yongtaof any one know this issue?
14:15 kedmison yongtaof: please use @paste for big dumps of data like this.
14:15 kedmison @paste
14:15 glusterbot kedmison: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
14:15 yongtaof sorry for span
14:16 VIP-ire joined #gluster
14:16 yongtaof run gluster volume quota limit-usage crashes glusterd process
14:16 kedmison I'm still a gluster newbie so unfortunately I can't help you, but having it in a paste site makes it easier for those who can help you.
14:16 VIP-ire hi there
14:16 yongtaof 0x0000003182214365 in data_destroy (data=0x7f26dd329b24) at dict.c:135
14:17 jskinner_ yongtaof, try pasting your output at fpaste.org, and then copying the link here
14:17 yongtaof ok thank you for your advice
14:17 jskinner_ makes it much nicer :)
14:18 VIP-ire can anybody points me to a documentation on GlusterFS 3.4 ? I dont find anything here: http://www.gluster.org/community/​documentation/index.php/Main_Page (the full Admin Guide is only for GlusterFS 3.3)
14:18 glusterbot <http://goo.gl/eAVvs> (at www.gluster.org)
14:19 yongtaof Yes here's the log
14:19 yongtaof http://fpaste.org/27205/74589165/
14:19 VIP-ire I've just upgraded a replicated volume from 3.3 to 3.4. It's used to host qemu images. Performances in 3.4 are much (much much much....) better, but there's a conflict with port allocation for qemu migration
14:19 glusterbot Title: #27205 Fedora Project Pastebin (at fpaste.org)
14:20 yongtaof actually it's the core file
14:20 yongtaof I find it hard to debug since I can't reproduce it on the test bed
14:20 yongtaof And I can't debug on the online servers
14:21 yongtaof the quota limit-usage command crashes glusterd process every time
14:21 vpshastry1 left #gluster
14:24 satheesh joined #gluster
14:30 kaptk2 joined #gluster
14:34 hybrid5122 joined #gluster
14:41 soukihei joined #gluster
14:45 harish joined #gluster
14:47 mooperd joined #gluster
14:50 ngoswami_ joined #gluster
14:52 hateya_ joined #gluster
15:00 bstr__ joined #gluster
15:00 bstr__ Hey guys
15:00 bstr i have a 2 node gluster setup, and one of the boxes crashed (unrecoverable) - this box had to be rebuilt
15:01 bstr is there a configuration file i can copy over to the newly rebuilt host and self-heal the filesystem?
15:01 bstr running gluster 3.3
15:01 bstr on fedora 18
15:03 bstr i *cannot* loose the data on the brick
15:05 VIP-ire I'm very new to gluster, but I think you first have to re-configure the old UUID (on the bricks which is still working, find the UUID in /var/lib/gluster/peers/ and restore that UUID on the new serveur in /var/lib/glusterd/glusterd.info)
15:05 VIP-ire then, restart glusterd on the new server
15:05 VIP-ire probe the other peer (gluster peer probe <still-working-server>)
15:06 VIP-ire and try to sync volume informations (gluster volume sync <still-working-server> all)
15:06 bstr only file i see in /etc/glusterfs is : /etc/glusterfs/glusterd.vol
15:06 VIP-ire not /etc/gluster, /var/lib/gluster
15:06 bstr VIP-ire : sorry, misread your text, let me take a look
15:07 VIP-ire if someone with more experience can confirm my steps.... (I've just built my first gluster install last week ^^)
15:11 ngoswami joined #gluster
15:12 bstr Ill hold off until someone confirms, thanks for the assist!
15:12 vpshastry joined #gluster
15:13 _pol joined #gluster
15:14 ujjain joined #gluster
15:15 MediaSmurf joined #gluster
15:16 VIP-ire if someone knows how to configure the port range used by glusterfsd in gluster 3.4, I'd be interested (using the RHEL6 rpm, and I'd prefer not re-compiling everything just to change the port range)
15:17 neofob left #gluster
15:18 bala1 joined #gluster
15:26 recidive joined #gluster
15:28 _pol joined #gluster
15:30 neofob joined #gluster
15:32 mmalesa joined #gluster
15:39 _pol joined #gluster
15:40 semiosis VIP-ire: what port range specifically?
15:41 VIP-ire since 3.4 it uses the IANA defined ephemeral port range (which starts at 49152)
15:41 VIP-ire the problem is that qemu live migration also uses this port range
15:42 VIP-ire and this prevent migration from time to time (not always, depend on the port glusterfsd and qemu choose in this range)
15:42 semiosis @forget undocumented options
15:42 glusterbot semiosis: The operation succeeded.
15:42 semiosis @learn undocumented options as Undocumented options for 3.4: http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented
15:42 glusterbot semiosis: The operation succeeded.
15:43 semiosis VIP-ire: you should probably file a bug about that
15:43 glusterbot http://goo.gl/UUuCq
15:43 semiosis there ^
15:43 VIP-ire ok, thansk, will do
15:43 spider_fingers left #gluster
15:46 sprachgenerator joined #gluster
15:50 semiosis JoeJulian: how's everything going in PDX?
16:02 vpshastry joined #gluster
16:04 piotrektt_alpha joined #gluster
16:05 manik joined #gluster
16:07 VIP-ire Bug opened (for port conflict with qemu live migration): https://bugzilla.redhat.com/show_bug.cgi?id=987555
16:07 glusterbot <http://goo.gl/SbL8x> (at bugzilla.redhat.com)
16:07 glusterbot Bug 987555: unspecified, unspecified, ---, rgowdapp, NEW , Glusterfs ports conflict with qemu live migration
16:13 daMaestro joined #gluster
16:14 hybrid5123 joined #gluster
16:18 NuxRo anyone played with the gluster Samba plugin/module? Is it usable?
16:20 TuxedoMan joined #gluster
16:20 CheRi joined #gluster
16:22 lalatenduM joined #gluster
16:29 bstr joined #gluster
16:30 manik joined #gluster
16:37 bulde joined #gluster
16:38 Kins joined #gluster
16:38 hagarth joined #gluster
16:39 _pol joined #gluster
16:41 failshell joined #gluster
16:42 mmalesa joined #gluster
16:46 recidive joined #gluster
16:48 satheesh joined #gluster
16:49 vpshastry left #gluster
16:57 xdexter if I have two bricks replicated between two servers and one of them is off the air for several hours, when that server unavailable to return, it will synchronize data with another server automatically?
16:58 jskinner_ does any one know off hand which version of CentOS has the correct versions of libvirt and qemu for utilizing libgfapi?
16:59 VIP-ire @jskinner_: AFAIK none. CentOS 6.4 (the latest) only provides qemu-kvm 0.12 and libvirt 0.10.2
16:59 jskinner_ Dang, will have to test with Fedora then. Thanks!
16:59 VIP-ire maybe the native gluster (with libgfapi) support will be backported for RHEL 6.5 (so CentOS 6.5), but I haven't find any information on this
16:59 jag3773 I posted last night, but to recap, if anyone has any ideas on why the gluster self-heal daemon would not be running after an upgrade from 3.3 to 3.4 I'd love to hear it.... I have a 3 brick replica, glustershd.log at http://paste.fedoraproject.org/27136/51759137/.  Thoughts?
17:00 glusterbot Title: #27136 Fedora Project Pastebin (at paste.fedoraproject.org)
17:00 jskinner_ that would be cool if that happened lol.
17:00 VIP-ire I agree (I'm just installing a dual node cluster for KVM using GlusterFS)
17:00 VIP-ire but anyway, it's working fine with the fuse mount point
17:02 semiosis jag3773: connection to 127.0.0.1:24007 failed (Connection refused) -- glusterd is not running?
17:02 VIP-ire performances are more than OK since GlusterFS 3.4 (I've doubled the write speed in my guests when I upgraded to 3.4, the network is the limit now, I only have 1Gb between my two nodes)
17:02 jag3773 semiosis,  root     12769     1  0 Jul22 ?        00:00:03 /usr/sbin/glusterd -p /var/run/glusterd.pid
17:03 samppah jskinner_: there is a bug report that mentions that libgfapi support will be in EL 6.5
17:04 jskinner_ Would be nice to see
17:04 semiosis jag3773: what does netstat -anp say about process 12769?
17:04 jag3773 but thanks semiosis, i need to enable the glusterd log and see what i get
17:04 semiosis iptables maybe?
17:04 jskinner_ thanks samppah
17:04 jag3773 no firewall
17:04 semiosis jag3773: can always try restarting glusterd
17:04 jag3773 tried that
17:04 jag3773 but i'lltry again when i enable logging...
17:04 jskinner_ trying to improve my performance for VM storage, and I think bypassing Fuse will be my best bet.
17:05 jag3773 it is listening semiosis : tcp        0      0 0.0.0.0:24007               0.0.0.0:*                   LISTEN      12769/glusterd
17:05 semiosis glusterd isn't logging?  how'd you manage that?  it usually logs by default into /var/log/glusterfs/etc-glusterfs-glusterd.log or similar
17:06 _pol joined #gluster
17:07 jag3773 the docs on that are conflicting then semiosis
17:07 semiosis what docs?
17:08 semiosis or what conflict?
17:08 jag3773 `glusterd --help` and /etc/sysconfig/glusterd
17:08 jag3773 they both say the default is /var/log/gluster/gluster.log
17:08 semiosis hmm is that new with 3.4?
17:09 jag3773 yes, 3.3 say /var/log/glusterfs/glusterfs.log
17:09 semiosis well thats news to me
17:10 jag3773 okay, so i was looking in the right place to begin with too ;)
17:10 semiosis must be an rpm/rhel/fedora thing
17:11 semiosis what distro are you using?
17:11 jag3773 this is amazon linux
17:11 jag3773 the RPM is from gluster-epel though
17:11 semiosis i see
17:14 jag3773 @paste
17:14 glusterbot jag3773: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
17:14 jag3773 semiosis: http://paste.fedoraproject.org/27257/59966713/
17:14 glusterbot Title: #27257 Fedora Project Pastebin (at paste.fedoraproject.org)
17:14 jag3773 does that help at all?  is rdma required to be installed?
17:15 jag3773 i have the following installed: glusterfs-fuse-3.4.0-2.el6.x86_64  glusterfs-3.4.0-2.el6.x86_64  glusterfs-server-3.4.0-2.el6.x86_64
17:16 semiosis try it & you tell me if rdma is required :)
17:16 semiosis idk
17:18 jebba joined #gluster
17:19 jag3773 I also see op_ctx modification failed in the log
17:19 jag3773 not sure if that is related
17:20 jag3773 anyway to troubleshoot the self-heal daemon?
17:20 semiosis what more do you want to troubleshoot?  clearly it wants to communicate with glusterd but can not
17:20 mooperd joined #gluster
17:20 jag3773 haha, that's hardly a prognosis... how about the why?
17:22 semiosis can you telnet localhost 24007?
17:23 jag3773 yep
17:23 semiosis then i would tcpdump on lo to see whats really going on between those two
17:23 xdexter if I have two bricks replicated between two servers and one of them is off the air for several hours, when that server unavailable to return, it will synchronize data with another server automatically?
17:24 semiosis xdexter: it should
17:24 semiosis xdexter: since glusterfs 3.3
17:24 semiosis oh wait i misunderstood
17:25 semiosis glusterfs will not automatically replace one brick with another
17:25 semiosis but when the down brick returns to service it should automatically sync up
17:25 xdexter exact
17:25 xdexter ok
17:27 hagarth jag3773: do you see errors about privileged port etc. in your glusterd log?
17:29 jag3773 no hagarth, the last couple lines here show that it apparently can't start some service http://paste.fedoraproject.org/27257/59966713/
17:29 glusterbot Title: #27257 Fedora Project Pastebin (at paste.fedoraproject.org)
17:30 jag3773 oh, so rdma is for inifiband semiosis -- not for me ;)
17:30 semiosis right, at least until we see Amazon Elastic InfiniBand
17:31 semiosis which would be pretty sweet
17:31 _pol joined #gluster
17:33 hagarth jag3773: can you try gluster volume start <volname> force?
17:33 jag3773 mmm... the volume is running hagarth... would that have a negative effect?
17:34 hagarth jag3773: no, it just attempts starting services that are not running.
17:36 jag3773 no change there hagarth
17:37 hagarth jag3773: the same error message in log file?
17:37 jag3773 I see op_ctx modification failed
17:37 jag3773 which was there before too
17:40 hagarth that should not cause a problem. Does volume status list self-heal daemon as offline?
17:41 Humble joined #gluster
17:43 jag3773 crap, no i'm getting "no volumes present"
17:45 hagarth jag3773: something is wrong.. can you fpaste your glusterd log?
17:45 jag3773 oddly, this just came up: [2013-07-22 20:00:30.100419] I [client-handshake.c:1636:sele​ct_server_supported_programs] 0-supportgfs-client-2: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
17:45 jag3773 that was in the self heal log
17:46 jag3773 hagarth, http://paste.fedoraproject.org/27270/74601570/
17:46 glusterbot Title: #27270 Fedora Project Pastebin (at paste.fedoraproject.org)
17:49 jag3773 there are rpmsave files in /var/lib/glusterd/vols/supportgfs/ -- but i wouldn't think gluster would load those
17:49 jag3773 like supportgfs.172.26.177.180.media-ephem​eral0-supportgfs-readonly.vol.rpmsave
17:50 jag3773 would a reboot help here hagarth ?
17:51 hagarth jag3773: hang on, things look ok in glusterd logs.
17:53 hagarth jag3773: does gluster volume info also fail?
17:54 mooperd joined #gluster
17:54 jag3773 ok, phew... i restarted glusterd on this node and i'm getting responses now
17:55 jag3773 well... still not for the volume status thoough
17:55 jag3773 volume info completes hagarth , volume status returns " Unable to obtain volume status information."
17:56 semiosis have you tried restarting the instance?
17:56 hagarth jag3773: and the same op_ctx modification failed in the logs?
17:56 jag3773 yes
17:57 jag3773 the times in the gluster logs are way off from the system time, i'm not sure if that is related at all
17:57 hagarth jag3773: the log file timestamps are in UTC.
17:57 jag3773 that's what i was hoping ;)
17:59 hagarth jag3773: can you fpaste the new glusterd log?
18:00 jag3773 http://paste.fedoraproject.org/27277/60241713
18:00 glusterbot Title: #27277 Fedora Project Pastebin (at paste.fedoraproject.org)
18:01 jag3773 note that there is an extra node that i was trying to probe, but it is in a wonky state, so i just not issued a detach command for that node
18:01 jag3773 that's why you see those detach related lines at the end of that
18:01 hagarth and even now the detach fails?
18:02 hagarth rather volume status fails?
18:02 jag3773 correct
18:04 semiosis have you upgraded all servers in the cluster?
18:04 jag3773 hagarth, do you think it would help to stop all glusters in the cluster and then start them afresh?
18:05 jag3773 yes semiosis
18:05 hagarth what happens if you execute volume status from a different node in the cluster?
18:05 pono joined #gluster
18:05 jag3773 same thing
18:06 hagarth does gluster peer status list all nodes as connected?
18:06 jag3773 I could try a rolling restart on these nodes too
18:06 jag3773 no hagarth the new node i was trying to add is messed up
18:06 jag3773 Hostname: 172.26.177.232
18:06 jag3773 Uuid: 604480be-1e67-476f-9586-31aceb51bf58
18:06 jag3773 State: Probe Sent to Peer (Connected)
18:07 jag3773 that entry is only on the node that i tried to probe from -- the other cluster members don't list .232
18:07 hagarth how many servers do you have in the cluster?
18:07 jag3773 originally 4 bricks... one brick died and so i removed that fromthe cluster leaving me with a 3 brick replica
18:07 jag3773 that was all on 3.3 -- then i upgraded to 3.4
18:07 jag3773 eventually i need to add the 4th node back in, but that's not urgent
18:08 jag3773 part of the reason why i upgraded to 3.4 is because 3.3.0 has a peer probe problem
18:08 jag3773 at least i've hit this problem on two different clusters
18:08 hagarth the new node that is being added - is that a 3.4 installation as well?
18:08 jag3773 yes hagarth
18:09 jag3773 ip-172-26-177-179: glusterfs-fuse-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-177-179: glusterfs-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-177-179: glusterfs-server-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-177-232: glusterfs-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-177-232: glusterfs-fuse-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-177-232: glusterfs-server-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-178-13: glusterfs-server-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-178-13: glusterfs-fuse-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-178-13: glusterfs-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-178-244: glusterfs-server-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-178-244: glusterfs-fuse-3.4.0-2.el6.x86_64
18:09 jag3773 ip-172-26-178-244: glusterfs-3.4.0-2.el6.x86_64
18:09 jag3773 ^that's all the gluster packages
18:09 jag3773 232 is the new node
18:09 hagarth can you attempt peer detach 232 force
18:10 jag3773 in progress...
18:13 jag3773 that worked, except 232 is still trying to establish
18:13 jag3773 run detach on 232 too?
18:13 jag3773 hagarth, ^
18:13 hagarth you can try detach force on 232 too.
18:14 jag3773 hagarth: [root@ip-172-26-177-179 glusterfs]# gluster volume info
18:14 jag3773
18:14 jag3773 Volume Name: supportgfs
18:14 jag3773 Type: Replicate
18:14 jag3773 Volume ID: 89460534-6d2d-4e4e-921e-71bddd533e4e
18:14 jag3773 Status: Started
18:14 jag3773 Number of Bricks: 0 x 4 = 3
18:14 jag3773 Transport-type: tcp
18:14 jag3773 Bricks:
18:14 jag3773 Brick1: 172.26.178.13:/media/ephem​eral0/supportgfs-readonly
18:14 jag3773 Brick2: 172.26.177.179:/media/ephe​meral0/supportgfs-readonly
18:14 jag3773 Brick3: 172.26.178.244:/media/ephe​meral0/supportgfs-readonly
18:15 jag3773 Options Reconfigured:
18:15 jag3773 auth.allow: 172.26.*
18:15 jag3773 performance.cache-size: 128MB
18:15 jag3773 oops, i meant to fpaste that
18:15 jag3773 that is my volume info hagarth, also at http://paste.fedoraproject.org/27282/03277137
18:15 glusterbot Title: #27282 Fedora Project Pastebin (at paste.fedoraproject.org)
18:15 jag3773 looks like it thinks it should have 4 bricks
18:16 hagarth looks weird. is this consistent on all nodes?
18:16 jag3773 no volumes present when run on the other nodes
18:18 jag3773 hagarth, i have to run to a meeting... the cluster is operational, but i'm a bit nervous about the lack of output in these commands now... I may try a rolling reboot later today, i'll be back on here later today
18:19 hagarth jag3773: i will also be afk now.. ttyl
18:19 jag3773 if you or anyone else has any ideas, please shoot me an email at jesse@tummy.com
18:19 jag3773 thanks
18:30 Recruiter joined #gluster
18:30 _pol joined #gluster
18:34 Technicool joined #gluster
18:43 jclift joined #gluster
18:48 bulde joined #gluster
18:49 _pol joined #gluster
18:55 Technicool joined #gluster
19:04 TuxedoMan joined #gluster
19:05 Technicool JoeJulian, https://bugzilla.redhat.com/show_bug.cgi?id=987624
19:05 glusterbot <http://goo.gl/rBmJ1W> (at bugzilla.redhat.com)
19:05 glusterbot Bug 987624: low, unspecified, ---, kaushal, NEW , Feature Request: Color code replica or distribute pairings output from gluster volume info
19:06 semiosis Technicool: he seems to be afk today, i think he's busy with the gluster pdx/oscon event
19:07 semiosis pretty cool idea in that bug report!
19:21 puebele joined #gluster
19:29 TuxedoMan joined #gluster
19:31 ujjain2 joined #gluster
19:35 bfoster joined #gluster
19:36 bstr anyone know how to replace a crashed server without loosing data on the existing brick (Two node setup)
19:37 semiosis @replace
19:37 glusterbot semiosis: (replace [<channel>] <number> <topic>) -- Replaces topic <number> with <topic>.
19:37 semiosis ,,(replace)
19:37 glusterbot Useful links for replacing a failed server... if replacement server has different hostname: http://goo.gl/4hWXJ ... or if replacement server has same hostname:
19:37 glusterbot http://goo.gl/rem8L
19:38 semiosis @forget replace
19:38 glusterbot semiosis: The operation succeeded.
19:38 semiosis @learn replace as Useful links for replacing a failed server... if replacement server has different hostname: http://web.archive.org/web/20120508153302/h​ttp://community.gluster.org/q/a-replica-nod​e-has-failed-completely-and-must-be-replace​d-with-new-empty-hardware-how-do-i-add-the-​new-hardware-and-bricks-back-into-the-repli​ca-pair-and-begin-the-healing-process/ ... or if replacement server has same hostname: http://goo.gl/rem8L
19:38 glusterbot semiosis: The operation succeeded.
19:38 semiosis ,,(replace)
19:38 glusterbot Useful links for replacing a failed server... if replacement server has different hostname: http://goo.gl/nIS6z ... or if replacement server has same hostname:
19:38 glusterbot http://goo.gl/rem8L
19:43 VIP-ire joined #gluster
19:46 bfoster joined #gluster
19:47 bstr semiosis : im using gluster 3.3, i would use the same proccess?
19:48 _pol joined #gluster
20:00 semiosis probably
20:00 semiosis what exactly are you trying to do?
20:03 bstr i have (had) a two node gluster setup in production, and one of the serveres died, causing me to rebuild it.
20:04 bstr i would like to bring the newly rebuilt node back into the gluster cluster without loosing any data off the brick
20:05 semiosis i generally advocate using hostnames and doing the same-hostname replacement procedure glusterbot linked
20:05 semiosis when that sort of thing happens
20:05 bstr that would be prefered for myself as well (currently setup with same hostname)
20:06 semiosis afaik that procedure should work with all versions of glusterfs since 3.1
20:06 semiosis if not, we'll update the page!
20:07 bstr looks like the tutorial on the gluster page is for 3.2 -- 3.3 does not look to have the same contents unter /etc/glusterd/*
20:07 bstr http://bit.ly/12gwzy0
20:07 glusterbot Title: Gluster 3.2: Brick Restoration - Replace Crashed Server - GlusterDocumentation (at bit.ly)
20:08 semiosis ok that's now /var/lib/glusterd, since 3.3 -- i'll fix the doc
20:10 semiosis fixed, thank you
20:11 edong23 joined #gluster
20:13 bstr semiosis : Thanks!
20:13 semiosis yw
20:19 jag3773 joined #gluster
20:24 _pol joined #gluster
20:38 _pol joined #gluster
20:39 mooperd joined #gluster
20:39 mmalesa joined #gluster
20:43 _br_ joined #gluster
20:46 vincent_vdk joined #gluster
20:53 bit4man joined #gluster
20:54 jag3773 so i'm still having the same weird issues as mentioned before.. the most interesting thing about it now is: Number of Bricks: 0 x 4 = 3
20:55 jag3773 that doesn't make sense, the full output may be seen at http://paste.fedoraproject.org/27282/03277137
20:55 glusterbot Title: #27282 Fedora Project Pastebin (at paste.fedoraproject.org)
21:00 jag3773 phew... well i'm back to where i started this morning... the self-heal daemons are not running but everything is returning as expected
21:03 nightwalk joined #gluster
21:10 MugginsM joined #gluster
21:19 bstr semiosis : restoring that brick worked flawlessly, thanks again!
21:20 bstr MUCH easier than i was expecting
21:25 MugginsM joined #gluster
21:31 _pol joined #gluster
21:39 StarBeas_ joined #gluster
21:50 jebba joined #gluster
22:12 chirino joined #gluster
22:19 _pol joined #gluster
22:20 hchiramm_ joined #gluster
22:26 soukihei joined #gluster
22:32 mooperd left #gluster
22:32 TuxedoMan joined #gluster
22:33 _pol joined #gluster
22:45 _pol joined #gluster
22:45 TuxedoMan joined #gluster
23:09 semiosis bstr: glad to hear that document was helpful.  /cc JoeJulian
23:10 txdoman joined #gluster
23:26 MugginsM joined #gluster
23:27 hchiramm_ joined #gluster
23:33 mmalesa joined #gluster
23:34 fidevo joined #gluster
23:36 ProT-0-TypE joined #gluster
23:40 basicer joined #gluster
23:41 basicer How does gluster determin if a peer is up?  Ive got a server offline, and my other server seems to think it is still connected.
23:46 StarBeast joined #gluster
23:51 duerF joined #gluster
23:54 hchiramm_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary