Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-10-10

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 cjanbanan joined #gluster
00:07 siel joined #gluster
00:08 plarsen joined #gluster
00:20 calisto joined #gluster
00:27 siel joined #gluster
00:32 Pupeno joined #gluster
00:32 ivok_ joined #gluster
00:33 tdasilva joined #gluster
00:37 ivok joined #gluster
01:11 bala joined #gluster
01:19 justinmburrous joined #gluster
01:22 justinmburrous joined #gluster
01:31 haomaiwa_ joined #gluster
01:45 calisto joined #gluster
01:46 haomai___ joined #gluster
01:57 haomaiwa_ joined #gluster
02:18 diegows joined #gluster
02:22 ACiDGRiM joined #gluster
02:22 kdhananjay joined #gluster
02:34 xleo joined #gluster
02:36 bharata-rao joined #gluster
02:36 doubt joined #gluster
02:48 haomai___ joined #gluster
02:56 kanagaraj joined #gluster
03:05 msmith_ joined #gluster
03:30 justinmburrous joined #gluster
03:33 _Bryan_ joined #gluster
03:40 shubhendu joined #gluster
03:46 atinmu joined #gluster
03:51 frayz joined #gluster
03:51 itisravi joined #gluster
03:52 frayz has anyone seen a problem with `mount.glusterfs` not working while `gluster` works fine?  Since mount.glusterfs is a shell script, I printed out the cmd_line and when run interactively it works fine (if I copy/paste it into a shell) but from the script it doesn't work
03:53 frayz typo correction: the `glusterfs` command works fine when run by hand
03:56 coredump joined #gluster
03:58 frayz btw, i am running mount.glusterfs as root in case that makes a difference
03:59 nbalachandran joined #gluster
04:01 cjanbanan joined #gluster
04:04 john_locke joined #gluster
04:04 frayz trying to understand why /usr/sbin/glusterfs can mount a volume interactively but when run with the same arguments via mount.glusterfs it fails to mount
04:07 john_locke Guys, anyone who is on Elance? I'm in need of someone who can deploy a gluster-cluster with good performance with Windows Clients. We have good hardware every server with 2 10gb nics, 12 4U Centos servers, more than 1 Petabyte in space. Someone who can show experience with may windows clients, please contact me. Thanks in advance!
04:08 frayz interesting... if i prefix the cmd_line in mount.glusterfs with "strace" the mount works
04:16 gildub joined #gluster
04:26 frayz the problem i'm having seems to be a bug in the check whether the mount was successful.  it is successful but the volume is unmounted when the checks fail so it appeared that i couldn't mount the volume
04:28 anoopcs joined #gluster
04:31 rafi1 joined #gluster
04:31 Rafi_kc joined #gluster
04:35 frayz here is the solution, if someone would like to take the patch: http://pastebin.com/Tj0pYi8Q    Just a slight delay between mounting the volume and doing a stat on the mount point
04:35 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
04:36 jiffin joined #gluster
04:38 rjoseph joined #gluster
04:38 lalatenduM joined #gluster
04:39 kshlm joined #gluster
04:42 spandit joined #gluster
04:45 kdhananjay joined #gluster
04:55 nishanth joined #gluster
05:00 glusterbot New news from newglusterbugs: [Bug 1151303] Excessive logging in the self-heal daemon after a replace-brick <https://bugzilla.redhat.com/show_bug.cgi?id=1151303>
05:02 xleo joined #gluster
05:04 bala joined #gluster
05:06 prasanth_ joined #gluster
05:11 maxxx2014 joined #gluster
05:12 shubhendu joined #gluster
05:16 panpanfeng joined #gluster
05:30 glusterbot New news from newglusterbugs: [Bug 1151308] data loss when rebalance + renames are in progress and bricks from replica pairs goes down and comes back <https://bugzilla.redhat.com/show_bug.cgi?id=1151308>
05:34 overclk joined #gluster
05:35 justinmburrous joined #gluster
05:36 harish joined #gluster
05:46 LebedevRI joined #gluster
05:48 RaSTar joined #gluster
05:54 kdhananjay joined #gluster
05:56 LebedevRI joined #gluster
06:00 haomaiwa_ joined #gluster
06:01 glusterbot New news from newglusterbugs: [Bug 1147236] gluster 3.6.0 compatibility issue with gluster 3.3 <https://bugzilla.redhat.com/show_bug.cgi?id=1147236>
06:03 zerick joined #gluster
06:04 soumya_ joined #gluster
06:13 kshlm joined #gluster
06:14 Philambdo joined #gluster
06:16 haomaiw__ joined #gluster
06:18 atalur joined #gluster
06:19 saurabh joined #gluster
06:24 rgustafs joined #gluster
06:25 lalatenduM joined #gluster
06:25 rgustafs joined #gluster
06:26 raghu joined #gluster
06:30 Fen2 joined #gluster
06:32 RaSTar joined #gluster
06:36 coredump joined #gluster
06:38 Fen2 Hi, all :)
06:42 ndarshan joined #gluster
06:46 pkoro joined #gluster
06:49 lalatenduM joined #gluster
06:55 ctria joined #gluster
07:00 JoeJulian vertex: If you're shutting down for weeks, I would probably stop the volume. If you're not using it, though, it's safe to just pkill -f gluster
07:01 glusterbot New news from newglusterbugs: [Bug 1146902] Stopping or restarting glusterd on another node when volume start is in progress gives error messages but volume is started <https://bugzilla.redhat.com/show_bug.cgi?id=1146902>
07:01 JoeJulian frayz: if I were a guessing man, I'd look at selinux.
07:05 lalatenduM joined #gluster
07:06 ricky-ti1 joined #gluster
07:08 ricky-ticky2 joined #gluster
07:10 rolfb joined #gluster
07:15 justinmb_ joined #gluster
07:29 cjanbanan joined #gluster
07:36 fsimonce joined #gluster
07:47 ACiDGRiM will gluster support solaris bricks or is that a non-starter since it's managed by redhat?
07:48 Fen2 JoeJulian: Is the replicate volume a technology preview !?
07:48 sickness I'm still trying to understand if it would be possible to compile and run bricks under cygwin =_)
07:50 ACiDGRiM last info I could find is that 3.1.x supported solaris, but I can't get 3.1.7 to compile any further than 3.6 or below
07:52 cjanbanan joined #gluster
07:53 haomaiw__ joined #gluster
07:54 fubada joined #gluster
08:02 AtuM joined #gluster
08:04 liquidat joined #gluster
08:06 ivok joined #gluster
08:07 Fen2 Why in the red hat official guide they say replicate and stripe volume are just preview ?
08:07 Fen2 https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/index.html
08:07 glusterbot Title: Administration Guide (at access.redhat.com)
08:17 haomaiwa_ joined #gluster
08:19 AtuM does one "append" new servers into the volume definition in order to expand it (in replica mode) ?
08:19 AtuM I'm new to this.. reading the "howto" and wondering :)
08:22 AtuM does gluster provide any "verify" feature in order to check if the data across the replicas is valid?
08:25 rgustafs joined #gluster
08:26 gildub joined #gluster
08:31 haomai___ joined #gluster
08:31 glusterbot New news from newglusterbugs: [Bug 1151343] DHT: Rebalance process crash after add-brick and `rebalance start' operation <https://bugzilla.redhat.com/show_bug.cgi?id=1151343>
08:41 Norky joined #gluster
08:45 anands joined #gluster
08:57 hypnotortoise_ joined #gluster
09:00 Fen2 joined #gluster
09:00 hypnotortoise_ I'm playing around with diagnostics.client-log-level DEBUG to see, what is getting into the read cache / what is getting removed from it
09:00 hypnotortoise_ can someone point me in the right direction what to search for
09:02 hypnotortoise_ I suspect the read cache is not properly used. using gluster version 3.5.2
09:05 hypnotortoise_ also, how do I check, that read-cache is used properly on server & client? I see two .vol files ([volname]-fuse.vol & trusted-[volname]-fuse.vol) in /var/lib/glusterd/vols/[volname]. are those the relevant vols for the client?
09:09 vimal joined #gluster
09:11 FarbrorLeon joined #gluster
09:12 nellemann joined #gluster
09:13 RaSTar joined #gluster
09:13 spandit joined #gluster
09:24 rjoseph joined #gluster
09:25 Slashman joined #gluster
09:34 xleo joined #gluster
10:02 glusterbot New news from newglusterbugs: [Bug 1151384] Rebalance fails to complete - stale file handles after 202,908 files <https://bugzilla.redhat.com/show_bug.cgi?id=1151384>
10:05 ivok joined #gluster
10:05 gildub joined #gluster
10:08 Philambdo joined #gluster
10:09 gildub joined #gluster
10:09 glusterbot New news from resolvedglusterbugs: [Bug 1151343] DHT: Rebalance process crash after add-brick and `rebalance start' operation <https://bugzilla.redhat.com/show_bug.cgi?id=1151343>
10:11 shubhendu joined #gluster
10:25 atinmu joined #gluster
10:28 calisto joined #gluster
10:31 XpineX_ joined #gluster
10:32 glusterbot New news from newglusterbugs: [Bug 1151397] DHT: Rebalance process crash after add-brick and `rebalance start' operation <https://bugzilla.redhat.com/show_bug.cgi?id=1151397>
10:41 atalur joined #gluster
10:46 coreping joined #gluster
10:49 calisto joined #gluster
10:51 Sunghost joined #gluster
10:52 gildub joined #gluster
10:52 Sunghost Hey guys
10:53 Sunghost i use debian wheezy with stable kernel 3.2.0-4-amd64 and want to upgrade to newer backport version - i asked myself if glusterfs recommend a special kernel or have problems with newer ones?
11:00 aravindavk joined #gluster
11:01 atinmu joined #gluster
11:10 nellemann left #gluster
11:14 rjoseph joined #gluster
11:14 spandit-mtg joined #gluster
11:19 virusuy joined #gluster
11:31 frayz JoeJulian: fyi, selinux is disabled on my cluster
11:32 glusterbot New news from newglusterbugs: [Bug 1147252] Pacemaker OCF volume Resource Agent fails when bricks are in different domain to the system hostname. <https://bugzilla.redhat.com/show_bug.cgi?id=1147252>
11:38 cjanbanan Would there be no split-brain situations at all if I specify that favorite-child option? Would it be possible to stop the volume change favorite-child and restart it without causing any problems?
11:43 ira joined #gluster
11:45 klaxa joined #gluster
11:50 Fen1 joined #gluster
11:54 diegows joined #gluster
12:01 anands joined #gluster
12:05 soumya_ joined #gluster
12:16 soumya_ joined #gluster
12:16 itisravi_ joined #gluster
12:16 RaSTar joined #gluster
12:17 rjoseph joined #gluster
12:39 giannello joined #gluster
12:40 theron joined #gluster
12:49 virusuy joined #gluster
12:49 virusuy joined #gluster
12:55 edward1 joined #gluster
12:56 vstokes joined #gluster
12:57 vstokes Would someone please help me understand the brick logs. What does this mean:
12:57 vstokes [2014-10-10 12:48:44.399466] E [posix.c:193:posix_lookup] 0-d01-posix: buf->ia_gfid is null for /d01/.glusterfs/51/71/51718fdc-69c8-4fb0-8db8-883395eb7f37/archive/ECI_IFTSTA_20140717025850_147392286_1569806242.xml
13:04 msmith_ joined #gluster
13:08 virusuy joined #gluster
13:12 julim joined #gluster
13:36 calum_ joined #gluster
13:41 rwheeler joined #gluster
13:45 virusuy joined #gluster
13:45 virusuy joined #gluster
13:46 XpineX_ joined #gluster
13:54 ctria joined #gluster
13:55 P0w3r3d joined #gluster
14:00 vstokes left #gluster
14:05 mojibake joined #gluster
14:07 plarsen joined #gluster
14:09 mojibake Is there a Console(Dashboard) available for community GlusterFS similar to the RedHat Storage Console?
14:11 jbautista- joined #gluster
14:15 lmickh joined #gluster
14:18 cjanbanan Which file should I edit to set the favorite-child option? It isn't supported by the CLI and there are a couple of files to choose from.
14:21 fattaneh1 joined #gluster
14:23 plarsen joined #gluster
14:24 rjoseph joined #gluster
14:26 frayz i'm just new to glusterfs but when in doubt, i'd recommend that you fetch the gluster source code and search it for whatever option you're not sure about to see how it handles that option
14:27 JoeJulian Fen1: No idea why Red Hat says that. This channel is upstream from Red Hat.
14:27 bennyturns joined #gluster
14:28 JoeJulian hypnotortoise_: The volname-fuse.vol file is the one that's fed to the clients.
14:30 JoeJulian cjanbanan: If you want to eliminate split-brain, implement quorum.
14:31 cjanbanan quorum is not an option for us I'm afraid. Our two bricks has to be available at all times.
14:32 calisto joined #gluster
14:35 cjanbanan As far as I understand favorite-child is the perfect solution for us. I'm so happy that I found it, so I really want to try it out.
14:41 shubhendu joined #gluster
14:45 JoeJulian cjanbanan: Let us know how that works out. Do data loss is acceptable in your use case?
14:45 JoeJulian s/Do/So/
14:45 glusterbot What JoeJulian meant to say was: cjanbanan: Let us know how that works out. So data loss is acceptable in your use case?
14:46 cjanbanan Yes we accept data loss. We only care about our two bricks being in sync.
14:46 TPU joined #gluster
14:47 TPU hi, i'm currently resyncing a large volume and it eats up all disk io on all nodes, making our application unresponsive. any ideas how to slow down the resync?
14:47 cjanbanan In fact, we could always copy in one direction. But this option of only losing data in case of a split-brain is even better.
14:47 JoeJulian TPU: What version?
14:48 cjanbanan Which file shall I edit to set this option?
14:48 jobewan joined #gluster
14:48 TPU JoeJulian: 3.5.2
14:48 JoeJulian TPU: It runs self-heal at a lower priority already. Are you in swap maybe?
14:49 TPU JoeJulian: how to verify if i'm in swap? (you mean disk cache getting messed up with all the useless io blocks?)
14:51 TPU i see glusterfsd using like 300-800% cpu time in top (8 core/ht)
14:51 TPU it's already niced down
14:51 JoeJulian cjanbanan: Since you can't set it from the cli, you would need to use a filter. http://www.gluster.org/community/documentation/index.php/Glusterfs-filter
14:52 JoeJulian TPU: "free" will show you memory and swap usage.
14:53 bene joined #gluster
14:53 TPU swap is almost completely full (but only using 1GB because our app will never swap). Mem:         31975      28477       3497        124          4       8587
14:53 TPU try turning off swap ?
14:55 TPU this is a mechanical hdd btw
14:55 JoeJulian If that's showing 3 gig free, that shouldn't be it.
14:57 TPU i'm seeing teh same behaviour when one node is coming back after reboot and healing its storage, killing the app on all the remaining nodes
14:58 cjanbanan Thanks! I guess that the filter has to insert the line at the correct position within the vol file (within the replicate context), right?
14:58 JoeJulian If you need other things to also run on storage servers, I would recommend learning about and implementing conrol groups http://en.wikipedia.org/wiki/Cgroups
14:58 glusterbot Title: cgroups - Wikipedia, the free encyclopedia (at en.wikipedia.org)
14:58 JoeJulian cjanbanan: yes
15:01 TPU JoeJulian: any thoughts on reducing performance.io-thread-count due to mechanical hdd ?
15:01 TPU disk util is 90-99% almost all the time.. during normal usage it's like 10% only
15:02 rjoseph joined #gluster
15:03 JoeJulian Depends on your comfort with longer MTRR. While recovery is happening your data is at risk. I think most people's preference is to get back in to a replicated state as quickly as possible. It sounds, to me, like your issue is not that your hardware is at maximum usage as much as that your application is then resource starved. If you pre-allocate resources for your application using cgroups, that won't happen.
15:05 xleo joined #gluster
15:05 JoeJulian Alrighty, gents. I've got to get packed up and get on a plane to head for home. Good luck.
15:05 pasqd joined #gluster
15:06 pasqd hi, i have replicated storage for two nodes used as storage for openstack virtual machines, is it possible to make local cache on openstakc machine?
15:07 TPU joe: we have 3 replicas, 1 is resyncing, so no risk of data loss
15:07 TPU and yes it's starving both other replicas, not just one
15:08 cjanbanan Thanks! I'll be back with the result when I finish the script.
15:09 TPU if i limit the glusterfs daemon using cgroups it wont solve the problem i think. gluster will just have less io total available and still starve the application (which is using files on the replicas)
15:20 harish_ joined #gluster
15:28 fattaneh1 joined #gluster
15:33 tdasilva joined #gluster
15:34 harish_ joined #gluster
15:36 _Bryan_ joined #gluster
15:47 bene3 joined #gluster
15:50 justinmburrous joined #gluster
16:09 diegows joined #gluster
16:24 soumya joined #gluster
16:35 zerick joined #gluster
16:37 plarsen joined #gluster
16:40 msmith_ joined #gluster
16:40 doo joined #gluster
17:04 ninkotech joined #gluster
17:10 pasqd I have replicated storage for two nodes used as storage for openstack virtual machines, is it possible to make local cache on openstakc machine?
17:13 lpabon joined #gluster
17:15 ricky-ticky joined #gluster
17:24 fattaneh1 left #gluster
17:35 vipulnayyar joined #gluster
17:40 ekuric joined #gluster
17:42 fattaneh1 joined #gluster
17:54 nueces joined #gluster
17:54 msmith_ joined #gluster
17:55 MacWinner joined #gluster
17:55 virusuy joined #gluster
17:55 virusuy joined #gluster
17:58 nated joined #gluster
18:07 natgeorg joined #gluster
18:16 zerick joined #gluster
18:45 ThatGraemeGuy joined #gluster
18:53 ninkotech joined #gluster
18:58 theron joined #gluster
19:03 nellemann joined #gluster
19:17 ninkotech joined #gluster
19:21 theron joined #gluster
19:22 calum_ joined #gluster
19:25 ninkotech joined #gluster
19:29 quique doing testing: i have four nodes setup with replica 4, replacing a node (killing it and bringing up a new vm with the same dnsed hostname different ip) very often results in a State: Sent and Received peer request (Connected) wi
19:31 theron joined #gluster
19:41 fattaneh1 joined #gluster
19:52 semiosis quique: ,,(replace)
19:52 glusterbot quique: Useful links for replacing a failed server... if replacement server has different hostname: http://web.archive.org/web/20120508153302/http://community.gluster.org/q/a-replica-node-has-failed-completely-and-must-be-replaced-with-new-empty-hardware-how-do-i-add-the-new-hardware-and-bricks-back-into-the-replica-pair-and-begin-the-healing-process/ ... or if replacement server has
19:52 glusterbot same hostname: http://goo.gl/rem8L
19:52 semiosis quique: restart glusterd
19:52 quique semiosis: i did
19:52 quique that doesn't solve it
19:53 quique the only way i found
19:53 quique is to stop glusterd on all affected nodes
19:53 quique replace the state=5 in the peer file
19:53 quique to state=3
19:53 quique and restart glusterd
19:54 quique per this bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1004546
19:54 glusterbot Bug 1004546: unspecified, unspecified, ---, gluster-bugs, NEW , peer probe can deadlock in "Sent and Received peer request" for both servers after server build
19:54 semiosis wow
19:54 quique which is a year old
19:54 quique and doesn't look like it's been fixed
19:54 quique it's really annoying
20:03 XpineX joined #gluster
20:20 calisto1 joined #gluster
20:22 pkoro joined #gluster
20:25 msmith_ joined #gluster
20:30 cjanbanan joined #gluster
20:34 glusterbot New news from newglusterbugs: [Bug 1004546] peer probe can deadlock in "Sent and Received peer request" for both servers after server build <https://bugzilla.redhat.com/show_bug.cgi?id=1004546>
20:38 R0ok__ joined #gluster
20:46 twx joined #gluster
20:46 rshott joined #gluster
20:48 rshott Hey Everyone, quick question. I'm trying to mount a gluster share inside of a kickstart script, but it fails everytime with the same error message "Mount failed. Please check the log file for more details.". That is also with log-level=DEBUG. I've tried both /sbin/mount.glusterfs and /bin/mount. Can anyone tell me if they've successfully mounted a glusterfs share during a kickstart?
20:49 rshott Also the log file (install.log.syslog, or install.log) does not contain anything useful/relevant
20:51 rshott I know the shares work, because I echo into fstab and upon reboot they're there.
20:55 theron_ joined #gluster
21:10 twx joined #gluster
21:19 twx joined #gluster
21:32 vipulnayyar joined #gluster
21:40 ira joined #gluster
21:42 nshaikh joined #gluster
21:45 Guest73219 joined #gluster
21:47 twx joined #gluster
22:01 navid__ joined #gluster
22:04 firemanxbr joined #gluster
22:12 siel joined #gluster
22:28 ira joined #gluster
22:46 siel joined #gluster
23:05 siel joined #gluster
23:38 sauce joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary