Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2013-11-28

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:13 ron-slc joined #gluster
00:24 glusterbot joined #gluster
00:29 NuxRo kkeithley_: hi, would you know how come the gluster.org rpms and the EL 6.5 ones conflict? also, which ones should I be using now? ... I also asked on the ml but nobody answered
00:38 glusterbot joined #gluster
00:54 dbruhn__ NuxRo, which version are you trying to use?
01:04 _polto_ joined #gluster
01:37 satheesh joined #gluster
02:05 NuxRo dbruhn__: existing 3.4.0 from gluster.org EL6 RPMs vs glusterfs-3.4.0.36rhs in CentOS 6
02:06 dbruhn__ use the gluster.org ones and exclude the 3.4.0 ones from the redhat repo
02:07 dbruhn__ create your exclusion in /etc/yum/pluginconf.d/rhnplugin.conf
02:07 dbruhn__ and then install the epel repo from gluster.org
02:07 dbruhn__ and you should be able to install the 3.4.1
02:08 dbruhn__ the redhat repos don't include glusterfs-server
02:08 dbruhn__ which is really annoying
02:08 NuxRo I was waiting for 3.4.2, but glusterfs-3.4.0.36rhs seems to incorporate a lot of fixes
02:09 NuxRo ie, it's tempting to use that one :)
02:09 dbruhn__ 3.4.1 is much better than 3.4.0 from my understanding
02:09 dbruhn__ I am still using 3.3.2 so I can't really speak to the fixes in it
02:09 NuxRo aha
02:10 dbruhn__ but the redhat repos do not include glusterfs-server so they... well... suck
02:10 dbruhn__ lol
02:10 dbruhn__ they only have enough packages to use the fuse client as a client to connect to redhat storage platform
02:15 NuxRo ah, you're right, it's not the whole thing
02:16 NuxRo i had EPEL enabled and obviously a yum search was listing glusterfs-server and co (the old 3.2 version)
02:16 dbruhn__ I have been fighting with it all week on a new system
02:16 dbruhn__ I have to run 3.3.2 because RDMA is either not working or not stable in 3.4 yet
02:16 NuxRo yeah, i heard
02:16 NuxRo anywya, imo it's a good thing rhel ships a glusterfs client in Base now
02:17 dbruhn__ I am hoping ones this one is set up I can take some time and start testing 3.4.1/2 on my test RDMA hardware
02:17 dbruhn__ yeah for sure
02:17 NuxRo the epel one is ancient
02:17 NuxRo well, then that's out of the way then, thanks :)
02:17 dbruhn__ np
02:19 NuxRo well, going to bed, good night :)
02:20 dbruhn__ night
02:21 harish joined #gluster
02:21 diegows joined #gluster
02:38 bala joined #gluster
02:44 satheesh1 joined #gluster
02:46 diegows joined #gluster
02:52 saurabh joined #gluster
03:04 nueces joined #gluster
03:22 jag3773 joined #gluster
03:25 bulde joined #gluster
03:27 bharata-rao joined #gluster
03:35 kanagaraj joined #gluster
03:35 shubhendu joined #gluster
03:37 jesse joined #gluster
03:47 dbruhn joined #gluster
03:49 RameshN joined #gluster
03:51 itisravi joined #gluster
03:51 DV joined #gluster
04:01 purpleidea Can someone point me to the exact (and complete) list of input parameters that go into building the DHT elastic hash?
04:12 kshlm joined #gluster
04:13 shyam joined #gluster
04:13 rjoseph joined #gluster
04:14 dbruhn purple idea, looks like it's based off of the path/filename
04:14 dbruhn http://moo.nac.uci.edu/~hjm/fs/An_Introduction_To_Gluster_ArchitectureV7_110708.pdf
04:14 glusterbot <http://goo.gl/rDFkyZ> (at moo.nac.uci.edu)
04:16 dbruhn although Joe's blog says just the name
04:16 dbruhn http://joejulian.name/blog/dht-misses-are-expensive/
04:16 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
04:18 DV joined #gluster
04:33 davidbierce joined #gluster
04:39 ndarshan joined #gluster
04:57 kshlm joined #gluster
04:57 dusmant joined #gluster
04:58 kshlm joined #gluster
05:12 bulde joined #gluster
05:13 MiteshShah joined #gluster
05:20 aravindavk joined #gluster
05:23 vpshastry1 joined #gluster
05:24 bala joined #gluster
05:36 anands joined #gluster
05:43 CheRi joined #gluster
05:50 raghu joined #gluster
05:50 hagarth joined #gluster
05:52 vpshastry1 joined #gluster
05:53 sgowda joined #gluster
05:54 kaushal_ joined #gluster
05:54 kaushal_ joined #gluster
05:58 nullck joined #gluster
06:05 dusmant joined #gluster
06:17 krypto joined #gluster
06:22 kevein joined #gluster
06:23 shruti joined #gluster
06:25 vshankar joined #gluster
06:30 vpshastry1 joined #gluster
06:37 _polto_ joined #gluster
06:38 vpshastry joined #gluster
06:39 vimal joined #gluster
06:41 kshlm joined #gluster
06:53 ricky-ti1 joined #gluster
06:53 hagarth joined #gluster
07:01 dusmantkp_ joined #gluster
07:16 hagarth joined #gluster
07:25 jtux joined #gluster
07:28 harish joined #gluster
07:39 glusterbot New news from newglusterbugs: [Bug 1035586] gluster volume status shows incorrect information for brick process <http://goo.gl/zo72Bt>
07:53 dusmantkp_ joined #gluster
08:03 ctria joined #gluster
08:03 eseyman joined #gluster
08:04 vpshastry1 joined #gluster
08:05 harish joined #gluster
08:08 purpleidea joined #gluster
08:08 purpleidea joined #gluster
08:15 pawkor joined #gluster
08:42 Staples84 joined #gluster
08:42 geewiz joined #gluster
08:44 JordanHackworth joined #gluster
08:44 fkautz joined #gluster
08:47 ekuric joined #gluster
08:48 dusmantkp_ joined #gluster
08:51 purpleidea joined #gluster
08:52 pkoro joined #gluster
09:10 dusmantkp_ joined #gluster
09:16 bulde joined #gluster
09:19 meghanam joined #gluster
09:27 dusmantkp_ joined #gluster
09:30 StarBeast joined #gluster
09:31 harish joined #gluster
09:37 jiphex joined #gluster
09:51 dylan_ joined #gluster
09:55 sgowda joined #gluster
09:55 harish joined #gluster
09:57 purpleidea joined #gluster
09:57 purpleidea joined #gluster
10:02 hurl joined #gluster
10:04 hurl Hi all. When I add a new brick to my replica volume, the data does not seem to sync to this brick. Am I missing something ? Regarding docs I need to run a "rebalance", but my gluster client want to rebalance only distributed volumes.
10:14 delhage you need to add a multiple of the replica count
10:20 hurl I have a little weird setup: I've created a replica 2 vol, removed a brick, and re-added a new one. so the vol move from replica 2 to dht, then again to replica 2
10:20 hurl so maybe the problem is just going from distributed to replica
10:21 delhage hm
10:21 delhage you didn't use replace-brick?
10:23 hurl nop, I've maybe done it wrong. I maybe should have shutdown on node when I wanted to reorganize partitions.
10:23 hurl Instead of that I've done a remove brick. surely not the best I guess now
10:26 RicardoSSP joined #gluster
10:26 RicardoSSP joined #gluster
10:27 hurl I've played with the healing command, but it has done
10:42 _polto_ joined #gluster
10:43 _polto_ joined #gluster
10:44 krypto trying to install 3.4.1 on el6.5 but its showing this error while trying to start process http://pastebin.com/xvH43CAk
10:44 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
10:54 calum_ joined #gluster
11:15 gdubreui joined #gluster
11:19 krypto fixed by installing libs_3.4.1 thanks
11:23 social krypto: sounds like dependancies are quite broken in rpm then, where did you get rpm from? epel?
11:23 social krypto: would you mind opening a ticket on this?
11:24 muhh joined #gluster
11:24 krypto social:following http://www.gluster.org/community/documentation/index.php/Getting_started_install downloaded rpm from  http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/epel-6.4/x86_64/
11:24 glusterbot <http://goo.gl/chDN9> (at www.gluster.org)
11:25 social krypto: dunno who's responsible for those rpms probably kkeithley_ ??
11:26 krypto using rhel 6.5,as per document we dont need to download glusterfs-libs-3.4.1-3.el6.x86_64.rpm but with out that it was showing error.. may be a documentation mistake
11:27 krypto trying on 6.5 but rpms for 6.4 :)
11:32 purpleidea joined #gluster
11:32 purpleidea joined #gluster
11:33 bulde joined #gluster
11:35 harish joined #gluster
11:38 ProT-0-TypE joined #gluster
11:50 sjoerd_ joined #gluster
11:57 hurl joined #gluster
12:06 itisravi joined #gluster
12:09 ctria joined #gluster
12:12 _polto_ joined #gluster
12:12 _polto_ joined #gluster
12:17 edward1 joined #gluster
12:39 getup- joined #gluster
12:39 andreask joined #gluster
12:47 vpshastry joined #gluster
12:49 vpshastry left #gluster
12:51 ctria joined #gluster
12:51 satheesh joined #gluster
12:59 getup- joined #gluster
13:13 sjoerd_ hi all
13:14 FooBar joined #gluster
13:16 FooBar I'm having some trouble with directories marked as split-brain (http://paste.sigio.nl/pr8e4vie3) ... any hints ?
13:16 glusterbot Title: Sticky Notes (at paste.sigio.nl)
13:17 FooBar specifically 'the root' directory
13:18 cyberbootje op diezelfde bak?
13:18 FooBar hmm?
13:19 FooBar the root of the gluster filesystem/export, not the root of the linux system ;)
13:20 kshlm joined #gluster
13:42 kshlm joined #gluster
13:52 dbruhn Foobar, check all of your bricks and make sure the permissions and meta data are the same
13:52 dbruhn then check the extended attributes on the directory
13:53 dbruhn you will see one of them is different, once you've found the different one, reset it back to 0x00 and it will stop throwing that split brain error
13:54 dbruhn but you have to make sure the directory has the same directories under all of them, and make sure that the permissions are the same
13:54 FooBar how do I check that ?
13:54 FooBar nice... 24 bricks to check :)
13:55 FooBar (it's a distributed and replicated volume)
13:57 dbruhn Yeah, it's not fun
13:57 dbruhn let me see if I can find the commands
13:57 FooBar can I ignore the contents of .gluster directory ?
13:59 dbruhn getfattr -m . -d -e hex /var/ENTV06EP/aa
13:59 dbruhn replacing /var/ENTV06EP/aa with the brick directory
14:00 dbruhn the contents of the .glusterfs is part of the file systems mapping
14:00 dbruhn http://joejulian.name/blog/what-is-this-new-glusterfs-directory-in-33/
14:00 glusterbot <http://goo.gl/j981n> (at joejulian.name)
14:00 dbruhn here is a writeup on it
14:02 FooBar dbruhn: ok, found the different one...
14:04 FooBar can I take a node out of the gluster, wipe it, and have it self-heal ?
14:04 dbruhn what is different about it
14:04 FooBar 7~trusted.afr.gv0-client-2=0x000000000000000400000000
14:04 FooBar with only 0's on 1 onther node
14:05 FooBar and some less 'gv0-client-x' lines
14:06 dbruhn if the permissions are the same, and the directories are the same you can just set the extended attribute back to match
14:06 _polto_ joined #gluster
14:06 _polto_ joined #gluster
14:09 FooBar dbruhn: ok, got it... lets see if it worked...
14:10 nullck joined #gluster
14:12 kshlm joined #gluster
14:13 FooBar md5sum to the rescue.... that compares a lot quicker ;)
14:15 dbruhn hahah
14:15 dbruhn Did your errors clear up?
14:21 FooBar I think so yeah... there were still quite a lot in some testing directories
14:21 FooBar but I just wiped those directories on all the bricks
14:21 FooBar a 'volume heal info' responds with just a few of those test-files now
14:21 FooBar and now clean
14:22 FooBar lets see if my mounted filesystem still has 3 '.' and '..' directories ;)
14:23 FooBar hmm.. still there
14:23 khushildep joined #gluster
14:24 FooBar http://paste.sigio.nl/ppy9f4kun
14:24 glusterbot Title: Sticky Notes (at paste.sigio.nl)
14:24 dbruhn what version are you running?
14:26 dbruhn I've had this issue myself.
14:26 FooBar 3.4.1-3
14:27 FooBar gluster volume heal gv0 info -> reports no issues
14:27 FooBar gluster volume heal gv0 info split-brain -> reports a ton...
14:27 FooBar is that historic, or are those still in split-brain ?
14:27 dbruhn check your bricks .glusterfs/00/00
14:28 dbruhn there should be a soft link for 00000000-0000-0000-0000-000000000001 to ../../..
14:28 dbruhn if I remember right, I had one of those had turned into a file instead of a soft link on me
14:29 FooBar I've got a few blinking red there (pointing to a file, not to ../../..)
14:29 FooBar should I remove and re-create those ?
14:29 dbruhn no those are fine
14:29 dbruhn the only one you are concerned about is the 00000000-0000-0000-0000-000000000001 one
14:31 FooBar ok, re-created those links
14:32 dbruhn I would say restart the glusterd services and then check
14:32 dbruhn make sure nothing is cached
14:33 FooBar one at a time, or all at the same time ?
14:33 dbruhn Well... if you can the whole system down, and then the whole system up.
14:33 dbruhn understood if you can't because of production
14:33 dbruhn it's more just a safety measure
14:34 FooBar \o/ victory :)
14:34 dbruhn *high five*
14:34 FooBar seeing 1 set of '.' '..' and my directories again
14:34 FooBar thnx
14:34 dbruhn no problem, glad I could help
14:35 FooBar yeah, we just went live with this setup yesterday, but it's offline due to some other problems...
14:35 dbruhn You should create a bug report on it
14:35 FooBar hopefully get it back online later today
14:36 dbruhn there should be some sort of safety mechanism to protect against that
14:36 FooBar dbruhn: I'm guessing it was caused by 2 systems having a wrong mapping from brick to path
14:36 FooBar /dev/sdc1 was mounted on /brick/sda1
14:36 dbruhn ahh
14:36 FooBar order of disks changed after reboots... so now I mount them on UUID
14:36 dbruhn ugh, yeah that's messy
14:37 FooBar first boot, ssd's first, hdd's second, and then after a reboot, HDD's first, ssd's second
14:37 FooBar hopefully that remains resolved now
14:37 dbruhn if you've done it via UUID it should be now
14:38 dbruhn what does your partition table look like on the machines?
14:40 FooBar single partition in fdisk for GPT, single GPT partition of whole-disk with xfs filesystem on them
14:40 FooBar Number  Start   End     Size    File system  Name              Flags 1      1049kB  4001GB  4001GB  xfs          Linux filesystem
14:40 dbruhn Here is a couple of learned the hard way warnings.
14:41 dbruhn make sure /var/log is in it's own partition and at least 30GB
14:42 dbruhn the dynamic config files are stored at /var/lib, if the partition that contains them fills up, it will corrupt your volume files, and make your life hell
14:43 FooBar dbruhn: yeah, i was only talking about the gluster disks...
14:43 FooBar the main system is running on normal non-shared disks, with sane layout ;)
14:43 FooBar and being monitored ;)
14:43 dbruhn Also out of the box the gluster logs don't rotate properly
14:43 dbruhn /var/log/glusterfs/*.log {
14:43 dbruhn daily
14:43 dbruhn rotate 7
14:43 dbruhn copytruncate
14:43 dbruhn }
14:43 dbruhn the system will stop working if the log dir fills up
14:44 dbruhn that log rotate script will correct the issue, example being using copytruncate
14:44 FooBar thnx, handy to know
14:44 hagarth joined #gluster
14:45 dbruhn Always trying to help people not experience the same issues i've had with my system
14:45 dbruhn lol
14:52 Spiculum joined #gluster
14:55 diegows joined #gluster
15:19 geewiz joined #gluster
15:23 zerick joined #gluster
15:31 sroy joined #gluster
15:47 failshell joined #gluster
15:53 _polto_ joined #gluster
15:53 _polto_ joined #gluster
16:16 hurl I have a maybe basic question, but how to be sure replication is working correctly ? does md5sum-ing brick/.glusterfs content is a reliable solution ?
16:28 pdrakeweb joined #gluster
16:31 chirino joined #gluster
16:31 bala joined #gluster
16:34 dbruhn joined #gluster
16:48 nueces joined #gluster
16:48 ababu joined #gluster
16:52 social has anyone hit this issue? I have mount which is owned by apache:root, when I start geo-replication it changes to root:root
17:09 zerick joined #gluster
17:27 vpshastry joined #gluster
17:27 vpshastry left #gluster
17:35 XpineX_ joined #gluster
17:47 calum_ joined #gluster
17:49 micu1 joined #gluster
18:07 redbeard joined #gluster
18:13 kanagaraj joined #gluster
18:20 davidbierce joined #gluster
18:23 redbeard joined #gluster
18:33 coxy23 joined #gluster
18:37 sroy joined #gluster
18:45 kanagaraj joined #gluster
18:53 diegows joined #gluster
18:56 nullck joined #gluster
19:22 dylan_ joined #gluster
19:30 failshell joined #gluster
19:33 psyl0n joined #gluster
19:34 glanzi joined #gluster
20:00 Sweeto joined #gluster
20:00 dylan_ left #gluster
20:00 dylan_ joined #gluster
20:00 Sweeto left #gluster
20:01 achuz joined #gluster
20:01 achuz hi
20:01 glusterbot achuz: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
20:01 kl4m joined #gluster
20:02 achuz when a gluster server goes down I can see a delay in switching ..
20:02 achuz to the secondary where replication is setup
20:02 achuz is it acceptable?
20:37 davidbierce joined #gluster
20:46 badone joined #gluster
20:51 badone joined #gluster
21:06 andreask joined #gluster
21:12 vpshastry joined #gluster
21:19 MarkR_ joined #gluster
21:25 MarkR_ I have a problem with Gluster mounts. Some servers have connection problems after some time. It looks like it has something to do with iptables/connection tracking:
21:25 MarkR_ Nov 28 22:23:53 app12 kernel: [420392.846984] [INPUT] dropped IN=eth0 OUT= MAC=aa:01:60:00:87:d5:aa:01:60:00:78:82:08:00 SRC=10.243.0.23 DST=10.243.0.72 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=24007 DPT=1021 WINDOW=14480 RES=0x00 ACK SYN URGP=0
21:25 MarkR_ In /proc/net/ip_conntrack:
21:25 MarkR_ tcp      6 118 SYN_SENT src=10.243.0.72 dst=10.243.0.23 sport=1021 dport=24007 src=10.243.0.23 dst=10.243.0.72 sport=24007 dport=1021 mark=0 use=2
21:26 MarkR_ lsof revealse:
21:26 MarkR_ glusterfs 31161       root    6u     IPv4            3735680        0t0                  TCP 10.243.0.72:1021->10.243.0.23:24007 (SYN_SENT)
21:28 MarkR_ glusterfs 31120       root    6u     IPv4            3736801        0t0                  TCP 10.243.0.72:1023->10.243.0.23:24007 (SYN_SENT)
21:28 MarkR_ glusterfs 31120       root   10u     IPv4             590752        0t0                  TCP 10.243.0.72:1018->10.243.0.23:49153 (ESTABLISHED)
21:28 MarkR_ glusterfs 31161       root    5u     IPv4            3736644        0t0                  TCP 10.243.0.72:1022->10.243.0.23:24007 (SYN_SENT)
21:32 MarkR_ I really have no idea where to look for a fix. The SYN_SENT's seem to start showing up after a network outage.
21:33 vpshastry joined #gluster
21:34 MarkR_ If I umount/mount the shares, all connections are restored properly. But I want to prevent the broken connections. Btw, iptables logs *a lot* dropped packets from port 24007 to the client 10.243.0.72.
21:53 khushildep joined #gluster
22:08 MarkR_ Found something interesting (after enabling /proc/sys/net/netfilter/nf_conntrack_log_invalid):
22:08 MarkR_ Nov 28 23:06:59 app12 kernel: [422978.061683] nf_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.243.0.23 DST=10.243.0.72 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=24007 DPT=1022 SEQ=174558485 ACK=1477362210 WINDOW=14480 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080AAFD03F55064C638B01030306)
22:11 agd joined #gluster
22:12 MarkR_ 23:12:26.149188 IP 10.243.0.72.1021 > 10.243.0.23.24007: Flags [S], seq 2564494691, win 14600, options [mss 1460,sackOK,TS val 105751288 ecr 0,nop,wscale 6], length 0
22:12 MarkR_ 23:12:26.149255 IP 10.243.0.23.24007 > 10.243.0.72.1021: Flags [S.], seq 3020408797, ack 2564494692, win 14480, options [mss 1460,sackOK,TS val 2949742274 ecr 105747529,nop,wscale 6], length 0
22:16 agd Hi! I need to have a CIFS share replicated via a WAN connection of 10 Mbps. It's possible to do that with Glsuter ?
22:18 Remco agd: You can probably do that with geo-replication
22:18 Remco MarkR_: Were you in here before with this problem?
22:19 * Remco remembers someone that had this problem
22:20 Remco I suggest making a bug with this data
22:20 MarkR_ Last week I send a mail to the gluster mailing list with this problem, but got no follow-up.
22:21 Remco It's holiday weekend in the US now, so don't expect it to be picked up immediately
22:32 khushildep joined #gluster
22:35 agd Remco: geo-replication is a Gluster feature ?
22:37 Remco Yes
22:39 gdubreui joined #gluster
22:39 agd Remco: ok, thanks.
22:40 dbruhn joined #gluster
22:47 hybrid5121 joined #gluster
22:56 davidbierce joined #gluster
22:58 MarkR_ This seems to help: sudo sysctl net.ipv4.netfilter.ip_conntrack_tcp_be_liberal=1
23:18 _polto_ joined #gluster
23:20 khushildep joined #gluster
23:20 fkautz joined #gluster
23:21 MarkR_ Yes, after net.ipv4.netfilter.ip_conntrack_tcp_be_liberal=1, the stalled SYN_SENT connections to port 24007 turn into ESTABLISHED!
23:22 MarkR_ Not sure if this is a Gluster of Ubuntu/iptable/conntracker issue, but we only got this issue with Gluster.
23:29 StarBeast joined #gluster
23:35 davidbierce joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary