Camelia, the Perl 6 bug

IRC log for #gluster, 2013-05-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 yinyin joined #gluster
00:38 premera joined #gluster
00:56 majeff joined #gluster
01:08 kshlm joined #gluster
01:16 majeff joined #gluster
01:23 brian___ joined #gluster
01:23 brian___ left #gluster
01:24 kevein joined #gluster
01:34 bala joined #gluster
01:54 jclift_ joined #gluster
01:54 awheeler joined #gluster
01:59 ninkotech joined #gluster
02:00 ninkotech__ joined #gluster
02:10 majeff joined #gluster
02:12 badone joined #gluster
02:22 mikemar10 joined #gluster
02:23 mikemar10 I just recently started using gluster but have had trouble finding the answer to this question.  When setting up a replica cluster, is it necessary to use a floating VIP for the connect string or is it fine to just use any member's IP/hostname?
02:23 mikemar10 I've found that when I choose an arbitrary member everything works fine, but I'd like to know the best practices
02:37 m0zes mikemar10: I use rrdns, iirc the mount will fallback and try the other entries in the rrdns record if the first fails.
02:38 mikemar10 m0zes: are you using NFS to interface with gluster or the fuse mount? I should have been more specific, I'm referring to the fuse mount
02:39 m0zes the fuse mount. with nfs you'd want keepalived (or something like it) and a floating VIP. (I also think it needs to send a reset to the clients to make sure they re-connect)
02:39 sprachgenerator joined #gluster
02:39 awheeler joined #gluster
02:41 mikemar10 cool so the reason to use rrdns would be because on connection failure it will try to reconnect to the same hostname it was already connected to but will hit one of the secondary members right?  Without rrdns if I understand correctly it would eventually reach out to a different host in the cluster but only after some timeout has expired
02:41 mikemar10 or at least that is what seems to be occurring from my failover tests
02:41 m0zes there is also a -o backupvolfile-server=<hostname> option to the fuse mount if you don't want rrdns (i've not tested this, though)
02:42 mikemar10 ah I forgot about that option, I remember seeing it in the docs
02:42 m0zes only during the initial mount process does the hostname specified matter. with fuse each client downloads a volfile to get info to talk to every server.
02:44 mikemar10 so the rrdns is to prevent an issue if the hostname you provided in the mount was unavailable at mount time.  Once mounted it shouldn't matter
02:44 mikemar10 just restating to make sure I'm grokking properly :)
02:44 m0zes right
02:44 m0zes the timeout will still happen if a server goes down, there is an option to lower that timeout, but I wouldn't lower it too much. the re-connection operation is relatively expensive because it drops and recreates all the locks.
02:45 m0zes the default is 42 seconds. http://gluster.org/community/documentation/index​.php/Gluster_3.2:_Setting_Volume_Options#network.ping-timeout
02:45 glusterbot <http://goo.gl/NugxI> (at gluster.org)
02:46 mikemar10 gotcha, I probably won't mess with the timeout at all, I just wanted to be sure that I was setting up my persistent mounts properly since I'm trying to setup HA and would hate to see it fail during a failover
02:47 m0zes yep, remember, replication happens client-side with the FUSE mount. so each client writes its data to each replica.
02:48 mikemar10 ah I didn't realize that.  I thought the data was uploaded to one and then the cluster members synced amongst each other.  I've been having trouble finding any decent technical documentation on gluster everything I've found so far has been mainly regarding setting it up and then high level
02:49 mikemar10 I'd love to see a breakdown of the native protocol
02:49 mikemar10 but I suppose I'll have to dig into the source for that :P
02:52 m0zes there are many in this channel that know the native protocol much better than I. this is stuff I've gleaned from here and learned first hand :)
02:53 m0zes On the plus side, this is the most stable and best documented distributed/cluster filesystem I've dealt with :)
02:53 mikemar10 haha yes I definitely agree with you there, I was very pleasantly surprised at how easy it was to set gluster up and how resilient it was
02:53 m0zes granted, I've only actually installed this and ceph. but the horror stories from lustre were enough to scare me off that :)
02:55 awheeler joined #gluster
02:56 mikemar10 what are your thoughts on ceph? it's next on my list to be evaluated but it seemed a tad too beta for me still
02:56 mikemar10 from just peeking at the website
03:00 m0zes documentation looked pretty good at first, unfortunately they don't tag any versions with it, so it is often out of date. they've apparently moved to a new ceph-deploy tool that is ubuntu specific (even though it grabs and compiles things from git) which is annoying in my gentoo environment.
03:01 m0zes the ceph.conf file doesn't auto-replicate between servers/peers when changes are made (and brought all writes to a screaming halt in the middle of a final project for a class this was used for)
03:01 mikemar10 that sounds… frustrating
03:02 m0zes we've found that it wasn't intelligently replicating some files (both copies on the same server) and some it decided not to replicate at all.
03:04 m0zes it looks like a great project, I am not sure I would put it in production yet. luckily this was a "test" environment that could be brought down and fixed if need be. test in this case means it wasn't supporting the research I am *actually* paid to support.
03:07 bharata joined #gluster
03:08 FyreFoX joined #gluster
03:08 war|chil1 joined #gluster
03:08 neofob1 joined #gluster
03:08 jskinner joined #gluster
03:08 jcastle joined #gluster
03:09 vex_ joined #gluster
03:09 foster_ joined #gluster
03:09 hflai_ joined #gluster
03:09 phix_ joined #gluster
03:12 Ramereth|home joined #gluster
03:12 twx_ joined #gluster
03:14 awheeler joined #gluster
03:14 vex joined #gluster
03:16 lkoranda joined #gluster
03:18 jiqiren joined #gluster
03:33 majeff joined #gluster
03:33 awheeler joined #gluster
03:36 majeff joined #gluster
04:04 vex joined #gluster
04:04 vex joined #gluster
04:05 awheeler joined #gluster
04:08 sgowda joined #gluster
04:08 awheeler joined #gluster
04:17 kevein joined #gluster
04:19 anands joined #gluster
04:24 yinyin joined #gluster
04:26 pithagorians_ joined #gluster
04:32 shylesh joined #gluster
04:41 hagarth joined #gluster
04:50 saurabh joined #gluster
05:00 mohankumar joined #gluster
05:03 MaRoe joined #gluster
05:08 vpshastry joined #gluster
05:10 MaRoe left #gluster
05:19 awheeler joined #gluster
05:27 hagarth joined #gluster
05:37 hchiramm__ joined #gluster
05:54 rgustafs joined #gluster
06:01 raghu joined #gluster
06:02 bala joined #gluster
06:05 andreask joined #gluster
06:05 ramkrsna joined #gluster
06:05 ramkrsna joined #gluster
06:06 hagarth joined #gluster
06:08 lalatenduM joined #gluster
06:13 glusterbot New news from newglusterbugs: [Bug 963537] Fix misleading log messages of the form "Node responded to " <http://goo.gl/qatuS>
06:14 bala joined #gluster
06:17 satheesh joined #gluster
06:23 jtux joined #gluster
06:25 guigui1 joined #gluster
06:26 StarBeast joined #gluster
06:35 ricky-ticky joined #gluster
06:35 jag3773 joined #gluster
06:36 ollivera_ joined #gluster
06:37 bulde joined #gluster
06:38 andreask joined #gluster
06:43 glusterbot New news from newglusterbugs: [Bug 963541] glusterd : 'gluster volume status ' some times do not show active task and sometimes it shows tasks which are not active. <http://goo.gl/TcNOj>
06:47 vshankar joined #gluster
06:49 ekuric joined #gluster
06:52 puebele joined #gluster
06:53 vpshastry1 joined #gluster
06:56 vpshastry1 joined #gluster
07:05 ctria joined #gluster
07:11 puebele joined #gluster
07:19 shylesh joined #gluster
07:26 hybrid512 joined #gluster
07:32 rudimeyer joined #gluster
07:32 ngoswami joined #gluster
07:33 vpshastry joined #gluster
07:34 andrei_ joined #gluster
07:35 rudimeyer Running Gluster 3.3.1 I have the impression that doc are outdated, like this goo.gl/H5XZI (version 3.2 based). can any one guide to to some 3.3 docs on how to replace a crashed server?
07:35 rudimeyer The guide talk about /etc/glusterd/glusterd.info and similar files that are not present in 3.3?
07:45 al joined #gluster
07:48 Staples84 joined #gluster
07:50 ujjain joined #gluster
07:50 wgao__ rudimeyer: here is a doc for gluster 3.3 "Gluster_File_System-3.3.0-A​dministration_Guide-en-US", if you like, I will send to you.
07:52 rudimeyer wgao__: Thank you, but it does not cover the handling of different failure scenarios - like a failed server. I have the docs
07:52 wgao__ OK
07:53 rudimeyer wgao__: Have you tried recovering a failed Gluster server on a 3.3 system?
07:53 wgao__ yeah, but I also lack the docs.
07:54 rudimeyer wgao__: But you succeded
07:54 rudimeyer Can you give some pointers?
07:56 wgao__ I just do it follow start guides, here you recreate volume  you should clean the env firstly.
07:57 wgao__ rudimeyer: cd /export/bricks ;  for i in `attr -lq .` ; do setfattr -x trusted.$i . ; done
08:05 aravindavk joined #gluster
08:21 dxd828 joined #gluster
08:23 xymox joined #gluster
08:37 SteveCoo1ing hi guys.. how important is it to have all nodes and clients resolve reverse dns?
08:38 jclift_ Fundamental
08:39 Nagilum_ depends on what you used for peer probes
08:39 SteveCoo1ing my setup runs mDNS right now for naming in my storage segment
08:39 jclift_ From memory, you can cheat using /etc/hosts overrides for everything though
08:39 jclift_ Can be a pita to setup, but if reverse DNS isn't possible that might help
08:40 SteveCoo1ing but it seems to slow down nonexistent real ip resolves, which kinda breaks my app
08:40 jclift_ How many nodes do you have, and which OS are the clients?
08:40 jclift_ (or OS's)
08:40 SteveCoo1ing all are rhel5 and 6
08:40 SteveCoo1ing 4 nodes
08:41 jclift_ Sounds like it technically wouldn't be too hard to use /etc/hosts overrides then?
08:41 jtux joined #gluster
08:42 jclift_ i.e. On each host just have the IP addresses for all of the hosts in /etc/hosts
08:42 SteveCoo1ing yes i understand
08:42 SteveCoo1ing just trying to see if there is a better solution
08:43 jclift_ np
08:43 SteveCoo1ing the real problem is clients not having reverse DNS set up correctly
08:43 SteveCoo1ing getent hosts <clientip> now takes 5 seconds to time out because of the mdns4 thing in nsswitch.conf
08:44 * jclift_ hasn't touched mdns yet :/
08:45 jclift_ No idea if it's possible to setup negative lookups or something with that
08:45 jclift_ SteveCoo1ing: Any chance you can update the /etc/hosts (or equiv) on the clients too?
08:46 SteveCoo1ing I'm sorry, I was being unclear...
08:47 SteveCoo1ing I'm seeing this problem from the gluster client perspective. This is a webserver. It has a thing that does reverse DNS for logging upon authenticating web service requests.
08:48 SteveCoo1ing The reverse resolve now takes 5 seconds because the traditional DNS failed resolving, and it takes 5 seconds to time out the mDNS resolve.
08:48 SteveCoo1ing So in this case the "client" is in fact my real world paying customer, not the GlusterFS client :)
08:48 jclift_ k.  Sorry, I have no idea how to fix that. :)
08:49 jclift_ Hmmm, can you cheat and disable the reverse lookups?  i.e. tell the logger to just use ip's?
08:49 jclift_ Meh, I'm not suggesting anything real useful here am I...
08:49 SteveCoo1ing that is one option :(
08:49 SteveCoo1ing i mean :)
08:50 SteveCoo1ing also i could just put this particular clients static ip in /etc/hosts for now
08:50 SteveCoo1ing while i marinate :)
08:50 jclift_ Heh, that would work too
08:51 jclift_ Out of curiosity, why mdns?
08:51 SteveCoo1ing zero configuration naming... no central point of failure
08:52 jclift_ Interesting.
08:52 jclift_ Maybe I should look into it later on
08:52 vpshastry1 joined #gluster
08:52 jclift_ Easier to manage than replicated db backed dns?
08:52 SteveCoo1ing the idea works good enough. but it should not even try to look up thins in other domains than .local in this setting
08:52 * jclift_ has been mucking around with PowerDNS recently, but not really gotten into the replication side of things with it yet
08:53 jclift_ k
08:53 SteveCoo1ing our authorative name servers are sql backed with replication, but i need the lowest parts, like storage to work without any services running already :)
08:54 SteveCoo1ing they are after all the basis for the services :)
08:54 jclift_ :)
08:57 saurabh joined #gluster
09:12 soukihei joined #gluster
09:15 duerF joined #gluster
09:21 soukihei joined #gluster
09:26 manik joined #gluster
09:34 andrei__ joined #gluster
09:38 lh joined #gluster
09:38 lh joined #gluster
09:39 lanning_ sounds like you should reverse your lookup. normal DNS first. that failing, mDNS.
09:41 lanning_ optimize for the common case
09:43 vpshastry1 joined #gluster
09:52 jskinner joined #gluster
09:53 saurabh joined #gluster
09:58 lanning_ oh, didn't read enough scroll back.  issue with NXDOMAIN response
09:59 lanning_ if you are using avahi, there is the wide-area feature for only .local lookup on mDNS
10:21 karoshi joined #gluster
10:24 shylesh_ joined #gluster
10:26 rudimeyer joined #gluster
10:30 majeff joined #gluster
10:39 jskinner joined #gluster
10:44 hchiramm__ joined #gluster
11:01 bleon joined #gluster
11:03 dustint joined #gluster
11:08 hchiramm__ joined #gluster
11:15 kkeithley1 joined #gluster
11:17 piotrektt_ joined #gluster
11:21 rotbeard joined #gluster
11:24 hagarth joined #gluster
11:24 jskinner joined #gluster
11:26 shireesh joined #gluster
11:30 tziOm joined #gluster
11:32 rudimeyer joined #gluster
11:35 tziOm How do I make the config file?
11:41 rudimeyer joined #gluster
11:44 mynameisbruce__ after upgrade from 3.4 alpha to 3.4 beta it get following error
11:44 mynameisbruce__ mount /mnt/sanlock
11:44 mynameisbruce__ mount.nfs: requested NFS version or transport protocol is not supported
11:44 glusterbot mynameisbruce__: make sure your volume is started. If you changed nfs.disable, restarting your volume is known to work.
11:49 NuxRo tziOm: you don't make a config file, glusterd will take care of that
11:52 tziOm NuxRo, Ok, so when I want to set http://www.gluster.org/community/documentatio​n/index.php/Translators/performance/io-cache ?
11:52 glusterbot <http://goo.gl/li7SQ> (at www.gluster.org)
11:52 vpshastry1 joined #gluster
11:59 tziOm I can set performance.io-cache on via gluster volume set ... but how do I set the cache-timeout option?
12:09 jskinner joined #gluster
12:11 mynameisbruce__ nice hint by glusterbot
12:12 mynameisbruce__ but after restart i get the same error message....do i have to permanently enable nfs anyway ?
12:12 mynameisbruce__ i dont get it
12:12 mynameisbruce__ the only workaround ...at the moment ... is to stop volume....start volume....mount nfs
12:12 edward1 joined #gluster
12:13 mynameisbruce__ any hint?
12:23 hagarth joined #gluster
12:26 andrei__ joined #gluster
12:27 yosafbridge joined #gluster
12:28 vpshastry joined #gluster
12:28 ricky-ticky joined #gluster
12:29 robos joined #gluster
12:39 awheeler joined #gluster
12:44 glusterbot New news from newglusterbugs: [Bug 781285] [faf9099bb50d4d2c1a9fe8d3232d541b3f68bc58] improve replace-brick cli outputs. <http://goo.gl/6mwh7> || [Bug 848556] glusterfsd apparently unaware of brick failure. <http://goo.gl/rIjjW> || [Bug 858732] glusterd does not start anymore on one node <http://goo.gl/X7NsZ> || [Bug 865327] glusterd keeps listening on ipv6 interfaces for volumes when using inet familly address <http://g
12:47 andrei__ hello guys
12:48 andrei__ i was wondering if someone could help me with nfs over glusterfs?
12:48 andrei__ i am currently having issues with several clients mounting nfs
12:48 andrei__ and it seems that the reason for relates to -o vers=3 flag
12:48 andrei__ i can't use this option on some of the clients
12:49 mynameisbruce__ any error message?
12:49 kkeithley| what's the OS on the clients where vers=3 doesn't work?
12:49 andrei__ it's ubuntu
12:50 andrei__ sorry, not ubuntu,
12:50 andrei__ it's debian
12:50 andrei__ it's a cloudstack secondary storage vm
12:50 andrei__ which is created automatically
12:50 andrei__ and if i login to it i can successfully mount glusterfs volume
12:51 andrei__ if I am specifying -o vers=3 option
12:51 andrei__ however, this vm is created automatically and mounting is also done automatically without using this option
12:51 andrei__ so, every time this vm is created by cloudstack I have to manually login and manually mount adding -o vers=3
12:52 andrei__ without this option the volume is not mounted
12:52 andrei__ so, what I am asking is do you know what config options do I change on the server side so that the clients can mount volumes without using -o vers=3 option?
12:54 jskinner_ joined #gluster
12:58 mohankumar joined #gluster
12:58 melaur joined #gluster
12:58 melaur Hello
12:58 glusterbot melaur: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
13:01 melaur I want to make an private video portal i have 2 servers, one for web/db and second for delivery video content file via web ... / can gluster help me ?
13:02 Foo1 joined #gluster
13:05 Foo1 Hi there! I'm having some problems with the fail over of the new GlusterFS beta, it takes about 45 seconds before it is possible to write anything to the Gluster storage. With the 3.3 version it was instant :O, does anyone have the same problems or know a solution?
13:05 kshlm joined #gluster
13:05 ndevos andrei__: some users have reported success by setting the gluster nfs-server to listed on port 2049 - nfs clients would fallback on NFSv3 if NFSv4 gets an appropriate error back
13:06 SteveCoo1ing lanning_: i ended up patching the nss_mdns_minimal library to do reverse lookups for my internal ip range. no more full mdns resolving failing regular DNS
13:07 awickham joined #gluster
13:08 andrei__ ndevos: thanks, I will try that!
13:12 andrei__ ndevos: many thanks, that has helped!!!!!
13:12 ndevos andrei__: cool!
13:14 Foo1 joined #gluster
13:14 glusterbot New news from newglusterbugs: [Bug 844584] logging: Stale NFS messages <http://goo.gl/z72b6> || [Bug 786007] [c3aa99d907591f72b6302287b9b8899514fb52f1]: server crashed when dict_set for truster.afr.vol-client when compiled with efence <http://goo.gl/i10pz> || [Bug 808073] numerous entries of "OPEN (null) (--) ==> -1 (No such file or directory)" in brick logs when an add-brick operation is performed <http://goo.gl/zQN2F>
13:19 MrNaviPacho joined #gluster
13:21 plarsen joined #gluster
13:38 rwheeler joined #gluster
13:39 jskinner joined #gluster
13:43 hagarth joined #gluster
13:46 ngoswami joined #gluster
13:46 Foo1 joined #gluster
13:46 bennyturns joined #gluster
13:47 rotbeard joined #gluster
13:51 jbrooks joined #gluster
13:53 ngoswami joined #gluster
14:03 bugs_ joined #gluster
14:03 mohankumar joined #gluster
14:15 andrei__ has anyone used glusterfs as a secondary storage in cloudstack?
14:15 andrei__ i am having some troubles
14:20 jtux joined #gluster
14:24 jskinner joined #gluster
14:25 Foo1 joined #gluster
14:26 vpshastry joined #gluster
14:30 andrewjs1edge joined #gluster
14:33 daMaestro joined #gluster
14:39 failshell joined #gluster
14:47 m0zes joined #gluster
14:48 tqrst is fix-layout supposed to be logged anywhere in 3.3.1? I just launched a targeted fix-layout in a folder, and all I've seen so far is "I [dht-common.c:2337:dht_setxattr] 0-bigdata-dht: fixing the layout of /foobar".
14:49 tqrst "rebalance volname status" shows all stopped/not started
14:50 tqrst and a related question: is 'setfattr -n distribute.fix.layout -v "anything" /bad/directory' recursive, or will it only fix the layout of that specific directory?
14:50 thebishop joined #gluster
14:52 failshell im curious, how many of you backup your data that's in gluster?
14:53 Supermathie failshell: That falls under the category of "Mirroring is not backup". :)
14:53 failshell yeah we had that conversation before ;p
14:53 thebishop gluster peer detach'
14:53 Supermathie tqrst: I'm guessing that it only does a single one
14:54 failshell Supermathie: i wonder if our regular VM backups are enough
14:54 thebishop 'gluster peer detach' will not let me remove a localhost ip
14:54 failshell Supermathie: we do SAN level snapshots
14:54 thebishop i get the error '<ip> is localhost', but i can't create a volume because it says the host is already in a cluster
14:54 failshell i wonder if replicating the data in gluster to a single VM for backups consistency is really needed
14:54 Supermathie failshell: Ah, you mean "backup from above the gluster layer" as opposed to "backup below the gluster layer"
14:54 Supermathie ?
14:55 failshell all gluster bricks are backed up by our vmware solution with SAN snapshots. but i also copy the data to another VM. to have a full instance of the data.
14:55 failshell that's the last part im not sure about
14:55 failshell because i end up with 3x the data
14:56 failshell plus sending it to tape
14:56 Supermathie failshell: mmmmm you need dedupe vtl ;)
14:56 failshell what's that?
14:57 jthorne joined #gluster
14:57 Supermathie To be honest, the only place I'm gluster in prod is on a 1x4 mirror setup... haven't thought a lot about backups.
14:57 Supermathie The other use case would be Oracle, and that's Rather Special for backups.
14:57 lpabon joined #gluster
14:58 Supermathie failshell: virtual tape library
14:59 ekuric joined #gluster
14:59 abyss^ joined #gluster
14:59 failshell i dont know much about that kind of stuff. i dont really handle backups
15:00 Supermathie failshell: idea is you backup to virtual tape drives that are really provided by an appliance with a bunch of disk behind it running an inlinen block-based dedupe engine. Then you can easily export from that to real tape.
15:01 failshell that's probably what we have
15:01 failshell i know we have some high end stuff
15:01 failshell another team takes care of that
15:02 Supermathie hmmmm with dedupe in place it really isn't so bad backing up the data 3x. Without it, I'd probably try and not backup the bricks but run the backup above gluster
15:03 failshell above gluster?
15:03 failshell you mean sync the data off the cluster?
15:03 Supermathie i.e. back up the gluster volume as opposed to back up the bricks
15:04 failshell that's the problem with VM + SAN snapshots backups
15:04 failshell you can't just say, backup /path
15:04 failshell we're moving off TSM
15:04 Supermathie failshell: You could perhaps organize things so that all bricks are on their own LUNs...
15:05 thebishop i can't create a volume because a node i want to use "not a friend" so i try to peer probe and it's "already part of another cluster" so I try to peer detach and it won't let me because the node "is localhost"... what now?
15:09 jskinner joined #gluster
15:10 jack_ joined #gluster
15:13 semiosis thebishop: are these servers clones?
15:14 thebishop semiosis, no
15:14 semiosis thebishop: try probing in the reverse direction
15:15 thebishop semiosis, i get it in either direction
15:15 thebishop "already part of another cluster"
15:15 Supermathie thebishop: can you wipe all state and start again?
15:15 thebishop and in both cases, peer status shows the machine's wall ip as a peer in cluster (connected)
15:16 thebishop Supermathie, is there a way to do that other than peer detach?
15:17 Supermathie thebishop: I mean blow away all the gluster state - stop gluster, nuke contents of /var/lib/glusterd (or move it out of the way) and restart from scratch? If you're just starting out and things got confused that may fix things.
15:18 Supermathie Or go through your config files and find something that doesn't make sense.
15:19 thebishop Supermathie, looks like it helped.  gluster volume create is blocking now, so hopefully it's doing the right thing
15:21 thebishop the quickstart guide says to make a folder like /export/brick1/sdb1, but the block device i'm using with gluster is different on my 2 machines.  is it safe to just use /export/brick1 for both servers in volume create, or should i make a folder with the appropriate device name on both?
15:21 m0zes joined #gluster
15:22 Supermathie It's safe to do ANYTHING, just pick somethign that makes sense. Also, make sure your gluster brick directory is a directory under the filesystem root, not the filesystem root itself.
15:22 Supermathie thebishop: In my one setup, I use /export/bricks/DISKWWN/glusterdata
15:22 thebishop Supermathie, ok thanks for clarifying
15:23 tqrst +1 on not being the root
15:23 semiosis ~brick naming | thebishop
15:23 glusterbot thebishop: http://goo.gl/l3iIj
15:23 thebishop semiosis, 404
15:23 semiosis http://gluster.helpshiftcrm.com/q/how-shoul​d-i-name-bricks-what-does-server-ext1-mean/
15:23 glusterbot <http://goo.gl/O5ia5> (at gluster.helpshiftcrm.com)
15:23 semiosis @forget brick naming
15:23 glusterbot semiosis: The operation succeeded.
15:24 semiosis @learn brick naming as http://gluster.helpshiftcrm.com/q/how-shoul​d-i-name-bricks-what-does-server-ext1-mean/
15:24 glusterbot semiosis: The operation succeeded.
15:24 tqrst Supermathie: (and judging by the lack of any kind of activity, my guess is also that fix-layout is not recursive)
15:25 sgowda joined #gluster
15:25 andrei__ hello guys once again
15:25 andrei__ i am having issues with automatically starting nfs service after a server reboot
15:25 andrei__ glusterfsd starts okay
15:26 Supermathie tqrst: Oh yeah, I doubt it. time for find -exec :)
15:26 andrei__ however, it is not starting nfs services
15:26 andrei__ even thought nfs.disable is set to off
15:26 semiosis andrei__: check the nfs.log file?
15:26 andrei__ any idea how to set gluster to autostart nfs services?
15:26 semiosis andrei__: it will autostart by default
15:26 semiosis unless you disable it, which you say you did not
15:26 tqrst Supermathie: or I could try to figure out why launching a rebalance always leads to a memory leak and segfaults all around, instead of fix-layouting by hand ;p
15:27 andrei__ if i run gluster volume set nfs.disable off after a server reboot it starts nfs service without any issues
15:27 anands joined #gluster
15:31 Supermathie andrei__: anything in the log?
15:31 andrei__ i will try to restart the server now and take a look
15:32 andrei__ restarging the glusterfs-server keeps nfs services running
15:34 semiosis andrei__: two people have asked you about the nfs.log file already.  help us help you.
15:34 andrei__ semiosis: i am going to reboot the server now to check the logs
15:35 andreask joined #gluster
15:35 andrei__ just a sec, need to make a few things happen before I can reboot
15:35 andrei__ almost finished
15:35 semiosis might as well trucate it first then
15:35 Supermathie andrei__: ... I usually use 'less', it's quicker.
15:35 semiosis pastie.org it for maximum support
15:35 andrei__ will do
15:36 chirino_m joined #gluster
15:37 andrei__ okay i've rebooted the server
15:37 puebele1 joined #gluster
15:37 andrei__ gluster peer status is showing the server as Connected
15:38 andrei__ showmount -e localhost
15:38 andrei__ clnt_create: RPC: Program not registered
15:38 andrei__ will check the logs now
15:38 Supermathie andrei__: also paste output of 'rpcinfo'
15:39 bulde joined #gluster
15:39 andrei__ http://pastie.org/7916516
15:39 glusterbot Title: #7916516 - Pastie (at pastie.org)
15:40 andrei__ that's the rpcinfo output from the server
15:40 lbalbalba joined #gluster
15:41 andrei__ very strange
15:41 andrei__ the last entry in the nfs.log file is almost an hour old
15:42 semiosis then check the etc-glusterfs-glusterd.log
15:42 Ramereth random question: is it possible to exclude a directory from being synced when using geo replication? (I know this is somewhat a silly question)
15:42 semiosis andrei__: also, what version of glusterfs, what distro/version?
15:43 semiosis Ramereth: never heard of such a feature, doubtful
15:43 Ramereth what I figured from what I could gather researching
15:43 andrei__ ubuntu 12.04 glusterfs 3.4beta1
15:43 andrei__ from ppa repos
15:43 Ramereth thanks!
15:43 semiosis andrei__: interesting. perhaps a bug in the new version?  if you havent resolved the issue later today I will try to reproduce on my VM lab
15:46 thebishop Supermathie, semiosis after blowing out /var/lib/glusterd, i followed the steps again with no errors, but when i do  "gluster volume info" it says No Volumes Present
15:46 * semiosis ducks
15:46 * Supermathie quacks
15:46 thebishop haha
15:47 Supermathie thebishop: Can you pastie your entire session history? all the commands you ran?
15:50 andrei__ this is so strange
15:51 andrei__ i can see that the log file modification time is pretty recent. however, the actual timestamps in the log entries are almost an hour old
15:51 andrei__ how could this be?
15:51 vpshastry joined #gluster
15:52 jack_ joined #gluster
15:52 andrei__ could it be that gluster has a different time zone?
15:53 Supermathie andrei__: if you start gluster (or any processes) and then change the system time zone.
15:54 andrei__ it was a clean server install without any time zone changes
15:54 andrei__ here is a log output after the server reboot
15:54 andrei__ http://pastie.org/7916598
15:54 glusterbot Title: #7916598 - Pastie (at pastie.org)
15:54 jskinner joined #gluster
15:54 semiosis andrei__: someone's messing with your logs?
15:55 andrei__ strange as the system time is one hour ahead of the glusterfs time reported in the log files
15:55 semiosis thats odd
15:55 andrei__ can't be messing as it's a dev lab
15:55 andrei__ noone has access to those servers
15:55 andrei__ and it was a clean install
15:56 andrei__ looking at a volfile  options, it's strange that it's using transport socket,rdma as I've specified transport tcp during volume creation
15:57 puebele1 joined #gluster
15:57 andrei__ the error with timestamp: [2013-05-16 15:51:51.760868] - probably relates to the network interface not ready at the time of glusterfsd start
15:59 Supermathie andrei__: "[2013-05-16 15:51:48.873658] W [common-utils.c:2330:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier" I've seen that a bunch in here, every single time it was networking (firewall blocking ports) related.
16:00 Supermathie 192.168.168.200:24007
16:00 Supermathie 192.168.168.200:24007 failed (No route to host)
16:00 vpshastry1 joined #gluster
16:00 andrei__ i think this relates to the start condition. I think glusterfs server starts prior to the network interface becoming available
16:01 andrei__ that's why you see the error messages
16:01 andrei__ but it does eventually connect as you see at the end of the log
16:01 andrei__ or am I not correct?
16:01 Supermathie sounds sensible
16:01 andrei__ could this be the reason why nfs is not enabled at start?
16:02 Supermathie yeah
16:02 andrei__ i am looking at the glusterfs-server init script in ubuntu
16:03 andrei__ and it seems to just start on runlevel 2345
16:03 andrei__ without specifying any requirements
16:04 jskinner joined #gluster
16:04 Supermathie doesn't say 'depends on X?'
16:05 andrei__ nope ((
16:05 aliguori joined #gluster
16:06 Supermathie for comparison, here's my ubuntu 12.04 init script from 3.3.1: http://pastie.org/7916644
16:06 glusterbot Title: #7916644 - Pastie (at pastie.org)
16:06 Supermathie s/init/upstart/
16:06 glusterbot Supermathie: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
16:06 andrei__ if I manually restart the glusterfs-server it still doesn't start the nfs service
16:07 andrei__ http://pastie.org/7916648
16:07 glusterbot Title: #7916648 - Pastie (at pastie.org)
16:07 andrei__ this is what I have with 3.4-beta1
16:07 andrei__ on ubuntu 12.04.2
16:10 andrei__ the strange thing is that it doesn't start the nfs services after I stop/start glusterfs-server
16:10 andrei__ when the network interface is ready and working
16:14 lbalbalba andrei__: no socket error in the log anymore ?
16:15 andrei__ still some errors
16:15 andrei__ one sec
16:16 andrei__ http://pastie.org/7916689
16:16 glusterbot Title: #7916689 - Pastie (at pastie.org)
16:16 andrei__ okay, i've stopped glusterfs server
16:16 andrei__ killed glusterfsd processes that were still running
16:16 andrei__ made sure no glusterfs processes were running on the server
16:17 andrei__ started glusterfs-server
16:17 andrei__ the logs above
16:17 andrei__ [2013-05-16 16:11:45.146903] - not sure what these errors are about?
16:18 andrei__ and also these Warnings: [2013-05-16 16:13:56.378153]
16:18 lbalbalba yeah, exactly . whats a 'Unknown key: brick-0'
16:18 thebishop joined #gluster
16:19 andrei__ i have a basic two server replication setup
16:19 andrei__ with 3 volumes
16:19 andrei__ no fancy stuff
16:19 lbalbalba and 2013-05-16 16:11:45.908856 says : connection attempt failed (Connection refused)
16:20 lbalbalba which means it received a tcp rst packet, right ?
16:20 andrei__ actually, very strange thing as the second server in the cluster also have the log timestamps off by an hour
16:23 lbalbalba i know you already mentioned this, but the log says  transport-type socket,rdma  ' when you specified tcp.   can you reset those to tcp ?
16:24 lbalbalba there are socket errors, after all ;)
16:24 Supermathie socket == tcp
16:24 lbalbalba ah
16:24 lbalbalba but why the rdma, then ?
16:25 lbalbalba rdma != tcp
16:27 andrei__ not sure
16:27 andrei__ how do i reset it?
16:30 andrei__ lbalbalba: from what I can see, the volume management information in the log file is taken from /etc/glusterfs/glusterd.vol file
16:30 lbalbalba this should work. it resets to the default, which appearantly is tcp:  gluster volume reset myvolume transport
16:30 andrei__ i think it's a default file
16:30 andrei__ which is not being used
16:30 Mo_ joined #gluster
16:30 lbalbalba ah
16:30 andrei__ could it show the default file prior to connecting to the real cluster?
16:32 lbalbalba dunno. sounds odd.
16:32 lbalbalba can you mv the file and restart yhe demon s ?
16:32 lbalbalba just guessing
16:33 lbalbalba ah, here we go : gluster volume set myvolume transport tcp
16:36 lbalbalba i dont even have a '/etc/glusterfs/' dir... and all my vol files in /var/lib/glusterd say 'option transport-type tcp', and not 'socket'
16:36 lbalbalba what version are you running. did you do an upgrade or new install ?
16:37 andrei__ one se
16:50 andrei__ without the /etc/glusterfs/glusterd.vol file my glusterfs server doesn't start at all
16:52 semiosis andrei__: that file launches glusterd
16:52 semiosis you need it
16:52 semiosis it never changes
16:52 semiosis just leave it alone
16:53 lalatenduM joined #gluster
16:53 andrei__ ah, okay
16:53 andrei__ in that case I don't really know why the nfs is not starting
16:54 andrei__ very odd indeed
16:54 semiosis andrei__: ,,(pasteinfo)
16:54 glusterbot andrei__: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
16:55 lbalbalba ah, there's my glusterd.vol: /usr/local/glusterfs/etc/glusterfs/glusterd.vol
16:55 andrei__ http://ur1.ca/dvwz8
16:55 glusterbot Title: #12584 Fedora Project Pastebin (at ur1.ca)
16:55 andrei__ here you go
16:58 semiosis attempting to reproduce
16:58 andrei__ thanks for your help
16:58 semiosis yw
16:59 andrei__ are you using 3.4beta1 as well ?
16:59 andrei__ or 3.3.1?
16:59 semiosis just installed the 3.4beta1
16:59 andrei__ cheers
17:00 rwheeler joined #gluster
17:00 andrei__ semiosis: are you going to be online in 20 mins? got to run home... newborn child and all, but will be online from home to continue
17:01 vpshastry1 joined #gluster
17:01 semiosis sure
17:02 lbalbalba andrei__: just curious does 'rpcinfo' show nfs ?
17:03 andrei__ http://pastie.org/7916516
17:03 andrei__ nope
17:03 glusterbot Title: #7916516 - Pastie (at pastie.org)
17:04 the-me joined #gluster
17:04 andrei__ i think i've found another bug with my setup
17:05 andrei__ when I am rebooting one of the servers, the nfs stops working even if I change the nfs mount point to a working server
17:05 semiosis andrei__: nfs started at boot on my clean install
17:05 andrei__ i am off, will resume in about 20 mins
17:05 semiosis now trying with your options
17:05 semiosis ok ttyl
17:11 rotbeard joined #gluster
17:13 thebishop joined #gluster
17:17 vpshastry joined #gluster
17:25 jskinner_ joined #gluster
17:28 saurabh joined #gluster
17:28 majeff left #gluster
17:30 plarsen joined #gluster
17:35 piotrektt_ hey. we have this setup in mind: 2 servers - 2 bricsk each (on separate NIC's) set to distributed replicated. Is that going to work?
17:35 andrei__ joined #gluster
17:36 andrei__ hello guys
17:36 andrei__ i am back online
17:36 andrei__ semiosis: did you manage to reproduce my issue?
17:37 semiosis piotrektt_: it could probably be made to work
17:37 piotrektt_ ok, and the real issue for me would be adding more servers to the equation
17:38 semiosis andrei__: nfs starts up fine for me
17:38 piotrektt_ when expanding volumes i would have to add 2 more servers for it to work, am i correct?
17:38 semiosis andrei__: you could try modifying /etc/init/glusterfs-server.conf like Supermathie did so that it doesnt start until your network interface is up
17:38 semiosis andrei__: worth a shot
17:39 semiosis andrei__: my VMs are very well behaved, no strange delays starting network interfaces
17:42 semiosis piotrektt_: well its possible to expand a volume by adding new bricks to existing servers....
17:42 failshell actually, you can use _netdev in /etc/fstab and that's going to delay mounting the volume
17:42 failshell at least with 3.2
17:44 thebishop joined #gluster
17:44 andrei__ semiosis: the trouble is it doesn't start even when I restart glusterfs-server
17:44 andrei__ following the system boot
17:44 andrei__ and interface being up
17:44 Supermathie semiosis: Pretty sure I didn't actually modify that... the packages installed it like that.
17:45 andrei__ i've also noticed one problem
17:45 andrei__ semiosis, do you still have your vms?
17:45 andrei__ i was wondering if you could test something for me
17:45 andrei__ what i've done is i've exported my three volumes as nfs to a client
17:45 andrei__ the client mounts nfs just fine
17:46 andrei__ i've got two servers - ibstorage1 and ibstorage2
17:46 andrei__ the client by default mounts nfs from ibstorage1
17:46 andrei__ mounting from ibstorage2 also works
17:46 andrei__ however, troubles start when I try restarting the servers
17:47 andrei__ when the client has a mount on ibstorage1, restarting ibstorage2 doesn't have any effect on the client
17:47 andrei__ nfs works fine, content is delivered to the client
17:47 andrei__ i remount client to ibstorage2
17:47 andrei__ and try to restart ibstorage1
17:48 andrei__ the mount point gives me: ls: cannot access /primary-kvm-centos-cluster1: Input/output error
17:48 andrei__ and i do not get a mountpoint until ibstorage1 is back up
17:48 andrei__ after that the mountpoint on ibstorage2 starts working okay
17:49 andrei__ i was wondering if you could test this on your setup?
18:02 vpshastry left #gluster
18:04 andrei__ semiosis: by the way, did you install gluster from the ppa?
18:05 andrei__ or from source?
18:09 neofob joined #gluster
18:26 andrei__ something odd is going on with gluster
18:27 andrei__ i've tried to reprode the problem with nfs mount and server restart and I can't do it anymore
18:27 andrei__ the mount point doesn't crash anymore
18:36 wushudoin joined #gluster
18:50 Supermathie yuck this error looks nasty
18:50 Supermathie [2013-05-16 14:50:31.801368] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 65f3181a, CREATE: NFS: 10006(Error occurred on the server or IO Error), POSIX: 117(Structure needs cleaning)
19:00 lbalbalba this test './tests/bugs/bug-823081.t.' keeps failing for me : http://fpaste.org/12617/36873064/
19:00 glusterbot Title: #12617 Fedora Project Pastebin (at fpaste.org)
19:00 neofob joined #gluster
19:00 lbalbalba something about not being allowed to use the system's root partition for storage backends
19:00 lbalbalba it says you can use 'force' to override it. i could hack up the test script to do that, but im not sure that that is the right 'solution' here ;)
19:01 ctria joined #gluster
19:07 rwheeler joined #gluster
19:07 hchiramm__ joined #gluster
19:07 nueces joined #gluster
19:08 plarsen joined #gluster
19:33 andrewjs1edge joined #gluster
19:36 lbalbalba strange. test ./tests/basic/posixonly.t should test, well, posix ;). and yet the gcov report says its not being hit: http://lbalbalba.x90x.net/lcov/gluster​fs/usr/local/glusterfs/lib/glusterfs/3​git/xlator/storage/posix.c.gcov.html
19:36 glusterbot <http://goo.gl/DV78R> (at lbalbalba.x90x.net)
19:53 andreask joined #gluster
20:07 thebishop When I run "gluster volume create gv0 replica 2 server01:/export/brick1/gluster_data server02:/export/brick1/gluster_data", it takes a few minutes, and then exits with no message.  afterward, /var/lib/glusterd/vols is empty and gluster volume list "No volumes present in cluster"
20:17 vigia joined #gluster
20:22 ninkotech__ joined #gluster
20:27 Nagilum_ thebishop: restart your glusterd's
20:30 thebishop Nagilum_, didn't help
20:30 melaur is using someone gluster for video storage server or image ?
20:30 thebishop Nagilum_, i found this blog post which might be relevant: http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
20:31 glusterbot <http://goo.gl/YUzrh> (at joejulian.name)
20:31 Supermathie thebishop: Have you looked at the logs? Also, check your firewall, I suspect glusterd can't talk to the bricks.
20:32 thebishop Supermathie, i can telnet to 24007 to/from both servers.  do i need any other ports?
20:32 Supermathie thebishop: Yep.
20:32 Supermathie ~ports
20:32 andrewjs1edge joined #gluster
20:32 thebishop Supermathie, "Ensure that TCP ports 111, 24007,24008, 24009-(24009 + number of bricks across all volumes) are open on all Gluster servers. If you will be using NFS, open additional ports 38465 to 38467. "
20:32 thebishop i'll try this
20:33 Supermathie ~ports | thebishop
20:33 glusterbot thebishop: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
20:33 Supermathie ^ that ;)
20:37 Supermathie @everybody: if you have weird errors creating volumes, turn off your firewall and try again ;)
20:38 andreask makes sense ;-)
20:41 lbalbalba @Supermathie: too bad the code doesnt check if the connections work before aqll hell breaks loose ;)
20:41 Supermathie lbalbalba: That could certainly be handled better
20:48 brian_ joined #gluster
20:51 brian_ Question: (This might be more of an infiniband question, but here goes…).. I had trouble with running the RDMA gluster client, so now I have made volume using the TCP transport. My question is: right now my infiniband interface is using the native kernel IB, and I'm going to switch it to IPoIB. Do I need to change anything with my gluster volume or will using a TCP transport type work fine using IPoIB instead of native IB?
20:52 jskinner_ joined #gluster
20:53 thebishop Supermathie, the nodes have all ports open between them
20:53 wN joined #gluster
20:53 thebishop Supermathie, and i'm seeing the same issue.  what should i look at in the logs?
20:53 jskinner_ joined #gluster
20:56 warthog9 joined #gluster
20:56 lkoranda joined #gluster
20:57 Supermathie thebishop: errmmm.... everything.
20:57 thebishop [2013-05-16 20:55:33.467683] E [rpc-transport.c:316:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
20:57 thebishop [2013-05-16 20:55:33.467700] W [rpcsvc.c:1356:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
20:58 semiosis [13:42] <failshell> actually, you can use _netdev in /etc/fstab and that's going to delay mounting the volume
20:58 semiosis failshell: not on ubuntu.
20:59 semiosis andrei__: good point about the nfsd not starting up on glusterfs-server start when the systems already booted. changing the upstart config wouldnt help with that at all.
21:00 semiosis andrei__: regarding your other problem, sounds like you have a brick not working right.  what does 'gluster volume status' say?  all bricks online?
21:00 jdarcy joined #gluster
21:01 majeff joined #gluster
21:01 semiosis thebishop: you can throw your log (etc-glusterfs-glusterd.log) up on pastie.org & give the link here, maybe someone will see something interesting
21:04 failshell semiosis: if you use ubuntu in production, you deserve the pain!
21:04 failshell :)
21:05 * failshell ducks
21:07 semiosis failshell: meh
21:07 failshell semiosis: it doesnt work with gluster 3.3 on RHEL
21:07 failshell but works with 3.2
21:07 failshell so i think its gluster that dropped support for it
21:07 failshell just like read-only mounts are broken in 3.3
21:08 semiosis _netdev is handled by startup scripts, it's an *fstab* option, not a *mount* option
21:08 semiosis afaik gluster just prints a spurious message saying it's ignoring the unknown option
21:09 semiosis if it doesnt work, that would most likely be a system issue
21:09 semiosis cant see how gluster vers would make a diff
21:09 failshell there's a bug opened for read-only mounts
21:09 failshell someone broke it
21:09 failshell _netdev, i dunno
21:09 semiosis separate issue
21:09 failshell i use 3.2
21:09 semiosis 3.1.7 here :)
21:09 failshell but if they broke read-only, maybe _netdev too?
21:09 failshell in any case, that works for me
21:10 semiosis your logic is fuzzy
21:12 duerF joined #gluster
21:15 thebishop semiosis, http://pastebin.com/fNtHucWs
21:15 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
21:16 Supermathie thebishop: looks like you have the same uuid on >1 host I think
21:17 thebishop Supermathie, should i wipeout /var/lib/glusterd again?
21:18 Supermathie Man, I got no idea at this point :) too tired.
21:20 badone joined #gluster
21:30 majeff joined #gluster
21:43 brian_ left #gluster
21:54 badone joined #gluster
22:08 fidevo joined #gluster
22:12 portante joined #gluster
22:19 y4m4 joined #gluster
22:36 jclift_ joined #gluster
23:16 StarBeast joined #gluster
23:22 robos joined #gluster
23:30 jag3773 joined #gluster
23:42 foster_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary