Camelia, the Perl 6 bug

IRC log for #gluster, 2013-04-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 sjoeboo joined #gluster
00:01 yinyin joined #gluster
00:01 ninkotech_ joined #gluster
00:02 juhaj joined #gluster
00:09 juhaj joined #gluster
00:17 _pol joined #gluster
00:21 juhaj joined #gluster
00:30 premera joined #gluster
00:30 genewitch joined #gluster
00:30 morse joined #gluster
00:30 flin_ joined #gluster
00:30 Azrael joined #gluster
00:30 Dave2 joined #gluster
00:30 jds2001 joined #gluster
00:32 juhaj joined #gluster
00:57 bala joined #gluster
01:17 d3O joined #gluster
01:22 d3O joined #gluster
01:27 xymox joined #gluster
01:30 lpabon joined #gluster
01:38 bennyturns joined #gluster
01:40 kevein joined #gluster
02:12 sjoeboo joined #gluster
02:35 berkc joined #gluster
03:13 chirino joined #gluster
03:18 sjoeboo joined #gluster
03:18 RobertLaptop joined #gluster
03:20 sjoeboo joined #gluster
03:22 mohankumar joined #gluster
03:47 itisravi joined #gluster
03:48 d3O joined #gluster
03:49 d3O joined #gluster
03:50 d3O joined #gluster
03:51 nickw joined #gluster
03:54 sgowda joined #gluster
04:21 d3O joined #gluster
04:23 itisravi joined #gluster
04:24 itisravi_ joined #gluster
04:27 bharata joined #gluster
04:30 itisravi joined #gluster
04:36 hagarth joined #gluster
04:54 sjoeboo joined #gluster
04:56 aravindavk joined #gluster
04:57 bala1 joined #gluster
05:02 CROS_ joined #gluster
05:04 bulde joined #gluster
05:05 domnic joined #gluster
05:06 saurabh joined #gluster
05:08 vpshastry joined #gluster
05:17 sgowda joined #gluster
05:19 lalatenduM joined #gluster
05:21 georges joined #gluster
05:31 mohankumar joined #gluster
05:32 d3O joined #gluster
05:33 d3O left #gluster
05:40 Cenbe joined #gluster
05:49 _pol joined #gluster
05:50 bharata joined #gluster
05:50 hagarth joined #gluster
05:52 vshankar joined #gluster
05:58 vimal joined #gluster
06:10 glusterbot New news from newglusterbugs: [Bug 918917] 3.4 Alpha3 Tracker <http://goo.gl/xL9yF>
06:15 stickyboy joined #gluster
06:17 stickyboy When I see updates, like 3.3.1-11 -> 3.3.1-12, should I just update and then restart all daemons, or should I bother with stopping volumes and unmounting from clients too?
06:18 stickyboy For the record, I see this: http://www.gluster.org/community/documentat​ion/index.php/Gluster_3.2:_Periodic_Updates
06:18 glusterbot <http://goo.gl/fsnMd> (at www.gluster.org)
06:18 stickyboy But the last item uses conflicting terminology so I'm not sure what the recommendation is.
06:19 Nagilum_ 3.3.1-12 is no good anyway, update to 3.3.1-13 instead! ;)
06:19 kevein joined #gluster
06:21 stickyboy Nagilum_: ah, it's 13 hehe.
06:22 stickyboy Yah, I just updated and it all seems fine.
06:22 stickyboy It seems the update stops the gluster processes, you just have to start them again.
06:22 stickyboy So the docs are right, it's just confusing if you haven't tried it.
06:23 samppah iirc there was some recommended way to either update clients or servers first
06:23 satheesh joined #gluster
06:23 samppah but i cant find documentation about it right away
06:23 stickyboy samppah: My clients are all on NFS so I guess it doesn't matter in my case. :D
06:23 samppah ahh :)
06:23 stickyboy Lucky me
06:28 14WAAPUDV joined #gluster
06:29 ujjain joined #gluster
06:35 mohankumar__ joined #gluster
06:40 glusterbot New news from newglusterbugs: [Bug 927648] volume status command not providing host names for NFS and SHD <http://goo.gl/AoK0w>
06:43 hagarth joined #gluster
06:44 deepakcs joined #gluster
06:46 ngoswami joined #gluster
06:54 ctria joined #gluster
06:55 ricky-ticky joined #gluster
06:59 raghu joined #gluster
06:59 ollivera joined #gluster
07:02 Nagilum_ joined #gluster
07:16 hybrid512 joined #gluster
07:18 shireesh joined #gluster
07:27 lh joined #gluster
07:34 tjikkun_work joined #gluster
07:34 rotbeard joined #gluster
07:39 stickyboy joined #gluster
07:39 stickyboy How do I unset a gluster volume option?
07:40 stickyboy Ah, `gluster volume reset volname performance.stat-prefetch`
07:40 glusterbot New news from newglusterbugs: [Bug 949406] Rebalance fails on all the nodes when glusterd is down on one of the nodes in the cluster <http://goo.gl/Q8dyW>
07:42 vpshastry1 joined #gluster
07:47 AD7six joined #gluster
07:50 AD7six hi. I have a folder which looks like this https://gist.github.com/AD​7six/3577e1c51334aeecbe8c - I'm not familiar with gluster, this is supposed to be an empty folder
07:50 glusterbot <http://goo.gl/Bcedv> (at gist.github.com)
07:50 AD7six what steps should I take to reoslve this?
07:50 samppah AD7six: what version you are using?
07:50 samppah @gfid
07:50 glusterbot samppah: The gfid is a uuid that's assigned to represent a unique inode that can be identical across replicas. It's stored in extended attributes and used in the .glusterfs tree. See http://goo.gl/Bf9Er and http://goo.gl/j981n
07:51 AD7six samppah: glusterfs 3.3.1 built on Oct 22 2012 07:54:24
07:51 rastar joined #gluster
07:54 AD7six hmz why would an empty folder have different gfid s
08:04 samppah AD7six: have you been always using 3.3.1 or is it possible that this is what has happened? http://community.gluster.org/a/alert-​glusterfs-release-for-gfid-mismatch/
08:04 glusterbot <http://goo.gl/uoyTN> (at community.gluster.org)
08:05 samppah not sure if it it's possible that something is causing split brains
08:05 samppah but iirc it should mention about it in log files
08:06 Rydekull joined #gluster
08:06 AD7six samppah: always 3.3.1
08:12 rb2k joined #gluster
08:16 spider_fingers joined #gluster
08:23 AD7six does "Please fix the file on all backend volumes" mean - run `gluster volume heal x full` on all gluster servers, or something different?
08:29 vpshastry1 joined #gluster
08:41 gbrand_ joined #gluster
08:44 rastar joined #gluster
08:47 bharata joined #gluster
09:03 stickyboy I have a java app that's really slow when my home is on nfs/fuse...
09:03 stickyboy A Java X11 app actually.
09:09 ngoswami joined #gluster
09:11 sonne joined #gluster
09:19 vpshastry1 left #gluster
09:19 vpshastry1 joined #gluster
09:41 glusterbot New news from newglusterbugs: [Bug 955546] TCP connections are stacking on master geo-replication side if the slave rejects the master IP. <http://goo.gl/n4ovx> || [Bug 955548] adding host uuids to volume status command xml output <http://goo.gl/rZS9c>
09:52 rastar joined #gluster
09:53 trapni joined #gluster
10:15 puebele joined #gluster
10:20 vpshastry joined #gluster
10:21 sonne joined #gluster
10:22 duerF joined #gluster
10:22 hagarth joined #gluster
10:23 nixpanic joined #gluster
10:23 nixpanic joined #gluster
10:32 ujjain joined #gluster
10:37 _ilbot joined #gluster
10:37 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
10:38 manik joined #gluster
10:51 sgowda joined #gluster
10:51 kkeithley1 joined #gluster
11:05 edward1 joined #gluster
11:09 andreask joined #gluster
11:09 ngoswami joined #gluster
11:11 glusterbot New news from newglusterbugs: [Bug 955588] Gluster volume info should return server uuid along with server ip and brick details. <http://goo.gl/ir68N>
11:12 jdarcy joined #gluster
11:13 16SABCFT8 joined #gluster
11:15 portante|ltp joined #gluster
11:23 rcheleguini joined #gluster
11:25 sgowda joined #gluster
11:31 ngoswami joined #gluster
11:35 jclift joined #gluster
11:50 manik joined #gluster
11:51 davis_ joined #gluster
11:55 sjoeboo joined #gluster
12:04 hagarth joined #gluster
12:10 yinyin_ joined #gluster
12:16 nickw joined #gluster
12:24 yongtaof joined #gluster
12:31 H__ my "mount -a -tglusterfs" stopped working on client nodes. Manual mounting the gluster volume works. Any ideas ?
12:36 ingard_ borked fstab?
12:48 manik joined #gluster
12:48 vpshastry joined #gluster
12:55 rastar1 joined #gluster
12:56 sjoeboo joined #gluster
12:56 H__ ingard_: not borked IMO, but gluster does not allow the perfectly valid noauto option
12:57 jclift H__: Hmmm, any interest in writing up a bug report (even super short) about that?
12:57 theron joined #gluster
12:57 H__ sure
12:59 nickw and the _netdev option is ignored in my env
13:00 rastar joined #gluster
13:01 ndevos H__: that could be Bug 827121 - but noauto should not abort mounting, the option just gets warned about and ignored
13:01 glusterbot Bug http://goo.gl/Rjhla unspecified, unspecified, ---, csaba, MODIFIED , [3.3.0] Mount options "noauto" and "_netdev" should be silently ignored
13:02 mohankumar joined #gluster
13:02 nickw 3.3.1 also this problem
13:02 nickw so i have to issued a 'mount -a' in my /etc/rc.local
13:04 ndevos _netdev should be interpreted by /etc/rc.sysinit and /etc/init.d/netfs, but you may have a race condition when glusterd starts, then netfs and then glusterd initializes completely and making the volumes available - netfs is too quick
13:04 jdarcy joined #gluster
13:09 yongtaof may be just chkconfig netfs on
13:10 yongtaof service netfs on will make it work
13:10 nickw i'll try it out
13:10 yongtaof at least in my test env I just run chkconfig netfs on
13:11 yongtaof then glusterfs will be mounted after boot
13:11 yongtaof you should try
13:11 yongtaof before chkconfig netfs on
13:11 yongtaof the auto mount fail due to varous problems even fail due to DNS look up
13:14 nickw anyone got the same problem that the log time used the UTC time but not the localtime in ver 3.3.2 and 3.4?
13:16 yongtaof jdarcy I want to learn more about .glusterfs/indices/xattrop directory can you help me?
13:17 yongtaof it is related to proactive self-heal, right?
13:17 yongtaof But I can't find any clue in glusterfs code, I only find that features/index xlator use it.
13:23 karoshi joined #gluster
13:25 andrewjsledge joined #gluster
13:25 karoshi I'm seeing a strange thing. In my test, I have one ext4 brick and an ext3 brick (I know, I certainly won't do this in production) creating a 2-brick replicated volume, mounted on the client. On the client, if I do a "find" over the volume mount point while rebooting one of the two bricks (the ext3 one), the find on the client terminates prematurely, but with no error: it just terminates as it would normally.
13:26 karoshi I somehow suspect it has to do with the fact that the two bricks use a different file system
13:26 karoshi if no brick is rebooted, the find on the client terminates normally
13:26 karoshi ie, listing all the volume contents
13:26 semiosis unlikely to do with a mix of filesystems
13:26 karoshi so what could it be?
13:27 semiosis don't ask me, check your log files
13:27 yongtaof the 2 brick is distribute or replica
13:27 semiosis yongtaof: good question
13:28 bennyturns joined #gluster
13:28 H__ jclift: I've added my release-3.3 branch noauto remark to Bug 827121 - [3.3.0] Mount options "noauto" and "_netdev" should be silently ignored
13:28 glusterbot Bug http://goo.gl/Rjhla unspecified, unspecified, ---, csaba, MODIFIED , [3.3.0] Mount options "noauto" and "_netdev" should be silently ignored
13:29 karoshi [2013-04-23 13:44:31.331164] W [fuse-bridge.c:2325:fuse_readdir_cbk] 0-glusterfs-fuse: 79144775: READDIR => -1 (Transport endpoint is not connected)
13:29 karoshi so I take it was reading from the one that goes away
13:29 karoshi but still, shouldn't that be transparent?
13:29 karoshi it's replica
13:29 semiosis maybe the other replica was never connected
13:30 karoshi ok, but the client has a volfile and knows where the data is, doesn't it?
13:30 karoshi it should failover
13:31 semiosis apparently there is a problem in your setup that prevents things from working the way they should
13:31 yongtaof use netstat to check if your client is connected to each brick
13:31 jclift H__: Cool. :)
13:31 semiosis remount the client, check the log, see if it fails to connect to any bricks
13:31 karoshi if instead of doing a find, I cat files all the time (for example), rebooting one or the other brick causes no problem whatsoever
13:31 semiosis or what yongtaof said
13:32 * jclift has been getting "Transport endpoint is not connected" way too much lately with rdma code and master head
13:33 jclift Going to look into that in next few days, after I get my head around debugging Gluster
13:33 jclift :)
13:34 nickw anyone got the same problem that the log time used the UTC time but not the localtime in ver 3.3.2 and 3.4?
13:35 H__ nickw: all my servers are in UTC
13:35 nickw while 3.3.1 uses the localtime
13:35 H__ so that I can compare timestamps between machines on different continents
13:35 karoshi it is indeed connected to both bricks
13:36 nickw i see. but i want the way that 3.3.1 deals with log file, any idea?
13:37 yongtaof besides I remember that glusterfs has a bug related to ext4, so you should not use ext4 as one of the ditributed brick
13:37 Supermathie yongtaof: or turn off dir_index on ext4
13:37 H__ yongtaof: depends on the kernel version, and latest gluster has it patched OK
13:38 yongtaof ah, good to know!
13:38 nickw seems 3.4 has the patch
13:38 karoshi this is 2.6.32, surely unaffected by the bg
13:38 yongtaof 2.6.32 is indeed affected by the bug
13:38 karoshi and in any case, the symptomps of the ext4 bug are different
13:38 yongtaof yeah, different
13:38 yongtaof the find will hang
13:39 yongtaof if use ext4 as ditributed brick
13:39 semiosis karoshi: glusterfs version?
13:39 karoshi on debian?
13:39 semiosis not on debian
13:39 karoshi rh has the bug in 2.6.32 because of their backport kludges
13:39 karoshi but this is debian
13:39 karoshi semiosis: 3.3.1
13:39 jdarcy Depends whose 2.6.32.  I believe the bug was backported into the RHEL6 kernel, but wouldn't be in the kernel.org one.
13:40 sandeen joined #gluster
13:40 karoshi also I noticed ext3 and ext4 readdir() return files in very different order
13:41 karoshi not sure what happens when client fails over from the ext3 brick to the ext4 or viceversa
13:41 karoshi while doing a find()
13:42 yongtaof jdarcy I can't find the self-heal related code regarding .glusterfs/indices/xattrop directory, do you know it? thank you.
13:43 semiosis karoshi: interesting. i ran with a mix of ext4 & xfs for a few weeks while i transitioned and didnt encounter any problems like that
13:43 yongtaof I can't find the benefit of mixing ext3 and ext4
13:43 semiosis karoshi: can you try the same experiment with same filesystems?
13:44 karoshi well, I'm doing kind of a stress test
13:44 karoshi that's what I was going to try
13:44 yongtaof :)
13:44 karoshi neither do I, but I happened to find myself with that situation
13:56 * karoshi is testing
13:57 karoshi now both are ext3
13:57 karoshi currently healing from the old ext3 brick to the new one
13:58 karoshi jdarcy: is it me or when starting with a fresh empty brick healing is much smoother than what I was describing yesterday?
13:58 karoshi perhaps it's just that that way things naturally spread over multiple dirs
14:02 bugs_ joined #gluster
14:03 bulde joined #gluster
14:12 aliguori joined #gluster
14:15 jdarcy karoshi: There should be a difference if you're doing full scans, but not if you're using the index (which is the default).
14:19 bala joined #gluster
14:21 karoshi so I guess it's just me
14:21 lh joined #gluster
14:21 lh joined #gluster
14:22 zykure joined #gluster
14:23 satheesh joined #gluster
14:23 karoshi just out of curiosity, how do I find out whether I'm using the index or doing full scans?
14:23 lh joined #gluster
14:23 lh joined #gluster
14:23 satheesh1 joined #gluster
14:28 nickw joined #gluster
14:30 Uzix joined #gluster
14:31 rwheeler joined #gluster
14:31 satheesh1 joined #gluster
14:34 H__ I auto-replaced part of configure.ac to get gluster report "glusterfs 3.3git-86acc0f" instead of [glusterfs 3.3git]
14:36 Staples84 joined #gluster
14:38 jbrooks joined #gluster
14:41 manik joined #gluster
14:46 nickw joined #gluster
14:54 vimal joined #gluster
14:55 dustint joined #gluster
14:58 dustint joined #gluster
14:59 daMaestro joined #gluster
15:00 zykure joined #gluster
15:01 spider_fingers left #gluster
15:02 jskinner_ joined #gluster
15:05 Supermathie OK... what would cause gluster to fail an NFS SETATTR call (truncate) to fail when seconds earlier, on the same file, on the same filehandle, it was perfectly OK?
15:05 nickw joined #gluster
15:06 Supermathie It may also be *that* causing gluster to think the files need healing
15:13 jrossi joined #gluster
15:14 dustint joined #gluster
15:15 jrossi I have a host that when probed will only show the IP address of the host that probed it. (4 in total cluster).  Doing and lsof -i shows that is speaking with all the other nodes but I can not get the server out of state: Sent and Received peer request (Connected).   I ahve looked into the logs and other sticks out what so ever.  All hosts are running gluster 3.3.1
15:16 jrossi does anyone have a suggestion of what to look for?  I have made sure that all dns is working the same from all hosts.  That they are all able to ping each other.
15:16 piotrektt joined #gluster
15:19 Supermathie oops no - the failing call is a SETATTR chmod... immediately prior to the call I see: "attributes  Regular File mode:0640 uid:500 gid:1000". Then the same client tries to 'chmod 440' as 500:1000 but I get a failure: "Status: NFS3ERR_ACCES (13)"
15:23 Supermathie (chmod and truncate)
15:24 Supermathie hmmmm..... if you truncate in the same NFS call as you chmod 440, does it chmod 440 the file first, then fail the truncate?
15:24 Supermathie I bet that's it.
15:43 xymox joined #gluster
15:44 hagarth joined #gluster
15:44 vpshastry left #gluster
15:46 zykure joined #gluster
15:49 bala joined #gluster
15:51 itisravi joined #gluster
15:56 soukihei joined #gluster
16:01 manik joined #gluster
16:04 rb2k joined #gluster
16:04 rb2k Hey. I'm currently trying to convert an older gluster setup to 3.3
16:05 rb2k the old vol files have several "volume" entries
16:05 rb2k with different types
16:05 rb2k "features/locks", "performance/io-theads", "storage/posix"
16:05 rb2k "protocol/server"
16:06 rb2k on 3.3. the preferred way to create volumes seems to be the commandline
16:06 rb2k "gluster volume create"
16:06 rb2k Do I somehow still have to add those different types, or is that now included in the default one?
16:06 _pol joined #gluster
16:07 _pol joined #gluster
16:09 manik1 joined #gluster
16:11 semiosis rb2k: since glusterfs 3.1 editing volfiles by hand is strongly discouraged
16:11 rb2k yeah, I'm trying to switch to the command line solution
16:11 semiosis rb2k: just use the command line
16:11 rb2k this is all a managed environment on about 4000 servers
16:11 rb2k so this has to be in puppet at some point
16:12 rb2k (not all in one FS)
16:12 rb2k I'm still reading up on the weird layout of our current volfiles
16:12 rb2k semiosis: do I still have to create/add those features separately?
16:13 semiosis wow thats a big deployment
16:13 semiosis you dont have to add those features manually
16:14 semiosis when you do 'gluster volume create...' the CLI will generate all necessary volfiles, with everything in place
16:14 rb2k semiosis: it's probably more at that point. we're probably at about 5000 machines :)
16:14 jclift rb2k: Are most of those clients, or servers?
16:14 jclift And do they all see each other?
16:14 rb2k jclift: it's usually client + server on each node
16:14 rb2k nope
16:15 rb2k it's just a LOT of small clusters
16:15 jclift Yeah.  My mind was boggling at potentially seeing 5000 glusterfsd's in ps -ef :D
16:15 rb2k I'd say up to 30 machines tops
16:15 rb2k :)
16:15 jclift Sure
16:15 rb2k but so far the vol file generation is pretty easily doable in a tempting system
16:15 jclift rb2k: Are you guys one of the known Gluster case studies?
16:15 rb2k we'll see how well the command line works
16:15 semiosis rb2k: curious, if you dont mind, what kind of application is this?  (just asking generally)
16:15 rb2k acquia
16:15 rb2k hosting
16:16 jclift Cool
16:16 manik joined #gluster
16:16 rb2k it's fun :)
16:17 rb2k but yeah, we have to upgrade quite a bit
16:18 Supermathie How do you add all the extra translators, etc from the command line?
16:18 jclift Heh, that's exactly the part of things I'm looking into now, trying to figure out how to do it easily.
16:18 jclift i.e. without having to hack volfiles
16:18 jclift So, totally curious myself too :)
16:18 semiosis you don't
16:19 Supermathie What I'd like to figure out is how to create two subvolumes, one being distributed-replicate, one being striped-replicate and then combine them with a translator that puts certain files onto a certain subvolume
16:19 semiosis Supermathie: thats not supported (yet)
16:20 semiosis something similar has been discussed, usually called "custom layouts."
16:21 Supermathie What I'm doing now is constructing two volumes from two different directories on the same brick and doin the layout at the client.
16:21 semiosis Supermathie: for now you'd need to have two separate gluster volumes
16:21 semiosis right
16:22 partner nice, just encountered first issues with our prod setup..
16:23 partner i was about to check why rebalance says the newer server has lesser disk and tried to ssh in: /bin/bash: Too many open files in system
16:24 partner seems i'm not the only one, googling around
16:25 partner wheezy with 3.3.1
16:25 partner stopped the rebalance and got in
16:30 Supermathie How do I inquire on a file's GUID?
16:30 semiosis you mean gfid?
16:30 Supermathie GFID yes :)
16:30 Supermathie OMG GUIDs EVERYWHERE
16:30 semiosis you can read its ,,(extended attributes)
16:30 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
16:31 jrossi left #gluster
16:31 Mo___ joined #gluster
16:32 semiosis @pathinfo
16:32 glusterbot semiosis: find out which brick holds a file with this command on the client mount point: getfattr -d -e text -n trusted.glusterfs.pathinfo /client/mount/path/to.file
16:32 partner VFS: file-max limit 813678 reached
16:33 Supermathie hmmmmm.... [nfs3.c:3408:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.10.3:35107) gv0 : 82c4c5ec-f3ad-4074-ac66-c5a455146d71
16:33 Supermathie buuuuuuuuuuuuuut that directory, as expected, has: trusted.gfid=0x82c4c5ecf3ad4074ac66c5a455146d71
16:34 Supermathie I think this may be due to glusterfs bugging out and files getting out of sync immediately prior to that error
16:34 semiosis partner: application not closing files properly
16:36 karoshi is port 24007 the one used to connect to the mount server?
16:39 JoeJulian @ports
16:39 glusterbot JoeJulian: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
16:40 karoshi thanks
16:43 jbrooks joined #gluster
16:44 partner semiosis: yeah, somehow this rebalance that i started yesterday triggered such behaviour
16:45 partner or timing was perfect to otherwise break, debugging...
16:45 semiosis ooh so could be rebalance that's not closing files?
16:45 semiosis not sure how that works
16:46 partner i can confirm my open files count was flat until i triggered rebalance yesterday 17:25 and since that moment the graph (of open files) went steadily up until it now reached the top
16:47 partner and it grew on both boxes having bricks running
16:47 jclift That sounds a bug needing filing. :/
16:48 semiosis partner: gluster version?
16:48 partner 3.3.1 running on top of wheezy
16:49 partner package is exactly version 3.3.1-1 from "your" repo
16:49 semiosis oh yeah you said that
16:49 semiosis "my" :)
16:49 partner you as the community or whatever, the official anyways :)
16:50 partner 735k files were rebalanced that far.. and guess if the graph shows 747k open files...
16:51 semiosis that sounds pretty incriminating
16:51 partner the "source" that is, a simple distributed setup, it was running only with one brick for a moment and yesterday added another server and brick from there and started to rebalance, nothing special there
16:53 mohankumar joined #gluster
16:54 mohankumar joined #gluster
16:57 partner if there's anything you want me to check out now would be good time to ask, i need to resurrect the production, not exactly even sure if something is still broken or not.. at least the source seems to have all still open..
16:57 partner used.value 756896
16:57 partner max.value 813618
16:57 partner the target recovered from that once i just stopped the rebalance
16:57 __Bryan__ joined #gluster
16:59 nueces joined #gluster
17:06 ctria joined #gluster
17:08 manik joined #gluster
17:12 glusterbot New news from newglusterbugs: [Bug 952693] 3.4 Beta1 Tracker <http://goo.gl/DRzjx>
17:18 saurabh joined #gluster
17:19 partner hmm
17:23 zaitcev joined #gluster
17:24 karoshi seeing the same thing as described earlier even with teo ext3 bricks
17:25 karoshi *two
17:27 karoshi client log: http://fpaste.org/t7Cy/
17:27 glusterbot Title: Viewing Paste #294135 (at fpaste.org)
17:27 JuanBre joined #gluster
17:28 semiosis karoshi: could you please remount that client, or make another client mount on the same client machine to a different mount point, and pastie the first 100 or so log lines?
17:28 semiosis that would show the initial connection attempt (and hopefully failure) which is usually very helpful
17:32 JuanBre I am having problems mounting via nfs
17:33 JuanBre here is the volume info, and the error message http://pastie.org/7704024
17:33 glusterbot Title: #7704024 - Pastie (at pastie.org)
17:33 karoshi semiosis: I'll do it tomorrow, thanks
17:35 JuanBre joined #gluster
17:41 JuanBre any advice?
17:43 glusterbot New news from newglusterbugs: [Bug 955751] /usr/sbin/glusterfsd coredump <http://goo.gl/0jVDh> || [Bug 955753] NFS SETATTR call with a truncate and chmod 440 fails <http://goo.gl/fzF6r>
18:01 semiosis JuanBre: i dont think <serverip>:38467/storage-ss is allowed.  afaik the gluster-nfs server requires <server>:/volume
18:04 johnmark semiosis: hmmm... I seem to recall trying to run an experiment using different ports for nfs
18:04 johnmark but I can't remember the outcome
18:05 johnmark the fact that I don't really remember probably indicates that it failed :)
18:05 semiosis oh i see, it's a port. didnt notice that before.  i've never seen that done (but have limited experience with nfs)
18:07 martin2__ joined #gluster
18:07 martin2__ good evening #gluster
18:08 semiosis good afternoon martin2__
18:11 JuanBre semiosis: http://pastie.org/7704189
18:11 glusterbot Title: #7704189 - Pastie (at pastie.org)
18:12 semiosis JuanBre: server iptables?
18:12 JuanBre semiosis: nop
18:12 semiosis see ,,(ports)
18:12 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
18:12 semiosis also ,,(nfs)
18:12 glusterbot To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
18:12 jskinner joined #gluster
18:12 semiosis JuanBre: check the log, /var/log/glusterfs/nfs.log, on the server you're trying to mount
18:13 semiosis access denied is better than no such file or directory, so progress
18:15 JuanBre semiosis:http://pastie.org/7704213
18:15 glusterbot Title: #7704213 - Pastie (at pastie.org)
18:16 semiosis JuanBre: please ,,(pasteinfo)
18:16 glusterbot JuanBre: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
18:16 semiosis oops never mind
18:16 semiosis you did that already :)
18:17 semiosis JuanBre: you have nfs.rpc-auth-allow: 192.168.1.245, is that the right IP for your client machine?
18:17 JuanBre semiosis: yes
18:18 semiosis hmm
18:20 martin2__ is there any best-practice available using glusterfs in xen/xenserver/xp dom0 with NFS?
18:21 martin2__ i already have it running but there seems to be some problems in the performance
18:25 JuanBre semiosis: I might have overwritten portmap configuration because I installed nfs-kernel-server accidentally some weeks ago. do you know how can I check if the nfs port is attached to the glusternfs server?
18:26 * semiosis scratches his head
18:26 semiosis been a while, let me check
18:26 JuanBre semiosis: by the way...I am using gluster 3.4 from your repositories...
18:27 semiosis oh i'm sorry about that :)
18:27 semiosis i have been remiss in my duties
18:27 semiosis the 3.4 repo is broken and i need to fix it
18:27 _pol Is it bad to have different brick sizes within a volume?
18:27 semiosis s/3.4 repo/3.4 ppa/
18:27 semiosis glusterbot: meh
18:27 glusterbot semiosis: I'm not happy about it either
18:27 glusterbot semiosis: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
18:28 _pol If I have a server with 10TB bricks, and I want to add another server that will contribute a 5TB brick, is that kosher?
18:29 _pol (Note, this is in a 2xReplicated, distributed set.  The replicate pairs are always the same size, but they might vary between servers 2x10 on one, 2x5 on another, etc)
18:29 semiosis _pol: brick size only matters when bricks fill up. then it matters a whole lot. you'd be setting yoruself up for one brick to fill up much faster than the rest
18:30 semiosis JuanBre: rpcinfo is the command
18:31 _pol semiosis: why does it matter, can't gluster just write to the one that isn't full?
18:31 JuanBre semiosis: I did post rpcinfo output in the first pastie...
18:31 semiosis partner: yes, but only for some definitions of 'write' :)
18:31 semiosis oops, not partner
18:31 semiosis _pol: yes, but only for some definitions of 'write' :)
18:31 _pol semiosis: and then, presumably, you mean that gluster doesn't take brick-size into account for the distributed hash algo. Is that true?
18:32 JuanBre semiosis: but now the output desnot show nfs
18:32 semiosis JuanBre: kill the glusterfs nfs server process -- see ,,(processes)
18:32 glusterbot information.
18:32 glusterbot JuanBre: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more
18:32 semiosis then restart glusterd, it will respawn it
18:32 semiosis that should get things working again
18:32 semiosis i hope
18:33 semiosis _pol: glusterfs will try to place new files on bricks that are not "full" but if you have files that continue to grow that is going to be aproblem, since they may be on the full brick, and wont get relocated
18:33 _pol semiosis: so, I am trying to scale-out, but within a server. I will have nodes half-full of drives, and then drives will get added.  Either I can have flexible brick sizes (so they can grow) or I can do multiple bricks per node.
18:34 semiosis basically, it gets complicated, and imho as a responsable admin you should plan your capacity and avoid that situation
18:34 semiosis responsible *
18:35 semiosis imho, expand existing bricks to add capacity, add new bricks to add performance (and capacity)
18:35 _pol semiosis: though if I am following you correctly, having different brick sizes is ok so long as they don't get full.
18:35 semiosis true
18:36 semiosis thats how i see it
18:36 semiosis gotta get back to java coding, bbl
18:36 _pol Is it not-terribly-difficult to force gluster to move files to a different brick if one is getting full?
18:36 JuanBre semiosis: I have the process /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/2122f93de9ec884c740468d3b5b35903.socket
18:37 lpabon joined #gluster
18:39 JuanBre semiosis: I killed that process...restarted glusterfs-server daemon (now the nfs appears at rpcinfo) but I still get permission denied
18:40 semiosis hmmm
18:40 semiosis idk
18:43 Supermathie Might it be best to rebalance often? (Does rebalance try to level % used or amount used?)
18:44 semiosis _pol: you can't force glusterfs to put files on one brick or another.
18:44 semiosis Supermathie: rebalance places files on bricks according to the elastic hash algorithm
18:44 Supermathie So it rebalances to try # of files?
18:45 Supermathie So it rebalances to try to level # of files on each brick?
18:45 semiosis files can be placed on the wrong brick (or replica set) for two reasons (usually) which are 1) file was renamed, and 2) bricks (or replica sets) were added to distribution
18:46 semiosis Supermathie: the elastic hash algorithm is what keeps the number of files on each brick level
18:46 semiosis Supermathie: and glusterfs has ways to deal with exceptions to the hash algorithm placement (link files, polling...)
18:47 semiosis Supermathie: rebalance resolves those exceptions so files are where they should be
18:48 puebele3 joined #gluster
18:49 Supermathie If you have one filesystem of size X and another of size 2X, might it make sense to create two bricks on top of the filesystem of size 2X so that the distribution algorithm places twice as many files on the larger filesystem as the smaller one?
18:49 semiosis Supermathie: sort of
18:50 semiosis Supermathie: it would be better if you split that 2x block device into two block devices (like with LVM) then gave each its own filesystem
18:50 semiosis so that space reporting would be sane
18:53 JuanBre semiosis: I think there is a bug related to the gluster nfs configuration....I have another volume in the same server with NO nfs.rpc-auth-allow configured but I can mount that volume perfectly
18:54 jdarcy joined #gluster
18:59 y4m4 joined #gluster
18:59 y4m4 joined #gluster
19:01 berkc joined #gluster
19:07 semiosis JuanBre: file a bug
19:07 glusterbot http://goo.gl/UUuCq
19:07 semiosis ooh actually there is a newer qa release than the one you have from my ppa
19:07 semiosis maybe it's fixed
19:07 semiosis i really need to get that PPA updated :(
19:10 JuanBre semiosis: I finally found the problem
19:11 semiosis what was it?
19:11 JuanBre semiosis: gluster volume option doc its wrong...nfs.rpc-auth-allow by default ALLOWS any client to connect
19:11 semiosis which doc exactly?
19:11 semiosis file a bug about that
19:11 glusterbot http://goo.gl/UUuCq
19:13 JuanBre semiosis: this doc http://gluster.org/community/documentation/ind​ex.php/Gluster_3.2:_Setting_Volume_Options#nfs.rpc-auth-allow
19:13 glusterbot <http://goo.gl/65I3J> (at gluster.org)
19:13 JuanBre semiosis: I know its for 3.2 but its the latest I could find
19:16 _pol Other than the bandwidth, is there any problem with adding several bricks of the same volume to a single server? Like server1 has brick1,brick3,brick5 and server2 has brick2,brick4,brick6, and all of them are in a single replicatedx2 distributed volume.  Is this ok?
19:16 semiosis JuanBre: check 'gluster volume set help' interactive documentation in 3.3+
19:17 lh joined #gluster
19:17 semiosis _pol: not even bandwidth. disks are often slower than networks, so you might need several bricks to get the most utilization.
19:17 semiosis _pol: theres no inherent problems with that setup
19:18 JuanBre semiosis: well...they changed that for the new versions...all connections are allowd..I think my problem was the rpcbind. but now its working
19:18 semiosis great
19:18 JuanBre semiosis: thanks for all your help. your repos are working really well for me
19:18 sandeen joined #gluster
19:19 semiosis yw, i'll get the 3.4 updated with the latest (alpha2 afaik) "real soon now"
19:19 semiosis the 3.4 ppa*
19:21 Jippi joined #gluster
19:22 rb2k joined #gluster
19:28 lh joined #gluster
19:28 lh joined #gluster
19:35 lh joined #gluster
19:35 lh joined #gluster
19:45 jag3773 joined #gluster
19:45 DEac- joined #gluster
19:46 bugs_ joined #gluster
19:49 JoeJulian What the... So if I have a bunch of bricks running and restart glusterd, now glusterd's not learning which port those bricks are listening on?
19:49 JoeJulian when did that start happening?
19:54 ricky-ticky joined #gluster
19:55 semiosis JoeJulian: did you see my PM earlier?
20:00 _pol_ joined #gluster
20:13 lh joined #gluster
20:13 lh joined #gluster
20:16 atrius joined #gluster
20:22 atrius joined #gluster
20:31 Uzix joined #gluster
20:36 atrius joined #gluster
20:39 hagarth joined #gluster
21:07 _pol joined #gluster
21:09 _pol joined #gluster
21:12 _pol_ joined #gluster
21:13 _pol joined #gluster
21:44 duerF joined #gluster
22:53 jag3773 joined #gluster
23:28 cyberbootje joined #gluster
23:29 rb2k joined #gluster
23:49 yinyin joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary