Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-29

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 TSM2 joined #gluster
00:16 nightwalk joined #gluster
00:18 jbrooks joined #gluster
00:32 mario_ joined #gluster
00:40 nightwalk joined #gluster
00:58 mario__ joined #gluster
01:01 robo joined #gluster
01:07 nightwalk joined #gluster
01:10 Psi-Jack joined #gluster
01:21 itamar_ joined #gluster
01:21 itamar_ hello guys
01:21 itamar_ why gluster is not accepting my commands.
01:22 JoeJulian because you typed them into a typewriter. Seriously, who keeps those around anymore anyway.
01:24 nightwalk joined #gluster
01:24 itamar_ JoeJulian, my typewriter runs linux
01:24 itamar_ and I have linux installed on them
01:25 JoeJulian Ok, fine.... gunna need more info then. What command, what error, have you checked the logs... is it plugged in and turned on...
01:27 itamar_ gluster volume quota VOLNAME enable
01:29 itamar_ hold
01:30 itamar_ gluster volume info returns something.
01:30 itamar_ but running gluster volume quota VOLNAME enable never ends
01:32 JoeJulian did you include [path] and [value]?
01:32 JoeJulian Or is that just for limit-usage.... hmm
01:32 JoeJulian never used quota myself.
01:33 itamar_ its just a example
01:33 itamar_ I am reading the manual
01:33 JoeJulian Well, check /var/log/glusterfs/etc-glusterfs-glusterd.vol.log for clues.
01:33 itamar_ To enable quota
01:33 itamar_ gluster volume quota VOLNAME enable
01:35 itamar_ I will show you what happens,hol
01:35 itamar_ hold
01:35 JoeJulian Don't paste here... use fpaste.org if you have output that you need to share.
01:35 itamar_ http://paste.stg.fedoraproject.org/2064/
01:35 glusterbot Title: #2064 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
01:37 mario_ joined #gluster
01:37 JoeJulian try restarting glusterd on both. It looks like some command left a stale lock somehow.
01:37 itamar_ the log keeps growing -> [2012-11-28 23:36:25.530759] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
01:46 kevein joined #gluster
01:50 ngoswami joined #gluster
01:51 nightwalk joined #gluster
01:52 itamar_ same problem after reboot
01:52 itamar_ http://paste.stg.fedoraproject.org/2065/
01:52 glusterbot Title: #2065 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
01:54 zhashuyu joined #gluster
01:57 mythzib_ joined #gluster
02:00 mythzib joined #gluster
02:00 JoeJulian ~pastestatus | itamar_
02:00 glusterbot itamar_: Please paste the output of "gluster peer status" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
02:01 JoeJulian ~pasteinfo | itamar_
02:01 glusterbot itamar_: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
02:02 itamar_ JoeJulian, I have 3 peers all with status connected
02:03 itamar_ b642d8c-a2fe-45fa-84ed-4094e36bf66e are the local machine.
02:10 sunus joined #gluster
02:16 lng joined #gluster
02:17 lng Hi! I have messages like this "[2012-11-29 02:16:03.739775] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-storage-client-6: remote operation failed: Stale NFS file handle. Path: /12050000/12055000/12055600/12055613/game.dat (f267c912-881e-4a3a-825f-687ee3654a15)". Should do something about it?
02:19 lng but these files seams to be ok
02:21 itamar_ one of my peers are "Peer Rejected"
02:24 semiosis go to the rejected peer, try using 'gluster volume sync <good-server> all'
02:25 semiosis rejected means the peer has volume config that is out of sync with the rest of the cluster
02:26 GLHMarmot Maybe I am missing something but is there a way to throttle the self-heal? It consumes large amounts of CPU and which causes the IO in some of my VM's to time out, causing the filesystem to go in to read-only mode.
02:26 GLHMarmot Yes, I am mounting the volume on the same machines that host the bricks.
02:27 itamar_ please delete all the volumes before full sync
02:27 semiosis itamar_: ok i though that might happen
02:28 GLHMarmot I am using the native fuse client with localhost:/volume (nfs with sync option does the same)
02:28 semiosis itamar_: so what you'll need to do is, on that rejected server, stop glusterd, move everything EXCEPT glusterd.info out of /var/lib/glusterd (so only glusterd.info remains) then start glusterd again
02:28 semiosis itamar_: then probe that server from a good one, probe the good one from the bad one, restart glusterd on the bad one, then you should be able to do the sync
02:29 lng who knows why some files have real path while others have gfid appended? <gfid:fdffd71b-fb1f-45d7-8869-6​91d2d40b311>/game.dat2012111612 vs /9760000/9762000/9762500/97​62503/16/game.dat2012111612
02:30 lng that's from `gluster volume heal storage info`
02:33 semiosis lng: idk
02:33 semiosis GLHMarmot: not sure but i doubt there's currently a way to throttle self heal
02:34 semiosis GLHMarmot: oh wait, yes there is!
02:34 semiosis GLHMarmot: gluster volume set <VOLNAME> cluster.background-self-heal-count 1
02:34 semiosis that restricts it to only healing one file at a time
02:34 lng semiosis: what is the correct to detect split-brain files? just detect first
02:35 semiosis default iirc is 16 in parallel
02:36 semiosis GLHMarmot: you can also set "cluster.data-self-heal-algorithm full" which could be less cpu intensive but more network intensive because it doesnt compare the files, just copies all the data
02:36 semiosis lng: i dont understand
02:36 lng semiosis: I need the list of split-brain files
02:37 y4m4 joined #gluster
02:37 semiosis lng: well i've not used 3.3 myself yet so idk much about the new features for that like volume status
02:37 semiosis in the old days of <=3.2 you would have to scan the xattrs of the files on the bricks
02:38 semiosis php sessions working on my dev & test but not on staging servers... what gives?!?!
02:38 lng is it possible to get real path by gfid? <gfid:c98bb368-1554-4c0e-8c91-cf2f5a7a9616>
02:39 lng semiosis: is it under ELB?
02:39 semiosis lng: hmm, i think there's a way to do it using find but i'm not sure
02:39 semiosis lng: ELB?  no, but my staging server is on ec2
02:39 lng semiosis: EC2 yes
02:40 semiosis it used to work, before one of my devs refactored this application :(
02:40 lng ELB is elastic load balancer
02:40 semiosis lng: oh right, i was thinking EBS! lol
02:40 semiosis lng: yeah it's under ELB but there's only one backend server for staging
02:40 lng we use redis for php sessions
02:40 lng it has php extension
02:40 semiosis lng: and I am using memcached (elasticache) session handler :)
02:41 semiosis hmmm
02:41 semiosis interesting
02:41 lng check your chaches
02:41 semiosis even though it *shouldn't* matter, i am enabling stickiness on the ELB
02:42 lng in this case, check memcached?
02:42 semiosis yeah i even tried going back to file-backed (default) session handler
02:42 semiosis tried a diff. browser with fresh cache
02:42 semiosis all same
02:43 lng hm
02:43 lng grep for session_start?
02:43 semiosis omfGaFhW&*(34ha9478W38GHA35
02:43 semiosis oh wait, wrong server
02:43 semiosis i thought sticky sessions fixed it
02:43 semiosis lol
02:43 lng heh
02:44 y4m4 joined #gluster
02:44 lng semiosis: did you mean `find` is able to resolve gfid to path?
02:44 semiosis lng: and the code works great on dev (localhost, file-backed sessions) and test (vm cluster in office with memcached sessions)
02:45 lng semiosis: check prod conf meyabe
02:45 semiosis lng: yeah, in the root of each brick there's a .glusterfs folder which has files named by gfid, those are hard links to the actual file in the brick, so you can use find (somehow) to locate the other hard links of the gfid file under .glusterfs
02:45 semiosis but i dont remember the command for that or where to find it :(
02:46 lng semiosis: I see
02:46 semiosis lng: thx for the help with php too :)
02:46 lng semiosis: ah NM
02:46 lng semiosis: are you PHP dev?
02:46 semiosis i dev all the things :)
02:46 semiosis except glusterfs hahaha, just a user
02:47 lng semiosis: I'm Server Engineer, working in Singapore on AWS
02:47 semiosis php, java, web stuff
02:47 semiosis cool
02:47 lng semiosis: nice to meet you!
02:47 semiosis likewise
02:47 JoeJulian "is it possible to get real path by gfid" yes. ls -i .glusterfs/c9/8b/c98bb368-1​554-4c0e-8c91-cf2f5a7a9616 will give you the inode number of gfid:c98bb368-1554-4c0e-8c91-cf2f5a7a9616
02:47 lng semiosis: I'm Ukraininan :)
02:47 semiosis :)
02:47 lng JoeJulian: hey!
02:48 lng cheerz!
02:48 JoeJulian Once you have the inode, find -inum $inode
02:48 lng great!
02:48 JoeJulian That, of course, only works on the indivdual server.
02:48 shireesh joined #gluster
02:48 itamar_ semiosis, thanks alot, I did what you say and worked pretty good.
02:48 semiosis awesome!
02:49 JoeJulian It'll have a different inode number on any other replica server.
02:49 lng JoeJulian: sure
02:50 lng I think after scaling my cluster, I have split-brain files - I can see that by file creation time
02:54 itamar_ semiosis, how can I prevent the nfs of glusterd being open to the internet ?
02:54 semiosis itamar_: iptables
02:55 semiosis ~ports | itamar_
02:55 glusterbot itamar_: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
02:55 semiosis there's also some ,,(options) for auth
02:55 glusterbot http://goo.gl/dPFAf
02:58 semiosis outta here
02:58 semiosis later all
02:58 semiosis good luck
02:59 itamar_ semiosis, before you go can you tell what options will help me ?
03:00 semiosis well i would look at the ones with nfs or auth in them
03:00 semiosis but i'm not sure
03:00 semiosis i am sure iptables would help you
03:00 * semiosis &
03:14 bharata joined #gluster
03:40 lng sometimes file being in split-brain failed to stat and sometimes not
03:48 hagarth joined #gluster
03:51 * m0zes tends to run something like 'file' on the files suspected of being split-brain. if it i/o errs it is split-brained.
03:52 ngoswami joined #gluster
03:55 shireesh joined #gluster
04:00 Ryan_Lane joined #gluster
04:00 Ryan_Lane I'm thinking I have too many volumes....
04:00 Ryan_Lane [2012-11-29 03:53:04.921723] W [socket.c:1842:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
04:01 JoeJulian Uh-oh...
04:01 Ryan_Lane any idea how to handle this?
04:01 Ryan_Lane fs.file-max in sysctl?
04:01 JoeJulian I think so. I was just looking that up myself. Last time I had this problem it was fixed in the kernel and I had to rebuild.
04:01 JoeJulian But that was ages ago.
04:03 Ryan_Lane well, let's see if that works.
04:03 JoeJulian Otherwise it might be nofile in http://linux.die.net/man/5/limits.conf
04:03 glusterbot Title: limits.conf(5) - Linux man page (at linux.die.net)
04:04 JoeJulian that would, of course, be per-process
04:04 Ryan_Lane yeah
04:04 Ryan_Lane I may need to do both
04:04 Ryan_Lane we'll see
04:04 Ryan_Lane I've restarted the process
04:04 Ryan_Lane not getting those errors yet
04:06 Ryan_Lane yeah. probably need limits.conf too
04:07 * m0zes gets those errors on occasion from a silly java app one of his users wrote. lets open 40,000 sockets and 20,000+ files.
04:07 Ryan_Lane wait. gluster is run by root...
04:07 Ryan_Lane ah
04:08 Ryan_Lane it's set very low
04:09 m0zes since gluster is run by root, wouldn't it make sense for the daemon to handle that kind of stuff automatically, say with 'setrlimit'? root processes are allowed to increase that ondemand in app, right?
04:10 Ryan_Lane yes
04:11 sripathi joined #gluster
04:11 m0zes also, prlimit should allow you to increase those limits on existing pids.
04:11 UnixDev I'm using 3.3.1, but I'm getting some strange kinds of hanging, simple volumes with 2 bricks in replicate 2 (1 brick per server on two servers)
04:15 m0zes UnixDev: what kind of hanging? what kind of i/o are you doing?
04:16 UnixDev hanging when trying to mount on server via fuse
04:17 UnixDev makes gluster hang, volume status fails
04:17 UnixDev volume info works
04:18 m0zes what is your mount command?
04:18 m0zes UnixDev: ,,(pasteinfo)
04:18 glusterbot UnixDev: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
04:19 UnixDev m0zes: mount command works on 1 server not on the other both host 1 brick
04:21 m0zes the reason I asked for the mount command is because I have seen others /try/ to mount the client fs over the brick mount. it has caused similar sounding issues.
04:23 vpshastry joined #gluster
04:25 raghu joined #gluster
04:28 Ryan_Lane hm
04:28 Bullardo joined #gluster
04:30 Ryan_Lane I'm honestly at a loss for how to fix this
04:31 Ryan_Lane when gluster starts it really need to increase the file limit
04:34 Ryan_Lane seems upstart may have a way of handling this
04:35 m0zes Ryan_Lane: http://upstart.ubuntu.com/wiki/Stanzas#limit
04:35 glusterbot Title: Stanzas - Upstart (at upstart.ubuntu.com)
04:35 Ryan_Lane yep
04:35 * Ryan_Lane sighs
04:35 Ryan_Lane nope
04:35 Ryan_Lane that didn't work
04:35 Ryan_Lane limit nofile 10000
04:36 Ryan_Lane [2012-11-29 04:35:33.921900] W [socket.c:1842:socket_server_event_handler] 0-socket.management: accept on 8 failed (Too many open files)
04:36 * Ryan_Lane sighs
04:37 m0zes maybe write a wrapper for the gluster upstart then?
04:37 Ryan_Lane may need to
04:37 Ryan_Lane I see reports that it doesn't work properly on ubuntu lucid
04:38 m0zes Ryan_Lane: with prlimit check that nofile is set correctly.
04:38 m0zes prlimit --pid <gluster pid>
04:39 m0zes either way, if the limit stanza stuff is broken, one would think that would be a critical bug to fix...
04:41 sunus hi, i got massive [2012-11-29 04:39:38.700373] W [socket.c:399:__socket_rwv] 0-socket.management: readv failed (No data available) in my log, what could possibly cause this?
04:45 * m0zes found a user with a homedir with 48 million ~4K files directly in their root. I hope they never try to 'ls -lah ~/'
04:52 Ryan_Lane seems switching to a script stanza and putting a ulimit -n 40000 and exec call insideof it is working
04:52 Ryan_Lane hopefully
05:13 nightwalk joined #gluster
05:17 Bullardo joined #gluster
05:20 lng I have '<gfid:2c18324e-54b8-4f00-b5d0-8​24b95583572>/game.dat2012111608' in my split-brain info heal output. I can list the directory in .glusterfs, but not the file: '/storage/1a/.glusterfs/2c/18/2c1​8324e-54b8-4f00-b5d0-824b95583572 -> ../../6f/74/6f74ec44-a2f2-4​01b-9e97-5da42a7ff34e/16/' and 'ls: cannot access /storage/*/.glusterfs/2c/18/2c18324e-54b8-​4f00-b5d0-824b95583572/game.dat2012111608: No such file or directory'
05:20 lng is it normal?
05:21 lng I need to take care of '<gfid:2c18324e-54b8-4f00-b5d0-8​24b95583572>/game.dat2012111608'
05:24 lng JoeJulian: how do I get inode number of it?
05:24 itamar_ my glusterfsd is failing to start http://paste.stg.fedoraproject.org/2073/
05:24 glusterbot Title: #2073 Fedora Project Pastebin (at paste.stg.fedoraproject.org)
05:26 m0zes itamar_: are you really wanting to start glusterfsd manually versus calling 'mount -t glusterfs'?
05:26 m0zes @daemons
05:27 m0zes @services
05:27 itamar_ m0zes, do I need it for nfs ?
05:27 m0zes whoops, not 'mount -t glusterfs', versus letting glusterd handle starting them
05:28 gbrand_ joined #gluster
05:29 m0zes this is the sort of cmd glusterfsd expects: /usr/sbin/glusterfsd --xlator-option cybercommons-server.listen-port=24020 -s localhost --volfile-id cybercommons.electra.mnt-glusterfs-cybercommons -p /var/lib/glusterd/vols/cybercommons/run​/electra-mnt-glusterfs-cybercommons.pid -S /tmp/b8ba6eac668d70f8042158ac691d8cf0.socket --brick-name /mnt/glusterfs/cybercommons --brick-port 24020 -l /var/log/glusterfs/bricks/mn​t-glusterfs-cybercommons.log
05:29 m0zes or for starting the nfs server: /usr/sbin/glusterfs -f /var/lib/glusterd/nfs/nfs-server.vol -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
05:31 UnixDev how can i fix "Failed to get names of volume" ?
05:32 lng JoeJulian: `find -inum $inode` is very slow
05:37 bala joined #gluster
05:38 itamar_ m0zes, my showmount -e doesnt show all volumes.
05:42 Humble joined #gluster
05:47 sunus hi, i got massive [2012-11-29 04:39:38.700373] W [socket.c:399:__socket_rwv] 0-socket.management: readv failed (No data available) in my log, what could possibly cause this?
06:05 GLHMarmot semiosis: Thanks! I read through that documentation (the pdf) a few days ago but it obviously just didn't click.
06:08 UnixDev semiosis: why would I not be able to get volume status? it seems a lot like some recent bugs "Failed to get names of volume"
06:12 lng JoeJulian: have you tried ` find -samefile` instead of `find -inum`?
06:14 Ryan_Lane upping the ulimit by sticking ulimit -n 40000 into a script stanza worked
06:18 JoeJulian @query granular
06:19 glusterbot JoeJulian: Bug http://goo.gl/ojYnw unspecified, unspecified, ---, csaba, NEW , [FEAT] Granular syncing with Geo Replication
06:19 glusterbot JoeJulian: Bug http://goo.gl/smLL4 medium, unspecified, ---, pkarampu, ASSIGNED , [FEAT] granular entry self-heal
06:22 JoeJulian oldbug 2949
06:22 glusterbot Bug http://goo.gl/m3fyO is not accessible.
06:23 VisionNL_ joined #gluster
06:27 Daxxial_ joined #gluster
06:28 JoeJulian oldbug 3182
06:28 glusterbot Bug http://goo.gl/U9bJ1 is not accessible.
06:38 JoeJulian I knew I should have never started following the mailing list.
06:47 rgustafs joined #gluster
06:50 sripathi joined #gluster
07:00 manik joined #gluster
07:07 mohankumar joined #gluster
07:11 yinyin joined #gluster
07:16 sunus is there any wrong for write-behind xlator?
07:18 ankit9 joined #gluster
07:20 20WABM2SJ joined #gluster
07:25 lng I have a lot of messages like '2012-11-29 07:17:16 <gfid:d259e11a-a91d-43bd-ac​32-aa5e9fe2ce60>/game.dat' produced by 'gluster volume heal storage info heal-failed'. how can heal it? normally I delete file and gluster metadata, but in this case I have no filename and searching it by inode is very slow because I have a lot of small files
07:41 lkoranda joined #gluster
07:50 ctria joined #gluster
07:53 puebele joined #gluster
07:54 tjikkun_work joined #gluster
07:57 ekuric joined #gluster
08:05 yinyin joined #gluster
08:07 36DACIFTZ joined #gluster
08:20 mohankumar joined #gluster
08:22 webwurst joined #gluster
08:22 webwurst left #gluster
08:22 webwurst joined #gluster
08:25 ankit9 joined #gluster
08:25 Daxxial_1 joined #gluster
08:28 andreask joined #gluster
08:28 andreask left #gluster
08:40 Humble joined #gluster
08:45 duerF joined #gluster
08:45 dobber joined #gluster
08:49 guigui3 joined #gluster
08:52 lng joined #gluster
08:53 lng Hi! could anybody help me on this gfid to file translation issue please? http://serverfault.com/questions/453342/s​earching-a-file-by-inode-of-gfid-hardlink
08:53 glusterbot <http://goo.gl/qqVC4> (at serverfault.com)
08:58 Humble joined #gluster
09:03 gbrand_ joined #gluster
09:12 gbrand__ joined #gluster
09:13 lng what should I do when I get <gfid:d259e11a-a91d-43bd-ac32-aa5e9fe2ce60>/file instead of real path?
09:14 vpshastry joined #gluster
09:35 morse joined #gluster
09:35 berend joined #gluster
09:45 lng anybody knows how to handle <gfid:d259e11a-a91d-43bd-ac32-aa5e9fe2ce60>/file?
09:45 sjoeboo joined #gluster
09:47 Alpinist joined #gluster
09:48 yinyin joined #gluster
09:54 ankit9 joined #gluster
09:56 Azrael808 joined #gluster
09:59 bitsweat joined #gluster
10:09 rudimeyer_ joined #gluster
10:13 vpshastry joined #gluster
10:28 * jdarcy_ o_O
10:33 mooperd kkeithley: hi
10:34 mooperd Is it possible for UFS instances to share keys
10:37 Daxxial_ joined #gluster
10:40 jayeffkay joined #gluster
10:41 jayeffkay What characters are allowed in glusterfs filenames?
10:41 lkoranda joined #gluster
10:42 jayeffkay all bar \0 and '/' ?
10:44 glusterbot New news from newglusterbugs: [Bug 880241] Basic security for glusterd <http://goo.gl/rOZ3P>
10:46 rudimeyer__ joined #gluster
10:48 Norky ugh, gluster performance %Bsucks%B on my system
10:49 Norky and I cannot work out why
10:49 Norky using tcp over GbE, rdma over 5 or 10Gb IB, tcp over IPoIB, the best performance I get is 60MB/s
10:51 Norky I've tested the individual components (network, both Ethernet and InfiniBand, disk subsystem) in isolation, they're all much faster
10:55 bauruine joined #gluster
10:55 jdarcy_ How are you testing GlusterFS performance?
10:55 TheHaven joined #gluster
10:58 grzany joined #gluster
10:59 Norky stupid dumb tests like dd if=/dev/zero bs=1M count=10K of=/glusterfsmountpoint and "time cp somefile /glusterfs" and also bonnie++
10:59 Norky they all give fairly consistent results
11:00 jdarcy_ Does your normal workload only have one I/O thread?
11:01 Norky as it stands, normally, yes
11:03 Norky I'm testing gluster (4 nodes, replicated, distributed) as the storage to replace some 'traditional NAS' for an HPC cluster of around 45 (8 or 12 core) nodes
11:03 twx_ replica count?
11:04 Norky the main application that runs on the cluster (STAR-CCM+, commerical CFD code) does apparently have support for parallel I/O, which I belive is one thread per core, and it can run across all 45*8 cores
11:04 Norky 2 replicas
11:05 jdarcy So even with one thread per node, you'd have 45 concurrent I/O threads?
11:05 Norky I think so, though we're not at the stage of testing with the real application yet
11:06 Norky I just wanted to see that I get better speed than bog-standard non-Gluster NFS over GbE before I start do 'real' tests. I don't
11:06 jdarcy So you might want to try running performance tests with more than one I/O thread too, e.g. iozone -l 24
11:06 unalt joined #gluster
11:06 jdarcy GlusterFS with replication is totally *latency* limited, bandwidth is hardly an issue.
11:07 jdarcy More parallelism overcomes the latency, and allows you to use more bandwidth.
11:09 andreask1 joined #gluster
11:09 Norky the latency of the interconnect betwixt the gluster servers, presumably?
11:10 jdarcy Replication is directly between client and N servers, so mostly that latency.
11:12 yinyin joined #gluster
11:13 torbjorn1_ are the details around failback documented somewhere ? .. Failover is working great, but I can't the client to failback, leaving me stranded with half the bandwidth
11:14 glusterbot New news from newglusterbugs: [Bug 881685] VM's were not responding when self-heal is in progress <http://goo.gl/ntb3Q>
11:16 jdarcy torbjorn1_: Are you talking about geo-sync?
11:17 torbjorn1_ jdarcy: no, this is a 2+2 distributed+replicated with 2 bricks on each server
11:17 torbjorn1_ jdarcy: running both boxes looks great, the reads are distributed among the two nodes
11:18 torbjorn1_ jdarcy: I block everything on one server with iptables, after 42 seconds the failover kicks in, and the operations continue
11:18 torbjorn1_ jdarcy: then I disable the firewall, log indicating that clients are reconnecting .. however, new reads are now only going to one server
11:19 jdarcy torbjorn1_: Is this all I/O to files that remained open throughout?
11:19 jdarcy torbjorn1_: Put another way, does I/O to *newly opened* files get distributed?
11:19 torbjorn1_ jdarcy: this is done doing dd, AFAIK all FDs to those files are closed between runs
11:19 mooperd Is it possible to do ranges?
11:20 jdarcy Hm.  I'd kind of expect this behavior for old fds, but not new ones.
11:20 mooperd gluster volume create gv0 replica 2 node{25..36}:/vdb
11:20 mooperd for instance
11:20 neofob joined #gluster
11:20 Norky mooperd, that would work if your shell is Bash
11:20 jdarcy mooperd: ISTRC that there is some support for that, but it's inconsistent (e.g. works for the path part but not the host part or vice versa).
11:21 Norky but that's a shell function, rather than gluster
11:21 torbjorn1_ jdarcy: should I inspect any gluster processes for open FDs to that file ?
11:21 sunus hi, do you guys see any romantic programming quotes?
11:21 jdarcy Roses are red, violets are blue, HDFS sucks and Ceph does too.
11:22 Norky mooperd, run "echo gluster volume create gv0 replica 2 node{25..36}:/vdb" - you will see what (if anything) your shell expands that to
11:22 sunus jdarcy: i love this:) any more?
11:22 * Norky awards sunus 5GBP for the most bizarre question in an irc support channel ever
11:23 jdarcy torbjorn1_: New fds *should* distribute I/O across both replicas, wouldn't even be aware of previous failures.
11:23 jdarcy torbjorn1_: If that doesn't happen, it's a bug.
11:23 mooperd gluster volume create gv0 replica 2 node{25..36}:/vdb
11:23 torbjorn1_ jdarcy: is there any way of inspecting the client state ? .. Can I ask stuff like "which servers are you seeing" ?
11:23 mooperd worked perfectly :)
11:23 mooperd I always get afraid when using brace expansion in anger
11:24 sunus Norky: hahahaha because i just want something like that but just couldn't find one:)
11:24 Norky mooperd, like I say, shell function rather than gluster, I used something like that myself
11:24 Norky and yes, if you're not sure what a glob will end up being, always test by prepending echo to your command line
11:25 jdarcy torbjorn1_: From the *client* I can't think of anything simple, but the I/O flows are what matter even (especially?) if both bricks are accessible.
11:26 sunus seriously, is there any wrong in write-behind xlator? because i've been debugging with qemu-gluserfs for week , and it just got segment fault, and when i turned of the write-behind, it work well
11:26 torbjorn1_ jdarcy: I was just thinking I might inspect the client state to see if it's actually aware of both servers now being available, although I belive thats the case
11:27 jdarcy sunus: A segfault in the app, or in one of the GlusterFS processes?
11:27 sunus jayeffkay: segfault in the qemu, which uses libgluster
11:27 jayeffkay huh
11:27 sunus jayeffkay: see those qemu-gluster intergeration
11:27 jdarcy torbjorn1_: You could use netstat to see if there are connections.
11:28 torbjorn1_ jdarcy: remounting on the client causes reads to go where I expected
11:28 jdarcy sunus: What's the bug-report URL?
11:28 Norky sunus, you're tab-completing to the wrong name jdarcy != jayeffkay
11:28 jayeffkay unless we're the same person
11:28 glusterbot New news from resolvedglusterbugs: [Bug 782088] [glusterfs-3.2.6qa1]: io-cache reconfigure gives warning messages for other xlator options also <http://goo.gl/FDnnK>
11:28 jdarcy jayeffkay: Are you breaking up with me?
11:29 jayeffkay jdarcy: it's not me, it's you
11:29 jdarcy LOL
11:30 jdarcy torbjorn1_: Seems like AFR is getting "stuck" on one replica.  I can see how that could happen, but it shouldn't.
11:30 jdarcy torbjorn1_: I suggest filing a bug report.
11:31 torbjorn1_ jdarcy: I'll experiment some more with it, and I'll file the bug report if I don't figure it out
11:31 torbjorn1_ general question: does the FUSE client keep any state between mounts ? .. Making stuff up here: Maybe a "volume info cache" or something like that ?
11:32 sunus jayeffkay: haha sorry, want a sec
11:33 jdarcy torbjorn1_: Nope.  Each mount basically starts from scratch.
11:34 torbjorn1_ jdarcy: thanks
11:38 andreask1 left #gluster
11:39 sunus https://lists.gnu.org/archive/html/​gluster-devel/2012-11/msg00033.html
11:39 glusterbot <http://goo.gl/Vvqye> (at lists.gnu.org)
11:39 sunus jdarcy: https://lists.gnu.org/archive/html/​gluster-devel/2012-11/msg00033.html
11:44 Norky jdarcy, I've just done a bit of testing of the interconnect between client and server. rdma_lat gives 1.35481 usec, rdma_bw gives Bandwidth average: 3244 MB/sec (local filesystem on the servers is around 1200MB/s sustained).  Forgive me, but it does not sound plausible that having two replicas gives me an effective bandwidth of 60MB/s
11:44 sunus jdarcy: so, i was wondering if there's something wrong in write-behind xlator, since i've been reading that part of code
11:45 jdarcy Norky: How about if you at least *try* multiple threads to see if it makes a difference before you say what's plausible or not?
11:46 Norky I will be testing with multiple concurrent threads/processes, but I just wanted to illustrate the huge difference in speeds between the individual components and the final result
11:47 jdarcy Norky: You can also use strace -T to see the actual syscall timings.
11:48 jdarcy sunus: It's entirely possible that something's awry in write-behind.
11:51 Norky ooh, I like that the clients see the increased space immediately after I grow the individual brick filesystems
11:58 Norky jdarcy, I've been testing with bonnie++, is iozone preferred (i.e. do you know it better)?
12:02 xavih joined #gluster
12:02 jdarcy Norky: I know both quite well, and I've seriously found that as a filesystem test it's more interesting to *build* bonnie++ than to run it.  Mostly it just tests memory-copy performance and libc (especially stdio) overhead.
12:03 jdarcy I used to have a script at my last job that would build bonnie++ ten times in parallel to test Lustre.  Found lots of bugs.  Never found a single one by running it.
12:03 Norky heh, righto, I'll let this bonnie++ test complete then install iozone
12:04 Norky I'll try fio as well
12:24 Alpinist joined #gluster
12:31 kkeithley1 joined #gluster
12:56 bulde joined #gluster
13:11 edward1 joined #gluster
13:19 toruonu joined #gluster
13:20 toruonu I have an odd state where I cannot delete a directory on glusterfs (mounted over nfs)
13:20 toruonu $ rm -Rf crab_0_121129_151837
13:20 toruonu rm: cannot remove directory `crab_0_121129_151837/share': Directory not empty
13:20 plarsen joined #gluster
13:20 toruonu doing ls -lR doesn't seem to show files
13:20 toruonu http://fpaste.org/DRFd/
13:20 glusterbot Title: Viewing crab_0_121129_151837: total 16 drwxr ... 2 mario HEPUsers 16384 Nov 29 15:19 ... crab_0_121129_151837/share: total 0 (at fpaste.org)
13:21 toruonu ah interesting, there's a .nfs… stale file
13:21 toruonu http://fpaste.org/MDxB/
13:21 glusterbot Title: Viewing crab_0_121129_151837/: total 48 drwx ... hare crab_0_121129_151837/share: tot ... 9 15:18 .nfs83fff0ff36d4e3490000000f (at fpaste.org)
13:21 plarsen joined #gluster
13:24 itamar_ joined #gluster
13:27 itamar_ how can I remove and re-create my bricks without loosing data ?
13:31 aliguori joined #gluster
13:38 itamar_ xx/yy or a prefix of it is already part of a volume
13:38 glusterbot itamar_: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
13:47 thekev joined #gluster
13:49 arusso joined #gluster
13:51 JordanHackworth joined #gluster
14:05 robo joined #gluster
14:06 mohankumar joined #gluster
14:08 hagarth joined #gluster
14:24 lh joined #gluster
14:26 toruonu guys … still stuck with this .nfs…. file issue...
14:27 toruonu the files are SQLite databases and it only happens at certain times, not always… when it does the file isn't deletable
14:27 lh joined #gluster
14:27 lh joined #gluster
14:32 tru_tru joined #gluster
14:48 Norky jdarcy, I neglected to say thank you - so thanks for your advice :)
14:48 balunasj joined #gluster
14:49 mohankumar joined #gluster
14:50 shireesh joined #gluster
14:57 stopbit joined #gluster
15:00 theron joined #gluster
15:07 asou joined #gluster
15:13 asou I have one out of 4 bricks 100% full
15:13 asou is it possible to distribute its data to other bricks ?
15:13 lkoranda_ joined #gluster
15:13 asou I have tried rebalancing but i does not work
15:16 yinyin joined #gluster
15:19 lkoranda joined #gluster
15:23 jbautista joined #gluster
15:26 Daxxial_1 joined #gluster
15:30 Humble joined #gluster
15:33 mooperd kkeithley: hi, are you there?
15:35 guigui3 joined #gluster
15:38 johnmark w00t...3.4-qa3 is released - http://bits.gluster.org/pub/​gluster/glusterfs/3.4.0qa3/
15:38 glusterbot <http://goo.gl/A5mli> (at bits.gluster.org)
15:45 puebele1 joined #gluster
15:45 Teknix joined #gluster
15:56 * toruonu waves … still got .nfs23986287342873642 files that are undeletable...
16:04 ekuric joined #gluster
16:13 H__ gluster volume rebalance vol01 status ...rebalance failed :( After 2 months of rebalancing that is.
16:21 daMaestro joined #gluster
16:22 tqrst H__: only two months?
16:22 tqrst lucky
16:26 MalnarThe hey peeps. i've been considering using gluster to store virtual machine VHDs, does it work well with large files that are kept constantly open?
16:27 MalnarThe more accuratelly, i need to re-export the files via iSCSI, so i wanted to setup 2TB files as loopback devices and export that
16:36 aliguori joined #gluster
16:42 Daxxial_ joined #gluster
16:45 Humble joined #gluster
16:46 nueces joined #gluster
16:47 Humble joined #gluster
16:58 rudimeyer___ joined #gluster
16:58 blendedbychris joined #gluster
16:58 blendedbychris joined #gluster
17:04 m0zes MalnarThe: that doesn't sound like a good fit for glusterfs. not saying that it couldn't be done... I *do* use glusterfs to house /small/ raw vm disks, but use very limited i/o to those disks.
17:06 semiosis but don't lose hope
17:06 semiosis even though that use case isn't the *best* fit for glusterfs, it can be done, and lots of people do it
17:06 Norky MalnarThe, I've read very similar things to what m0zes is saying. Large files on a POSIX glusterfs for use as disk images for may not work brilliantly
17:07 semiosis the tricky part is getting HA to work well
17:07 Norky however work is being done on a qemu module that interfaces directly to gluster, and apparently this is likely to perform better and more stably
17:08 semiosis if you're using replication and want the VMs to survive (migrate away from) a failed glusterfs server, there's some work to be done to avoid ,,(split-brain)
17:08 glusterbot (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
17:08 Humble joined #gluster
17:08 Norky of course if your VM host is something other than kvm/qemu that;s not much sue to you...
17:08 toruonu semiosis, now that you're online .. have you seen/heard of .nfs345345… files that stick around and are undeletable? I've noticed they always prop up when a tool is creating its SQLite db file, but at the same time a lot of times the tool works perfectly. Once the file is created I can't remove it...
17:09 toruonu neither can I remove the directory
17:10 semiosis toruonu: never heard of that
17:10 the-dude joined #gluster
17:10 JoeJulian https://uisapp2.iu.edu/confluence-prd/​pages/viewpage.action?pageId=123962105
17:10 glusterbot <http://goo.gl/OZbJ8> (at uisapp2.iu.edu)
17:10 bfoster joined #gluster
17:10 hackez joined #gluster
17:11 ron-slc joined #gluster
17:11 wN joined #gluster
17:11 robos joined #gluster
17:11 mario_ joined #gluster
17:11 eightyeight joined #gluster
17:12 sjoeboo question: doing some bench marks. I'm seeing pretty bad performance for a replicated cluster....if i run teh benchmark on one of teh storage nodes, (native gluster mount of its own volume), its slightly faster..but the REALLY weird thing is...on the remote client w/ the worst performance....using the nfs mount is WAY faster
17:12 toruonu damn netsplit
17:12 sjoeboo but...shouldn't that be the same as the "client of itself?"
17:12 toruonu semiosis: how much did you get? the fpaste and the example of rm?
17:12 sjoeboo or does that never have to pop-out into user space?
17:12 semiosis toruonu: none of that
17:13 toruonu http://fpaste.org/M1da/
17:13 toruonu [mario@ied Configuration]$ rm -Rf crab_0_121129_151837
17:13 toruonu rm: cannot remove directory `crab_0_121129_151837/share': Directory not empty
17:13 glusterbot Title: Viewing crab_0_121129_151837: total 36 drwxr ... are crab_0_121129_151837/share: tota ... 9 15:18 .nfs83fff0ff36d4e3490000000f (at fpaste.org)
17:13 semiosis toruonu: s/Rf/rf/ ?
17:13 toruonu no diff
17:13 toruonu it's not happening all the time, but I've already had reports from 2 other users as well
17:14 toruonu so it's common enough to be a problem at some point
17:14 semiosis sjoeboo: how are you benchmarking?
17:14 sjoeboo this is me just doing simple iozone thoughput tests
17:14 JoeJulian What is this .nfs file and why can I not remove it? https://uisapp2.iu.edu/confluence-prd/​pages/viewpage.action?pageId=123962105
17:14 glusterbot <http://goo.gl/OZbJ8> (at uisapp2.iu.edu)
17:14 semiosis sjoeboo: whats the network latency like?
17:15 semiosis sjoeboo: replication is very sensitive to latency, especially for small writes
17:15 toruonu ah … so somehow a process kind of has it open still … interesting
17:15 sjoeboo 267 ms or so
17:15 semiosis sjoeboo: that is extremely high latency for glusterfs replication
17:15 sjoeboo oops .267ms that is
17:15 semiosis haha, ok, much better :)
17:16 toruonu JoeJulian: or this is not the case here…
17:16 sjoeboo yeah, this is all on some pretty ugly left over iscsi kit, i'm not a fan of the storage backend for this, but just trying to rule out slow spots, teh nfs thing was a "why not" test, really supprised me.
17:16 toruonu first of all I cannot remove this file
17:16 toruonu [root@ied share]# rm .nfs83fff0ff36d4e3490000000f
17:16 toruonu rm: remove regular file `.nfs83fff0ff36d4e3490000000f'? y
17:16 toruonu rm: cannot remove `.nfs83fff0ff36d4e3490000000f': No such file or directory
17:16 toruonu and lsof doesn't show it in use
17:17 JoeJulian Maybe on another client?
17:18 JoeJulian bbiab... heading in to the office.
17:19 dberry joined #gluster
17:19 pdurbin joined #gluster
17:20 dberry joined #gluster
17:20 pdurbin i hear sjoeboo is making noise in here
17:20 semiosis sjoeboo: does the NFS client performance change if you add mount options "noac,sync" ?
17:20 sjoeboo i can try!
17:22 _Bryan_ @terms
17:22 glusterbot _Bryan_: I do not know about 'terms', but I do know about these similar topics: 'time'
17:22 _Bryan_ err..
17:22 semiosis ,,(glossary)
17:22 glusterbot A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
17:22 _Bryan_ thanks
17:23 sjoeboo semiosis: seemingly, yes, its slower (still running)
17:23 MalnarThe thats for the info. alas, my VM host is...shudder...hyper-v
17:24 MalnarThe but the idea was 'bulk' block level storage presented via iSCSI, guess i have to much about with GFS2 :/
17:24 MalnarThe gluster is soo nice otherwise
17:26 rudimeyer__ joined #gluster
17:28 sjoeboo yeah w/ sync,noac the nfs benchmark is slower than native gluster, as i'd expect
17:30 semiosis great, so... problem solved?
17:37 sjoeboo eh....sure....i'm still trying to figure out why this is SO slow, could be the iscsi, could be networking between the gluster servers and my test hosts, etc etc
17:38 semiosis all that latency adds up
17:39 sjoeboo indeed
17:43 semiosis not too familiar with iozone but generally speaking some benchmarks perform pretty poorly on glusterfs... imho best to do perf testing with actual work, not artificial benchmarks
17:47 Jippi joined #gluster
17:52 sjoeboo semiosis: yeah, been doing that too (not as thorough, but a bit). my real world test is "provision a vm onto the storage"
18:04 mooperd http://bugs.gluster.com/ is down
18:04 mooperd kkeithley: hey
18:09 semiosis mooperd: file a bug
18:09 glusterbot http://goo.gl/UUuCq
18:09 semiosis ^^^
18:11 mooperd semiosis: aha
18:11 mooperd Is redhat selling gluster now?
18:12 elyograg mooperd: as I understand it, redhat bought the company.  this happened some time ago.
18:13 mooperd elyograg: yea. but they havent had a gluster based product for a while
18:13 mooperd I think storage server is gluster based
18:20 Humble joined #gluster
18:26 johnmark mooperd: ?
18:26 mooperd johnmark: hello
18:26 johnmark RHS is a packaging of glusterfs
18:26 mooperd johnmark: thoughtso
18:26 johnmark heh :)
18:27 johnmark also, bugs.gluster.com should forward to bugzilla.redhat.com
18:27 johnmark if it's not, I need to file a bug
18:27 glusterbot http://goo.gl/UUuCq
18:28 mooperd johnmark: who moderates gluster-users?
18:28 johnmark mooperd: I do
18:28 johnmark although I mostly let it roll
18:28 mooperd johnmark: I got a bounce
18:28 johnmark oh?
18:29 mooperd Post by non-member to a members-only list
18:30 johnmark mooperd: are you a.holway@...?
18:30 mooperd johnmark: yep
18:32 lh joined #gluster
18:32 lh joined #gluster
18:33 tqrst @automount
18:34 johnmark mooperd: ok, done
18:34 mooperd johnmark: thanks man
18:35 mooperd johnmark: did you subscribe me?
18:35 mooperd I have this horrible feeling that Ill be posting a lot to that list
18:35 tqrst anyone here use autofs with the native gluster client?
18:36 johnmark mooperd: I added you to the accept list
18:36 johnmark do you already get messages from that list?
18:36 tqrst just curious if there are caveats
18:36 johnmark I assumed you were subscribed with a different address
18:37 johnmark tqrst: if you're talking about boot-time mounting, I've heard there is sometimes a need to insert sleep()
18:37 mooperd johnmark: no
18:37 johnmark tqrst: otherwise, I think there are people who use autofs
18:37 johnmark mooperd: ok. just send a message to gluster-users-request@gluster.org with "subscribe" in the subject
18:38 tqrst johnmark: I was considering using it for boot time mounting, yes
18:38 tqrst because adding _netdev to fstab isn't enough when I'm rebooting the whole cluster, since the server that line points to might not even be up yet when the other machines come up
18:39 mooperd johnmark: urgh. did my posts get posted? or should I re post em?
18:39 tqrst mooperd: it got through
18:39 tqrst s/it/they
18:39 mooperd tqrst: I am having a stupid day today
18:39 johnmark mooperd: they're posted
18:39 johnmark I pushed them through
18:40 johnmark heh :)
18:40 bauruine joined #gluster
18:40 mooperd Right. Im going to bloogy bed
18:47 genewitch joined #gluster
18:52 xymox joined #gluster
19:03 wN joined #gluster
19:04 _Bryan_ With 3.2.5 has anyone ever seen where a self heal completes successfully but doe snot heal any of the files?
19:05 _Bryan_ I have a small 3.2.5 glsuter that is hosting home directories...and it does not seem to heal any of the files...but the log files say it completes correcting all of them
19:07 gbrand_ joined #gluster
19:07 duerF joined #gluster
19:09 y4m4 joined #gluster
19:12 kkeithley1 mooperd: I'm not aware of any 3500 file limit in a container. I'll have to dig into the code, but I'm on a conf call atm.
19:23 kkeithley1 oops, didn't see that he left
19:33 semiosis tqrst: only times i've seen people here asking about autofs it's because they couldn't get fstab to work normally & are seeking a workaround
19:33 semiosis is there a genuine use-case for autofs with glusterfs?  maybe, but idk what it is
19:34 copec I could see it with the hekafs translators
19:35 semiosis furthermore, except for edge cases like bridged ethernet interfaces & other exotic things like that, fstab should work, though it may need a little tweak here or there
19:35 semiosis though recent packages have that stuff working out of the box afaik
19:38 tqrst semiosis: my use case is bringing my whole cluster up at the same time
19:39 tqrst semiosis: /etc/fstab currently has 'some_node:/myvol           /mnt/myvol            glusterfs defaults,auto,_netdev      0 0'
19:39 tqrst this will wait until the network connection is up, but it might still try to mount before some_node is up
19:39 tqrst whereas autofs will only attempt to mount when someone accesses /myvol, which is almost always going to be after some_node is up
19:40 semiosis ooh, interesting
19:41 semiosis fwiw, you can use ,,(rr-dns) to improve availability of the mount server
19:41 glusterbot I do not know about 'rr-dns', but I do know about these similar topics: 'rrdns'
19:41 semiosis ,,(rrdns)
19:41 glusterbot You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
19:41 tqrst I was just going to ask about that
19:41 tqrst thanks :)
19:41 semiosis yw
19:44 _Bryan_ any feedback on gluster falsing reporting the data was self healed?
19:44 _Bryan_ s/falsing/falsely/
19:44 glusterbot What _Bryan_ meant to say was: any feedback on gluster falsely reporting the data was self healed?
19:45 semiosis how do you know it's not healed?
19:46 _Bryan_ the file does not mirror the other side..
19:46 _Bryan_ and it continues to show up in subsequent self heals
19:46 _Bryan_ btw...I have self-heal count set to 1
19:46 sjoeboo for rrdns, isn't is best to use fetch-attempts=X when mounting? that way if a node in teh cluster is down,the next one (provides by rrdns) is picked ?
19:47 semiosis JoeJulian: didn't you have an issue with things getting stuck healing over & over again?
19:47 semiosis sjoeboo: doesn't it automatically try all the records?
19:48 _Bryan_ I have tried to stop the entire volume...restart it and re run self heal..but same files show up over and over
19:48 semiosis _Bryan_: could a brick be read-only?  maybe brick logs say something useful...
19:48 sjoeboo semiosis: does it? i don't know. i always use fetch-attempts  = to the ndoe count to be safe...
19:49 sjoeboo semiosis: i'm thinking of the case where the first system in the rr is down when a client tries to mount and get teh vol info
19:49 semiosis well tbh, idk
19:49 semiosis but never heard of that option before :)
19:49 _Bryan_ sjoeboo: you setup backup server volume file in the mount options..
19:50 _Bryan_ backupvolfile-server={fqdn backup server to get vol info from}
19:51 _Bryan_ basically another host in the glsuter volume
19:51 sjoeboo _Bryan_: hm, never done that. fetch attempts works well for us here...all nodes are a backup volefile server basically...
19:51 _Bryan_ this is what my mount looks like....it has primary locaiton for vol info information adn then backup
19:51 _Bryan_ this is on 3.2.5 fyi
19:51 _Bryan_ gfsxxxxx:/gfs-vol0 /gfsnet glusterfs defaults,_netdev,backupvolfile-server=gfsxxxx 0 0
19:52 sjoeboo i like the idea of rrdns + "try to get the volfile X number of times" , the X just needs to grow as your nodes do..
19:52 * semiosis fires up the test vm
19:53 _Bryan_ semiosis: will go verify..but I swear I have looked at that
19:55 semiosis nothing is needed
19:55 _Bryan_ goign to stop voluem and bounce all the gluster daemons..
19:55 _Bryan_ and see if it helps
19:55 semiosis i created an A record with 8 addresses, 7 of which are non-routable on my lan, and the 8th is 127.0.0.1
19:56 DaveS joined #gluster
19:56 semiosis i used 'host' to fetch the rr-set on the gluster client machine, several times, to make sure the rr-set was returned in an order where the localhost ip would be down near the bottom, not the first address
19:56 semiosis (it is round-robin, after all :)
19:57 semiosis did a basic mount, with no options, and it went through the first 6 non-routables before connecting & mounting successfully from the real one
19:57 semiosis no retries, nor backup server needed
19:58 semiosis oh that was with glusterfs 3.3.1 btw, so ymmv on older versions
20:03 semiosis @learn peer-rejected as http://community.gluster.org/​q/how-do-i-fix-peer-rejected/
20:03 glusterbot semiosis: The operation succeeded.
20:08 tqrst that's a 404 for me
20:08 tqrst with https-everywhere anyway, and a "please log in" for some other browser
20:09 tqrst oh. If you omit the trailing / in http://community.gluster.org/​q/how-do-i-fix-peer-rejected, it asks to login
20:10 glusterbot <http://goo.gl/YhQAP> (at community.gluster.org)
20:10 tqrst ...interesting
20:15 semiosis tqrst: looks like the SSL cert is for *.helpshift.com which of course isn't valid for c.g.o
20:18 atrius joined #gluster
20:38 badone joined #gluster
20:41 bitsweat left #gluster
20:49 tc00per joined #gluster
20:52 aliguori joined #gluster
20:53 y4m4 joined #gluster
21:01 DaveS joined #gluster
21:18 y4m4 joined #gluster
21:20 duerF joined #gluster
21:26 edward1 joined #gluster
21:31 pdurbin jdarcy: 5b176fdb ?
21:33 pdurbin i'm getting a 404 from https://github.com/gluster​/glusterfs/commit/5b176fdb
21:33 glusterbot <http://goo.gl/a7vSC> (at github.com)
21:42 gbrand_ joined #gluster
21:42 johnmark doh
22:23 nueces joined #gluster
22:25 cyberbootje joined #gluster
22:30 tqrst any idea why the 'Port' entry for two of my bricks is 'N/A' instead of an actual port number? The bricks are online and work fine as far as I can tell.
22:30 tqrst in the 'volume status' output, that is
22:31 cyberbootje joined #gluster
22:32 cyberbootje1 joined #gluster
22:34 tqrst see lines 37 and 39 in http://fpaste.org/yIk5/
22:34 glusterbot Title: Viewing Paste #256155 (at fpaste.org)
23:07 wnl_work joined #gluster
23:08 wnl_work am i losing my mind or was there a "volume status" command in 3.1? it doesnt appear to exist in 3.2
23:09 semiosis new in 3.3
23:09 semiosis so, losing your mind i guess :)
23:09 wnl_work oh. not in 3.1?
23:09 duerF joined #gluster
23:10 semiosis right
23:10 wnl_work now im really confused
23:10 wnl_work either that or i loaded gluster from the wrong repo. which would be really sad.
23:10 semiosis there were "peer status" and "volume info" but not "volume status"
23:10 wnl_work crap
23:10 TSM2 joined #gluster
23:10 wnl_work crap crap crap. theres an hour of work down the drain. i didnt put the epel-gluster repo in place before installing.
23:11 wnl_work okay, thanks. at least i saw this before laoding the data in to the volume
23:11 semiosis yw
23:12 wnl_work well, maybe not an entire hour. but too late to try to fix up now.
23:16 ctria joined #gluster
23:19 glusterbot New news from newglusterbugs: [Bug 881997] gluster volume geo-replication config checkpoint now works but returns command failed <http://goo.gl/OiUKK>
23:37 aliguori joined #gluster
23:38 robo joined #gluster
23:59 cyberbootje joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary