Camelia, the Perl 6 bug

IRC log for #gluster, 2013-03-07

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 gbrand_ joined #gluster
00:12 NcA^__ joined #gluster
00:15 NcA^ joined #gluster
00:26 Ryan_Lane joined #gluster
00:26 Ryan_Lane I needed to restart my glusterd processes, since they were all hung
00:27 Ryan_Lane now nfs isn't working
00:27 Ryan_Lane it shows the exports via showmount -e
00:27 Ryan_Lane and the relevant services are working via rpcinfo -p
00:27 Ryan_Lane mount simply hangs, though
00:27 Ryan_Lane and no, I have no firewalls
00:27 Ryan_Lane any ideas?
00:34 _pol joined #gluster
00:36 Ryan_Lane mount.nfs: mount(2): Permission denied
00:36 Ryan_Lane mount.nfs: access denied by server while mounting labstore1.pmtpa.wmnet:/keys
00:36 * Ryan_Lane sighs
00:36 Ryan_Lane nfs.rpc-auth-allow: *
00:47 Ryan_Lane anyone?
00:47 Ryan_Lane showmount also shows it as shared to the world
00:48 JoeJulian Ryan_Lane: Just got back home from a Dr. Appt and some errands. Have you found anything yet?
00:49 Ryan_Lane no. I can't figure out why it's denying access
00:50 JoeJulian Nobody's currently successfully using nfs, right?
00:50 Ryan_Lane right
00:50 Ryan_Lane totally dead since glusterd restart
00:50 JoeJulian pgrep -f gluster/nfs | xargs kill
00:50 JoeJulian Then restart glusterd again
00:51 JoeJulian That should restart the nfs server process
00:52 JoeJulian Hopefully that should get you up and running. I would look in /var/log/glusterfs/nfs.log once it's fixed to see if there's some clue as to why it failed.
00:52 Ryan_Lane I didn't see any errors in the nfs log
00:53 Ryan_Lane hm. one of the gluster servers wasn't running portmap
00:53 Ryan_Lane that said, it's not the one that was trying to be accessed
00:54 JoeJulian afaik, that should only have affected that one server then.
00:54 Ryan_Lane yep
00:54 Ryan_Lane just in case I started it
00:55 Ryan_Lane that seems to have worked
00:55 Ryan_Lane killing all daemons and restarting all again
00:55 Ryan_Lane though only when I specify the version....
00:56 Ryan_Lane precise defaults to vers 4, though, so that makes some sense
00:59 Alknelt joined #gluster
01:00 hagarth joined #gluster
01:00 Alknelt I'm back for help again. Running a new cluster, but I seem to have broken it. I can't successfully mount the fs. I can't run gluster volume status <vol-name>, just stalls out.
01:02 Alknelt Starting or stopping the volume fails...
01:02 Alknelt peer status looks good.
01:02 Alknelt volume info looks okay
01:04 yinyin joined #gluster
01:05 tryggvil joined #gluster
01:08 vpshastry joined #gluster
01:09 JoeJulian Check the logs for clues. If those other commands succeed, I don't know why status would fail.
01:15 Alknelt I can't find any clues in the logs… I'm brand new to gluster
01:17 JoeJulian Well paste them on fpaste or something so I can take a look then. We're probably looking for the cli.log and etc-glusterfs-glusterd.vol.log
01:18 Ryan_Lane JoeJulian: thanks again for the help!
01:18 JoeJulian You're welcome.
01:18 Ryan_Lane you always seem to help me figure out the issue :)
01:23 Alknelt http://pastie.org/6408293
01:23 glusterbot Title: #6408293 - Pastie (at pastie.org)
01:24 akshay hey JoeJulian
01:24 akshay i have a question regarding the logs
01:24 Alknelt http://pastie.org/6408299
01:24 glusterbot Title: #6408299 - Pastie (at pastie.org)
01:24 JoeJulian Ryan_Lane: I'm glad I can help. :)
01:24 akshay is there a way by which we can truncate the logs or rotate them
01:25 JoeJulian use copytruncate in logrotate.
01:25 akshay ok
01:27 Alknelt I do see some Lock held information by a UID....
01:28 Alknelt but it just seems to be locked by itself already...
01:33 JoeJulian Alknelt: Is the volume started? The last command I see seems to have stopped the volume.
01:34 Alknelt "volume status" returns operation failed. volume start <vol> returns operation failed
01:36 Alknelt http://pastie.org/6408366
01:36 glusterbot Title: #6408366 - Pastie (at pastie.org)
01:36 Alknelt The etc-gluster log is full of this ^
01:37 Alknelt disconnecting now
01:37 JoeJulian How about "gluster volume info"
01:37 JoeJulian Actually, that does look like a race. Restart both/all glusterd and try again.
01:39 Alknelt Restarted both glistered services. volume info returns great
01:39 JoeJulian Can I see the output of that?
01:40 Alknelt http://pastie.org/6408384
01:40 glusterbot Title: #6408384 - Pastie (at pastie.org)
01:41 Alknelt stop volume, start volume, success. Then Volume status….. pukes
01:41 JoeJulian hmm... this is 3.3.1, right?
01:42 Alknelt yes, fresh install
01:42 Alknelt Well not exactly fresh. Earlier today I had ext4 bricks. Now I have xfs bricks..
01:42 JoeJulian selinux maybe?
01:44 JoeJulian It shouldn't make any difference, but I see one thing that doesn't get a lot of attention so may still have bugs.
01:44 JoeJulian Distributed-Stripe
01:44 JoeJulian Do you know /why/ you want stripe?
01:45 JoeJulian I'm not asking for an explanation, but unless you already understand gluster and know /why/ you want to use it, you usually don't want to.
01:45 Alknelt I'd picked stripe to try to get a little more speed out of gluster. I'm currently running Ibrix, and gluster is a little slow in comparison
01:45 JoeJulian @stripe
01:45 glusterbot JoeJulian: Please see http://goo.gl/5ohqd about stripe volumes.
01:46 Alknelt Just straight up distributed would be better then?
01:46 tg2 @alknelt mount it with nfs if you want better performance, I found this essential for good small-file and non-parallel write workloads
01:47 tg2 copying a 1gb flat file from/to the same volume
01:47 tg2 was about 10-15x faster with it mounted as nfs
01:47 tg2 vs fuse
01:47 Alknelt tg2: good to know
01:47 JoeJulian not usually
01:47 tg2 try it
01:47 JoeJulian I have
01:47 JoeJulian been doing it for 3 years.
01:48 tg2 I found a huge difference in testing
01:48 tg2 I recommend trying both to see which works best
01:48 JoeJulian The fuse client offers better throughput. The nfs client offers stat caching.
01:48 tg2 ext4 underlying system on centos
01:48 tg2 was faster for using nfs mount in every single use case
01:48 JoeJulian @ext4
01:48 Alknelt ext4 already broke.. so switched to xfs
01:48 glusterbot JoeJulian: Read about the ext4 problem at http://goo.gl/PEBQU
01:48 tg2 yeah
01:49 tg2 you have ot use an older version of the kernel to get ext4 to work
01:49 tg2 not game changing, I am pushing the devs to work through the 32 bit patch so we can use the newest and greatest
01:49 tg2 but yes @ Joe
01:49 tg2 single-file copy operation with fuse client, on a single node with 45 drives (4tb ultrastars) was about 80-120MB/s
01:49 tg2 with nfs it was about 1.2GB/s
01:50 tg2 that is with 4 bricks (r5 arrays of 12 drives each + 1 hot spare)
01:50 Alknelt Currently using Ibrix for home Directories and data storage. Was thinking of Fuse for the Data, and NFS the home DIRs for throughput of small files… all on the same volume.
01:50 JoeJulian Wait... single client local access? That's your use case?
01:50 tg2 xfs might be different
01:50 tg2 also on a distributed remote client over 10gbe
01:50 tg2 we saw the same
01:50 tg2 for concurrent parallel writes
01:50 Alknelt Use is HPC
01:51 tg2 the overall throughput was about the same with many threads writing
01:51 tg2 but single threaded read/write was faster with nfs mounts, and small file performance was better
01:51 JoeJulian tg2: Something's odd there, but hey, if it's working for you, don't fix it. :)
01:51 tg2 using native infiniband or 10gbbe
01:51 yinyin joined #gluster
01:51 tg2 just stating a use case!  not saying it is set in stone, but definitely try it out to see which is better
01:52 Alknelt I was planning on looking at NFS. But first I wanted to prove gluster could even hold its own. I'm having problems at that level still.
01:52 tg2 what are you using it for, vm's?
01:52 tg2 db, etc?
01:52 Alknelt High Performance COmputing
01:53 tg2 waht is your read/write load
01:53 Alknelt Digital Signal Processing
01:53 tg2 map/reduce stuff?
01:53 tg2 are you doing many small concurrent reads/writes
01:53 Alknelt Usually small writes larger reads
01:53 tg2 dumping the data to it, then parsing it in chunks (what is the read chunk size)
01:53 Alknelt 4Mb
01:53 tg2 writing the results back to the volume? if so what is the write load (size, frequency)
01:54 tg2 If you're just testing it out still
01:54 tg2 try an older kernel prior to ext4 bug introduction
01:54 Alknelt Engineers usually create small meta files before submission to the cluster
01:54 tg2 so distributed setup works properly
01:54 JoeJulian Answering your earlier question: distribute-only provides no redundancy. If that fits within your risk analysis then yes.
01:54 tg2 and try it with nfs mount
01:55 Alknelt Distribute only is sufficient if the filesystem itself will not corrupt files
01:55 flrichar joined #gluster
01:55 Alknelt The Bricks are mounted Luns from HP EVA 8000s in a Vraid5 configuration
01:55 JoeJulian should be good then.
01:56 Alknelt Each individual disk will never loose data… essentially 8 disk raid6 behind each Brick
01:57 tg2 just be careful for actual disk rot
01:57 tg2 zfs checks for this and fixes it at the fs level
01:57 Alknelt and approximately 144 LUNs
01:57 tg2 but if having a bit switched somewhere is not the end of thew orld
01:57 Alknelt Raid 6 accounts for bit rot
01:57 tg2 r6 checks parity on read?
01:58 Alknelt 2x parity
01:58 tg2 i thought i depended on the implementation (controllers)
01:58 Alknelt true, raid6 is still not standardized
01:58 Alknelt hence HPs Vraid5 with double parity ~ raid6
01:59 tg2 I'd read up on it, some providers do offer it but sometimes it has to be explicitly set and not by default
01:59 Alknelt I'll switch this up again to a standard distributed Gluster and try to break it again...
02:00 Alknelt I've been running EVAs for 6 years with great sucess
02:00 tg2 yeah they are up there with netapp
02:00 Alknelt Tell you what, I'd still like to get my hands on 3PAR
02:01 Alknelt Thanx JoeJulian, Thanx tg2
02:02 purpleidea joined #gluster
02:02 purpleidea joined #gluster
02:04 tg2 Does anybody know if there are some large-scale zfs-underlying glusterfs deployments in the wild?
02:14 JoeJulian I'm not sure. There's been a lot of passers-by in this channel that have at least looked at doing that.
02:21 tg2 http://eugenyho.blogspot.ca/​2012/04/new-big-picture.html
02:21 glusterbot <http://goo.gl/d8sUS> (at eugenyho.blogspot.ca)
02:21 tg2 that but with no R5
02:21 tg2 just zfs raid
02:21 tg2 would be a decent starting point
02:22 tg2 ho hardare r5 *
02:22 * JoeJulian shudders at the mention of drbd
02:22 tg2 lol
02:22 JoeJulian drbd is how I found gluster in the first place.
02:23 tg2 HA block level, WHAT COULD GO WRONG
02:23 JoeJulian !!!
02:23 tg2 putting zfs on hardware R5 sort of counters any benefits you'd have from it
02:23 tg2 you could to this with just zfs + gluster
02:24 tg2 without all the extra crap
02:24 JoeJulian And if you could use the fuse client, no heartbeat or virtual ip
02:24 tg2 yeah i wish the fuse performance was optimized a bit
02:24 tg2 had some weird issues with .nfsxxxx files being created
02:25 JoeJulian It's improving.
02:25 tg2 when mounting via nfs
02:25 tg2 as our application grows its becoming more parallel anyway in terms of file transfers and patterns
02:25 tg2 the only killer was single-threaded copy speed
02:25 JoeJulian fuse has been improving with patches, at least in part from avati.
02:26 tg2 any 3.4alpha benchmarks?
02:27 JoeJulian iirc, isn't proxmox basically RHEL?
02:27 JoeJulian If there are, I either haven't seen them or have actively blocked them from my memory. ;)
02:28 tg2 QEMU optimizations look dcent
02:29 JoeJulian Yes, and that avoids fuse entirely.
02:29 tg2 lol
02:29 tg2 sort of side stepping the problem :D
02:29 JoeJulian It's a smart way to go.
02:29 tg2 for that use case yeah
02:30 JoeJulian Damn. I can't remember which database johnmark mention was going to do that as well. A direct library interface that bypassed fuse.
02:30 tg2 shit that would be nice
02:30 tg2 i just use mongodb anyway, it works well
02:30 JoeJulian I /think/ it was postgres.
02:31 tg2 yeah postgres hooks to bypass fuse in this case would be a pretty big step up in usability, I shudder to think about hosting a heavy transactional db on glusterfs over fuse
02:32 tg2 you could just run it inside a vm that takes advantage of libgfapi as a workaround
02:33 JoeJulian With mysql, if you use innodb, don't use file_per_table and name your innodb shards carefully so they're balanced across the distribute set, it's not too bad. (too bad being someone complains). It works for my workload.
02:33 tg2 i have a distributed/replicated volume set up accross ssd's on each node, onto which the vm's are put
02:33 tg2 it works well
02:34 tg2 tbh I haven't built anything with a relational db in a while
02:34 JoeJulian My work data for my vms are all on glusterfs mounts.
02:34 tg2 i've been using key/val for the last year
02:34 tg2 but I do have a painful partitioned mysql cluster to maintain
02:34 JoeJulian I have a lot of legacy stuff I have to support.
02:34 tg2 replication breaking on a weekly basis
02:34 tg2 etc
02:34 tg2 40,000 qps
02:35 tg2 my esx hosts are mounted to the same gluster share (the ssd distributed/replicated one) as they need it for HA/vmotion
02:35 tg2 haven't had a problem yet
02:35 JoeJulian Since glusterfs is posix complient, theoretically I think you could run multiple servers from the same files.
02:36 JoeJulian I do the same with libvirt/kvm.
02:37 tg2 one of my devs wrote a hillarious workaround into a file reading system, he figured out that the local storage on each node could be accessed directly, not via the common mount, and he wrote his application to failover to the local storage if it had an issue reading the glusterfs volume mount
02:37 tg2 lol
02:37 JoeJulian I was hoping to finish my openstack deployment this week but I don't think it's going to happen.
02:37 tg2 how do you like openstack?
02:37 akshay left #gluster
02:37 JoeJulian That's why I make /data mode 0700. Keeps fingers off the bricks.
02:38 JoeJulian OpenStack's going to make my puppet configurations a lot simpler. I only have to say, "make sure this machine is running" and openstack will handle the rest.
02:39 tg2 hm
02:39 JoeJulian It's a nice abstraction layer.
02:39 tg2 kvm is quemu too, yeah i guess that could take advantage of this version too
02:39 JoeJulian yep
02:39 tg2 i love vmware
02:39 tg2 but i hate the license fees
02:40 tg2 its gotten to the point where I only use it for development/prototyping to keep everyting on 1 server
02:40 tg2 then scale out into bare metal
02:40 JoeJulian vmware's on the board for openstack, so it won't be long before you see that supported (if it's not already)
02:40 tg2 tooe xpensive
02:40 tg2 plus 48 core servers = vmware can't even assign that many vcores to a vm
02:40 JoeJulian :/
02:41 tg2 I think you an run esx with openstack now
02:41 tg2 since 4.1/1
02:41 JoeJulian I last looked at vmware about six or seven years ago and settled on xen. Then when RH announced the kvm acquisition (and we run centos) I figured I'd better at least try it.
02:42 tg2 bare metal hypervisor was the attraction to vmware
02:42 tg2 illumos is doing some crazy shit with zones tho
02:42 JoeJulian my services ran faster under kvm than they did in xen, so I was happy with the change.
02:43 tg2 when did RH acquire kvm?
02:43 JoeJulian September 2008
02:44 tg2 160 virtual CPUs per virtual machine
02:44 tg2 hmm
02:44 tg2 they've been busy on it since I checked last
02:44 elyograg joined #gluster
02:45 tg2 wonder if thats just RHEV
02:47 tg2 running windows on RHEV
02:47 tg2 lol
02:47 JoeJulian :)
02:47 JoeJulian Someone came in here once asking if they could run the GlusterFS server on windows. I said "Sure, just run linux in a vm."
02:48 tg2 that is a legitimate solution for me now
02:48 tg2 lol
02:48 tg2 somebody wants to do some development testing
02:48 tg2 sure let me set up a vm on your desktop...
02:49 JoeJulian I had a headhunter contact me this week asking for someone to support their existing system while rolling out a new puppetized linux infrastructure. Their existing system had windows servers...
02:50 tg2 hmmmm
02:50 tg2 one of my good friends does consolidation consulting for banks and government
02:50 tg2 you would be amazed
02:50 tg2 at the infrastructures that run our banking system and government :(
02:50 JoeJulian Thanks for contacting me, but unfortunately in a big-data/big-storage infrastructure I have no desire to drag along the dead weight of a windows based operating system. Since you're looking for someone to work with that existing system, I'm not interested.
02:51 tg2 I use windows for 2 things in my office
02:51 tg2 exchange server
02:51 tg2 and active directory
02:51 tg2 thats about it
02:52 JoeJulian I have a windows partition on this box here at my home office. It has one application installed. Firefall.
02:52 tg2 hah
02:52 JoeJulian I don't even like exchange server. It's bloated, slow, has crappy spam filtration...
02:52 tg2 its just for organizational ease with active directory
02:52 tg2 acl is simple
02:52 tg2 createa  new group
02:52 tg2 add users to the group
02:52 tg2 done
02:53 tg2 fire somebody -> remove them from the groups, done
02:53 tg2 scales horizontally decently
02:53 JoeJulian I do the same with ldap.
02:53 tg2 well, i used to work for HP
02:53 JoeJulian Since that's essentially all AD is anyway.
02:53 tg2 for internal IT infrastructure
02:53 tg2 at the time (about 10 years ago)
02:53 tg2 they had 180 exchange servers
02:53 tg2 and ~80 active directory servers
02:53 tg2 and those "servers" were clusters in some cases
02:54 tg2 at that scale
02:54 tg2 there is very little that works other than microsoft
02:54 tg2 lotus notes
02:54 tg2 is the other option
02:54 tg2 lol
02:55 tg2 you can tell that hp had some influence on the way active directory and exhange were built at microsoft
02:55 tg2 just because fo their sheer size
02:55 tg2 it was when they acquired compaq
02:55 tg2 well, before and leading up to
02:55 JoeJulian Of course. Between HP and Boeing, MS doesn't give a crap about most other users.
02:55 tg2 yeah
02:55 tg2 that being said
02:56 tg2 i recently tried microsoft thread management gateway
02:56 tg2 and laughed to myself while reformatting the windows 2008 server whos tcp/ip stack it completely hosed
02:56 JoeJulian Hehe
02:57 tg2 I'd wager that most conventional big companies > 10 years old and > 50 employees, are running active directory and exhcange at their core
02:57 JoeJulian I really need to learn to stop bashing MS in here.... I know it's the wrong forum for my opinions.
02:57 tg2 anyway, thanks for the advice I'll look into kvm more seriously
02:58 JoeJulian You're welcome.
02:58 JoeJulian Dang, it's nearly 7:00 and it's my day to make dinner.
02:58 JoeJulian ... I wonder why nobody's complained yet...
03:05 pipopopo joined #gluster
03:28 duffrecords joined #gluster
03:28 nueces joined #gluster
03:30 hagarth joined #gluster
03:31 duffrecords one of my four Gluster boxes has a high load average and I see a bunch of "unable to self-heal contents of gfid… possible split-brain" errors in the glustershd.log.  how do I determine which files are affected?
03:33 JoeJulian duffrecords: First stat the gfid file. It's located on the brick in the .glusterfs/##/##/ directory, where the ##/## are the first four digits of the gfid.
03:35 JoeJulian Check the number of links. If there's only one, then it's probably safe to delete the gfid file. If there's more, then take the inode and find -inum {inode number} on the brick to find which filename is associated with that inode.
03:38 duffrecords there are many of them.  is the inode number the number that's in the gfid tag?
03:39 twx joined #gluster
03:40 JoeJulian The gfid is a false inode number that's consistent across replicas...
03:40 JoeJulian http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
03:40 glusterbot <http://goo.gl/j981n> (at joejulian.name)
03:47 vpshastry joined #gluster
03:51 duffrecords I'm not really clear on how to get the inode number from something like '<gfid:a549fb60-07cf-45ac-82c9-68e2a0763374>'
03:51 satheesh joined #gluster
03:52 JoeJulian stat .glusterfs/a5/49/a549fb60-0​7cf-45ac-82c9-68e2a0763374
03:52 bala joined #gluster
03:53 bulde joined #gluster
03:54 mohankumar joined #gluster
03:54 pai joined #gluster
03:56 elyograg does anyone know the magic incantation required to install windows 8 onto a brand new system I just built with intel raid on the Z77 chipset?
04:01 mohankumar joined #gluster
04:11 lala joined #gluster
04:18 duffrecords @JoeJulian: I looked up one of the inodes and there are many files associated with it, all within .glusterfs/indices/xattrop/
04:18 duffrecords do
04:18 duffrecords I need to delete those
04:19 niv joined #gluster
04:20 juhaj joined #gluster
04:21 yinyin joined #gluster
04:21 JoeJulian Hmm, I don't know anything about that directory...
04:21 en0x joined #gluster
04:22 JoeJulian I would guess that those indicate xattr operations that are pending, and it sounds like those are failing. So my guess is yes. Perhaps back them up somewhere?
04:22 aravindavk joined #gluster
04:22 _br_ joined #gluster
04:23 duffrecords I think I'll try that tomorrow.  all the VMs that are running on the Gluster array are fine (after a brief outage this evening).  perhaps the load average will subside by then.  hopefully it doesn't get worse
04:24 JoeJulian I'll try to do some code reading and figure out what the indicies directory structure is all about.
04:25 _br_ joined #gluster
04:30 duffrecords I started seeing tons of "readv failed" warnings in the #3 Gluster box's nfs.log.  then as we were cloning a disk image it seemed to freeze and that's when I saw "structure needs cleaning" in the same log.  so I rebooted it into single user mode and repaired the XFS brick and brought it up to run level 3.  at that point the #1 Gluster box freaked out (there is a HA address between the two, managed by UCARP)
04:30 _br_ joined #gluster
04:31 duffrecords but I guess I'll wait it out and see what happens overnight
04:35 _br_ joined #gluster
04:37 duffrecords left #gluster
04:41 Humble joined #gluster
04:45 _br_ joined #gluster
04:45 shylesh joined #gluster
04:47 Ryan_Lane joined #gluster
04:48 _br_ joined #gluster
04:49 timothy joined #gluster
04:50 sripathi joined #gluster
04:52 vpshastry joined #gluster
04:54 sgowda joined #gluster
04:55 _br_ joined #gluster
04:59 _br_ joined #gluster
05:04 yinyin joined #gluster
05:04 _br_ joined #gluster
05:09 _br_ joined #gluster
05:12 _br_ joined #gluster
05:17 sgowda joined #gluster
05:19 _br_ joined #gluster
05:21 yinyin joined #gluster
05:26 _br_ joined #gluster
05:27 vshankar joined #gluster
05:27 aravindavk joined #gluster
05:28 _br_ joined #gluster
05:33 _br_ joined #gluster
05:34 glusterbot New news from newglusterbugs: [Bug 903396] Tracker for gluster-swift refactoring work (PDQ.2) <http://goo.gl/wiUbE>
05:40 _br_- joined #gluster
05:44 vshankar joined #gluster
05:45 _br_ joined #gluster
05:47 raghu joined #gluster
05:47 vpshastry joined #gluster
05:48 satheesh joined #gluster
05:49 alex_ joined #gluster
05:49 alex_ left #gluster
05:52 _br_ joined #gluster
05:55 _br_ joined #gluster
05:55 shireesh joined #gluster
06:00 ramkrsna joined #gluster
06:04 _br_ joined #gluster
06:06 _br_ joined #gluster
06:08 vpshastry joined #gluster
06:12 _br_ joined #gluster
06:14 aravindavk joined #gluster
06:15 _br_ joined #gluster
06:23 _br_ joined #gluster
06:26 rastar joined #gluster
06:27 hagarth joined #gluster
06:30 _br_ joined #gluster
06:34 medhu joined #gluster
06:34 _br_ joined #gluster
06:34 ngoswami joined #gluster
06:36 medhu hi. while trying to mount glusterfs via nfs i get a permission denied by server while mounting <volume>
06:36 medhu error
06:36 medhu sorry. access denied error
06:37 medhu semiosis: hi
06:38 _br_ joined #gluster
06:38 pai joined #gluster
06:42 _br_ joined #gluster
06:42 venkatesh_ joined #gluster
06:45 ThatGraemeGuy joined #gluster
06:47 _br_ joined #gluster
06:54 _br_ joined #gluster
06:54 phase5 joined #gluster
06:59 mooperd joined #gluster
07:01 _br_ joined #gluster
07:09 bcc joined #gluster
07:10 Nevan joined #gluster
07:12 _br_ joined #gluster
07:18 edong23 joined #gluster
07:19 jtux joined #gluster
07:19 _br_- joined #gluster
07:21 sripathi joined #gluster
07:24 edong23 joined #gluster
07:28 _br_ joined #gluster
07:30 _br_- joined #gluster
07:32 sgowda joined #gluster
07:34 bulde joined #gluster
07:37 _br_ joined #gluster
07:39 shireesh joined #gluster
07:41 _br_ joined #gluster
07:46 _br_ joined #gluster
07:50 _br_ joined #gluster
07:50 sripathi joined #gluster
07:53 yinyin joined #gluster
07:55 _br_ joined #gluster
07:57 ctria joined #gluster
08:00 joeto joined #gluster
08:00 dobber_ joined #gluster
08:01 jtux joined #gluster
08:03 _br_ joined #gluster
08:05 tjikkun_work joined #gluster
08:05 _br_- joined #gluster
08:06 johnmark greetz
08:07 johnmark :O
08:10 lala joined #gluster
08:11 _br_ joined #gluster
08:15 y4m4 joined #gluster
08:15 bala joined #gluster
08:19 _br_- joined #gluster
08:20 Staples84 joined #gluster
08:21 _br_ joined #gluster
08:23 venkatesh_ joined #gluster
08:29 tjstansell joined #gluster
08:30 _br_ joined #gluster
08:33 _br_ joined #gluster
08:36 gbrand_ joined #gluster
08:38 _br_ joined #gluster
08:39 bala joined #gluster
08:39 morse joined #gluster
08:46 _br_ joined #gluster
08:48 Staples84 joined #gluster
08:49 _br_ joined #gluster
08:57 _br_ joined #gluster
08:59 mooperd joined #gluster
09:00 _br_- joined #gluster
09:01 shireesh joined #gluster
09:02 mohankumar joined #gluster
09:02 andrei joined #gluster
09:02 bulde joined #gluster
09:03 _br_ joined #gluster
09:03 timothy joined #gluster
09:03 aravindavk joined #gluster
09:05 rastar1 joined #gluster
09:05 glusterbot New news from newglusterbugs: [Bug 918917] 3.4 Beta1 Tracker <http://goo.gl/xL9yF>
09:06 mooperd joined #gluster
09:09 ProT-0-TypE joined #gluster
09:11 rotbeard joined #gluster
09:12 _br_ joined #gluster
09:14 _br_ joined #gluster
09:15 vpshastry joined #gluster
09:20 _br_ joined #gluster
09:24 mooperd joined #gluster
09:25 _br_- joined #gluster
09:25 Nagilum joined #gluster
09:29 glusterbot New news from resolvedglusterbugs: [Bug 850006] qemu integration with libgfapi <http://goo.gl/ZDBGe>
09:30 _br_ joined #gluster
09:37 _br_ joined #gluster
09:38 shruti joined #gluster
09:38 overclk joined #gluster
09:40 _br_- joined #gluster
09:43 ekuric joined #gluster
09:43 rgustafs joined #gluster
09:45 manik joined #gluster
09:46 _br_ joined #gluster
09:59 jdarcy joined #gluster
09:59 jdarcy left #gluster
09:59 _br_ joined #gluster
10:02 jdarcy joined #gluster
10:05 glusterbot New news from newglusterbugs: [Bug 918944] Deadlock in lk calls of stripe subvolume <http://goo.gl/5KEtn>
10:05 _br_ joined #gluster
10:08 _br_ joined #gluster
10:18 _br_ joined #gluster
10:21 shireesh joined #gluster
10:26 _br_ joined #gluster
10:29 _br_ joined #gluster
10:33 shruti joined #gluster
10:37 _br_ joined #gluster
10:38 _br_- joined #gluster
10:43 sripathi joined #gluster
10:48 _br_ joined #gluster
11:01 cw joined #gluster
11:08 tryggvil joined #gluster
11:12 manik1 joined #gluster
11:14 tryggvil joined #gluster
11:15 sripathi joined #gluster
11:17 manik joined #gluster
11:21 bala joined #gluster
11:24 tryggvil joined #gluster
11:26 jdarcy joined #gluster
11:32 pai_ joined #gluster
11:34 aravindavk joined #gluster
11:42 lala_ joined #gluster
11:42 timothy joined #gluster
12:01 tryggvil joined #gluster
12:05 glusterbot New news from newglusterbugs: [Bug 919007] After some time, transfer is slow and all writes are 4kb. Re-opening fds brings back fast transfer <http://goo.gl/9rIAH>
12:06 jclift joined #gluster
12:16 edward1 joined #gluster
12:17 rastar joined #gluster
12:19 lala_ joined #gluster
12:22 al joined #gluster
12:25 hagarth joined #gluster
12:25 andrei joined #gluster
12:26 tryggvil joined #gluster
12:32 vshankar joined #gluster
12:37 dustint joined #gluster
12:38 lpabon joined #gluster
12:45 al joined #gluster
12:49 bala joined #gluster
12:52 vpshastry joined #gluster
12:57 aliguori joined #gluster
13:00 rotbeard joined #gluster
13:00 bala1 joined #gluster
13:03 shruti joined #gluster
13:15 timothy joined #gluster
13:16 bulde joined #gluster
13:19 vpshastry joined #gluster
13:20 shireesh joined #gluster
13:20 yinyin_ joined #gluster
13:25 hagarth joined #gluster
13:30 rotbeard left #gluster
13:34 manik joined #gluster
13:35 16WAAKL8L joined #gluster
13:41 manik joined #gluster
13:42 mooperd joined #gluster
13:46 bennyturns joined #gluster
13:46 manik1 joined #gluster
14:00 edward1 joined #gluster
14:22 theron joined #gluster
14:23 semiosis @latest
14:23 glusterbot semiosis: The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
14:29 Staples84 joined #gluster
14:34 phase5 left #gluster
14:35 shireesh joined #gluster
14:38 en0x hi
14:38 glusterbot en0x: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:40 en0x i have a question about gluster fs and samba shares... http://gluster.org/community/docume​ntation/index.php/Gluster_3.2:_Expo​rting_Gluster_Volumes_Through_Samba . so if my server with samba will go down then the whole thing will be unavailabe on windows right? is there a way to prevent that?
14:40 glusterbot <http://goo.gl/k1Tv9> (at gluster.org)
14:41 rgustafs joined #gluster
14:47 Norky en0x, that's not really a gluster problem
14:48 Norky the same woudl be true if you had another distributed filesystem, or some shared storage, underneath Samba
14:48 Norky you can run more than one Samba server
14:49 Norky or set Samba up in a HA server cluster
14:50 Nagilum assuming three identical nodes, is it possible to create a gluster that has the storage of two nodes combined and able to withstand the loss of any one of the three nodes?
14:51 Staples84_ joined #gluster
14:52 Staples84 joined #gluster
14:53 mooperd joined #gluster
14:54 stopbit joined #gluster
14:59 jdarcy joined #gluster
15:02 Norky Nagilum, what do you mean?
15:02 Norky " the storage of two nodes combined "?
15:03 Nagilum Norky: if each node hat 1TB I want a glusterfs of 2TB
15:03 Nagilum s/hat/has/
15:03 glusterbot What Nagilum meant to say was: assuming three identical nodes, is it possible to create a gluster thas has the storage of two nodes combined and able to withstand the loss of any one of the three nodes?
15:03 Nagilum hehe, good bot .-)
15:04 Norky but you also want replica 2, so that every file is replicated in two places?
15:04 Nagilum yes
15:04 Norky errr, I *think* you can do that with a linked-list brick setup
15:05 Norky 2 bricks of 500GB on each server
15:05 Norky http://pthree.org/2013/01/25/g​lusterfs-linked-list-topology/
15:05 glusterbot <http://goo.gl/0HHCK> (at pthree.org)
15:06 Nagilum thanks, I think the important missing piece for my understanding was: " The number of replicas is the number of times you want a file to be replicated."
15:07 Norky yep
15:07 Nagilum that sentence should be part of the documentation, multiple times..
15:07 edward1 joined #gluster
15:07 Staples84 joined #gluster
15:07 Norky or, replace "file" with "given brick"
15:07 Nagilum right
15:07 jruggiero joined #gluster
15:07 tjstansell isn't the replica number the number of bricks in a replica set? which is the # of copies it will make.
15:08 Norky yes
15:08 tjstansell so you can't have 3 bricks, but 2 copies floating between the three.  which is why you need 6 bricks, each set being paired up ...
15:08 Norky that's what I said
15:08 aliguori joined #gluster
15:09 Norky <Norky> 2 bricks of 500GB on each server
15:09 tjstansell yes, i know.
15:09 Nagilum Norky and I speak the same language, tjstansell doesn't ;)
15:11 tjstansell i was just trying to word things differently, i guess.  there's a difference between X copies of data in a group of bricks and X copies in a replica set, with Y number of sets.
15:12 Norky fair enough :)
15:12 tjstansell i'm fairly new to gluster so this concept wasn't immediately obvious to me at first...
15:12 nemish joined #gluster
15:13 bugs_ joined #gluster
15:13 NcA^ joined #gluster
15:14 Norky nor me
15:18 bennyturns joined #gluster
15:20 shylesh joined #gluster
15:21 _pol joined #gluster
15:26 mooperd joined #gluster
15:26 anon1234 joined #gluster
15:27 NcA^ Does geo-replication ALWAYS require an authenticated user with SSH access between clusters ?
15:31 hagarth joined #gluster
15:34 flakrat joined #gluster
15:40 Norky NcA^, geo-replication is currently based on a version of rsync. How like standard rsync it is, and whether it will support 'direc't rsync:// interface I dont' know
15:41 akshay joined #gluster
15:41 phase51 joined #gluster
15:43 anon1234 joined #gluster
15:44 NcA^ cheers Norky , that was my understanding. Currently have 2x geographically separate environments that I'd like to tie together with geo-rep. I have a simple setup of 3x 10Gb replicated bricks in each and want to enable 2-way sync between them
15:44 Norky I have never done geo-rep, however I don't beleive it does two-way
15:45 Norky only master -> slave
15:45 Norky proper two-way repolication is on the roadmap I beleive
15:47 chouchins joined #gluster
15:51 tryggvil joined #gluster
15:53 aliguori_ joined #gluster
15:54 H__ Norky: yes, for 3.5
15:56 lh joined #gluster
15:56 lh joined #gluster
15:58 NcA^ H__, any idea whether that will be based on the current methodology (rsync)? Or a native gluster implementation?
15:59 torbjorn1_ If I'm doing a synced write to a file on gluster, say with dd using oflag=sync, should the write-behind translator be able to aggregate the write requests together ? .. Will dd even issue a new request before the "current" one has been flushed ?
16:00 H__ NcA^: sorry, don't know.
16:00 torbjorn1_ My setup drops in xfer rate from 25MB/s to 3MB/s after some hours, and I'm trying to figure out if the higher xfer rate is the one that's "wrong"
16:02 hagarth joined #gluster
16:04 tryggvil_ joined #gluster
16:13 vpshastry joined #gluster
16:14 _pol joined #gluster
16:17 timothy joined #gluster
16:21 bulde joined #gluster
16:22 stmaher joined #gluster
16:22 stmaher hi guys,
16:22 stmaher Very quick question..
16:22 stmaher How do I export a brick as read only
16:22 stmaher I have tried several combinations that dont work..
16:22 stmaher the daemon wont start saying invalid config
16:22 stmaher any link would be appreciated.. as google isnt helping me :-
16:22 stmaher :-(
16:23 tryggvil joined #gluster
16:23 torbjorn1_ stmaher: Could it be possible to mount Gluster read-only, ie. handling it on the client ?
16:24 stmaher yeah I dont want to handle it on the client as I dont trust them :-)
16:24 stmaher I know you can add ro to the mount point in fstab.. however.. I would prefer to do this from the glusterfsd side
16:25 anon1234 2
16:26 tjstansell i only see nfs.volume-access as a volume option.  so if you are using nfs, you could do that.
16:27 stmaher yeah Im using the glusterfs
16:27 stmaher but will give nfs a look
16:27 stmaher Many thanks for those ideas guys!.. Anyone else have any other suggestions?
16:27 tjstansell mount your bricks read-only on the server... but that would likely cause all sorts of glusterfs issues
16:28 stmaher LOL
16:28 stmaher good idea in theory.......
16:28 stmaher LOL
16:31 torbjorn1_ stmaher: It could be that writing your own read-only xlator for Gluster is kind of easy, for varying levels of "kind of" .. still, quite a bit more complicated than editing a configuration file
16:33 stmaher torbjorn1_ I hear you ..
16:33 stmaher I wish i had the knowledge do add to this awesum daemon.. but I dont :-(
16:33 hagarth joined #gluster
16:33 stmaher for something thats well established I would have expected it to be an included feature
16:34 mjc_ joined #gluster
16:36 tryggvil_ joined #gluster
16:40 manik joined #gluster
16:41 ndevos stmaher: earlier versions had a filter xlator, I think that supported read-only exports, but I am not sure if its coming back or is being replaced
16:42 akshay joined #gluster
16:45 Staples84 joined #gluster
16:47 stmaher ndevos thank you
16:47 stmaher LMAO
16:47 stmaher mount -r -t glusterfs /etc/glusterfs/glusterfs.vol /var/www
16:47 stmaher still mounts as rw
16:48 stmaher also mount -o ro ... does the same
16:49 ndevos stmaher: are you on 3.4 already? See Bug 853895
16:49 glusterbot Bug http://goo.gl/xCkfr medium, medium, ---, csaba, ON_QA , CLI: read only glusterfs mount fails
16:50 phase51 left #gluster
16:50 stmaher DAM.. im on 3.0.5 debian
16:50 stmaher ndevos thank you for the heads up on that!
16:51 ndevos stmaher: oh, thats an *old* release! you really should think of updateing
16:51 ndevos @latest
16:51 glusterbot ndevos: The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
16:51 vpshastry joined #gluster
16:51 stmaher ndevos your right..
16:52 stmaher debian is abit behind the times alright.. just time pressure on projects..
16:52 stmaher argh!
16:52 stmaher Thank you guys for the support.. its a great community and software..
16:52 stmaher Thanks again1
16:53 semiosis stmaher: debian backports has 3.2.7, http://packages.debian.org/search?keywo​rds=glusterfs&amp;searchon=names&amp;se​ction=all&amp;suite=squeeze-backports
16:53 glusterbot <http://goo.gl/MhmTz> (at packages.debian.org)
16:53 semiosis stmaher: but you can find 3.3.1 packages at the link ndevos & glusterbot gave
16:53 wN joined #gluster
16:53 torbjorn1_ stmaher: the packages for 3.3.1 from http://download.gluster.org/pub/gl​uster/glusterfs/3.3/3.3.1/Debian/ seems to work very well
16:53 glusterbot <http://goo.gl/AwJsw> (at download.gluster.org)
16:53 semiosis +1
16:53 stmaher Thanks guys!
16:54 torbjorn1_ stmaher: I've only tested the Squeeze version, though
16:54 aliguori joined #gluster
16:55 torbjorn1_ stmaher: I'n on mostly Debian stable, running Xen managed by Ganeti .. I hit https://bugzilla.redhat.com/show_bug.cgi?id=919007, and it's blocking me pretty hard right now. If you hit something like that and manage to fix it, I would be very grateful if you documented a fix or work-around
16:55 glusterbot <http://goo.gl/0Fzk7> (at bugzilla.redhat.com)
16:55 glusterbot Bug 919007: unspecified, unspecified, ---, amarts, NEW , After some time, transfer is slow and all writes are 4kb. Re-opening fds brings back fast transfer
16:56 stmaher torbjorn1_ Thanks for the heads up on that too
16:56 stmaher torbjorn1_ are you stuck to using fuse?
16:56 mjc_ joined #gluster
16:56 stmaher torbjorn1_ what about a native NFS client?
16:56 stmaher does it do that same?
16:57 chlunde joined #gluster
16:57 torbjorn1_ stmaher: AFAIK, the failover story isn't as nice on NFS ? .. I can loose one storage server with no service interruption, is that the case on Gluster-NFS ?
16:58 stmaher torbjorn1_ Im new to gluster.. so unfortunately im not all that helpful.. just something I spotted
17:02 torbjorn1_ stmaher: I looked over the "Admin guide" PDF and some quick googling, and it seems you have to implement HA yourself if using NFS for client access, for example on http://supercolony.gluster.org/piperma​il/gluster-users/2012-June/033570.html
17:02 glusterbot <http://goo.gl/P8ndT> (at supercolony.gluster.org)
17:06 gbrand_ joined #gluster
17:07 dowillia joined #gluster
17:08 nueces joined #gluster
17:09 ekuric left #gluster
17:12 JoeJulian That's right. nfs does not have any protocol for failing over.
17:15 JoeJulian torbjorn1_: That's a strange bug... I wonder what's chunking the writes...
17:15 torbjorn1_ JoeJulian: Do you think the fast or slow case is the correct way of operating ?
17:17 torbjorn1_ I'm worried that the fast case is actually not synced, that it might cause data corruption if the timing is just right
17:18 jdarcy joined #gluster
17:19 Gilbs joined #gluster
17:19 torbjorn1_ If I toggle write-behind, I see pretty much the same xfer rate as the slow case
17:20 JoeJulian I know that the sync flags are honored all the way to the disk, so I would say the 32k chunks are correct.
17:20 torbjorn1_ looking at https://github.com/gluster/glusterfs​/blob/master/xlators/performance/wri​te-behind/src/write-behind.c#L1348, I'm thinking you might get into a state where write-behind is disabled (warning: wild guessing is going on)
17:20 glusterbot <http://goo.gl/0a58Z> (at github.com)
17:21 mynameisbruce joined #gluster
17:21 mynameisbruce_ joined #gluster
17:21 torbjorn1_ I have to admit to not even knowing how write-behind would work in the oflag=sync case
17:22 JoeJulian A debug log would certainly show if you are correct. And that function is fd specific.
17:23 jag3773 joined #gluster
17:23 torbjorn1_ JoeJulian: My behavior seems to be specific to a fd, I can have a slow one and a fast one at the same time
17:24 JoeJulian Oh, that's cool. That function ensures that if any fd has the file open with O_SYNC, that every fd has that file open that way.
17:25 foster torbjorn1_: sorry I haven't gone through your mails yet, been distracted recently...
17:26 foster torbjorn1_: FWIW, I did trace w/ loop and wb on and off and noticed a difference in the size of _some_ writes
17:26 foster it just wasn't a large enough number of writes to affect performance
17:27 vpshastry joined #gluster
17:27 torbjorn1_ foster: No reason for anything remotely like an apology here, I'm grateful to the Gluster community for providing such cool software even if it doesn't work 100% correctly for me right now
17:28 foster torbjorn1_: I think what needs to happen is somebody to take a look at wb and establish what the expected behavior is (it seems somewhat strange that sync writes would aggregate)
17:28 foster torbjorn1_: have you filed a bug?
17:28 torbjorn1_ JoeJulian: You mentioned "debug log", would that mean adjusting diagnostics.{brick,client}-log-level to DEBUG ?
17:28 JoeJulian yes
17:28 torbjorn1_ foster: I filed a bug on  https://bugzilla.redhat.com/show_bug.cgi?id=919007
17:28 glusterbot <http://goo.gl/9rIAH> (at bugzilla.redhat.com)
17:28 glusterbot Bug 919007: unspecified, unspecified, ---, amarts, NEW , After some time, transfer is slow and all writes are 4kb. Re-opening fds brings back fast transfer
17:29 foster cool, thanks
17:30 torbjorn1_ JoeJulian: thanks, I'll do that right now .. should obviously have done that when I started looking into this
17:32 jbrooks joined #gluster
17:34 torbjorn1_ stmaher: Regarding your read-only case, have you looked into the "feature.read-only" volume setting ?
17:35 torbjorn1_ I saw it now by coincidence, looking at something else in the docs
17:38 msmith_ joined #gluster
17:39 NcA^____ joined #gluster
17:39 gbrand__ joined #gluster
17:40 Mo____ joined #gluster
17:40 zoldar_ joined #gluster
17:40 foster_ joined #gluster
17:41 Zenginee1 joined #gluster
17:44 vshankar joined #gluster
17:45 msmith_ I seem to be having a hell of a time getting my gluster 3.3.1 working reliably.  I have 6 servers, 1 brick per server (10 drives raid 6 with xfs) in a replica 2 setup (3x2).  I'm using NFS mounts instead of FUSE.  Storage is used as storage for dovecot mail servers.  I'm constantly seeing the dovecot.quota file go I/O error.
17:46 msmith_ when I check the bricks, I see the dovecot.quota file on server 3 and 4 with the correct permissions and file size, but then I also see it on servers 1 and 2 with 0 bytes and ---------T for the file permissions
17:49 msmith_ I'm stumped and google isn't being too helpful as this doesn't appear to be identified as an actual split-brain condition.  gfid on server 1 and 2 are identical, gfid on server 3 and 4 are identical, but they are different between pair 1 and pair2.
17:51 semiosis msmith_: the --------T is a ,,(link file)
17:51 glusterbot semiosis: Error: No factoid matches that key.
17:51 semiosis heh
17:51 stopbit joined #gluster
17:52 semiosis it is a reference telling glusterfs the file is located on another brick.  this is usually the result of a rename
17:52 aliguori joined #gluster
17:52 semiosis you can check the ,,(extended attributes) of the linkfile which should refer to the other brick pair
17:52 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
17:52 stmaher torbjorn1_  didnt spot that..
17:52 stmaher torbjorn1_ will take a look thanks
17:53 kevein joined #gluster
17:54 semiosis msmith_: do your nfs clients use noac,sync options?  if you have many clients trying to lock or write to a file you'll probably want to disable the attribute caching & use synchronous writes
17:55 msmith_ semiosis: will look into the attributes.  not currently using those options, but will give them a try.
17:55 semiosis hope that helps
17:55 msmith_ i'm using dovecot proxy to ensure that a user is accessing their files from a single host though
17:56 semiosis not too familiar with dovecot... is the dovecot.quota file global or is there one per account?
17:56 msmith_ per user
17:56 semiosis hmm
17:57 mjc_ joined #gluster
17:57 semiosis and they all exhibit this problem?
17:58 semiosis brb
17:59 msmith_ given enough time yes.  However it seems to really become an issue when gluster locks up (apparently in the kernel), which has happened 3 times in 12 hours, whenever I try to migrate a mailbox in to the new system.
17:59 timothy joined #gluster
18:01 msmith_ I don't know quite how to describe the lockup, it doesn't just affect the gluster processes and the nfs mount, but even shell sessions running top.
18:01 _br_ joined #gluster
18:08 _br_ joined #gluster
18:14 JoeJulian msmith_: Are you mounting from localhost?
18:16 y4m4 joined #gluster
18:21 _br_ joined #gluster
18:23 semiosis that would explain it
18:23 duffrecords joined #gluster
18:26 jag3773 joined #gluster
18:27 Gilbs setting up geo-replication on ubuntu, ran into this bug:  https://bugzilla.redhat.com/show_bug.cgi?id=895656    Is there anything else I need to do besides make the symbolic link files on the master and slave?  I keep getting the following errors and my geo-rep connection stays faulty
18:27 Gilbs IOError: [Errno 13] Permission denied: '/var/log/glusterfs/geo-replication-sl​aves/370cc05a-f62d-48df-bdf8-920c5c30b​20c:file%3A%2F%2F%2Fmnt%2Fgluster.log'
18:27 Gilbs [ Popen: ssh> failed with IOError.
18:27 glusterbot <http://goo.gl/ZNs3J> (at bugzilla.redhat.com)
18:27 glusterbot Bug 895656: unspecified, unspecified, 3.4.0, csaba, ON_QA , geo-replication problem (debian) [resource:194:logerr] Popen: ssh> bash: /usr/local/libexec/glusterfs/gsyncd: No such file or directory
18:27 _br_ joined #gluster
18:29 duffrecords I had an outage last night in my 4-node distributed-replicated system which caused glusterfsd on the first node to do lots of processing (the load average was extremely high for several hours).  it's finished doing whatever it was doing now but I'm seeing the following error every 10 minutes: E [afr-self-heal-data.c:763:afr​_sh_data_fxattrop_fstat_done] 0-guests-replicate-0: Unable to self-heal contents of '<gfid:a549fb60-07cf-45ac-82c9-68e2a0763374>' (p
18:30 disarone joined #gluster
18:32 duffrecords in fact, now I'm seeing that error in both nodes of a replicated pair
18:34 msmith_ joejulian: yes, mounts are done at the localhost level
18:37 Ryan_Lane joined #gluster
18:38 msmith_ btw, I seem to be able to fairly reliably cause the lockup when doing 'doveadm import' to migrate mail from the old system
18:46 JoeJulian There's a known race with nfs mounts from localhost. Either use fuse or mount from the other server.
18:46 torbjorn1_ msmith_: you are experiencing a full system hang there ? .. Given that it's all on localhost, could it be that you are seeing memory contention and deadlocks ?
18:47 semiosis torbjorn1_: ^^^ yes that's probably whats happening
18:49 semiosis http://community.gluster.org/a/nfs-per​formance-with-fuse-client-redundancy/ <-- seemed like a good idea but turned out to be awful, as explained by avati
18:49 glusterbot <http://goo.gl/WGUrr> (at community.gluster.org)
18:50 msmith_ i'll give that a try, once I figure out how to clean up all the file problems that resulted
18:51 msmith_ all the dovecot.quota files I'm having problems with have lines like "remote operation failed: Permission denied. ... Key: trusted.glusterfs.dht.linkto
18:51 msmith_ failed to get the 'linkto' xattr Permission denied
18:51 msmith_ and   /0C/1000000012/dovecot.quota => -1 (Structure needs cleaning)
18:51 JoeJulian selinux?
18:52 msmith_ disabled
18:57 msmith_ question on the nfs/localhost issue.  would that be eliminated if I moved the dovecot service ( and resulting nfs mount) to a guest vm on the same box, and have the nfs mount connect to it's vm host?
18:59 torbjorn1_ msmith_: wild guess: yes. Especially if you cap the memory on the VM so that you have some elbow room on the VM host
18:59 semiosis msmith_: yes probably
19:08 msmith_ thoughts on the best/quickest way to clean up these i/o error / permission denied linkto issues, short of rebuilding the cluster?
19:10 semiosis msmith_: could try deleting the -----T linkfiles from the bricks.  that should be harmless to try, but idk for sure if it will help
19:16 duffrecords I've identified the files referenced by all the "unable to self-heal gfid" errors in my logs.  if I delete the file on one replica node will that allow it to self-heal?
19:17 cw joined #gluster
19:17 semiosis duffrecords: see ,,(self heal)
19:17 glusterbot duffrecords: I do not know about 'self heal', but I do know about these similar topics: 'targeted self heal'
19:17 semiosis oh
19:17 semiosis duffrecords: see ,,(split brain)
19:17 glusterbot duffrecords: I do not know about 'split brain', but I do know about these similar topics: 'split-brain'
19:17 semiosis duffrecords: see ,,(split-brain)
19:17 glusterbot duffrecords: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
19:17 semiosis #2
19:19 tjstansell JoeJulian: since you cc'd yourself on my bug, i'll pick on you. :)  out of curiosity, do you think I should expect anyone to be looking at this timestamp bug any time soon, especially given the summit that's going on? I just want to set my own expectations accordingly ...
19:20 duffrecords semiosis: thanks.  that looks like exactly what I need.
19:20 semiosis great :)
19:21 duffrecords the tough part is figuring out which files are the good ones.  I did an md5sum on one and it's not the same across the replica
19:22 semiosis indeed that is the tough part
19:22 semiosis consider using quorum to prevent split brain
19:22 duffrecords would a quorum require more than 2 servers in a replica set?
19:24 semiosis odd, cluster.quorum-type doesnt show up in 'gluster volume set help'
19:26 semiosis duffrecords: just tried & it seems to work fine with replica 2.  when i killed one of the bricks my client turned read-only like i expected
19:27 semiosis brick restored & client turned read-write again :)
19:27 semiosis http://gluster.org/community/documentation/index​.php/Gluster_3.2:_Setting_Volume_Options#cluster.quorum-type
19:27 glusterbot <http://goo.gl/dZ3EL> (at gluster.org)
19:27 duffrecords that's using the GlusterFS client, I presume?  we're mounting the volume over NFS
19:27 semiosis heh, ok, let me try :)
19:29 semiosis hmm
19:29 semiosis client turned read-only when a brick died
19:29 semiosis but seems to be having trouble turning read-write again :/
19:30 duffrecords well, read-only is good.  that can allow an admin to step in and fix things before an entire VM image gets corrupted
19:30 semiosis just took time, but worked ok in the end :)
19:30 Gilbs I'm setting up geo-replication on ubuntu 12.04, ran into this bug:  https://bugzilla.redhat.com/show_bug.cgi?id=895656    Is there anything else I need to do besides make the symbolic link files on the master and slave?  My poor geo-rep stays at faulty.
19:30 glusterbot <http://goo.gl/ZNs3J> (at bugzilla.redhat.com)
19:30 glusterbot Bug 895656: unspecified, unspecified, 3.4.0, csaba, ON_QA , geo-replication problem (debian) [resource:194:logerr] Popen: ssh> bash: /usr/local/libexec/glusterfs/gsyncd: No such file or directory
19:31 Gilbs glusterbot knows my pain!  :)
19:31 duffrecords thanks.  once I get this split brain sorted out I'll set up quorum
19:31 semiosis Gilbs: and glusterbot feels ,,(meh)
19:31 glusterbot Gilbs: I'm not happy about it either
19:31 semiosis Gilbs: sorry idk what to say about geo-rep
19:33 Gilbs Bummer, logs not giving me too much love.
19:37 chlunde Hmm, is gluster support Red Hat "Resilent Storage Add-On", or is that GFS only?  https://www.redhat.com/wap​ps/store/allProducts.html
19:37 glusterbot <http://goo.gl/Ky8J0> (at www.redhat.com)
19:37 semiosis @commercial
19:38 glusterbot semiosis: Commercial support of GlusterFS is done as Red Hat Storage, part of Red Hat Enterprise Linux Server: see https://www.redhat.com/wapps/store/catalog.html for pricing also see http://www.redhat.com/products/storage/ .
19:38 semiosis or if not, please let us know so we can update glusterbot's links :)
19:39 chlunde catalog.html: Can't find it.  products/storage/: "Contact sales" :)
19:40 en0x joined #gluster
19:46 ThatGraemeGuy joined #gluster
20:06 Ryan_Lane joined #gluster
20:11 duffrecords semiosis: I followed the instructions on that "fixing split brain" document on a single file, to test it out.  now instead of saying "unable to self-heal" it says:
20:11 duffrecords E [afr-self-heal-data.c:1311:afr_sh_data_open_cbk] 0-guests-replicate-0: open of <gfid:b1305bdc-dff6-4ee0-8eb3-a30e74c7594a> failed on child guests-client-1 (No such file or directory)
20:13 semiosis is that a problem?
20:13 duffrecords it didn't replicate the file to the server where I deleted it
20:14 manik joined #gluster
20:15 semiosis can you access the file from a client mount?
20:15 duffrecords yes
20:16 Gilbs left #gluster
20:21 duffrecords wait, never mind.  looks like it just took a bit longer to replicate.  md5sums on both servers match now
20:22 semiosis \o/
20:22 duffrecords however, the number of entries when I run "gluster volume heal VOLUME info split-brain" hasn't changed
20:26 semiosis idk about that & have to go afk for a while
20:26 duffrecords ok.  thanks for your help
20:26 semiosis yw
20:27 _pol joined #gluster
20:28 jdarcy joined #gluster
20:34 atrius_ joined #gluster
20:38 manik1 joined #gluster
20:39 elyograg duffrecords, semiosis: I haven't dealt with split brain alot in my testbed yet, but when I've seen it, I have also seen that the info list never shrinks even after the problem is fixed.
20:41 andrei joined #gluster
20:41 duffrecords so I should probably assume the problem is fixed once I stop seeing errors in the glustershd.log
20:43 jdarcy joined #gluster
20:44 Ryan_Lane joined #gluster
20:45 duffrecords and once that happens, do you know of a way to remove them from the info list?
20:46 tqrst is there a way to get the status of a fix-layout? "gluster volume rebalance myvol fix-layout status" doesn't work (prints out usage), contrarily to what the cli help implies.
20:47 elyograg tqrst: what happens if you just ask for the rebalance status without fix-layout? is that a no-go?
20:47 tqrst elyograg: shows all zeros
20:47 tqrst elyograg: (even though I can see activity in the rebalance log file)
20:48 elyograg tqrst: ok, i'm out of ideas now.
20:50 tqrst elyograg: thanks anyhow :)
20:50 tqrst guess I'll just track the rebalance logs
21:02 atrius joined #gluster
21:19 minor joined #gluster
21:19 jdarcy_ joined #gluster
21:23 JoeJulian Yay! I finally got my nvidia card replaced with a radeon card at the office and can use my desktop again! :)
21:23 JoeJulian That's why I haven't been as responsive lately...
21:25 H__ odd, i prefer it the other way around.
21:27 JoeJulian radeon = open-source friendly with working drivers. nvidia = thinks their drivers have monetary value and the open-source hacks cause my nv50 to lock up under the latest kernels.
21:31 minor well, congrats there =)
21:31 H__ oh i'm sure you have good reasons :)
21:35 minor I just created an new volume and I'm trying to test it... I keep getting this "Transport endpoint is not connected" error
21:43 akshay left #gluster
21:51 Ryan_Lane joined #gluster
21:53 _pol joined #gluster
21:54 _pol joined #gluster
22:02 mattr01 joined #gluster
22:04 mattr01 left #gluster
22:04 mattr joined #gluster
22:05 mattr left #gluster
22:08 tqrst the recommende inode size for xfs is 512. Is it the same for ext4?
22:08 tqrst s/recommende/recommended
22:15 JoeJulian tqrst: I haven't seen that recommendation for ,,(ext4) but it would seem to make sense to do that.
22:15 glusterbot tqrst: Read about the ext4 problem at http://goo.gl/PEBQU
22:16 tqrst JoeJulian: (yep, aware of the ext4 glitch, but I don't have the time to switch ~50 hard drives over)
22:16 tqrst JoeJulian: yeah, I was just curious if there was something specific to the way ext4 handles inodes that could make the recommended size different
22:18 mooperd joined #gluster
22:18 JoeJulian No, in fact from ext3 to ext4 the default size changed from 128 to 256 to stone nanosecond timestamps and inode version information. That change sped up the extended attribute operations 3 to 7 times.
22:19 JoeJulian s/stone/store/
22:19 glusterbot What JoeJulian meant to say was: No, in fact from ext3 to ext4 the default size changed from 128 to 256 to store nanosecond timestamps and inode version information. That change sped up the extended attribute operations 3 to 7 times.
22:20 Nagilum strange..I thought 512 would be the minimum
22:22 elyograg I created mine with 1024. Someone in the company may want extended ACLs, or more info may get stored by future filesystem improvements.
22:22 JoeJulian +1
22:23 elyograg I wouldn't be surprised to find that they'll learn about this whole xattr thing and say "we can store some metadata there ourselves!"
22:24 tqrst you can also stuff the file itself in there if it's small enough
22:24 JoeJulian I wonder if there would be any advantage to 4k inodes so they would be cylinder aligned...
22:25 tqrst (inline_data in ext4)
22:38 nemish joined #gluster
22:41 tryggvil joined #gluster
22:46 mjc_ joined #gluster
23:07 nueces joined #gluster
23:52 DWSR joined #gluster
23:52 DWSR joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary