Camelia, the Perl 6 bug

IRC log for #gluster, 2013-07-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 RicardoSSP joined #gluster
00:01 RicardoSSP joined #gluster
00:05 JoeJulian T0aD: No bad luck... I have a volume with 7002 quotas set.
00:05 T0aD JoeJulian, and no problem ?
00:07 JoeJulian no problem
00:07 T0aD hmpf.
00:07 T0aD try it with 10,000
00:07 JoeJulian Which version are you testing?
00:08 T0aD 3.4.0 baby
00:08 T0aD and now im on the git
00:17 JoeJulian Ah, I see. I was looking for the wrong failure.
00:19 T0aD ?
00:32 T0aD tail -f /log/usr-local-etc-glusterfs-glusterd.vol.log -n 0 | awk '$0 ~ /bleh/'
00:32 T0aD lol trying for 2 minutes with different awk combinations 'why it doesnt work ?' haha.
00:35 bdperkin joined #gluster
00:35 dblack joined #gluster
00:43 brosner joined #gluster
00:53 yinyin joined #gluster
00:54 JoeJulian pfft... found it. Your volfile is overflowing "static char oldvolfile[131072];"
00:54 T0aD haha
00:55 T0aD nice catch dude.
00:55 * JoeJulian will file a bug
00:55 glusterbot http://goo.gl/UUuCq
00:55 T0aD kewl
01:11 T0aD hmm too bad gluster quota list command just reports the quotas set in the file
01:14 JoeJulian T0aD: Add yourself to bug 986100
01:14 glusterbot Bug http://goo.gl/6LCvr unspecified, unspecified, ---, amarts, NEW , Extreme numbers of quotas can cause segfault
01:14 T0aD pff
01:14 T0aD thats not extreme
01:14 T0aD im planning on 200,000
01:15 JoeJulian Heh... not sure how efficient that quota check is going to be... :/
01:15 T0aD Could this be a security bug since you're overflowing buffers? yeah but a security bug exploited by admin.. so :)
01:15 T0aD yeah
01:15 T0aD well now its fine loading 20,000 quotas with a new way (limit-set to a configuration file with one quota per line)
01:16 T0aD but I really struggle to find where the quotas limit are set in memory
01:17 JoeJulian Not sure what you're asking.
01:17 T0aD the ctx->limit
01:17 T0aD well i dont know much of the internals of xlator, quite lost in it
01:19 JoeJulian So you were able to exceed your 6000 limit by adding them through the cli interface one at a time using an input file to the cli?
01:19 T0aD jesus my comment was saved
01:19 T0aD no
01:19 T0aD using the cli interface it takes 60 minutes to add 1600 quotas
01:19 T0aD <T0aD> well now its fine loading 20,000 quotas with a new way (limit-set to a configuration file with one quota per line)
01:19 T0aD i gfsquota.py show > ./quotas.gf
01:20 T0aD and quota_parse_limits() is opening the file, reading it and parsing each quotas
01:20 T0aD but the gluster volume quota users list doesnt work since its using limit-set to display the quotas...
01:22 JoeJulian So you're hacking the source then?
01:22 T0aD yup
01:22 JoeJulian Ah, ok. That makes more sense. :D
01:23 JoeJulian I'd best make some dinner and get back to my own project.
01:24 T0aD no !
01:24 T0aD you come back this instant to gluster
01:25 harish joined #gluster
01:38 yinyin joined #gluster
01:41 chirino joined #gluster
01:43 glusterbot New news from newglusterbugs: [Bug 986100] Extreme numbers of quotas can cause segfault <http://goo.gl/6LCvr>
01:50 puebele3 joined #gluster
01:50 sprachgenerator joined #gluster
02:12 harish joined #gluster
02:13 Bluefoxicy joined #gluster
02:13 Bluefoxicy quick question.
02:13 Bluefoxicy Glusterfs server running using local disks, on a VM server
02:14 Bluefoxicy the 'local disks' are underlying VMWare virtual disks across a 10GbE connection over iSCSI to a network SAN RAID array
02:14 Bluefoxicy would this be "your IO controller is a piece of shit and that cluster is going to be laughably slow storage"?
02:15 Bluefoxicy since my IO controller is multiple levels of virtual device emulation, network transport, and operating system monkey business
02:18 JoeJulian Only one way to find out, but you're just adding a layer of complexity if they're all on one server so it's certainly not going to perform any better.
02:19 Bluefoxicy well also if one OS fails the other is still up.
02:20 Bluefoxicy my current suggestion to my manager is to run our assets on a DASD with GFS2+CLVM mounting iSCSI over a dedicated 10GbE network
02:21 Bluefoxicy with directions in the middle including gluster backed by DASD iSCSI over 10GbE direct link (instead of vmware virtual disks); GlusterFS built on real hardware (with DASD); or GlusterFS with real hardware and high-quality 3ware RAID IO controllers in the actual box providing local disks
02:21 Bluefoxicy If we use Gluster it's probably best to use it to build a better SAN than what we have
02:22 Bluefoxicy which is probably a good idea because 1) server-level redundancy (we don't have it); and 2) we escape these expensive leasing maintenance contracts (it's insane, you buy a SAN and then you pay a yearly license fee per server accessing the SAN???)
02:23 Bluefoxicy all this "Build a SAN on top of a VM backed by a SAN" stuff is nuts :|
02:25 Bluefoxicy JoeJulian:  anyway we found out because we did it trying to back a 0.5Gb/s web server (yes) writing 180kb/s of logs each from two servers (yes) with incoming 250kb/s file uploads (yes) on GlusterFS on 2 VMs with 2CPU each (...) and 16GB RAM using "local disk" which is a VMWare Virtual disk on a data store backed by SAN iSCSI over 10GbE
02:25 Bluefoxicy that ... gluster was really slow.
02:26 Bluefoxicy except I'm beating that by about 10 times on two 32 core high-end storage servers with integrated RAID-6 controllers elsewhere
02:26 Bluefoxicy so I'm blaming the IO chain.
02:30 hagarth joined #gluster
02:49 JoeJulian seems pretty likely. the latency is usually the killer.
03:00 puebele1 joined #gluster
03:00 lalatenduM joined #gluster
03:01 Bluefoxicy nod
03:02 Bluefoxicy I think GlusterFS is the best option we have out in our hosted data center between two servers (an odd use case--we have two servers providing service, both running Gluster locally to keep files synchronized; in this case we're bounded to just these two, and building separate storage and mounting that onto service machines would be worse)
03:03 Bluefoxicy What we have in our corporate data center would replicate a lot of work for, near as I can tell, protection against file system corruption and the availability of a host of features we're not currently using
03:04 bharata joined #gluster
03:04 Bluefoxicy If I need extreme performance, I'll get 4xPCI-Express and throw in 3-ware cards, set up 4 RAID-6, have 2 servers like this with 8 bricks per volume, distributed across 4 bricks on each server and replicated to corresponding opposite bricks
03:04 Bluefoxicy boom, throughput of 4 high-speed IO buses and RAID arrays.
03:06 kshlm joined #gluster
03:06 Bluefoxicy As it stands, we have a SAN that can provide iSCSI over 10GbE, GFS2+CLVM is probably better in that environment, GlusterFS out in the other one.
03:22 bulde joined #gluster
03:40 raghug joined #gluster
03:46 semiosis hardest part of buliding deb packages tonight turns out to be generating enough entropy on the VM to build a gpg key :/
03:51 shylesh joined #gluster
04:00 elyograg semiosis: i like servers with hardware rng built in.
04:01 semiosis didn't see that checkbox in virtualbox machine config ;)
04:01 elyograg heh.
04:02 elyograg i don't deal much with virtual machines.  I once had my Solr installation on virtual machines, but it performs better on bare metal with multiple index cores per server.
04:04 hagarth joined #gluster
04:06 glusterbot New news from resolvedglusterbugs: [Bug 763046] portmapper functionality <http://goo.gl/oBaXT>
04:16 skyw joined #gluster
04:22 yinyin joined #gluster
04:23 sgowda joined #gluster
04:25 psharma joined #gluster
04:39 rastar joined #gluster
04:39 bulde joined #gluster
04:40 hateya joined #gluster
04:46 CheRi joined #gluster
04:49 rjoseph joined #gluster
04:52 semiosis huzzah! http://download.gluster.org/pub/glust​er/glusterfs/3.4/3.4.0/Debian/README
04:52 glusterbot <http://goo.gl/tfGS3> (at download.gluster.org)
04:52 semiosis a for real, working, apt repo
04:52 semiosis achievement unlocked
04:52 JoeJulian +1
04:53 semiosis now i just need to do that for 3.3.2, but not tonight
04:53 JoeJulian Holy cow.. I just realized how late you're up. :D
04:53 JoeJulian Even I'm thinking it's almost time for sleep.
04:54 semiosis yeaahh
04:54 JoeJulian This project I told you about is coming along nicely.
04:54 semiosis can't wait to see it in action
04:54 JoeJulian It would never work in the real world, but yeah....
04:54 semiosis awwww
04:54 JoeJulian Mostly because rebalance takes too long on large scale volumes.
04:54 semiosis try explaining that to people
04:55 JoeJulian And remove-brick isn't reliable
04:55 JoeJulian but other than that....
04:55 semiosis great idea in theory, until you realize you're turning your production infra (i.e. livelihood) over to skynet
04:55 semiosis then not so appealing
04:55 JoeJulian hehe
04:55 JoeJulian But at least the proof of concept will be ready when the software is.
04:56 semiosis maybe it will even inspire some people to get involved & make it work better
04:57 vpshastry joined #gluster
04:57 semiosis ok bed time
04:57 semiosis later
04:57 JoeJulian Goodnight
05:05 sgowda joined #gluster
05:08 CheRi joined #gluster
05:11 raghu joined #gluster
05:20 lalatenduM joined #gluster
05:23 edong23 joined #gluster
05:27 yinyin joined #gluster
05:35 edong23 joined #gluster
05:46 satheesh joined #gluster
05:49 ghftggfhhjj joined #gluster
05:55 satheesh joined #gluster
06:05 pkoro joined #gluster
06:14 satheesh joined #gluster
06:15 jtux joined #gluster
06:23 fidevo joined #gluster
06:28 Recruiter joined #gluster
06:30 dpaul joined #gluster
06:32 shireesh joined #gluster
06:32 shireesh joined #gluster
06:37 pkoro joined #gluster
06:51 guigui3 joined #gluster
06:57 ctria joined #gluster
07:04 ngoswami joined #gluster
07:08 jtux joined #gluster
07:23 _BuBU joined #gluster
07:25 rgustafs joined #gluster
07:25 andreask joined #gluster
07:31 ujjain joined #gluster
07:39 hagarth joined #gluster
07:43 ngoswami joined #gluster
07:51 dobber_ joined #gluster
07:56 tjikkun_work joined #gluster
07:58 ricky-ticky joined #gluster
08:05 ctria joined #gluster
08:09 piotrektt joined #gluster
08:09 piotrektt joined #gluster
08:15 glusterbot New news from newglusterbugs: [Bug 952029] Allow an auxiliary mount which lets users access files using only gfids <http://goo.gl/x5z1R>
08:22 mmalesa joined #gluster
08:23 tjikkun_work joined #gluster
08:25 odyssey4me joined #gluster
08:33 ctria joined #gluster
08:34 mmalesa joined #gluster
08:41 hagarth joined #gluster
08:45 glusterbot New news from newglusterbugs: [Bug 985390] Posix ACL not supported via fuse-client <http://goo.gl/Pp6sC>
08:46 bala joined #gluster
08:54 vimal joined #gluster
09:06 vshankar joined #gluster
09:08 shireesh joined #gluster
09:14 yinyin joined #gluster
09:41 Tangram joined #gluster
09:43 mooperd joined #gluster
09:54 mmalesa joined #gluster
09:59 raghug joined #gluster
10:01 kevein joined #gluster
10:03 mmalesa joined #gluster
10:05 raghug joined #gluster
10:28 hateya joined #gluster
10:45 ngoswami joined #gluster
10:56 bulde joined #gluster
11:03 lpabon joined #gluster
11:05 harish joined #gluster
11:12 ngoswami joined #gluster
11:15 hagarth joined #gluster
11:15 andreask joined #gluster
11:31 CheRi joined #gluster
11:48 satheesh joined #gluster
11:50 bulde joined #gluster
12:02 CheRi joined #gluster
12:12 spider_fingers joined #gluster
12:28 hateya joined #gluster
12:32 ctria joined #gluster
12:41 sac joined #gluster
12:52 edward1 joined #gluster
12:59 matiz joined #gluster
13:00 bennyturns joined #gluster
13:04 sprachgenerator joined #gluster
13:04 hagarth joined #gluster
13:05 ngoswami joined #gluster
13:11 spider_fingers joined #gluster
13:12 semiosis phox: http://download.gluster.org/pub/glust​er/glusterfs/3.3/3.3.2/Debian/README
13:12 glusterbot <http://goo.gl/Llb7L> (at download.gluster.org)
13:16 theron joined #gluster
13:22 elyograg left #gluster
13:27 semiosis @later tell phox http://download.gluster.org/pub/glust​er/glusterfs/3.3/3.3.2/Debian/README
13:27 glusterbot semiosis: The operation succeeded.
13:29 harish joined #gluster
13:39 bulde joined #gluster
13:43 failshell joined #gluster
13:55 jiku joined #gluster
13:59 Tangram joined #gluster
13:59 brosner joined #gluster
14:00 ctria joined #gluster
14:02 kaptk2 joined #gluster
14:10 vpshastry joined #gluster
14:11 vpshastry left #gluster
14:24 ctria joined #gluster
14:25 ricky-ticky joined #gluster
14:28 bdperkin joined #gluster
14:32 rcheleguini joined #gluster
14:38 manik1 joined #gluster
14:41 spider_fingers left #gluster
14:44 T0aD jeez still cant figure out where the limits are stored.
14:44 ccha about logrotate, messages write in etc-glusterfs-glusterd.vol.log.1 and not in etc-glusterfs-glusterd.vol.log
14:45 ccha should logrotate need to reload the daemon glusterfs ?
14:45 T0aD maybe fd is still open
14:45 T0aD and daemon was not closing correctly fds while shuting down
14:47 ccha there is no postrotate in /etc/logrotate.d/glusterfs-common
14:47 T0aD it seems its stored in trusted.limit.list
14:48 T0aD getfattr -m limit -d /home/users/home/04/00/00 pff no result.
14:49 T0aD aah i get it
14:50 T0aD its not stored, its hijacking it
14:51 semiosis T0aD: copytruncate
14:51 T0aD ?
14:51 semiosis logrotate's copytruncate option
14:51 T0aD ah ok it was ccha looking for it
14:52 semiosis oh oops
14:52 semiosis need moar coffee
14:52 T0aD im still diging the source
14:53 lpabon joined #gluster
14:53 T0aD root@web:/home/toad# ./attr_quota.py /var/gluster/users/home/04/00/00/
14:53 T0aD /var/gluster/users/home/04/00/00 trusted.limit.list 18 82283520,10485760
14:54 T0aD bingo.
14:55 duerF joined #gluster
14:56 TuxedoMan joined #gluster
14:56 txdoman joined #gluster
14:57 TuxedoMan joined #gluster
14:58 TuxedoMan joined #gluster
15:00 TuxedoMan joined #gluster
15:02 plarsen joined #gluster
15:04 ccha semiosis: sur but the logrotate file is from the deb package
15:04 semiosis which deb package?
15:04 ccha lucid
15:04 ccha 3.3.1
15:05 ccha there is not copytruncate option inside the logrotate file
15:05 brosner joined #gluster
15:07 recidive joined #gluster
15:08 brosner joined #gluster
15:15 phox ah-hah, Debian repo.
15:16 semiosis phox: let me know (in channel) how that works for you.  i tested installing from there and that was as far as I went with it
15:24 phox semiosis: I might wait and go 3.4.0
15:24 phox semiosis: we're going to move to it eventually anyways; this gets me fewer transitions
15:31 rcheleguini joined #gluster
15:35 dbruhn Grr, I have been battling lots of stupid errors that I can't seem to get on top of with one of my system.
15:35 dbruhn Input/output error (/mnt/ENTV04EP/root/data/1/549/113​4/13/1/1/9398/1916170.1369695646)
15:35 dbruhn We keep cleaning them up, and then come across more, and more.
15:35 dbruhn I had a RAID card go out on one of the servers in a replicant pair.
15:36 dbruhn and this has been clean up effort
15:41 T0aD i wish i could use gluster :D
15:42 klaxa joined #gluster
15:44 semiosis well same goes for the 3.4.0 debian packages on download.g.o
15:44 semiosis phox: ^
15:44 jebba joined #gluster
15:57 zaitcev joined #gluster
15:59 phox note that I read ".g.o" as ".gentoo.org"
15:59 phox :)
15:59 T0aD lets hang him
16:00 * T0aD gets a rope
16:05 sprachgenerator joined #gluster
16:08 zombiejebus joined #gluster
16:14 vpshastry joined #gluster
16:14 T0aD http://www.bpaste.net/show/jtBgUzKABmNrWbM5YKJg/ funny it wouldnt use a hash table
16:14 glusterbot <http://goo.gl/RCMcB> (at www.bpaste.net)
16:14 Guest63207 joined #gluster
16:16 T0aD thats probably an issue if 2 quotas are used like /home/toad and /home/toad/jack and the path is equal to /home/toad/jack/cool, what if /home/toad quota is defined first ?
16:16 vpshastry left #gluster
16:19 bala joined #gluster
16:22 Guest63207 hi, I have installed gluster on two VM's which have no ACL's between them. from node-1 I have successfully added node-2 as a peer.
16:22 Guest63207 however..
16:22 T0aD quota_lookup() on / toad
16:22 T0aD jesus.
16:22 Guest63207 a peer probe node-1 from node-2 fails with: errno 107
16:23 Guest63207 even though, when I peer probe node-1 from node-2, I run a tcpdump on node-1. I see node-1 and node-2 happily talking.
16:24 Guest63207 there isn't any useful debugging information in the cli.log other than the error 107 message.
16:24 T0aD developpers are no longer active on this channel ?
16:27 brosner joined #gluster
16:32 jyundt joined #gluster
16:36 semiosis terje: try restarting glusterd on both servers, then check 'gluster peer status' on both servers -- pastie.org the output if you want, that might help
16:41 terje semiosis: thanks.. I think it must be a network issue on my side.
16:41 terje debugging now.
16:41 semiosis terje: ok
16:44 kkeithley Since most of of the devs are in Bangalore, where it's 11:15 pm, I'm guessing the majority of them are not in the office or near a computer.
16:47 jag3773 joined #gluster
16:48 morse joined #gluster
16:50 jyundt joined #gluster
16:51 T0aD # grep seriously -r /log/ | wc -l
16:51 T0aD 14589631
16:51 T0aD lol. and i cut the process.
16:51 lpabon joined #gluster
16:52 duerF joined #gluster
16:54 bulde joined #gluster
16:55 hagarth T0aD: are you running quota with 3.4.0?
16:55 T0aD yup
16:55 T0aD well im not running till i dont patch
16:55 T0aD the whole quota process needs to be rethought, i saw the next pull but its not enough
16:56 hagarth T0aD: quota is being revamped, you can find some activity in forge.gluster.org.
16:56 T0aD hm link ?
16:56 T0aD so far i saw https://github.com/gluster/glusterfs/pull/16
16:56 glusterbot Title: Quota new design by vpshastry · Pull Request #16 · gluster/glusterfs · GitHub (at github.com)
16:56 T0aD which means i will have to port my patch later on :D
16:56 hagarth T0aD: https://forge.gluster.org/~vpshastry/glusterfs-​core/glusterfs-quota/commits/quota-improvements
16:57 glusterbot <http://goo.gl/sKeto> (at forge.gluster.org)
16:57 T0aD thx dude
16:57 hagarth rather quota-improvements branch in this repo: https://forge.gluster.org/~vpshast​ry/glusterfs-core/glusterfs-quota
16:57 glusterbot <http://goo.gl/PhfK8> (at forge.gluster.org)
16:57 T0aD yeah its the same person
16:57 T0aD so i guess the github points to the same code
16:58 jag3773 joined #gluster
16:58 T0aD but still
16:58 hagarth the github pull requests are dated
16:58 T0aD that wont do for a lot of quotas
16:58 T0aD actually im loading a complete external file to load quotas
16:58 hagarth what is the number of quotas you are trying to configure?
16:59 T0aD well working on 20,000
16:59 T0aD so i can see all the issues
16:59 T0aD but i will push it to 1M
16:59 hagarth what is your use case for 1M quotas?
16:59 T0aD none, Im just trying to have a stable system with 200,000 quotas
17:00 T0aD but with 1M i can see the problems ahead, i did the same with linux users and apache vhosts
17:00 T0aD i will need to understand whats purpose the lookup() hook
17:01 T0aD because its so bad its doing loops over all the quotas like 20,000 quotas * lookups
17:01 T0aD so 14M strcmp checks (and i cut the process) when I run gluster volume quota users list
17:02 guigui1 joined #gluster
17:03 hagarth it has not been designed to accommodate such scale, I would probably look at a much lower number to get a stable setup going.
17:03 T0aD i cant
17:03 T0aD im a massive free webhoster
17:03 T0aD so as a start there will be probably 12,000 quotas * 2 for migration
17:04 T0aD anyway it shouldnt be a problem, its just about getting some limit number
17:04 T0aD but yeah it definitely wasnt made for such use
17:04 hagarth do you have a single volume for all your hosting?
17:04 T0aD so far it will be the case
17:05 T0aD the problem is the fs will scale but not the quotas :D
17:05 T0aD wouldnt be a shame not to be to scale a FS because of the way its handling quotas ? :D
17:06 hagarth it might be ;) .. have you considered breaking up into multiple volumes and trying to scale quotas with more number of volumes?
17:06 T0aD once again, that would be ridiculous
17:06 T0aD actually glusterfs cant go higher than 7000 quotas without a heap overflow
17:07 T0aD so since i have like 200GB for 12,000 that would be already 2 volumes for that
17:07 T0aD 100 GB per volume.
17:08 semiosis why not limit usage at the application level?
17:08 T0aD the kernel was doing that
17:08 T0aD i dont control the application
17:08 semiosis ah
17:10 hagarth T0aD: one of the design goals of the quota revamp is to get to 64K quotas, so it would be worth trying more quotas on that repo.
17:10 T0aD really ?
17:11 T0aD damn didnt see the part of the code that was doing that
17:12 T0aD https://github.com/gluster/gl​usterfs/pull/16/files#L9R563 so far this loop still smells
17:12 glusterbot <http://goo.gl/UMHMX> (at github.com)
17:14 T0aD https://github.com/gluster/gl​usterfs/pull/16/files#L7R785
17:14 glusterbot <http://goo.gl/tZivU> (at github.com)
17:14 T0aD and the quota storage still use one single line
17:15 T0aD i doubt its achieving 64K
17:16 brosner joined #gluster
17:32 jclift_ joined #gluster
17:34 brosner joined #gluster
17:35 ThatGraemeGuy joined #gluster
17:35 T0aD haakon_, you re a dev ?
17:38 lmickh joined #gluster
17:46 Jasson joined #gluster
17:50 chirino joined #gluster
17:55 haidz joined #gluster
17:59 vpshastry joined #gluster
18:04 tg2 joined #gluster
18:11 vpshastry left #gluster
18:15 _pol joined #gluster
18:17 raghug joined #gluster
18:24 bulde joined #gluster
18:29 Recruiter joined #gluster
18:42 chirino joined #gluster
18:51 T0aD i meant hagarth
18:51 T0aD :D
18:53 Peanut I'm looking into glusterfs as a backend to host KVM virtual hosts on Linux (Ubuntu). Should I go with the glusterfs 3.2.7 that currently ships with Ubuntu, or are there major advantages (or risks) to using 3.4.0 ?
18:54 haidz Peanut, you might want to look at the new blockio support
18:54 T0aD its so complex to compile glusterfs im not sure what to answer !
18:55 Peanut T0aD: I just spent 3 days getting gcc-4.8.1 to compile on Solaris 10. Don't tempt me ;-)
18:55 T0aD haha
18:55 haidz http://www.gluster.org/community/do​cumentation/index.php/Planning34/BD
18:55 glusterbot <http://goo.gl/LglqL> (at www.gluster.org)
18:55 T0aD well glusterfs aint lustre, fairly easy to compile
18:55 T0aD and deploy
19:01 chirino joined #gluster
19:03 Peanut Does gluster run in userspace? I notice that the bricks are newfs'ed with xfs, and mounted in the OS?
19:05 T0aD its using fuse
19:05 T0aD so yeah
19:05 T0aD and its just exposes a real fs
19:05 jyundt Peanut: FWIW, I've been unable to get glusterfs (3.4.0) to work with KVM using XFS
19:05 T0aD like xfs, reiser, extfs, whateverfs
19:05 T0aD jyundt, working fine here
19:05 jyundt https://bugzilla.redhat.com/show_bug.cgi?id=958781
19:05 glusterbot <http://goo.gl/4Goa9> (at bugzilla.redhat.com)
19:05 glusterbot Bug 958781: medium, unspecified, ---, kparthas, NEW , KVM guest I/O errors with xfs backed gluster volumes
19:05 T0aD well for my 1GB virtual disk test :D
19:05 T0aD jeez and they advise to use xfs
19:06 bradfirj I like XFS for Gluster vols
19:06 jyundt toad: I'm not ruling out user error, but I haven't been able to explain the behavior I'm observing
19:06 Peanut T0aD: I was wondering about xfs too - then again, has ext4 finally gotten to the point where it can create FS > a few TB?
19:06 bradfirj I think it can, but that doesn't nesscesarily mean you should
19:07 T0aD well i dont have that much storage, but check out the wikipedia page about ext4fs limits, it should be huge
19:07 Peanut Oh, the FS can handle it, but for ages, there were no tools that could actually newfs such a FS as they still had all kinds of 32bit limits.
19:08 bradfirj Yeah some of the older tools might get confused and murder the FS just incase
19:09 T0aD Peanut, ah funny, didnt know that
19:10 Peanut We're into the fairly large storage business here (radio astronomy), so I've ran into it repeatedly over the last years.
19:11 Peanut So in glusterfs, using distribute, the translation is done at the file level, no smaller? Can't have a file bigger than the individual disks in your pool?
19:12 JoeJulian Peanut: That's the purpose for ,,(stripe).
19:12 glusterbot Peanut: Please see http://goo.gl/5ohqd about stripe volumes.
19:12 T0aD Peanut, nice
19:12 T0aD hi JoeJulian !
19:13 Peanut Thanks, JoeJulian
19:17 Peanut Anyway, weekend! Thanks so far.
19:17 glusterbot New news from newglusterbugs: [Bug 986429] Backupvolfile server option should work internal to GlusterFS framework <http://goo.gl/xSA6n>
19:32 mmalesa joined #gluster
19:37 guigui1 left #gluster
19:47 semiosis Peanut: yes gluster is all in userspace.  xfs is recommended mainly because it's what gluster is most commonly used & tested with, so a safe bet.  ext4 should work too though there was a bug affecting glusterfs on recent linux kernels just fixed in gluster 3.4
19:47 semiosis but with the ,,(latest) it should work
19:47 glusterbot The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
19:48 semiosis people have used zfs & other filesystems too, though much less commonly
19:59 andreask joined #gluster
20:11 odyssey4me is there a way to remove a brick from a volume in such a way to migrate the data automatically to the other bricks in the cluster?
20:11 tg2 I think gluster overtop of zfs would be amazing
20:11 tg2 personally
20:30 brosner joined #gluster
20:31 recidive joined #gluster
20:43 JoeJulian odyssey4me: That's what the "remove-brick...start" does since 3.3.
20:45 odyssey4me JoeJulian - aha, thank you
20:45 odyssey4me It seems that there's no way to get a general status of the cluster?
20:45 odyssey4me (sorry, noob here)
20:45 JoeJulian "gluster volume status" doesn't do it for you?
20:47 odyssey4me it doesn't show much in the way of performance, health or anything like that - that said, I guess I have a healthy cluster
20:48 foster joined #gluster
20:50 odyssey4me I am finding the enforced numbers when adding bricks a little annoying. Sometimes I'd like to add one brick before taking one away, and it won't let me. :p
20:50 tg2 ?
20:50 tg2 do replace-brick
20:51 tg2 you add a new brick, remove the old with remove-brick start
20:51 hchiramm_ joined #gluster
20:51 tg2 then commit it
20:51 odyssey4me well, realistically, for my testing - I want to add and remove bricks at will...
20:51 tg2 what is preventing you from doing that
20:52 odyssey4me let me give a bigger picture... I am generally looking at the concept of having each vm host being part of the storage cluster in order to provide a shared storage environment for kvm guests to facilitate simple guest moves, etc
20:53 odyssey4me in order to cater for possible failures I'd like at least one copy of the data to be somewhere else
20:54 odyssey4me right now I have 3 hosts to play with - I'm trying to use the RAID on the hosts as wisely as possible... but I can only add two at a time into the cluster with a 2 replica configuration
20:55 tg2 you can do that
20:55 tg2 but yeah
20:55 tg2 in distributed you have to add yoru bricks in pairs
20:55 tg2 in teh same volume that is
20:55 tg2 you can also have a non-replicated volume
20:55 JoeJulian You have to add bricks in replica quantities. Otherwise what's it supposed to replicate against?
20:55 odyssey4me ok, so I'll be trying a non-replicated volume too
20:56 T0aD pff i have no energy to continue patching glusterfs
20:56 T0aD its booze time.
20:56 tg2 do replica bricks have to be the same size?
20:56 tg2 or just the same count
20:56 odyssey4me JoeJulian - if I ask for two replicas, but have three servers in the cluster, I expect that it can make its own decisions
20:56 JoeJulian They don't "have" to, but they won't exceed the smallest.
20:56 JoeJulian @brick order
20:56 haidz beer oclock
20:56 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
20:56 tg2 for geo replication, can both sites have different sizes?
20:57 tg2 use case: one site has a distributed replica set, 4 nodes of 50tb = 100tb avilable in site 1
20:58 tg2 site 2 is just 2 nodes of 50tb but distribute donly
20:58 odyssey4me tg2 - good question... so if I understand it correctly, if each brick isn't the ame size then the volume will use a max size of brick data equal to the smallest brick JoeJulian?
20:58 tg2 ie: site2 is just an edge and doesn't need redundancy
21:00 JoeJulian georeplication will sync the volume from the master to the slave. The slave will need to be large enough to hold the data on the volume. The slave /can/ be another gluster volume.
21:00 JoeJulian odyssey4me: In a (not-geo) replicated volume, each replica set will only hold the files that will fit on the smallest of the set.
21:01 odyssey4me ok, so for geo the slave setup is essentially a volume on its own... brick size doesnt matter as long as the volume of data from the master can fit into the volume on the slave?
21:01 JoeJulian right
21:02 odyssey4me JoeJulian - ok, that makes sense... it's for the purpose of keeping redundancy
21:03 odyssey4me if I'm migrating from smaller disks to larger ones, I just need to ensure that I replace-brick on each small disk to ensure that all bricks in the volume are the same size?
21:04 odyssey4me can two volumes use the same brick?
21:04 JoeJulian no, but a brick is not a disk. I have 15 bricks per disk in my configuration.
21:05 JoeJulian A brick is simply a path that is used by glusterfs for a specific volume's storage.
21:05 jyundt JoeJulian: can I bend your ear for a second about my crazy xfs problem?
21:06 odyssey4me aha, so smart usage would be to partition your disks into the same sizes regardless of your disk size...
21:06 JoeJulian Sure... I've looked at your traces... It seems to be erroring on that readp suggesting that it's trying to read more bytes than exist.
21:06 JoeJulian odyssey4me: I do that with lvm.
21:06 bronaugh left #gluster
21:07 jyundt I've also looked at the traces and the code as well
21:07 jyundt and it looks super basic
21:07 odyssey4me JoeJulian - so configure the disks in a hardware RAID set, then VM them into same-sized volumes and assign each volume as a brick
21:08 jyundt I mean, I can't figure out why I'm getting an EINVAL
21:08 jyundt what can I do besides an strace to gather more data?
21:09 tg2 @joe is there a way to enforce file replication on write?
21:09 tg2 ie: i want to absolutely make sure a file has been replicated offsite before the write op returns
21:09 tg2 sort of like a write concern in mongodb
21:11 mooperd joined #gluster
21:20 odyssey4me is there a way to smartly use sdd's for a gluster configuration - perhaps for read or write caching in some way?
21:21 tg2 you can do it on the underlying ifle system
21:21 tg2 ie: ext4 ssd journaling
21:21 tg2 http://www.raid6.com.au/post​s/fs_ext4_external_journal/
21:21 glusterbot <http://goo.gl/Ekwcg> (at www.raid6.com.au)
21:21 tg2 read the disclosures
21:21 tg2 it would be really nice to have something like this built into gluster though
21:21 tg2 a cache brick
21:26 odyssey4me that's kinda what I was thinking
21:28 odyssey4me I suppose using something like flashcache could be a workaround.
21:36 odyssey4me what's the best way of mounting kvm disk images from a gluster volume?
21:36 tg2 flashcache is iffy
21:39 tg2 how big is your data set
21:39 tg2 on disk
21:39 tg2 and how big is your ssd
21:40 tg2 if you ahve a big data store
21:40 tg2 and a small ssd
21:40 tg2 you will ruin it's write endurance by reading through it
21:40 tg2 and can actually introduce bottlnecks throught he ssd
21:40 tg2 the idea is to use it for write journaling
21:40 tg2 so the data can then be written to the spinning disks in sequential chunks
21:41 tg2 reading through the ssd is probably a bad idea unless you have a bit of content that is frequently accessed
21:41 tg2 and a lot that is rarely accessed
21:41 odyssey4me I would imagine that you'd get best performance using something like that by implementing it on a different controller, and the use case would be when you have big, but slow disks
21:42 odyssey4me yes, journaling seems to be the most common use case
21:42 tg2 I have 2gb of write cahce on the controller, with battery backup unit (bbu)
21:42 tg2 this helps
21:42 tg2 the controller also supports "cachecade"
21:42 tg2 which is ssd caching
21:42 tg2 natively
21:42 tg2 transparent to the OS/fs
21:42 odyssey4me yes, I see that with mine too... I'll give that a whirl some time this w/end
21:44 odyssey4me ffs, I can see it's going to be a long one
21:45 odyssey4me while my gluster volume mounts, the raw disk image on it prevents the vm starting with a 'could not open disk image' error
21:49 odyssey4me oh well, that's something for tomorrow - g'night all
22:07 recidive joined #gluster
22:09 JonnyNomad joined #gluster
22:11 RicardoSSP joined #gluster
22:17 recidive joined #gluster
22:26 jebba joined #gluster
22:26 brosner joined #gluster
22:39 mooperd joined #gluster
23:17 Tangram_ joined #gluster
23:18 andreask1 joined #gluster
23:18 andreask joined #gluster
23:27 zombiejebus joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary