Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-01-27

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 sroy_ joined #gluster
00:29 jag3773 joined #gluster
00:32 quillo joined #gluster
00:45 geewiz joined #gluster
01:03 bennyturns joined #gluster
01:33 harish joined #gluster
01:46 geewiz joined #gluster
01:54 mojorison joined #gluster
02:16 asku joined #gluster
02:18 AtAM1 joined #gluster
02:25 AtAM1 hi everyone :-) I am considering gluster for file replication between 2 web servers and wanted to know whether it is possible to have the replica volumes setup to replicate without using clients ie. Server 1 reads/writes to a local brick which is then replicated with a brick on Server 2. Server 2 should do the same - key here is having both servers read/write locally whilst having the bricks/mount replicated in the background
02:31 AtAM1 I tried using the native and nfs client to mount the replicated volume but was only able to achieve anywhere between 102 and 114 MB/s on a 1Gbit interface as opposed to 980MB/s+ local reads/writes
02:33 bharata-rao joined #gluster
02:46 ilbot3 joined #gluster
02:46 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:22 kshlm joined #gluster
03:27 geewiz joined #gluster
03:28 satheesh joined #gluster
03:48 itisravi joined #gluster
03:56 shubhendu joined #gluster
04:01 kdhananjay joined #gluster
04:24 CheRi joined #gluster
04:27 shylesh joined #gluster
04:45 vpshastry joined #gluster
04:46 nshaikh joined #gluster
04:50 vpshastry joined #gluster
05:01 samppah AtAM1: umm, 100 MegaBytes/s is maxing out your 1Gbit/s link
05:06 AtAM1 samppah: It's actually 125MB/s for a 1Gbit link... I had the clients pointing to their local glusterfs instances
05:06 AtAM1 instead of client 1 to server 2 and vice versa
05:07 AtAM1 writes are not as important as reads
05:08 shyam joined #gluster
05:15 AtAM1 bbl
05:15 jporterfield joined #gluster
05:16 spandit joined #gluster
05:19 RameshN joined #gluster
05:20 rastar joined #gluster
05:21 prasanth joined #gluster
05:21 kanagaraj joined #gluster
05:22 vpshastry joined #gluster
05:28 jporterfield joined #gluster
05:28 hagarth joined #gluster
05:30 dusmant joined #gluster
05:32 lalatenduM joined #gluster
05:34 mohankumar joined #gluster
05:56 meghanam joined #gluster
05:56 meghanam_ joined #gluster
06:02 benjamin__ joined #gluster
06:06 vimal joined #gluster
06:11 saurabh joined #gluster
06:17 ricky-ticky1 joined #gluster
06:23 psharma joined #gluster
06:31 davinder joined #gluster
06:35 benjamin__ joined #gluster
06:39 shyam joined #gluster
06:43 mohankumar joined #gluster
06:51 ngoswami joined #gluster
06:52 jporterfield joined #gluster
06:53 kdhananjay joined #gluster
06:54 ndarshan joined #gluster
06:54 davinder joined #gluster
07:00 CheRi joined #gluster
07:00 dusmant joined #gluster
07:07 micu joined #gluster
07:07 raghu joined #gluster
07:12 TonySplitBrain joined #gluster
07:15 mohankumar joined #gluster
07:25 jkroon joined #gluster
07:30 b0e joined #gluster
07:34 b0e Hi everybody! I have a question concerning geo-replication. Is the parameter "geo-replication.indexing off" only available if i use a gluster-volume as slave?
07:37 b0e Right now i use ssh+path/file as the slave of my geo-replication and after starting the geo-replication the parameter indexing is set to on, even if i set it to off right before. And i cannot find any file with extended attributes on the slave.
07:38 shyam joined #gluster
07:41 vkoppad joined #gluster
07:43 ekuric joined #gluster
07:48 dusmant joined #gluster
07:52 kdhananjay joined #gluster
08:02 mick27 joined #gluster
08:03 eseyman joined #gluster
08:04 jporterfield joined #gluster
08:06 rastar joined #gluster
08:07 SuperJuice joined #gluster
08:11 Philambdo joined #gluster
08:12 franc joined #gluster
08:14 keytab joined #gluster
08:26 keytab joined #gluster
08:32 StarBeast joined #gluster
08:42 mohankumar joined #gluster
08:43 T0aD joined #gluster
08:44 yosafbridge joined #gluster
08:44 gork4life joined #gluster
08:47 meghanam_ joined #gluster
08:47 meghanam joined #gluster
08:50 blook joined #gluster
08:56 shylesh joined #gluster
09:01 ndarshan joined #gluster
09:09 harish joined #gluster
09:12 rastar joined #gluster
09:12 askb joined #gluster
09:12 fidevo joined #gluster
09:17 vpshastry1 joined #gluster
09:17 kaushal_ joined #gluster
09:27 droberts joined #gluster
09:31 jkroon if gluster volume status shows that a brick is not online - how do I go about re-adding the resyncing the brick?
09:33 [1]gork4life joined #gluster
09:34 eseyman joined #gluster
09:39 badone joined #gluster
09:40 navid__ joined #gluster
09:47 lilveg joined #gluster
09:48 kshlm joined #gluster
09:49 harish joined #gluster
09:50 lilveg Hi, I'm looking at a mailing list entry and having a bit of trouble figuring out how the suggested layout works. Does it have a name? Does it have an entry in the manual? How would it scale to 5 servers? I would appreciate any help! Here's the post: http://permalink.gmane.org/gmane.comp.file-systems.gluster.user/1257
09:50 glusterbot Title: hi, all! how to make GlusterFS wokr like RAID 5?? (at permalink.gmane.org)
09:52 mgebbe_ joined #gluster
09:53 purpleidea lilveg: have you read about distribute and replicate ?
09:54 92AAARLT0 joined #gluster
09:55 lilveg purpleidea, yes, but the example in the manual results in only 50% available storage
09:57 ells joined #gluster
09:59 lilveg Ideally, I would like 4/5th available storage and to be able to tolerate a single node failure, is that possible with this setup?
10:01 purpleidea lilveg: there are chained configurations for glusterfs, but they are really not recommended... short answer, no
10:02 purpleidea lilveg: normally you build bricks from raid 6 (or 5 if you want) devices, and then those become part of the pool, using a d-r setup.
10:05 vpshastry1 joined #gluster
10:09 purpleidea kshlm: you around?
10:11 aravindavk joined #gluster
10:14 lilveg purpleidea, I see. Why are they not recommended? I don't have high requirements for performance. Wouldn't a d-r setup still give only 50% effective data usage, on top of the loss for using raid 5 or 6?
10:18 kshlm purpleidea: i'm here
10:21 glusterbot New news from resolvedglusterbugs: [Bug 764966] gerrit integration fixes <https://bugzilla.redhat.com/show_bug.cgi?id=764966>
10:23 satheesh joined #gluster
10:26 badone_ joined #gluster
10:26 ndarshan joined #gluster
10:26 kdhananjay joined #gluster
10:33 CheRi joined #gluster
10:37 harish joined #gluster
10:46 TonySplitBrain joined #gluster
10:50 satheesh1 joined #gluster
10:51 delhage joined #gluster
10:53 dusmant joined #gluster
10:54 lilveg It looks like you would only get 50% of the storage space with this solution. Am I wrong or is the author of the mail incorrect? http://permalink.gmane.org/gmane.comp.file-systems.gluster.user/1257
10:54 glusterbot Title: hi, all! how to make GlusterFS wokr like RAID 5?? (at permalink.gmane.org)
10:57 kanagaraj joined #gluster
10:57 RameshN joined #gluster
10:58 shubhendu joined #gluster
10:58 ndarshan joined #gluster
10:59 khushildep joined #gluster
11:06 purpleidea kshlm ?
11:07 lilveg Is there even a way to get more than 50% data utilization while still having some redundancy?
11:07 purpleidea lilveg: yes
11:07 purpleidea no
11:08 purpleidea sorry i went afk for a bit, internet problem
11:08 diegows joined #gluster
11:20 meghanam_ joined #gluster
11:20 meghanam joined #gluster
11:25 lilveg purpleidea, ok, new idea: if I have two (equally large) volumes, one containing data and one empty, can they be combined with replication without losing the data?
11:27 rahul286 joined #gluster
11:28 mohankumar__ joined #gluster
11:28 kanagaraj joined #gluster
11:29 davinder2 joined #gluster
11:31 rahul286 hello all
11:31 rahul286 I saw http://www.gluster.org/community/documentation/index.php/Planning34
11:31 glusterbot Title: Planning34 - GlusterDocumentation (at www.gluster.org)
11:31 rahul286 contains  Multi-master GeoReplication - Csaba, jdarcy, venky
11:32 rahul286 I am already using GlusterFS 3.4 and sync replica across continents creating problem
11:32 rahul286 I wish to use Geo-replica (async) mainly but in master-master mode
11:32 purpleidea lilveg: what do you want to do...
11:32 rahul286 any help?
11:34 lilveg purpleidea, I have 5 nodes and one iSCSI drive. The iSCSI has the same amount of storage as the nodes combined. I want to distribute across the nodes, and replicate across the nodes and the iSCSI. The problem is the nodes are already more than 50% full, and the data needs to be preserved during the upgrade.
11:35 RameshN joined #gluster
11:39 purpleidea lilveg: have you setup a test gluster setup before?
11:41 lilveg purpleidea, I am doing so for the first time right now.
11:41 purpleidea lilveg: so this is not a good idea to do with your data if you don't know how glusterfs works
11:42 purpleidea lilveg: get comfortable with glusterfs using some vm's or ,,(vagrant) when you're comfortable, then we'll chat about how to do weird migrations, make sense for safety?
11:42 glusterbot lilveg: (#1) Part 1 @ https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/, or (#2) Part 2 @ https://ttboj.wordpress.com/2013/12/21/vagrant-vsftp-and-other-tricks/, or (#3) Part 3 @ https://ttboj.wordpress.com/2014/01/02/vagrant-clustered-ssh-and-screen/, or (#4) Part 4 @
11:42 glusterbot https://ttboj.wordpress.com/2014/01/08/automatically-deploying-glusterfs-with-puppet-gluster-vagrant/, or (#5) https://ttboj.wordpress.com/2014/01/16/testing-glusterfs-during-glusterfest/
11:42 ndarshan joined #gluster
11:42 shubhendu joined #gluster
11:43 dusmant joined #gluster
11:44 lilveg purpleidea, makes sense, but I would still like to know if it is at all possible before committing too much time to a system might work for my purposes
11:45 purpleidea lilveg: doing migrations without enough disk storage can be tricky things. they're possible, but complicated. far too complicated to explain without you having knowledge of some gluster terminology...
11:45 purpleidea which is why it would be too complicated to explain (for me) without you getting comfortable with glusterfs first.
11:46 purpleidea also, if you don't have backups, then you're in big trouble anyways! if you do have backups, then backup and use that as the storage when you build your setup
11:48 Slash joined #gluster
11:48 lilveg purpleidea, I see, thank you. I do have a pretty good overview of the teminology of Gluster, but I don't expect instructions - doing the actual work is my job, after all :P I do have a backup of the nodes on the iSCSI, but unfortunately, getting an extra 40TB of hard drive space for migration is not trivial...
11:49 dbruhn joined #gluster
11:51 hagarth joined #gluster
12:01 purpleidea lilveg: yw!
12:09 itisravi joined #gluster
12:17 ira joined #gluster
12:18 CheRi joined #gluster
12:26 aravindavk joined #gluster
12:27 edward2 joined #gluster
12:40 CheRi joined #gluster
12:50 meghanam joined #gluster
12:52 meghanam_ joined #gluster
13:01 B21956 joined #gluster
13:08 chirino joined #gluster
13:08 rastar joined #gluster
13:15 dusmant joined #gluster
13:17 shyam joined #gluster
13:23 tdasilva joined #gluster
13:25 overclk joined #gluster
13:28 plarsen joined #gluster
13:33 sroy_ joined #gluster
13:38 kshlm joined #gluster
13:49 lilveg I thought I could make a volume out of 5 small bricks, let's call it "dist", and then make a replicating volume containing dist and another brick of equal size... Is this not possible?
13:50 lilveg (I'm getting an error about it already being in a volume)
13:53 RameshN joined #gluster
13:54 primechuck joined #gluster
14:03 X3NQ joined #gluster
14:04 kkeithley you want to create a gluster volume called dist, and then use the dist volume as one of two bricks in another (replica) volume? No, you can't have "nested" gluster volumes like that. If you want a dist+replica volume like that then you create it like this: gluster volume create replica 2 distrep $node1:$path $node2:$path $node3:$path $node4:$path.
14:09 lilveg kkeithley, that might work, but then I would always have to have an even number of bricks..
14:10 kkeithley If you've got five small  drives on node1 and one big drive on node2 and you want to create a replica volume like that then I recommend using lvm or mdraid to consolidate the five into a single lvm volume
14:10 lalatenduM lilveg, yes thats right, with "replica 2" you need to have even number of bricks
14:13 jmarley joined #gluster
14:14 lilveg kkeithley, I see. I was hoping to be able to solve this using Gluster, the restriction seems somewhat artificial. The small drives are not on the same node.
14:15 kkeithley Not in any release to date you can't "nest" gluster volumes. It's been discussed as a possibility for a future release.
14:19 lilveg kkeithley, I understand. If I were to use a simple replicate-2 setup, would Gluster intelligently make sure that the larger brick was used to full effect? The large one is 40TB, while the smaller ones are around 7 TB.
14:20 kkeithley In a replica? gluster writes synchronously to both bricks in a replica pair. If the write to one fails because it's full then you no longer have replication. That kinda violates the replication "contract."
14:21 nullck_ joined #gluster
14:22 lilveg kkeithley, yes, but I have 5 small bricks and 1 large, so if each replica pair gets written once to a small brick and once to the large one, the they should fill up at about an equal rate
14:24 aravindavk joined #gluster
14:24 kkeithley What I would maybe recommend instead is that you use geo-replication for this.
14:25 meghanam joined #gluster
14:26 kkeithley do your primary writes to the 40TB volume. Set up geo-replication to replicate that (asynchronously) to the five brick gluster volume.  Regular replica 2 synchronous replication simply can't deal with the setup you're describing.
14:28 lilveg kkeithley, that seems like a great idea! I would probably do it the other way around, using the 5-brick volume as master, but this seems to fit my needs perfectly. I'll have to read up on it. Thanks!
14:29 meghanam_ joined #gluster
14:29 lilveg There would still be failover, right?
14:29 pixelgremlins hey - trying to create a replication using: gluster volume create www replica 3 apollo:/var/export/www chronos:/var/export/www atlas:/var/export/www
14:30 pixelgremlins Keeps saying unsuccessful : here's the log: https://gist.github.com/pixelgremlins/8649484
14:30 glusterbot Title: etc-glusterfs-glusterd-vol.log..1 (at gist.github.com)
14:30 tjikkun_work joined #gluster
14:30 kkeithley geo-rep fail-over  https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/ch11s05.html
14:30 glusterbot Title: 11.5. Triggering Geo-replication Failover and Failback (at access.redhat.com)
14:32 japuzzo joined #gluster
14:34 rwheeler joined #gluster
14:34 kkeithley why are you getting this [2014-01-27 09:23:09.566008] W [glusterd-utils.c:4920:mkdir_if_missing] 0-: Failed to create the directory /var/export/www   ?
14:35 bennyturns joined #gluster
14:36 pixelgremlins nvm, figured it out.. -- It wouldn't create nested folder structure--I made /var/export on all servers, then it worked
14:36 thefiguras joined #gluster
14:38 vpshastry joined #gluster
14:41 RameshN joined #gluster
14:41 khushildep_ joined #gluster
14:44 meghanam joined #gluster
14:45 gmcwhistler joined #gluster
14:47 CheRi joined #gluster
14:51 gmcwhistler joined #gluster
14:52 gmcwhistler joined #gluster
14:56 meghanam_ joined #gluster
14:56 Humble joined #gluster
15:02 mattappe_ joined #gluster
15:03 satheesh1 joined #gluster
15:10 jobewan joined #gluster
15:13 satheesh1 joined #gluster
15:14 benjamin__ joined #gluster
15:15 bugs_ joined #gluster
15:19 hybrid512 joined #gluster
15:22 satheesh4 joined #gluster
15:24 glusterbot New news from resolvedglusterbugs: [Bug 816447] RFE:object-storage: enhance the response by status command <https://bugzilla.redhat.com/show_bug.cgi?id=816447>
15:31 vimal joined #gluster
15:43 vpshastry left #gluster
15:50 benjamin__ joined #gluster
15:52 Technicool joined #gluster
15:53 failshell joined #gluster
15:57 failshell im looking at the output of gluster volume geo-replication status right now, and i see i have skipped files. how can i find out which ones and why?
15:59 jag3773 joined #gluster
16:00 aixsyd joined #gluster
16:00 aixsyd jclift: you there?
16:03 jclift Kinda.  Out sick today, but am abt to dial into a meeting anyway since I'm awake. ;)
16:17 mattappe_ joined #gluster
16:17 glusterbot New news from newglusterbugs: [Bug 1052759] The default value from help for option 'self-heal-daemon' is incorrect. <https://bugzilla.redhat.com/show_bug.cgi?id=1052759>
16:19 mattapp__ joined #gluster
16:20 purpleidea kshlm: how did it go? one last screencast to record and then i'm done
16:20 purpleidea (bbl)
16:25 ira joined #gluster
16:29 theron joined #gluster
16:29 hchiramm_ joined #gluster
16:30 kshlm purpleidea: the vms have been provisioned and have started up.
16:31 kshlm i'm now looking into puppet, i have no idea how puppet works and how to use it.
16:33 theron joined #gluster
16:35 rcaskey joined #gluster
16:36 mattappe_ joined #gluster
16:38 rcaskey Howdy, is qemu-glusterfs usable async as geo replication over low bandwidth connecitons/
16:39 davinder joined #gluster
16:40 purpleidea kshlm: you don't need to know anything about it to use the setup... but if you do have puppet questions, let me know if you like
16:46 tdasilva joined #gluster
16:47 kshlm purpleidea: any quickstart that you recommend for puppet?
16:58 zaitcev joined #gluster
16:58 mattappe_ joined #gluster
17:02 zerick joined #gluster
17:03 gmcwhistler joined #gluster
17:04 Amanda joined #gluster
17:05 lpabon joined #gluster
17:08 lilveg joined #gluster
17:11 purpleidea kshlm: actually, provision the puppet host that i gave you, and make changes in site.pp to experiment!
17:12 purpleidea depends on what your goal is too... if you want to be good at puppet, or if you just want to get puppet-gluster running...
17:12 purpleidea kshlm: https://ttboj.wordpress.com/ has lots of puppet related material, but it's more advanced level stuff...
17:13 primechuck Have you run your puppet-gluster on ubuntu or am I going to submitting patches to you soon :)
17:15 kshlm mainly to make use of puppet-gluster to do things i want. the vagrant cluster gives a prepackaged setup, i want to be able to roll my own custom setups.
17:16 kshlm i found some docs and if I have any questions I'll ask you :)
17:17 dbruhn joined #gluster
17:20 monotek joined #gluster
17:22 asku joined #gluster
17:23 purpleidea kshlm: the provided code (i think) makes it easy to setup pretty much any type of setup without knowing any puppet... if there's something you can't do, or you're trying to build, let me know!
17:23 aixsyd joined #gluster
17:23 aixsyd dbruhn: yo yo
17:25 purpleidea primechuck: i haven't tried on ubuntu yet, because i don't currently have a test environment to try any of it out. i do have some patches hiding in my dev tree, but they're untested, so not public yet. if you have access to hardware i can use or want to donate to help me work on this, let me know!
17:26 dbruhn aixsyd, sup bud?
17:27 aixsyd dbruhn: I got into my switch as was finally able to factory default - but all of the ports are going at 1x link speed. any idea why? iperf goes at 1gbps. i know the card can go at least 6-7gpbs
17:27 monotek joined #gluster
17:28 dbruhn what does ibstat show on the servers?
17:28 aixsyd 1x
17:28 dbruhn is the switch locking the traffic down to 1x?
17:29 monotek hi,
17:29 monotek if i do: "gluster volume heal gv9 info" i got several entrys like "<gfid:e7144256-b55f-4ea2-888d-b6ff80390e65>" which exists forever....
17:29 monotek how to fix that?
17:29 aixsyd dbruhn: it might be - theres an error that say it found 4x links but is running in 1x mode
17:29 dbruhn monotk, http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/
17:29 glusterbot Title: Fixing split-brain with GlusterFS 3.3 (at joejulian.name)
17:30 dbruhn aixsyd, that's where I would start
17:31 mick27 guys, is there a quick'n dirty way to do multipathing with cluster and nfs clients
17:31 mick27 one that does not require dns
17:31 aixsyd only thing I found was that its possibly a hardware failure of the cable or the cards - but i know neither of that is true
17:31 monotek thanks, but "gluster volume heal gv9 info split-brain" gives no (0 on every node) output?
17:32 monotek still split brain?
17:32 dbruhn monotek, maybe not. I would check your logs on each of the servers, and see if it thinks some of the directories are in split-brain
17:33 dbruhn the output for the info is the last 1023 entries from the log files, so rotating your log files and restarting the heal might resolve seeing them in the list
17:33 primechuck purpleidea: Excellent, I'll find your email somewhere.  I'll be getting to testing later this week or next week and it was continue to write our own or look into somethings else already done.  Until the logic can work its way into core :)
17:33 Mo__ joined #gluster
17:35 aixsyd dbruhn:  check this out... http://fpaste.org/72052/84410613/
17:35 glusterbot Title: #72052 Fedora Project Pastebin (at fpaste.org)
17:35 aixsyd "could be 4x"
17:35 dbruhn weird
17:37 cfeller I'm tarring up a large directory (274 G) that resides on a Gluster 3.4.2 volume (native FUSE mount), and I keep seeing these sporadic "tar: ... file changed as we read it" messages scroll by.  maybe once every 50 files.
17:37 cfeller There are no errors in the gluster mount log.
17:37 cfeller I've verified that nothing else is accessing these files (via lsof), so I'm a bit puzzled.
17:37 cfeller Is gluster reporting info that is confusing tar?
17:37 cfeller Should I be concerned?
17:37 cfeller I saw on old bug report on this: https://bugzilla.redhat.com/show_bug.cgi?id=762345
17:37 glusterbot Bug 762345: high, low, 3.1.1, aavati, CLOSED CURRENTRELEASE, tar keeps reporting "file changed as we read it" on random files on glfs/replicate
17:37 cfeller Gluster client and servers in my case are EL 6, gluster version is 3.4.2 on both.
17:38 vpshastry joined #gluster
17:39 gork4life I have gluster on two machines, I'm using heartbeat and pacemaker for high availability. Is it possible to create a virtaul ip on both machines with the same ip and create a nfs mount to attach to vsphere
17:39 dbruhn aixsyd, http://docs.oracle.com/cd/E18476_01/doc.220/e18478/fabric.htm
17:39 dbruhn section 14.2.4
17:40 cfeller ..and when tar completes, it does give a return value of 1.
17:41 purpleidea primechuck: the module will happily support other platforms, it just doesn't in git master at the moment. feel free to ping me about it.
17:43 aixsyd dbruhn: ibportstate: iberror: failed: smp set portinfo failed
17:43 jobewan joined #gluster
17:47 vpshastry left #gluster
17:50 StarBeast joined #gluster
17:53 tdasilva joined #gluster
17:53 rotbeard joined #gluster
18:00 diegows joined #gluster
18:01 aixsyd dbruhn: this is so alien to me.
18:02 mick27 as of today, should I go with ext4 or xfs for small files ? 500kb-ish
18:04 dbruhn aixsyd, I think you are going to have to figure out if your cards or your switch is stepping the speed down, but since it seems to be all of the cards, I would say it probably is a setting on the switch
18:05 aixsyd seemingly. and the documentation is horrific.
18:05 aixsyd there are no guides or tutorials anywhere. either.
18:07 dbruhn I can imagine, I hadn't started working with IB till post DDR stuff which is after some of the mergers and buyouts happened, so it seemed to become a bit more forward and clear.
18:07 aixsyd management is going to be super pissed they spent this money if, for some reason, we cant get past 1x link speeds
18:08 dbruhn maybe try booting the switch with nothing attached, and then trying bringing up a single host after the switch is fully up
18:08 dbruhn maybe its something in the negotiation
18:08 aixsyd just did that - nope
18:08 dbruhn Is there a log available on the switch side?
18:09 aixsyd sec
18:09 semiosis mick27: xfs is usually recommended for all gluster applications
18:10 semiosis it's the most widely used & heavily tested backend filesytem for glusterfs
18:13 SpeeR joined #gluster
18:14 KyleG joined #gluster
18:14 KyleG joined #gluster
18:28 mick27 semiosis: ok thx !
18:30 semiosis yw
18:32 aixsyd I'm about ready to send this swicth back. this is so maddening.
18:33 dbruhn Might be defective :/
18:33 aixsyd seems to work fine - is just... something is up.
18:33 aixsyd what is a "fabric"?
18:33 dbruhn the interconnect between nodes on the network
18:33 aixsyd "found a 4x link that operates in 1z" - fabric config error
18:33 aixsyd *1x
18:34 mattappe_ joined #gluster
18:34 dbruhn have you tried updating the firmware on the switch?
18:34 aixsyd theres nowhere to get firmware
18:34 dbruhn HP sold a version of that switch, the firmware is the same between them, you *might* be able to see if they have firmware
18:35 aixsyd i just know theres a 9024, 9024M, 9024D-M, 9024S, etc etc etc
18:35 aixsyd and imagine flashing it with wrong firmware v.v
18:36 dbruhn Understood
18:37 aixsyd i'd need to know the exact model number for HP
18:37 dbruhn voltaire of course was bought out by mellenox, and there was a voltaire badged version of those switches, an HP one, and a Mellenox one I believe
18:37 aixsyd yep - and every one of those was slightly different than the one I have. figures.
18:38 dbruhn Where the hell did your higher ups find this thing?
18:38 aixsyd ebay
18:38 mick271 joined #gluster
18:39 aixsyd of course, its non-refundable
18:40 aixsyd dbruhn: is it possible its the cables?
18:40 dbruhn How much did they spend? A lot of times those ebay retailers will take the stuff back if they can sell you something else, maybe get a different model or see if they can source one for you.
18:40 dbruhn Cables are always a possibility
18:41 aixsyd but the same cables can go 7gbps between two nodes directly
18:41 dbruhn It seems like something is defaulting the switch to a lower speed mode, it might be a transceiver compatibility issue or something like that
18:42 dbruhn similar to a 1GB switch auto negotiating at 10/100
18:42 aixsyd ugh
18:43 aixsyd so if I was to get another switch - whats the recomendation
18:44 dbruhn the ddr managed version is what I have in my test environment
18:45 dbruhn honestly with the 10GB stuff I have no idea
18:45 aixsyd maybe its just not worth having a switch in this environment
18:46 aixsyd just sucks that my VM's are hard-lining at 15mb/s due to 1gig ethernet
18:46 dbruhn It seems like you had it working the way you needed before, and the switch is just adding complexity, and a single point of failure
18:46 dbruhn Were you going to put IB cards into the vm servers?
18:46 aixsyd yes
18:46 aixsyd bought and did.
18:46 aixsyd so theres more wasted money v.v
18:47 dbruhn check the revision dates on all of your firmware, maybe you are at way to new of a version on your cards compared to the switch
18:47 dbruhn maybe it's not playing well there
18:48 dbruhn also the mellenox documentation if you can find it usually has a list of cards that are verified with the switches
18:48 dbruhn you might be able to call them and ask if they have any known issues, or if they can provide you some firmware
18:48 dbruhn even though it's not on the site
18:49 dbruhn since the last time you called them you were still fighting trying to get into it
18:49 dbruhn Everytime I talk to those guys the first thing they do is try and get me to upgrade my firmware on everything
18:49 aixsyd yet they dont provide firmware for any of this anymore - its long since EOL'd
18:51 ira joined #gluster
18:54 aixsyd dbruhn: i assume theres no better way to defeat 1gig ethernet than IB or 10GBE - which is way out of our price range
18:55 jobewan joined #gluster
19:03 failshell im looking at the output of gluster volume geo-replication status right now, and i see i have skipped files. how can i find out which ones and why?
19:16 ndk` joined #gluster
19:24 xavih joined #gluster
19:31 xavih joined #gluster
19:39 capncrunch4me joined #gluster
19:40 capncrunch4me Im really trying to justify deploying gluser in lieu of a SAN for block based storage with Cloudstack
19:41 capncrunch4me I have been told that a 4gb infiniband connection operates at considerably less latency that a 10gb ethernet connection, thoughts on that perspective vs going the ethernet route?
19:43 dbruhn capncrunch4me, the IB stuff can provide an increase in performance, but it doesn't come without struggles at times.
19:43 capncrunch4me can you deploy gluster over IB?
19:43 dbruhn Yep, you can use IPoverIB or RDMA
19:43 capncrunch4me I was under the understanding that gluster is pure ip
19:44 capncrunch4me ahh ok
19:44 dbruhn RDMA support is lacking in some of the newer versions or untested
19:44 capncrunch4me I need to teach my irc client to not autocorrect gluster into bluster
19:44 dbruhn hahah same issue here
19:44 capncrunch4me with cassandra, they shortened it to C*
19:44 capncrunch4me has bluster gone to G*?
19:44 capncrunch4me :)
19:46 purpleidea so I made these screencasts for puppet-gluster+vagrant, and the audio cut out part way in. I tried mixing the two together with openshot, but the quality of the video is now garbage. anyone have the special skills to do this?
19:46 purpleidea (i later re-recorded the audio)
19:46 capncrunch4me dbruhn: would you use 10GBe interconnects or IB?
19:47 dbruhn capncrunch4me; I use IB personally, but that's not to say it's the right choice for everyone. In either case your interconnect is only part of the puzzle.
19:49 dbruhn You are also locking yourself into a proprietary interconnect, something to keep in mind. While IB is a cheaper faster alternative today, in the future 10GB will probably come down in price
19:50 capncrunch4me dbruhn: we are going the virtualization route with cloudstack away from bare metal. With that comes going to blades and away from regular servers. The storage tier has to be highly redundant and reasonably fast. We don't plan on putting our database tier on anything other than DAS, but our VM storage will be all shared
19:50 capncrunch4me SSD only
19:51 dbruhn With the cloudstack primary storage architecture you can limit the risk because you can simply build another cluster when you want to add.
19:51 dbruhn The IB will help your SSD's perform to their peak ability
19:51 dbruhn johnmark, has been looking for some cloudstack use cases
19:52 dbruhn and gluster has QEMU support now, so even running IPoverIB you could see performance improvements and stay with more current versions
19:53 kkeithley 3.5.0beta2 RPMs for el6, el7, fedora 19, fedora 20, and fedora 21 (rawhide) are available at http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.0beta2/  (el5  available momentarily).   Gluster Test Week starts now.
19:53 glusterbot Title: Index of /pub/gluster/glusterfs/qa-releases/3.5.0beta2 (at download.gluster.org)
19:53 capncrunch4me so a 4Gb DDR IB connection will be faster than 10Gbe?
19:53 kkeithley johnmark: ^^^
19:54 kkeithley Debian and Ubuntu dpkgs coming soon too (I hope)
19:54 dbruhn capncrunch4me, no idea, I've never used 4GB IB, DDR IB is typically 20GB, and I never actually seen anything less than 10GB.
19:54 capncrunch4me ohhh, nm
19:55 capncrunch4me I meant 20gb
19:55 mick271 guys, is there a training/cert worth something in redhat world regarding gluster ?
19:55 capncrunch4me but, yes obviously it is faster. I was curious about latencies. it seems to have considerably less latency
19:56 dbruhn capncrunch4me, indeed latency reduced with IB. The wikipedia entry talks about it quite a bit. http://en.wikipedia.org/wiki/InfiniBand
19:56 glusterbot Title: InfiniBand - Wikipedia, the free encyclopedia (at en.wikipedia.org)
19:57 capncrunch4me dbruhn: is IB worth deploying instead of direct SAS or sata storage? Is it considerably slower?
19:58 dbruhn Capncrunch4me, that depends on your use case, direct attach storage is going to be faster. It's not shared storage, so in the case of cloudstack, you'll loose the advantages of shared storage.
19:58 capncrunch4me the database is highly IO sensitive..specificially IOPS sensitive
19:59 dbruhn understood, the best thing I can suggest is test test test.
19:59 gdubreui joined #gluster
20:00 KyleG1 joined #gluster
20:00 capncrunch4me dbruhn: honest answer, thanks
20:01 dbruhn Capncrunch4me, how many hypervisors are you planning on running, and how many DB servers?
20:01 capncrunch4me starting small, probably 64
20:01 capncrunch4me then go up to around 256
20:02 capncrunch4me all over c7000 blades
20:02 capncrunch4me so 4 chassis to start, going up to 16 or so
20:02 dbruhn You could set up one cluster with no redundancy and DAS for the DB and then another cluster with shared storage for your other gm's.
20:02 capncrunch4me I really want to go the gluster/ceph route. I have deployed both, like g the best because of the lack of mds
20:02 dbruhn KVM, Xen, or VMware?
20:03 capncrunch4me kvm
20:03 dbruhn limiting your cluster size to how many hypervisors?
20:03 capncrunch4me ?
20:03 semiosis kkeithley: thx for pointing that out.  i'll do the debs tonight
20:04 dbruhn in cloudstack you want your clusters to be limited to 16 hypervisors per cluster with KVM last I checked.
20:04 capncrunch4me by HV, you are talking about physical hosts?
20:04 * semiosis still hasn't solved the FTBFS on ubuntu precise
20:04 dbruhn yep
20:04 capncrunch4me pretty sure you can run hundred of cloudstack HVs
20:04 capncrunch4me hundreds, rather
20:05 dbruhn There are installs running 30,000 hv's out there with cloudstack
20:05 semiosis kkeithley: do they call it FTBFS in rpm land as well?
20:05 dbruhn They break down into clusters though
20:05 dbruhn regions/zones/pods/clusters
20:06 capncrunch4me ugh, its been nearly a year since I built CS from scratch, I can't remember its limitations, and/or if that has changed
20:06 dbruhn a cluster in a shared storage category always shares it's storage, that's how it fails a vm over if the hypervisor dies
20:23 mick271 some nfs expert around here ? what happens if I use a virtual ip on a dual member node, some clients are logged into the first one via the VIP, that one dies, will the connection be still up directly onto the second ?
20:23 mick271 (once the VIP migrate)
20:24 klaxa joined #gluster
20:30 [1]lilveg joined #gluster
20:35 _Bryan_ joined #gluster
20:36 purpleidea mick271: no
20:36 purpleidea at least not with tcp
20:37 purpleidea no idea how udp handles this
20:37 purpleidea think of the connection state for tcp... how would the new server know about what tcp was in the process of doing?
20:37 purpleidea there are things to sync this, but not at the nfs level...
20:37 purpleidea conntrackd is a cool thing to look at...
20:38 purpleidea and nfsv4+ apparently will help with this type of issue i think... not sure
20:38 AtAM1 joined #gluster
20:38 AtAM1 left #gluster
20:41 askb joined #gluster
20:42 mick271 purpleidea: thx
20:42 semiosis glusterfs training & cert, there's a novel idea
20:44 purpleidea semiosis: :P
20:46 samppah would choose that over red hat storage training :)
20:48 purpleidea mick271: yw
20:51 mick272 joined #gluster
20:55 KyleG joined #gluster
20:55 KyleG joined #gluster
20:58 mattappe_ joined #gluster
21:03 armiller left #gluster
21:07 atrius joined #gluster
21:19 ells joined #gluster
21:23 mattappe_ joined #gluster
21:26 ells joined #gluster
21:37 [1]lilveg joined #gluster
21:37 andreask joined #gluster
21:58 _pol joined #gluster
22:09 jobewan joined #gluster
22:09 ira joined #gluster
22:16 realdannys joined #gluster
22:17 realdannys Hi guys. Gluster server not starting on boot on my Centos 6.5 server, chkconfig gluster on is activated but for some reason it doesn't start and I have to log in and manually start the server if I reboot. Any ideas?
22:18 askb joined #gluster
22:23 realdannys Required-Start: $local_fs $network
22:23 realdannys is in the init.d file so I can't understand why its not starting after network
22:28 JoeJulian "chkconfig gluster on" or "chkconfig glusterd on"?
22:28 realdannys sorry yes, glusterd
22:29 realdannys auto correct
22:29 realdannys glusterd       0:off1:off2:on3:on4:on5:on6:off
22:29 realdannys what about glusterfsd ? should that be on too??
22:30 realdannys its not a service I ever start manually, if I SSH in I just run "service glusterd start" and its back up and running and my other machines work with it just fine again
22:30 JoeJulian some network cards take a long time to initialize. Could yours not be up despite network having been started?
22:30 _pol_ joined #gluster
22:33 lilveg joined #gluster
22:34 khushildep joined #gluster
22:34 realdannys @JoeJulian perhaps - I'm running an EC2 instance. If thats the case, how could I push gluster back a bit futher?
22:34 lilveg Does gluster need some special configuration to use geo-replication? I'm getting "unrecognized word" when running the following command: "gluster volume geo-replication dist test2:/nanook config". Gluster version 3.4.2.
22:35 JoeJulian Throw a sleep in the init file
22:35 realdannys is that like a manual delay?
22:35 JoeJulian realdannys: yeah. :(
22:35 realdannys Ok...well, if it works it works! Doesn't matter if it doesn't boot up for another 30 seconds really
22:36 realdannys how exactly would I go about adding that?
22:36 realdannys just something like - #!/bin/bash sleep 300
22:37 dbruhn you will want to add a line after #!/bin/bash
22:37 semiosis virtual nic should be up & running immediately
22:38 realdannys @dbruhn like this
22:38 realdannys #!/bin/bash
22:38 realdannys # sleep 100
22:38 realdannys ?
22:39 dbruhn # makes it a comment in a bash script, that means it will be ignored
22:39 dbruhn so no # at the beginning of "sleep 100"
22:39 realdannys ok
22:39 semiosis also 100 seconds is a bit long
22:39 semiosis maybe 5
22:39 realdannys 5 seconds? Thats a bit short isn't it
22:39 realdannys I mean, if I want it to go last
22:39 semiosis well we dont know what the problem is
22:39 JoeJulian realdannys: Look for the line: "daemon $GLUSTERD" and add a line with "sleep 5" right before it.
22:40 JoeJulian if that's what it is.
22:40 semiosis oeJulian's guess that the nic is slow to start up makes sense on real iron, not in ec2
22:40 JoeJulian right
22:40 semiosis s/oeJulian/JoeJulian/
22:40 glusterbot What semiosis meant to say was: JoeJulian's guess that the nic is slow to start up makes sense on real iron, not in ec2
22:40 realdannys yes I thought that too
22:41 realdannys I've go no problems with gluster at all - anymore, it works, it stops and starts fine, it just doesn't come on during boot
22:42 semiosis realdannys: check the glusterd log /var/log/gluster/etc-glusterfs-glusterd.log to see if it tried to start but failed during boot
22:42 semiosis brb, coffee
22:43 dbruhn realdanny, you might want to start chasing logs to see what's really up
22:43 plarsen joined #gluster
22:45 realdannys Heres my glusterd log, I'm struggling to see anything but you guys might have better trained eyes here - http://pastie.org/8673481
22:45 glusterbot Title: #8673481 - Pastie (at pastie.org)
22:47 atrius joined #gluster
22:47 JoeJulian btw, and this is becoming a common problem all over - I'm not sure why - it's easiest to look at a log if it only has one example of the failure. In the future, truncate the log, make it fail, post that.
22:48 JoeJulian Failed to remove /var/run/b8d41decead472daf7ab82e357d50627.socket error: Resource temporarily unavailable
22:49 JoeJulian I don't think that's normal.
22:49 semiosis there's a consistent pattern of glusterd shutdown a few seconds after it finishes starting up...
22:49 semiosis [2014-01-27 20:06:16.937452] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/brick1/data on port 49152
22:49 semiosis [2014-01-27 20:06:18.106663] W [glusterfsd.c:1002:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7fca82f8db6d] (-->/lib64/libpthread.so.0(+0x79d1) [0x7fca836209d1] (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xcd) [0x40533d]))) 0-: received signum (15), shutting down
22:49 semiosis or what JoeJulian said
22:50 realdannys right
22:50 realdannys although I've got no idea what that means..
22:50 realdannys or how to fix it
22:50 JoeJulian hmm, no, that's in my logs too.
22:50 JoeJulian Guess it's not really an E.
22:50 * JoeJulian grumbles about message log levels again...
22:51 JoeJulian SIGTERM... why? What's killing glusterd?
22:51 semiosis yeah
22:51 semiosis that cant be normal
22:52 realdannys is that on boot do you think? Im struggling to make sense of where boot even is on the logs
22:52 semiosis truncate the log & reboot
22:52 realdannys ok
22:53 semiosis or move the log somewhere else if you want to save it
22:53 realdannys i'll take the sleep out first
22:53 mattappe_ joined #gluster
22:59 zaitcev joined #gluster
23:01 realdannys Ok here we go, fresh reboot - http://pastie.org/8673522
23:01 glusterbot Title: #8673522 - Pastie (at pastie.org)
23:02 dbruhn rdma volume in EC2?
23:02 capncrunch4me left #gluster
23:02 KyleG left #gluster
23:02 semiosis dbruhn: there isn't
23:03 semiosis realdannys: no problem in that log
23:03 semiosis is glusterd running?
23:03 dbruhn I know, seeing it in the log.
23:03 JoeJulian dbruhn: It always tries to load the rdma library. That's normal.
23:03 iksik joined #gluster
23:03 dbruhn ahh ok
23:03 JoeJulian Yeah, looks like it's running to me, too.
23:04 semiosis JoeJulian: or there's more log we're not seeing
23:04 realdannys errr...it looks like its running to me as well this time ?!
23:05 semiosis "have you tried turning it off & on again?" ;)
23:05 dbruhn Love that show
23:05 realdannys pfff, its worked, mad
23:05 realdannys thats the first time ive ever rebooted and its been fine!
23:06 semiosis dbruhn: that phrase is older than the show
23:06 dbruhn I'm aware, just saying
23:06 dbruhn lol
23:06 realdannys maybe it only goes crazy if I reboot both servers
23:07 dbruhn back in a bit gotta drive home
23:12 LessSeen_ joined #gluster
23:12 LessSeen_ hi all
23:14 semiosis hi
23:14 glusterbot semiosis: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
23:14 semiosis LessSeen_: ^
23:18 ekis_isa_ joined #gluster
23:20 LessSeen_ ^^ for sure :) - i am busy here too -juggling a few things- one of which is testing gluster on some ec2 instances to see if it is a suitable solution for us.
23:20 mattappe_ joined #gluster
23:27 realdannys I'm going to try restarting both servers now - out of interested, should it matter if the gluster client comes up before the gluster server?
23:28 mattappe_ joined #gluster
23:32 mattapperson joined #gluster
23:32 _pol joined #gluster
23:36 mattappe_ joined #gluster
23:39 mattapperson joined #gluster
23:40 JoeJulian realdannys: Depends... if the client specifies a specific server to mount from and that server is not up at the time, the mount will fail. There are options to avoid this such as the "backupvolfile-server" mount option or using rrdns.
23:40 mattapperson joined #gluster
23:41 realdannys ah yes that makes sense
23:41 realdannys so really its best to boot gluster server first
23:41 realdannys and then any clients second really
23:42 pixelgremlins_ba joined #gluster
23:43 JoeJulian I suppose you could also just set fetch-attempts to some outrageously high number...
23:43 JoeJulian but yeah, have the servers up first.
23:44 mattapperson joined #gluster
23:45 lanning_ joined #gluster
23:46 hchiramm__ joined #gluster
23:49 klaxa joined #gluster
23:50 overclk joined #gluster
23:52 gdubreui joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary