Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-09-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 Mr_Psmith joined #gluster
00:07 zhangjn joined #gluster
00:14 dgandhi joined #gluster
00:15 plarsen joined #gluster
00:39 Mr_Psmith joined #gluster
00:41 nangthang joined #gluster
01:17 badone_ joined #gluster
01:25 anil joined #gluster
01:29 Lee1092 joined #gluster
01:39 nangthang joined #gluster
01:45 zhangjn joined #gluster
01:58 CyrilPeponnet Guys when did you plan to replace your outdated cert ?
01:59 CyrilPeponnet it breaks yum :(
02:00 EinstCrazy joined #gluster
02:01 baojg joined #gluster
02:06 CyrilPeponnet @purpleidea for now I have disabled https in puppet-gluster (private fork) as the outdated cert *.glusterfs.org was causing great pain to yum. FYI
02:07 thangnn_ joined #gluster
02:08 badone_ joined #gluster
02:12 rafi joined #gluster
02:26 kotreshhr joined #gluster
02:29 RameshN joined #gluster
02:29 rafi joined #gluster
02:39 badone__ joined #gluster
03:00 chirino joined #gluster
03:10 [7] joined #gluster
03:14 aravindavk joined #gluster
03:25 haomaiwa_ joined #gluster
03:30 kshlm joined #gluster
03:34 haomaiwa_ joined #gluster
03:34 RameshN joined #gluster
03:37 gem joined #gluster
03:37 shubhendu joined #gluster
03:39 atinm joined #gluster
03:42 raghu joined #gluster
03:42 sakshi joined #gluster
03:49 poornimag joined #gluster
03:53 overclk joined #gluster
03:58 nbalacha joined #gluster
03:58 calavera joined #gluster
03:59 itisravi joined #gluster
04:00 nishanth joined #gluster
04:01 haomaiwang joined #gluster
04:05 skoduri joined #gluster
04:06 ppai joined #gluster
04:10 yazhini joined #gluster
04:20 calavera joined #gluster
04:22 badone_ joined #gluster
04:24 calavera joined #gluster
04:27 aravindavk joined #gluster
04:28 kanagaraj joined #gluster
04:33 purpleidea JoeJulian: ^^ maybe you know who is responsible for cert issues?
04:33 vmallika joined #gluster
04:48 ramteid joined #gluster
04:51 RameshN joined #gluster
04:52 neha joined #gluster
04:58 rafi joined #gluster
05:01 haomaiwa_ joined #gluster
05:02 ndarshan joined #gluster
05:05 vmallika joined #gluster
05:06 RameshN joined #gluster
05:12 kshlm joined #gluster
05:15 jiffin joined #gluster
05:16 aravindavk joined #gluster
05:22 kshlm joined #gluster
05:26 Bhaskarakiran joined #gluster
05:29 skoduri joined #gluster
05:32 hchiramm_home joined #gluster
05:33 EinstCrazy joined #gluster
05:39 hgowtham joined #gluster
05:41 deepakcs joined #gluster
05:42 geoffr joined #gluster
05:43 kdhananjay joined #gluster
05:43 Bhaskarakiran joined #gluster
05:43 geoffr Hi, does anyone know how to fix / quiet the "0-rpc-service: RPC procedure 2 not available for Program GF-DUMP" error?
05:44 geoffr ah nevermind googling old irc logs shows this has been answered before :)
05:45 vimal joined #gluster
05:47 nishanth joined #gluster
05:47 shubhendu joined #gluster
05:47 Manikandan joined #gluster
05:54 hchiramm_home joined #gluster
06:01 haomaiwa_ joined #gluster
06:03 ashiq joined #gluster
06:06 raghu` joined #gluster
06:07 hagarth joined #gluster
06:11 itisravi_ joined #gluster
06:19 pppp joined #gluster
06:19 jtux joined #gluster
06:20 mhulsman joined #gluster
06:20 raghu joined #gluster
06:23 skoduri joined #gluster
06:53 Philambdo joined #gluster
06:54 haomai___ joined #gluster
06:57 atalur joined #gluster
06:58 rgustafs joined #gluster
06:59 shubhendu joined #gluster
06:59 jiffin1 joined #gluster
07:01 haomaiwa_ joined #gluster
07:05 papamoose1 joined #gluster
07:16 Bhaskarakiran joined #gluster
07:23 fsimonce joined #gluster
07:24 [Enrico] joined #gluster
07:25 anil joined #gluster
07:30 maveric_amitc_ joined #gluster
07:30 [Enrico] joined #gluster
07:34 jiffin1 joined #gluster
07:37 mattmulhern joined #gluster
07:43 itisravi_ joined #gluster
07:49 gildub joined #gluster
08:05 raghu joined #gluster
08:09 baojg joined #gluster
08:11 jwd joined #gluster
08:14 Simmo joined #gluster
08:15 Simmo Good morning from Salzburg! : )
08:18 SimmoTali I just met the amazing Gluster :)
08:19 SimmoTali I set-up two EC2 istances with 1 volume each formatted as suggested in Getting Started.
08:19 SimmoTali Volume and Bricks set-up.
08:19 SimmoTali I would need now your debugging skills :-/
08:20 SimmoTali As most of newbies I'm stuck: creating a file in one instance is not replicated in the other.
08:20 SimmoTali gluster-client is also configured
08:21 SimmoTali I would need to understand where I did something wrong in the setup. What I could check ? :_(
08:24 SimmoTali (Can you read me ? )
08:28 jwaibel joined #gluster
08:29 Bhaskarakiran joined #gluster
08:29 Bhaskarakiran joined #gluster
08:32 arcolife joined #gluster
08:38 ndevos welcome SimmoTali!
08:39 SimmoTali Hello ndevos!!!
08:39 ndevos SimmoTali: you need to write to a glusterfs mountpoint, not to the bricks
08:39 SimmoTali uh.. ok, that might explains something :)
08:40 SimmoTali Unfortunately, I just entered the topic so my knowledge is really low
08:40 ndevos SimmoTali: this quickstart should explain the different steps well: http://gluster.readthedocs.org/en/l​atest/Quick-Start-Guide/Quickstart/
08:40 glusterbot Title: Quick start Guide - Gluster Docs (at gluster.readthedocs.org)
08:41 Bhaskarakiran joined #gluster
08:41 ndevos I'm not sure if we have a guide that explains about the AWS EC2 specifics
08:41 SimmoTali Yes, you have : )
08:42 SimmoTali http://www.gluster.org/community/document​ation/index.php/Getting_started_setup_aws
08:42 SimmoTali Anyway, I try to follow your link now... I'm sure I messed with the mountpoints
08:43 SimmoTali Dummy question: both in the "server" and in the "client" I have to mount the bricks, and then write in the mounted dirs... ?
08:43 ndevos hmm, that is actually the old doc/wiki, the same guide is on http://gluster.readthedocs.org/en​/latest/Install-Guide/Setup_aws/
08:43 glusterbot Title: Deploying in AWS - Gluster Docs (at gluster.readthedocs.org)
08:44 ndevos also, this one contains a little more details: https://access.redhat.com/documentation/​en-US/Red_Hat_Storage/3/html-single/Admi​nistration_Guide/index.html#part-Red_Hat​_Storage_Administration_on_Public_Cloud
08:44 glusterbot Title: Administration Guide (at access.redhat.com)
08:44 SimmoTali Uh, ok.. not sure how I reach the old docs : )
08:44 csim ndevos: so the page should redirect to the new content ?
08:44 csim SimmoTali: likely some old link somewhere
08:44 haomaiwa_ joined #gluster
08:44 ndevos csim: yeah, and I think hchiramm was working on getting that done...
08:45 SimmoTali Likely, typing on google Gluster EC2
08:45 SimmoTali uh.. stand-up!
08:45 SimmoTali Trying your links after :)
08:45 * ndevos stands up to get a coffee :)
08:45 SimmoTali Thanks a lot for the help!
08:45 csim oh yea, coffee, good idea
08:45 csim ndevos++
08:45 glusterbot csim: ndevos's karma is now 22
08:47 LebedevRI joined #gluster
08:49 ndevos csim++ coffee++ SimmoTali++
08:49 glusterbot ndevos: csim's karma is now 1
08:49 glusterbot ndevos: coffee's karma is now 2
08:49 glusterbot ndevos: SimmoTali's karma is now 1
08:50 ndevos SimmoTali: you need to mount bricks on all the servers that provide storage for the Gluster volume, these bricks are local to the storage servers, not shared
08:51 anil joined #gluster
08:51 ndevos SimmoTali: with those bricks, you build the volume "gluster volume create/start ....", and then those bricks are made available through the volume
08:52 ndevos SimmoTali: after starting the volume, you can mount it with "mount -t glusterfs $storage_server:/$volume /mnt"
08:53 ndevos SimmoTali: the bricks would be mounted somewhere else, I like to have them as /bricks/volumename_n and use /bricks/volumename_n/data when creating the volume
08:57 SimmoTali Lol, unfortunately it was not a coffee-standup :D
08:58 SimmoTali but now that you mentioned it I got a tremendous wish for a coffee!
08:58 SimmoTali ndevos: thanks a LOT for your directions. I need to reiterate through the setup and fix my mistakes : )
09:00 SimmoTali Where are you chatting from, guys ? : )
09:00 ndevos SimmoTali: you're welcome! if you think the docs need some improvement, go and edit https://github.com/gluster/glusterdocs/​blob/master/Install-Guide/Setup_aws.md and all ;-)
09:00 glusterbot Title: glusterdocs/Setup_aws.md at master · gluster/glusterdocs · GitHub (at github.com)
09:01 ndevos SimmoTali: I'm in Zaandam, close to Amsterdam in The Netherlands
09:01 haomaiwa_ joined #gluster
09:02 csim Zaandam sound like a place in a sci-fi movie :p
09:03 SimmoTali Nice! I would like to come back and visit again The Netherlands: I 'only' visited Amsterdam loooong time ago.. such a beautiful city!
09:03 * csim is in Paris
09:03 SimmoTali nice to meet you, guys : )
09:03 David_Varghese joined #gluster
09:04 ndevos if you ever visit Amsterdam and would like to talk Gluster, let me know :)
09:05 SimmoTali ooooh, I will! Thanks ndevos : )))
09:07 ndevos hah! Zaandam also seems to be the name of a cruise ship, thats not where I live though
09:07 jcastill1 joined #gluster
09:11 spalai joined #gluster
09:11 kshlm joined #gluster
09:11 spalai left #gluster
09:12 jcastillo joined #gluster
09:16 spalai joined #gluster
09:17 atinm joined #gluster
09:20 jiffin joined #gluster
09:25 shubhendu joined #gluster
09:27 kotreshhr joined #gluster
09:37 SimmoTali AMAZING!
09:38 SimmoTali Understanding that bricks have to be mounted (somewhere) did the trick :_)
09:38 ppai joined #gluster
09:43 skoduri joined #gluster
09:45 nbalacha joined #gluster
09:47 SimmoTali I would like to check with you if Gluster makes sense for my scenarios:
09:47 SimmoTali I would like to scale horizontally a Web Application. That Application requires to manage some files and folders located on the filesystem.
09:47 hchiramm_home joined #gluster
09:47 SimmoTali Now, you can see the problem: if I run 2 istances of the Web Application on two different machines then I need to keep in sync those files and folders.
09:47 SimmoTali As first step, I would try to have 1 "Writer" (i.e. an instance enabled to create/update/delete those files) and N "Readers" (i.e. instances enabled only
09:47 csim so, is there locking or concurent access issue ?
09:47 SimmoTali to "read" those files).
09:48 SimmoTali The long plan would be to have multiple N Writer and N Reader :)
09:48 SimmoTali Yes, there are concurrent issue. I can manage those at application level
09:48 csim so yeah, it would work
09:48 SimmoTali it means, when a "writer" is managing a specific file(s) than the other cannnot access it
09:49 csim how do you do the locking ?
09:49 csim also, what about delay of propagation ( ie, I do a request on e the writer, then go to the reader )
09:50 SimmoTali I'm thinking at two solution: the first (more naive) is using an centralized db. The second (which I need still to evaluate) is using asynchronous queues.
09:51 SimmoTali the asynchronous queues would help also to notify the "Reader" that a files/folders have changed so that it can re-load the related resources
09:51 SimmoTali and,
09:51 SimmoTali > what about delay of propagation
09:51 SimmoTali that is really a good question: How can i know when the "diff" are propagated on all the nodes ? :(
09:52 csim well, you can't
09:52 csim so my best bet is to try to cope with failure as gracefully as possible :)
09:52 csim (or redirect client to the same node on the LB level)
09:52 SimmoTali I guess a dummy approach is to assume the worst delay
09:53 csim yeah
09:53 csim I think the delay is ok most of the time, but if a nde fail at the wrong time, the delay become infinite :)
09:53 SimmoTali According to your experience: what is the worst delay you ever notice? :)
09:53 SimmoTali true!
09:53 csim I have no experience, depend on the size of file, the load on network
09:54 SimmoTali The Network should not be a problem... I will use machines in the same AWS Cloud
09:54 csim like if you upload iso, and suddenly, you have a 100k network bandwidht because we have emergency backup to do...
09:54 SimmoTali so that the transfer rate will be impressive : )
09:54 csim I have faith on people being able to always fill the pipe
09:54 SimmoTali naaa.. luckly those files are really small
09:54 SimmoTali but are lot of them
09:55 csim and "network always work" is a fallacy in distributed computing :)
09:55 SimmoTali right :D
09:55 SimmoTali I need to study the failover in Gluster
09:55 nbalacha joined #gluster
09:55 SimmoTali i'm just grasping the tip of the iceberg now
09:55 SimmoTali : )
09:56 SimmoTali be patience with me: what do you mean with
09:56 SimmoTali > (or redirect client to the same node on the LB level)
09:56 SimmoTali the LB level ?
09:56 csim there is lots of good distributed computing ressource around
09:56 csim load balancer
09:56 SimmoTali ah so
09:56 SimmoTali sorry : )
09:57 csim I kinda assume you have one if you have multiple applications
09:57 csim I would recommend also to try to make a test env, and just kill systems randomly, etc to see what break
09:57 csim chaos monkey is quite nice for that, it seems
09:57 SimmoTali In this I have a nice advantage: I can have a load balancer and even more I know which requests require a "write" or a "read"
09:58 SimmoTali so I can forward them to the writer
09:58 SimmoTali or to the reader
09:58 SimmoTali accordingly
09:58 SimmoTali good tip|
09:59 SimmoTali Before reading the doc, may i ask what is Gluster's behaviour in a node failure case?
09:59 SimmoTali (overview)
09:59 * csim has no idea
09:59 SimmoTali Lol : )
09:59 csim I am just the sysadmin of gluster.org, but do not have a usecase requiring gluster so I mean it when i say I have no experience :)
10:00 SimmoTali thanks anyway... chatting with you I got already some nice tips : )
10:00 SimmoTali another staff that worries me is the NTP
10:00 csim why ?
10:01 SimmoTali on a plain linux machine it should be easy to setup
10:01 SimmoTali but on EC2 it seems a crazy action
10:01 SimmoTali because there is XEN in the back
10:01 SimmoTali which requires some more steps
10:01 haomaiwa_ joined #gluster
10:01 SimmoTali I also need to go deep in that part :(
10:02 SimmoTali EC2 instances in the same area can already have several minutes of difference
10:02 csim http://slaptijack.com/system-administr​ation/ntp-and-your-xen-based-linux-vm/ ?
10:02 SimmoTali bah!
10:02 glusterbot Title: NTP and Your Xen-based Linux VM | Slaptijack (at slaptijack.com)
10:02 csim seems that you need to be root on the dom0 to say "each VM have a separate clock"
10:02 SimmoTali uh! Here looks way simpler
10:02 csim so either amazon did, or you can't fix
10:02 SimmoTali :D
10:03 csim but I might have misunderstood
10:03 SimmoTali I'll tell you in several weeks ; )
10:04 csim otherwise, there is non xen based cloud hosting company out there
10:04 csim I do use rackspace, that's quite good, and I had nice result on digital ocean too
10:05 csim (didn't tried yet gce)
10:05 SimmoTali Good idea... I will evaluate also those
10:05 SimmoTali Since my application is a nodejs app
10:05 SimmoTali do you know if Heroku gives you complete access to the machine ?
10:06 SimmoTali or any other provider
10:06 csim openshift give a ssh access
10:06 csim but you are not root
10:06 SimmoTali doh!
10:06 csim and the v3 give you a complete docker container, where you are root, but the container is contained so you can't do everything
10:07 csim not sure if heroku give access, depend mostly on what you need and want
10:07 csim ( if that's "running gluster", the answer is likely "no" :p )
10:07 SimmoTali indeed, it is ;)
10:09 SimmoTali Ok, thanks again! I go back to my studies : )
10:10 csim see you
10:14 baojg joined #gluster
10:18 gildub joined #gluster
10:22 neha joined #gluster
10:27 plarsen joined #gluster
10:33 teknologeek joined #gluster
10:33 teknologeek hi all
10:34 itisravi joined #gluster
10:35 teknologeek i have serious error messages with xlator/cluster/distribute.so that says dht_layout_set invalid argument inode
10:35 teknologeek any clue about this message ?
10:39 kshlm joined #gluster
10:44 Bhaskarakiran joined #gluster
10:47 Mr_Psmith joined #gluster
10:55 ira joined #gluster
10:55 atinm joined #gluster
11:01 64MADTH3B joined #gluster
11:01 vmallika joined #gluster
11:02 zhangjn joined #gluster
11:02 Mesh23_ joined #gluster
11:04 Mesh23_ I'm trying to decide which filesystem to use for a shared storage for VM images. I'm thinking about glusterfs but I'm wondering: how does glusterfs handle a node failure? Will it re-sync ALL data when the dead node comes back? (I read that somewhere)... This could take a long time with big data...
11:08 firemanxbr joined #gluster
11:08 samppah Mesh23_: it crawls through files and syncs data that has been changed
11:09 Mesh23_ samppah: for vm images, which change frequently, will it have to retransmit the whole vm images or will it be able to just transfer the changes parts?
11:09 Mesh23_ changed*
11:10 samppah Mesh23_: by default it transmits only the changed parts
11:10 samppah although i think you can change it to transfer whole files and it can be even faster in some cases
11:11 Mesh23_ okay, but why would that be faster? The VM images are like 50Gb or something while changes might be only 500mb in size ?
11:12 samppah afaik it compares the file part by part and that can take some time
11:13 samppah i'm not sure if newer versions have some kind of log of what has changed inside the file
11:14 Mesh23_ If it doesn't keep track of changes, it's always going to read all files on both nodes and send traffic over the network to compare them? so it's going to be very CPU and network hungry while recovering from a failed node?
11:16 samppah true, except it only sends checksum of data that its comparing
11:17 Mesh23_ okay, but still it's going to be heavy on the disks :)
11:17 Mesh23_ I'm not sure if glusterfs is the right choise for us then... :) I'm still trying to decide between glusterfs or ceph
11:18 samppah i have been using glusterfs for vm storage in production use for couple years now
11:18 Mesh23_ And no problems with it?
11:19 samppah 2.7 TB used about 80 vms i think
11:19 samppah nope.. we had some hardware problems with one node and didn't have any problems syncing the data between nodes after that
11:20 Mesh23_ And performance wise, everything is fine too?
11:22 samppah there are some minor issues with performance but haven't got any complaints from customers so i think it's fine that way
11:23 samppah still running version 3.5 and i think that 3.6 and 3.7 has lots of improvements that may address issues we have seen
11:23 Mesh23_ What exactly is the problem with performance you're seeing? I've got about 10 VM's running pretty big databases and are doing pretty heavy disk i/o.
11:26 samppah this is bit hard to explain but this is something i have been hitting only when trunning testing tool like fio.. when doing lots of writes performance is good for a while and then it hits somekind of barrier which forces data to be written to disks and during that time all writes are stopped..
11:27 samppah just a sec.. i'll see if i can create an example
11:27 Mesh23_ okay :)
11:34 samppah Mesh23_: http://pastie.org/10412157 random write with 4k block size.. it's fast enough for a while but then write speed drops to 0 for couple of seconds
11:34 glusterbot Title: #10412157 - Pastie (at pastie.org)
11:34 samppah i have doing lots of debugging for this but i still have no clue what's causing it
11:35 samppah i have been doing..
11:35 samppah but this is something that has never been a problem in production use for us
11:36 samppah disks underneath and network are fine and should be able to handle the load
11:37 Mesh23_ It's indeed strange. Did you test with other filesystems to rule out any possibility for a bad disk/other hardware issue ?
11:38 LebedevRI joined #gluster
11:38 Mesh23_ which hypervisor are you using?
11:38 samppah oVirt
11:38 samppah sorry i did lots of testing for this long time ago and can't remember all the things i did after all :)
11:38 Mesh23_ okay :)
11:39 Mesh23_ I was thinking about Proxmox for the hypervisor and either glusterfs or ceph for shared storage. But it seems I have to do some tests myself and hopefully won't have the write issues you're having in gluserfs
11:39 Mesh23_ glusterfs seems a lot easier that ceph to install, so I think I'm going to give glusterfs a try first
11:40 samppah yeah, it's pretty easy to install
11:40 Mesh23_ does glusterfs read parallel from both storage nodes? Any idea?
11:41 samppah at least per file
11:41 samppah for example vm1 from storage1, vm2 from storage2 etc
11:42 Mesh23_ okay ;)
11:44 RameshN joined #gluster
11:48 Bhaskarakiran joined #gluster
11:57 aravindavk joined #gluster
11:58 shubhendu joined #gluster
12:00 B21956 joined #gluster
12:01 haomaiwa_ joined #gluster
12:01 Slashman joined #gluster
12:02 SimmoTal_ joined #gluster
12:05 plarsen joined #gluster
12:15 TvL2386 joined #gluster
12:16 jtux joined #gluster
12:35 ashiq- joined #gluster
12:35 johnmark joined #gluster
12:39 unclemarc joined #gluster
12:39 cabillman joined #gluster
12:40 calisto joined #gluster
12:48 rgustafs joined #gluster
13:05 sghatty joined #gluster
13:09 shyam joined #gluster
13:21 aravindavk joined #gluster
13:27 nbalacha joined #gluster
13:28 dgandhi joined #gluster
13:29 ashiq- joined #gluster
13:29 hgowtham joined #gluster
13:30 skoduri joined #gluster
13:32 rwheeler joined #gluster
13:35 tdasilva joined #gluster
13:39 nbalacha joined #gluster
13:42 harold joined #gluster
13:44 Mesh23_ Anyone here using gluserfs in combination with balance-rr bonding? I'm looking for a cheap way to increase network performance using multiple nic's
13:45 Mesh23_ I'm wondering if this works pretty well, or if out-of-order delivery of packets (caused by balance-rr) is making it useless?
13:46 maveric_amitc_ joined #gluster
13:47 dijuremo Mesh23_: 10Gbps copper nics are affordable now....
13:48 64MADTJHW joined #gluster
13:48 Mesh23_ dijuremo: and are there any cheap 10G switches not too expensive? I would need to connect 10 servers
13:51 dijuremo Mesh23_: *cheap* is a relative term...
13:51 devilspgd joined #gluster
13:52 hchiramm_home joined #gluster
13:54 _Bryan_ joined #gluster
13:54 dijuremo Mesh23_: Do you trust Netgear? Is $1500 cheap? If so -> http://www.amazon.com/NETGEAR-ProSAFE-12-P​ort-10GBase-T-XS712T-100NES/dp/B00BWBLL6S/
13:56 ashiq- joined #gluster
13:58 spalai left #gluster
14:01 haomaiwa_ joined #gluster
14:03 papamoose1 left #gluster
14:03 Gill_ joined #gluster
14:07 dijuremo Mesh23_: You could also just try lacp on gigabit switches that support it, but are you sure the network is the bottleneck?
14:09 natarej joined #gluster
14:10 Bhaskarakiran joined #gluster
14:10 Bhaskarakiran joined #gluster
14:10 Slashman joined #gluster
14:15 Bhaskarakiran joined #gluster
14:21 hagarth joined #gluster
14:27 baojg joined #gluster
14:32 mhulsman1 joined #gluster
14:34 leeyaa joined #gluster
14:34 leeyaa hello
14:34 glusterbot leeyaa: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:35 Yoel joined #gluster
14:35 leeyaa if i use a VM host to test glusterfs setup and use that host's single raid1 array will this penalise gluster's performance ? because usually I am getting X MBs and when testing gluster mounts I get around twice lower performance
14:35 leeyaa well more like 3 times
14:36 leeyaa I presume I will need to use dedicated arrays for each brick to get best performance ?
14:36 Yoel left #gluster
14:36 leeyaa and maybe ditch the arrays, but use the physical disks insteasd
14:36 hagarth leeyaa: what kind of tests are being run?
14:37 leeyaa hagarth: just regular dd reads and writes
14:37 hagarth leeyaa: what is the block size being used?
14:38 leeyaa hagarth: for example time dd if=/dev/zero of=delme bs=10024k count=1k
14:38 leeyaa hagarth: you mean sector size ?
14:38 Yoel joined #gluster
14:38 hagarth leeyaa: what is the volume type in use with gluster?
14:39 hagarth leeyaa: no, I meant blocksize with dd. seems to be 1024k in your case.
14:39 leeyaa yeah default
14:39 leeyaa and by volume type you mean fs ?
14:39 hagarth leeyaa: no, output of type in gluster volume info
14:40 leeyaa hagarth: this https://bpaste.net/show/4ba03fac63cc
14:40 leeyaa i think performance will be a lot lower than using dedicated disks
14:41 leeyaa had similar behaviour when testing ceph
14:41 hagarth leeyaa: since you are using a replicated volume type, the write performance will be slower than normal as 2 writes have to be committed before an acknowledgement can be sent back to dd
14:42 leeyaa hagarth: yep thats what i thought thanks
14:42 leeyaa hagarth: and if i want to scale writes, how much bricks i need to add? (each on dedicated disk)
14:42 leeyaa how many*
14:44 hagarth leeyaa: normally more bricks will help if you have more clients or multi-threaded applications operating on multiple files. the aggregate throughput will tend to be better then.
14:45 leeyaa hagarth: so scaling is not exactly linear like the web says ? what do you have to do if you want better performance?
14:46 yoli joined #gluster
14:46 hagarth leeyaa:  linear scaling is dependent on workload, would recommend looking into various layers of the stack to determine what is causing the performance hit.
14:50 shaunm joined #gluster
14:51 leeyaa well atm the hardware is my issue as it is just a test lab. single host with vms. the real setup consists of 3 hosts with ssd only drives. i guess ill have to make more tests there.
14:51 leeyaa thanks hagarth
14:53 qubozik joined #gluster
14:54 zhangjn joined #gluster
14:55 zhangjn joined #gluster
15:01 haomaiwa_ joined #gluster
15:02 maveric_amitc_ joined #gluster
15:07 _maserati joined #gluster
15:08 leeyaa is it ok to use mdadm and lvm with gluster ?
15:12 magamo leeyaa: I believe LVM is required to make use of certain features of gluster (snapshots come to mind)
15:13 amye1 joined #gluster
15:13 leeyaa magamo: hm so i can use raid0 for performance boost with lvm and gluster for redundancy?
15:15 magamo That sounds like a valid strategy to me.  However, I'll differ to the experts.
15:16 CyrilPeponnet yes, on my own, we are using hardware raid 10 and the volume is then exposed to gluster
15:17 CyrilPeponnet not sure if snapshots are using lvm
15:17 CyrilPeponnet but I don't use them
15:17 magamo gluster snapshots in 3.7.X require that the bring be on a thin provisioned LV.
15:18 leeyaa CyrilPeponnet: why raid10 if gluster gives you the redundancy
15:18 magamo .... The bricks, even. Bricks!
15:18 CyrilPeponnet For speed access and also we have some cases with only 1 brick
15:18 CyrilPeponnet because the more the brick you have the longer it takes to write
15:18 shyam joined #gluster
15:20 leeyaa i see
15:21 hagarth leeyaa: raid10 provides redundancy for disk failures and gluster provides that for node failures
15:21 atalur joined #gluster
15:23 squizzi joined #gluster
15:26 leeyaa one more, does it matter if you use different or same size bricks?
15:27 CyrilPeponnet same size is better
15:27 CyrilPeponnet or it will show you a warning during the creation
15:27 semiosis purpleidea: i'm still lurking but haven't been active in a bit
15:33 qubozik_ joined #gluster
15:35 purpleidea semiosis: still running gluster anywhere?
15:35 semiosis nope :/
15:36 semiosis people are using the ,,(java) project though!
15:36 glusterbot https://github.com/semiosis​/glusterfs-java-filesystem
15:36 purpleidea semiosis: doing good? what's up lately?
15:36 semiosis so my legacy lives on
15:36 semiosis yes, doing great, thanks.  busy busy with work and life
15:37 purpleidea semiosis: oh shit, why you got 3 clause bsd license instead of 2? isn't this gpl _incompatible?_
15:37 semiosis hah!  no one's mentioned that yet.  is it?  i dont know
15:37 semiosis what should I do?
15:37 purpleidea semiosis: yeah! _nobody_ uses this license anymore, but i can double check
15:37 purpleidea big detractor to using your code...
15:38 purpleidea one sec
15:38 qubozik joined #gluster
15:38 CyrilPeponnet You will need to contact each person that have done PR or commit and ask them if they agree to change the licence, if not you will need to remove their code from your master as long as you update the licence model (this is a pain)
15:38 semiosis i wanted the most permissive license to encourage people to use it & contribute.  guess i missed the mark on that one
15:38 purpleidea i might be wrong about compat though, let me just check first
15:39 CyrilPeponnet apache is great for that
15:39 purpleidea there was an advertising clause in one of the bsd licenses that was a problem
15:39 purpleidea i agree with CyrilPeponnet that alv2 would have been a better choice then
15:41 purpleidea semiosis: my bad, this version of the license is okay
15:41 purpleidea semiosis: alv2 offers some extra patent protections, but you're good in so far as what you wanted is true and IANAL, but i'm fairly sure this is gpl compat. so you're good
15:41 * purpleidea turns off legal convo :P
15:42 semiosis :-D
15:42 purpleidea sorry brain fuck
15:42 CyrilPeponnet licensing is a pain
15:42 purpleidea i had an off by one error: it's four clause bsd that's bad, but i briefly thought it was 3 clause bsd :P
15:42 CyrilPeponnet even for lawyers
15:43 semiosis 2 clauses is not enough.  4 is too many.  i think 3 is juuuuuust right
15:43 purpleidea semiosis: in any case, i recommended you for a gluster conf. in case you're interested
15:43 JoeJulian What's the clause that gives away everthing to everyone but only once a year?
15:43 semiosis i'd love to do another conference but budgeting...
15:43 semiosis purpleidea: send me the info please
15:44 JoeJulian Santa Clause!
15:44 semiosis oh no
15:44 purpleidea JoeJulian: say whaaat?
15:44 JoeJulian semiosis: I've been told, once again, that someone's got a budget to make that happen. We'll see if it's true this time.
15:45 CyrilPeponnet btw you still haven't found yet how to renew your *.gluster.org cert?
15:45 semiosis CyrilPeponnet: whats that about?
15:45 rwheeler joined #gluster
15:46 CyrilPeponnet https://download.gluster.org/pub/gluster/​glusterfs/3.6/3.6.4/CentOS/epel-6/x86_64/ cert is expired
15:46 CyrilPeponnet this cause yum to crash badly actually
15:46 JoeJulian CyrilPeponnet: The certificate request was sent.
15:46 CyrilPeponnet awesome
15:46 csim we did ask for a new one, waiting on it getting signed by our provider
15:47 CyrilPeponnet great
15:50 purpleidea g2g
15:53 cholcombe joined #gluster
16:00 shyam joined #gluster
16:01 haomaiwang joined #gluster
16:06 natarej so someone pointed me to this https://shellycloud.com/blog/2013/09/why-glust​erfs-should-not-be-implemented-with-openstack
16:06 natarej i wonder how on earth its one of the top results on google
16:07 maveric_amitc_ joined #gluster
16:07 natarej he didn't read the comments, and it's the second time i've seen it referenced by someone that didn't read it thoroughly
16:09 CyrilPeponnet well don't use openstack and you are good :p
16:09 qubozik_ joined #gluster
16:12 skoduri joined #gluster
16:13 natarej i think the only point he makes in the entire blog that isn't factually incorrect is that gluster lacks copy on write
16:14 JoeJulian Right, but if you make a bd volume you can still get that with lvm.
16:15 JoeJulian I still like my comment. :D
16:15 JoeJulian I read it and I'm like, damn. That guy really gave it to him good. Oh, that's me!
16:16 natarej oh thats you!
16:16 calavera joined #gluster
16:16 natarej i don't read names
16:16 natarej on blogs / comment sections
16:16 natarej haha yeah you really did
16:16 jp joined #gluster
16:19 jp Hello guys, I am trying to use GlusterFS (I have got 2 HDDs in one physicaly server), each HDD = 1 brick and I have got distributed volume with these 2 bricks. The volume is mounted via NFS and now I am testing performance. I am copying 1000 empty files via rsync to my mounted gluster volume, but the time is 30 - 35 sec. When I run the same direct
16:19 jp to disk (no to the mounted storage), time is 1-5 sec. Can you help me to improve GlusterFS system to performance tons of small files?
16:19 purpleidea JoeJulian: i was reading it, and at the end i was like damn, two cool guys beat him up about the nonsense... then i was like, oh, it's JoeJulian and purpleidea :P
16:19 jp I was read tons of manuals and tips, but ... its still too slow :/
16:20 natarej jp, gluster doesn't handle small files very well
16:20 ir8 morning guys.
16:21 jp natarej - yes I know, I read this is many discussions ... is here any other solution / or have you got some tips, how to build a storage for small files?
16:21 * JoeJulian hates the term "small files".
16:21 jp because gluster storage is really simple and looks good. its only too slow :(
16:21 CyrilPeponnet plain nfs is working better for us
16:21 JoeJulian It's slowly working its way in to becoming either a phobia, or a trigger to a psychotic break.
16:22 CyrilPeponnet for example git status on a repo with 60k files takes 30min with gluster backend, and few seconds without (either with nfs or native glusterfs mount)
16:23 CyrilPeponnet that's the only caveat of gluster
16:23 CyrilPeponnet IMPOV
16:23 jp yes, when I will gluster via fuse.gluster, its ok, the time fot that 1000 files is 8 sec, but speed for large files is under 10 MB/s (80 - 150 MB/s for NFS) and CPU is too loaded
16:24 jp *when I will mount gluster volume via fuse.gluster ....
16:25 semiosis jp: if you only have one server, perhaps mdadm or lvm would be better?
16:25 jp because I need some storage system for webhosting servers backup and I found gluster and its looks good ... but its Not applicable with this speeds
16:26 CyrilPeponnet web hosting and gluster is not a good idea
16:26 jp cyril - yep :/ lots of people says it
16:29 JoeJulian bullshit
16:30 JoeJulian I've hosted plenty of web on gluster. Your application is the problem.
16:32 natarej eg most PHP applications?
16:32 JoeJulian Some.
16:33 JoeJulian I've only encountered a few that I can't do anything about.
16:33 JoeJulian If they load plugins with open, you're screwed. If it's include/require, it's not that big of a deal.
16:34 jp I dont need gluster volume to host my files, I need it only for backup volume, so all files from production servers will be backuped here
16:34 jp but it means, that I need to copy milions of files (thousands and thousands of files every day)
16:34 jp and in this case is glusterfs really slow
16:34 JoeJulian If that's all it's for, I would probably use some sort of rsync/inotify tool. I'm sure there's dozens.
16:36 CyrilPeponnet @JoeJulian just putting /mnt/my_vol with glusterfs fuse containing 200 files takes a lot of time to display.
16:36 qubozik joined #gluster
16:37 CyrilPeponnet (the first time usually, afterward it hits the cache and get faster)
16:38 CyrilPeponnet no php just exposing the mount point through http
16:38 JoeJulian Sure, that's why most everyone who has high volume services preloads their caches.
16:38 CyrilPeponnet how to do that ?
16:38 JoeJulian Before you add that server to the load balancer, do a find on the mountpoint.
16:39 JoeJulian Or a wget mirror
16:39 CyrilPeponnet hmm this volume is 4TB with lot of subfolders
16:39 CyrilPeponnet and lot of files
16:39 JoeJulian Or maybe keep a log of the top N requested files and just preload those.
16:40 JoeJulian It's about strategy not tools.
16:40 CyrilPeponnet I was wondering if putting a squid or varnish caching proxy could help
16:40 JoeJulian Sure, know what all the tools are that are available to you. Think outside the box and apply the tools you have to the goals and see what fits best.
16:40 JoeJulian Even then, you'll probably want it preloaded.
16:41 CyrilPeponnet what size is the cache ?
16:41 JoeJulian Facebook has an hour long preload routine before their server clusters go into the load balancer.
16:42 CyrilPeponnet as a second question is it possible to do the caching on client side when using glusterfs-fuse
16:42 JoeJulian If you keep the files/directories open. When you close the fd, its resources are released.
16:43 JoeJulian The cache size... "gluster volume set help" shows the defaults.
16:43 JoeJulian I forget what it is off the top of my head.
16:43 JoeJulian @php
16:43 glusterbot JoeJulian: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
16:43 glusterbot JoeJulian: --fopen-keep-cache
16:43 CyrilPeponnet ok, so for example I have in the volume a bin folder with plenty of binary. This is added to PATH
16:43 JoeJulian Those mount flags in that factoid will also help.
16:44 CyrilPeponnet from time to time hitting tab hang because it takes time to parse the files on the volume
16:44 JoeJulian and HIGH is just some number.
16:44 stickyboy joined #gluster
16:44 CyrilPeponnet I was wondering if I could preload the bin in client cache to avoid to hit the backend each time
16:44 hagarth joined #gluster
16:45 JoeJulian If you know the filename that you're loading, it should be instant.
16:45 JoeJulian Like I said, "echo *"
16:45 JoeJulian It's instant because it doesn't read the attributes for every file.
16:45 CyrilPeponnet yes
16:46 JoeJulian /bin/ls will probably also be instant (not the "ls" alias that we usually use).
16:46 CyrilPeponnet yeah I know that the color thing make it way slower
16:47 JoeJulian So what's probably happening is the tools that exposes that directory is reading the attributes so it can also add decorations.
16:47 CyrilPeponnet btw gluster vol set help showing twice performance.cache-size one with default value at 32MB the other 128MB
16:48 JoeJulian Heh, I thought I filed that before... maybe I never got around to it. file a bug
16:48 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
16:48 natarej would consolidating storage to one brick per server reduce the replication overhead enough to get a good speed increase on small files?
16:49 CyrilPeponnet and by the way gluster vol heal info heal-failed is still showed in help but not available anymore
16:49 CyrilPeponnet :p
16:49 JoeJulian natarej: It will reduce hash misses if you try looking up files that don't exist a lot.
16:50 JoeJulian conference call in a few minutes. I'll be back later.
16:53 Rapture joined #gluster
17:01 haomaiwa_ joined #gluster
17:02 nangthang joined #gluster
17:07 qubozik joined #gluster
17:19 ir8 is there a to get a current config dump form gluster to save to written configs?
17:22 ro_ joined #gluster
17:28 harold joined #gluster
17:32 ro_ Hey guys - is there an easy/accepted way to age off data? We are looking into using gluster but want to age data off every 5 days.
17:32 ro_ alternatively I can script it, just didn't know if there was a way to configure gluster to do that by default
17:39 haomaiwa_ joined #gluster
17:41 qubozik joined #gluster
17:46 calavera joined #gluster
17:46 shyam joined #gluster
17:46 Trefex joined #gluster
17:53 cc1 joined #gluster
17:54 cc1 What would cause the following when starting georeplication? "Invalid slave name [Invalid argument]"
18:00 Pupeno joined #gluster
18:07 rotbeard joined #gluster
18:12 calisto joined #gluster
18:13 qubozik joined #gluster
18:17 qubozik joined #gluster
18:20 calavera joined #gluster
18:20 theron joined #gluster
18:22 jwd joined #gluster
18:22 jwaibel joined #gluster
18:31 atalur joined #gluster
18:42 jp joined #gluster
18:43 jp hello
18:43 glusterbot jp: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
18:43 jp guys, do you know any solution for small file cluster? GlusterFS is too slow for this operation
18:49 yawkat joined #gluster
18:50 yawkat hello! will gluster recognize files i manually copy into one of the bricks?
18:52 togdon joined #gluster
18:58 JoeJulian yawkat: no
18:59 yawkat hm okay thanks
18:59 JoeJulian yawkat: Consider it akin to dd'ing to some place on a hard drive and expecting xfs to know what to do with it.
18:59 yawkat yea it's reasonable, i just thought it might do some magic to allow that :P
19:00 JoeJulian cc1: Seen that before, never did find out why. We redid it with a different target that worked.
19:01 JoeJulian yawkat: You can, of course, mount the volume locally and write to that.
19:01 yawkat yea thats what im doing, it's just slower
19:02 cc1 JoeJulian: was a pebkac issue. I was used to the host:/path/to/brick syntax instead of slave::volume
19:02 JoeJulian If you're looking to cause inconsistency and see if gluster's heal is acceptable, just kill the remote brick.
19:03 JoeJulian You'll lose consistency but gain write speed.
19:03 JoeJulian The you can start the other brick and self-heal will fix it.
19:05 yawkat it's only one brick currently :P
19:07 lkoranda_ joined #gluster
19:08 marbu joined #gluster
19:10 cc1 Here's a theoretical. I setup geo-replication from data center 1 to data center 2. If we are going to make dc2 primary, in theory, i should be able to create a secondary geo-rep setup from dc2->dc1 and just start geo-rep when i cut over, shutting down geo-rep from dc1->dc2, right?
19:10 TSM joined #gluster
19:11 TSM can anyone point me to a gluster vs ceph, ie why you would use one over the other
19:12 JoeJulian TSM: I haven't seen one I would recommend and nobody has accepted my paper to speak about it so I haven't written it. :D
19:13 JoeJulian There are differences in performance, consistency, resiliency. One
19:13 JoeJulian is a filesystem, the other a block device.
19:13 JoeJulian Sure, cephfs exists, but it hasn't been blessed by sage as production ready yet.
19:14 TSM i can see that gluster only requires the servers ie bricks and you have raid on each server, ceph no raid just single osds and require seperate mon servers.
19:14 JoeJulian Also there's a whole aspect of "when it breaks" which makes gluster much more manageable than ceph.
19:15 TSM i read that gluster had problems with different size nodes, has that been sorted or does this need to be taken into consideration when the cluster is built?
19:15 JoeJulian We just had the raid discussion yesterday. Raid is *only* redhat's corporate recommendation which is not consistent with upstream.
19:15 TSM gluster seems to be far simpler as you still have access to the raw files on each brick if it was to all go titsup
19:16 JoeJulian That was my bit tipping point when I chose gluster all those years ago.
19:16 JoeJulian s/bit/big
19:16 TSM without raid are you pushing the idea that each disk is a seperate brick? or just use MD/lvm?
19:16 JoeJulian but i also use ceph
19:16 JoeJulian brick per disk
19:17 JoeJulian unless you need the speed that a single spindle isn't going to provide.
19:17 JoeJulian Then you may with to have replica 3 and raid0
19:17 JoeJulian Or not.
19:17 TSM so gluster can handle node/brick domains like ceph to prevent things to go on one node?
19:17 JoeJulian Depends on your use case.
19:17 JoeJulian @brick order
19:17 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
19:19 TSM for the most part we are an media library, lots of images/video, hi-res/thumbnails etc, ceph I like the idea of the RADOS S3 GW, I read that gluster will have that and if so may be a good plus point for me
19:19 JoeJulian #swiftonfile
19:21 JoeJulian radosgw doesn't handle tiering the same way as swift, but that can theoretically be managed with the crush map. If I was doing a cross-datacenter CDN I would use swift.
19:23 TSM teiring, you mean like performance/slow  teirs etc
19:24 TSM we dont really have need for that unless it self managed itself
19:26 JoeJulian If you're within a single datacenter, then radosgw will be perfectly adequate.
19:29 TSM I really need to build up a couple of test clusters and then figure out the best way to migrate from what we have now to where we need to go.
19:37 lkoranda joined #gluster
19:39 JoeJulian And test whether where your going does what you need. :)
19:46 plarsen joined #gluster
19:53 calavera joined #gluster
20:17 David_Varghese hello, i have 6 vm and replicate to all 6. is it that a bad practive to have gluster client on all 6 vm? im using it to LB web using haproxy
20:17 qubozik joined #gluster
20:21 cc1 left #gluster
20:22 qubozik_ joined #gluster
20:28 qubozik joined #gluster
20:33 togdon joined #gluster
20:35 dlambrig joined #gluster
20:40 shaunm joined #gluster
20:42 lkoranda joined #gluster
20:54 Pupeno joined #gluster
21:09 TSM joined #gluster
21:09 Pupeno joined #gluster
21:16 Pupeno joined #gluster
21:31 amye joined #gluster
21:35 calavera joined #gluster
21:39 Pupeno joined #gluster
21:46 shyam joined #gluster
22:02 amye left #gluster
22:35 dlambrig joined #gluster
22:57 calisto left #gluster
23:04 kkeithley joined #gluster
23:05 jockek joined #gluster
23:07 lezo joined #gluster
23:08 twisted` joined #gluster
23:09 amye joined #gluster
23:09 fyxim joined #gluster
23:09 billputer joined #gluster
23:20 zhangjn joined #gluster
23:24 chirino joined #gluster
23:27 zhangjn joined #gluster
23:28 neofob joined #gluster
23:42 zhangjn joined #gluster
23:44 Pupeno joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary