Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-06-08

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:38 ninkotech_ joined #gluster
00:39 ninkotech joined #gluster
00:46 David_H_Smith joined #gluster
00:49 David_H_Smith joined #gluster
01:12 Javezim Anyone know how to get a brick back online?
01:12 Javezim Brick cb-03:/data/gv0/brick4/brick      N/A       N/A        N       N/A
01:12 Javezim Brick cb-04:/data/gv0/brick4/brick      49157     0          Y       2212
01:21 shyam joined #gluster
01:48 ilbot3 joined #gluster
01:48 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:49 PatNarciso Javezim, have you tried restarting glusterd on cb-03?
01:49 Lee1092 joined #gluster
01:49 PatNarciso Javezim, also, it maybe a good idea to review the logs -- and identify why it went down.
01:51 Javezim Found it
01:52 Javezim Entire Brick folder is not found
01:52 Javezim d????????? ? ?    ?       ?            ? brick4
02:10 DV joined #gluster
02:34 arcolife joined #gluster
02:57 JoeJulian Ouch, that would do it.
03:04 RameshN joined #gluster
03:13 om joined #gluster
03:13 om2 joined #gluster
03:25 sage joined #gluster
03:28 nishanth joined #gluster
03:56 atinm joined #gluster
03:58 itisravi joined #gluster
04:00 nbalacha joined #gluster
04:04 shubhendu joined #gluster
04:04 aravindavk joined #gluster
04:05 RameshN joined #gluster
04:05 ppai joined #gluster
04:08 nehar joined #gluster
04:14 alghost joined #gluster
04:18 rafi joined #gluster
04:24 ramky joined #gluster
04:24 armyriad joined #gluster
04:34 aravindavk_ joined #gluster
04:34 skoduri joined #gluster
04:37 gowtham joined #gluster
04:46 raghug joined #gluster
04:48 Siavash joined #gluster
04:48 Siavash joined #gluster
04:50 aravindavk joined #gluster
04:59 kovshenin joined #gluster
05:07 hchiramm joined #gluster
05:07 gem joined #gluster
05:11 ashiq joined #gluster
05:14 Javezim Ran a xfs_repair and it seemed to do the trick
05:16 kotreshhr joined #gluster
05:18 Bhaskarakiran joined #gluster
05:19 jiffin joined #gluster
05:22 David_H_Smith joined #gluster
05:27 ndarshan joined #gluster
05:29 spalai joined #gluster
05:31 Apeksha joined #gluster
05:32 gowtham joined #gluster
05:39 Manikandan joined #gluster
05:44 aspandey joined #gluster
05:47 prasanth joined #gluster
05:49 hgowtham joined #gluster
05:51 kshlm joined #gluster
05:52 Saravanakmr joined #gluster
06:04 Gnomethrower joined #gluster
06:11 karnan joined #gluster
06:13 overclk joined #gluster
06:20 kdhananjay joined #gluster
06:21 om joined #gluster
06:21 om2 joined #gluster
06:24 kshlm joined #gluster
06:25 atalur joined #gluster
06:31 anil joined #gluster
06:34 pur__ joined #gluster
06:35 jtux joined #gluster
06:36 ppai joined #gluster
06:44 om2 joined #gluster
06:45 om3 joined #gluster
06:46 om3 left #gluster
06:56 Siavash joined #gluster
06:56 Siavash joined #gluster
06:58 RameshN joined #gluster
07:05 [Enrico] joined #gluster
07:11 deniszh joined #gluster
07:11 jri joined #gluster
07:14 harish joined #gluster
07:15 mbukatov joined #gluster
07:33 RameshN joined #gluster
07:42 rastar joined #gluster
07:44 ctria joined #gluster
07:49 hackman joined #gluster
07:54 jri joined #gluster
08:00 [Enrico] joined #gluster
08:04 ivan_rossi joined #gluster
08:07 om joined #gluster
08:08 ahino joined #gluster
08:08 om2 joined #gluster
08:08 om3 joined #gluster
08:11 om3 left #gluster
08:14 nix0ut1aw joined #gluster
08:15 nix0ut1aw joined #gluster
08:27 level7_ joined #gluster
08:33 Slashman joined #gluster
08:49 glafouille joined #gluster
08:52 jri joined #gluster
08:55 level7 joined #gluster
08:56 rafi joined #gluster
08:57 lanning joined #gluster
08:58 kshlm joined #gluster
09:03 ramky joined #gluster
09:06 poornimag joined #gluster
09:10 om2 joined #gluster
09:10 Saravanakmr joined #gluster
09:12 Gugge joined #gluster
09:12 kotreshhr joined #gluster
09:14 om3 joined #gluster
09:22 nishanth joined #gluster
09:23 ndarshan joined #gluster
09:23 karnan joined #gluster
09:29 ramky joined #gluster
09:37 Manikandan joined #gluster
09:44 kshlm joined #gluster
09:47 Saravanakmr joined #gluster
09:49 ivan_rossi left #gluster
09:52 Jules- joined #gluster
09:59 ramky joined #gluster
10:02 hchiramm joined #gluster
10:05 RameshN kshlm, Do you know from where I can download gluster master bits ? is there any nightly build available in download.gluter.org ?
10:27 Manikandan joined #gluster
10:31 kaushal_ joined #gluster
10:34 shubhendu joined #gluster
10:35 rastar joined #gluster
10:40 ndarshan joined #gluster
10:40 msvbhat_ joined #gluster
10:46 kotreshhr joined #gluster
10:46 atinm joined #gluster
10:47 aravindavk joined #gluster
10:47 nishanth joined #gluster
10:48 Saravanakmr joined #gluster
10:49 kshlm joined #gluster
10:54 nottc joined #gluster
11:01 msvbhat_ joined #gluster
11:26 atinm joined #gluster
11:31 DV joined #gluster
11:31 ramky joined #gluster
11:32 om joined #gluster
11:42 Debloper joined #gluster
11:42 micw joined #gluster
11:42 micw hi
11:42 glusterbot micw: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
11:44 micw can i easily set up 4 identical gluster nodes and tell it to have 3 copies of my files in the cluster and gluster manages automatically on which node my files are?
11:44 [Enrico] joined #gluster
11:46 jri joined #gluster
11:46 micw hm. the architecture doc says for replicated volume "Number of bricks must be a multiple of the replica count"
11:46 micw seems not to suport my use case
11:46 msvbhat_ joined #gluster
11:50 johnmilton joined #gluster
11:51 ndevos micw: you can have multiple bricks on each storage server
11:52 micw can they share a disk?
11:53 ndevos they can, we recommend to use LVM on the disks in any case, that makes it possible to do volume snapshots
11:53 ndevos the bricks can even be in the same filesystem, as long as they are in seperate directories
11:53 micw i'd use zfs in this case
11:54 ndevos and with multiple bricks on 4 servers, you can create something like a chained configuration
11:54 micw sounds a bit ... hacky ;-)
11:54 ndevos micw: zfs is local only right? it would not know how to coordinate creating snapshots of all the bricks to create a volume snapshot
11:55 micw how is it done with lvm? that's local as well
11:55 ndevos oh, it is a commonly used setup, there even is a blog post about is describing the details
11:56 ndevos gluster knows how to coordinate creating lvm snapshots, but it does not know about zfs
11:56 micw what i'd like: start with 3 servers, 8 TB each, giving me a storage of 8 tb with 3x redundancy. if i add a new node with 8 TB, i'd like to have my files re-distributed so that I still have 3x redundancy and ~2.6TB extra storage
11:57 micw next node brings additional 2.6tb, next again...
11:57 micw but what I've read let me guess, i always have to add 3 nodes at once
11:59 kshlm Weekly community meeting starting now in #gluster-meeting
12:00 ndevos you have to add 3 bricks at the same time, but yes, with such a setup adding a storage server is a little more tricky
12:01 micw ndevos, i'm doing some drawing to visualize it for me ;-)
12:02 micw i can start with 3 nodes (a,b,c), 1 brick each, replicated a1+b1+c1
12:02 micw if i add a node, a add a few more bricks:
12:03 micw distributed (replicated a1+b1+c1)+(replicated a2+b2+d2)+(replicated a3+c3+d3)+(replicated b4+c4+d4)
12:04 micw with next node i'd need one more brick on each
12:05 micw with next (6th) node, i could switch to distributed (replicated a1+b1+c1)+(replicated d1+e1+f1)
12:05 ndevos micw: I always try to visualize it like (a1+b1+c1) + (b2+c2+d2) + (c3+d3+a3) + (d4+a4+b4)
12:05 micw as I did (but i added info about replicated vs. distributed)
12:06 ndevos yes, it is the same, but I like the way to shift a brick on every iteration
12:07 micw ?
12:10 MrAbaddon joined #gluster
12:14 plarsen joined #gluster
12:18 ahino1 joined #gluster
12:26 karnan joined #gluster
12:42 spalai left #gluster
12:46 robb_nl joined #gluster
12:47 PaulCuzner joined #gluster
12:49 level7 joined #gluster
12:55 julim joined #gluster
12:57 rastar joined #gluster
12:57 jri joined #gluster
12:58 PaulCuzner joined #gluster
13:01 jbrooks joined #gluster
13:06 NuxRo joined #gluster
13:09 msvbhat_ joined #gluster
13:11 level7 joined #gluster
13:14 nishanth joined #gluster
13:19 Guest96239 joined #gluster
13:22 jiffin joined #gluster
13:28 Manikandan joined #gluster
13:29 jri_ joined #gluster
13:37 MrAbaddon joined #gluster
13:40 nbalacha joined #gluster
13:41 hagarth joined #gluster
13:57 dgandhi joined #gluster
13:57 MrAbaddon joined #gluster
13:59 dgandhi joined #gluster
14:01 dgandhi joined #gluster
14:02 dgandhi joined #gluster
14:03 ben453 joined #gluster
14:05 arcolife joined #gluster
14:06 ben453 Has anyone ever tried to do a replace-brick on a replicate only volume using a non-empty directory as the new brick? I'm wondering if I can replace a brick without having to have gluster heal/copy all of the data to the new brick.
14:08 squizzi joined #gluster
14:10 Gnomethrower joined #gluster
14:14 shyam joined #gluster
14:15 squizzi joined #gluster
14:15 Siavash joined #gluster
14:15 Siavash joined #gluster
14:17 MrAbaddon joined #gluster
14:21 post-factum JoeJulian: ^^
14:22 level7 joined #gluster
14:23 ahino joined #gluster
14:26 morgan joined #gluster
14:29 nishanth joined #gluster
14:39 monotek joined #gluster
14:40 julim joined #gluster
14:44 dnunez joined #gluster
14:51 wushudoin joined #gluster
14:52 wushudoin joined #gluster
14:56 msvbhat_ joined #gluster
15:00 hagarth joined #gluster
15:03 monotek joined #gluster
15:07 sage joined #gluster
15:08 kpease joined #gluster
15:09 dgandhi joined #gluster
15:12 shubhendu joined #gluster
15:13 arcolife joined #gluster
15:16 karnan joined #gluster
15:17 nishanth joined #gluster
15:18 ikla joined #gluster
15:18 ikla Does gluster work well for replication between two hosts for high-availability?
15:20 monotek joined #gluster
15:20 JoeJulian ikla: gluster is a clustered filesystem, not a synchronization service. My advice is to try to alter your mindset and re-analize what that means to your use case.
15:20 julim joined #gluster
15:20 ikla JoeJulian, it does mirroring right?
15:21 JoeJulian Not in the way you're thinking, no.
15:21 julim joined #gluster
15:21 shyam joined #gluster
15:22 ikla thanks
15:22 ikla Do you recommend anything out there?
15:22 JoeJulian It does replication, but it's for itself. The bricks are not "yours". Once assigned, they're used for gluster's storage. You access a gluster volume through a client. That client presents the same filesystem to as many clients as you may need.
15:22 julim joined #gluster
15:23 JoeJulian So it *is* HA, and may work for your needs.
15:23 Guest96239 joined #gluster
15:26 squizzi joined #gluster
15:31 haomaiwang joined #gluster
15:33 ben453 I asked this earlier, does anyone know the answer: can you do a replace-brick on a replicate only volume using a non-empty directory as the new brick? I'm wondering if I can replace a brick without having to have gluster heal/copy all of the data to the new brick.
15:35 ben453 Or is an empty directory for the new brick required since gluster has to re-build all of its metadata on the new brick?
15:39 nbalacha joined #gluster
15:41 JoeJulian ben453: I've not had success with that in the past and have always gone with the empty brick ever since.
15:41 F2Knight joined #gluster
15:42 julim joined #gluster
15:44 julim joined #gluster
15:46 julim joined #gluster
15:46 ben453 @JoeJulian thanks for the info
15:47 DV joined #gluster
15:47 julim joined #gluster
15:49 monotek1 joined #gluster
15:59 nishanth joined #gluster
16:01 sage joined #gluster
16:02 shyam joined #gluster
16:11 julim joined #gluster
16:13 julim joined #gluster
16:24 squizzi joined #gluster
16:26 dlambrig joined #gluster
16:28 Guest96239 joined #gluster
16:29 gowtham joined #gluster
16:43 monotek1 joined #gluster
16:44 Siavash joined #gluster
16:48 ahino joined #gluster
17:06 monotek joined #gluster
17:11 dnunez joined #gluster
17:12 dlambrig joined #gluster
17:18 RameshN joined #gluster
17:20 gowtham joined #gluster
17:30 dlambrig joined #gluster
17:32 dnunez joined #gluster
17:48 hagarth joined #gluster
17:48 jri joined #gluster
17:49 rwheeler joined #gluster
17:49 cliluw joined #gluster
17:57 PaulCuzner joined #gluster
18:05 arcolife joined #gluster
18:05 Elmo_ joined #gluster
18:06 Elmo_ Hi there, don't want to get into a debate or anything. I looked everywhere and can't seem to find a clear distinction between GlusterFS and Ceph (with CephFS lets say)
18:07 JoeJulian Elmo_: Ceph is a clustered block storage design where cephfs is an object store with an abstracted name space on top of it.
18:08 JoeJulian I haven't seen any directo performance comparisons, but in ceph block storage vs gluster file storage hosting VM images, gluster performed significantly better.
18:08 JoeJulian ceph automatically handles outages better, mostly.
18:09 Elmo_ I see, so for stability, Ceph is better, for performance, GlusterFS is the way to go?
18:10 JoeJulian No
18:10 JoeJulian If you're using replica 3 (each one recommends 3 replica) you're about even for stability.
18:11 JoeJulian The only thing ceph block storage has over gluster is the zero-copy cloning. You want to spin up 100 vms with the same image, it's instant. With gluster you have to actually copy stuff.
18:11 Elmo_ OK, so I was already leaning towards GlusterFS but thats the only thing I was wondering. I got confused tons especially since even Red Hat does not properly explain their two offerings.
18:12 JoeJulian imho, I like gluster better. We just had a failure with a ceph cluster - not ceph's fault at all, it's all a hardware failure.
18:12 Elmo_ Because of Gluster's hashing algorithm?
18:12 JoeJulian With gluster, I could have pulled whole files off of the bricks that make up the cluster and saved the data.
18:13 JoeJulian With ceph, I'd have had to figure out which osds held the blocks I needed and stiched them together.
18:13 JoeJulian All the data was lost.
18:13 Elmo_ So, let me turn that question the other way then ;-) What is better in Ceph than GlusterFS?
18:13 Elmo_ Because beforehand like you say, GlusterFS seems lot better
18:13 Elmo_ easier to configure also (I already did it to test some stuff)
18:14 JoeJulian It's popular and you look cool using it.
18:15 Elmo_ Hahahaha
18:15 JoeJulian I've got nothing bad to say about ceph. It's a good tool with some really smart people building it.
18:15 Elmo_ Any real world usage of GlusterFS you might think of? Like big companies using it?
18:16 JoeJulian I've just found, for most offerings that our company wants to provide, gluster either fits better or fits as well. By standardizing we simplify our technical asset requirements and ease our hiring needs.
18:17 Elmo_ Funny story, I read a bit on you Joe (I was curious)
18:17 Elmo_ ends up that I looked into GlusterFS for the same reason you started implementing it
18:17 JoeJulian Real world: facebook, a huge handfull of big telcos, disney, cern...
18:17 Elmo_ DRBD issues :P
18:17 JoeJulian ugh
18:18 Elmo_ I thought Facebook were using HDFS
18:18 JoeJulian They use a lot of stuff, but they do have a gluster (internal) offering as well that gets a lot of use.
18:19 Elmo_ I was looking online for such a list of "current customers" yet I couldn't find it
18:20 Elmo_ If I was in the GlusterFS crew, I would push on that :-)
18:21 Elmo_ So, Ceph stuff is harder to retrieve upon failures because of OSDs, harder to configure, harder to maintain (it seems)
18:21 JoeJulian Meh, maintenance isn't all that bad.
18:21 Elmo_ From the look on the docs of it also, it seems a very powerful toolset yet more targeted for engineers in grid research
18:21 JoeJulian And it will automatically replicate across racks or rows if you so configure it.
18:22 Elmo_ My application is rather small scale at the moment though
18:22 Elmo_ Basically just needs file system based replication
18:22 Elmo_ more scallable than DRBD (which isn't.. at all)
18:22 Elmo_ and, very reliable
18:23 Elmo_ (I know, DRBD isn't file replication, but has its share of issues ;-))
18:23 JoeJulian "very reliable" is a number. 99.9%? 99.99%?
18:24 JoeJulian That number is calculated using the MTBF and the MTTR of your components, including software.
18:25 Elmo_ I mean that lost of data should not happen because of the underlying tech
18:25 JoeJulian With a replica 3 volume, your typical MTTR of a complete server failure (goes up in smoke) is 42 seconds if you count recovery as data availability.
18:25 JoeJulian If the DC floods, there are no guarantees.
18:25 Elmo_ let me google MTTR and MTBF ;-)
18:26 JoeJulian Mean Time To Recovery and Mean Time Between Failure
18:26 Elmo_ Ha thanks :-)
18:26 JoeJulian For instance: ,,(ping-timeout)
18:26 glusterbot instance: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. With an average MTBF of 45000 hours for a server, even just a replica 2 would result in a 42 second MTTR every 2.6 years, or 6 nines of uptime.
18:26 glusterbot The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. With an average MTBF of 45000 hours for a server, even just a replica 2 would result in a 42 second MTTR every 2.6 years, or 6 nines of uptime.
18:26 * JoeJulian whacks glusterbot
18:27 JoeJulian I still recommend replica 3 to help prevent split-brain.
18:27 Elmo_ Thats when losing 1 of the 3 bricks in the replica?
18:27 JoeJulian Though replica 2 with an arbitor is the next best thing.
18:29 Elmo_ that 42 seconds
18:29 Elmo_ when can that happen?
18:29 JoeJulian We get focused on eliminating worst-case scenarios, but most service level agreements (SLA) only require a percentage of uptime per year or month.
18:29 Elmo_ not when only 1 brick falls right?
18:30 JoeJulian When a server fails without having a chance to close the TCP connection, the clients will wait for the ping-timeout to expire before marking that server's bricks as failed.
18:30 Elmo_ Ha! Hear you
18:30 hagarth joined #gluster
18:31 Elmo_ Thats the worst case of course right?
18:31 JoeJulian Right
18:31 Elmo_ That TCP connection is opened only when the client tries to open an FD
18:31 JoeJulian It's open at mount, and stays open until unmount.
18:31 Elmo_ if the server was taken down before that, no delay
18:31 Elmo_ ha I seee
18:32 JoeJulian If the server is shut down, it closes the tcp connection
18:32 Elmo_ Ha understood
18:32 Elmo_ so thats really worst of worst cases like hard failures (which is most of the time the failure that will occur)
18:32 Elmo_ Question though
18:32 Elmo_ (sorry for asking so many)
18:33 Elmo_ is there a monitoring service available also in GlusterFS? Aside from manually doing "gluster peer probe" or "gluster peer status"
18:33 Elmo_ and.. is there something like that on client side?
18:34 JoeJulian Not really, no.
18:34 JoeJulian People generally use nagios
18:34 Elmo_ nagios with "gluster peer status" sorta?
18:35 JoeJulian @nagios
18:35 JoeJulian damn
18:35 JoeJulian I know that people have made gluster plugins for nagios.
18:35 Elmo_ ha ok nice
18:36 Elmo_ You've been quite useful Joe. :-)
18:36 Elmo_ Thanks a lot
18:37 JoeJulian You're welcome.
18:39 Elmo_ @sage, are you the real Sage Weil?
18:39 Elmo_ like.. the Ceph guy
18:40 JoeJulian That would be him. I think he should be asleep right now though.
18:41 Elmo_ So both the Ceph and GlusterFS community are walking hand in hand?
18:41 Elmo_ How come there is no "merge" between both
18:41 JoeJulian Sure, we share a lot of the same goals.
18:41 Elmo_ like
18:41 Elmo_ an Hybrid monster :-)
18:42 JoeJulian They're different approaches to the same problem but they do it in completely different ways.
18:42 Elmo_ Supporting Object Storage and File System storage all in one solution
18:42 Elmo_ hear you
18:42 JoeJulian But there totally could be a gluster translator to use an rbd backend. It's possible.
18:43 Elmo_ That would drop the need for CephFS would it
18:47 JoeJulian It could, but it would then need a posix filesystem translator which would be a huge undertaking that I can't imagine anyone would want to do.
18:48 post-factum Elmo_: personally me (throw some stones in my garden) considers ceph stack to be more considered
18:48 post-factum Elmo_: i enjoy the way the logic of ceph architecture
18:49 post-factum Elmo_: but there are lots of "but"
18:50 Elmo_ post-factum: keep me honest, but they way I see it from a high level, is the way Ceph does replication looks more like DRBD in that its at lower level than GlusterFS does
18:51 post-factum Elmo_: with its architecture ceph solves many tasks naturally
18:52 post-factum Elmo_: if you take some look at further glusterfs development, some parts of ceph come here as well
18:52 post-factum Elmo_: like journal based replication
18:53 BitByteNybble110 joined #gluster
18:53 post-factum Elmo_: the only weak place of ceph is cephfs backups
18:53 post-factum Elmo_: that is why i use gluster for file storage
18:53 Elmo_ is CephFS still not production ready?
18:53 Elmo_ last status, it wasn't
18:53 post-factum last status, it is
18:53 Elmo_ That scared me a bit
18:53 JoeJulian They announced a couple months ago that it is.
18:53 post-factum but there may be dragons
18:53 Elmo_ lol dragons
18:54 Elmo_ The reason I ask is that the component that will use the storage, will use it to store very important data
18:54 Elmo_ And, I can't really afford to lose data because of, lets say, the "CephFS" layer on top of Ceph
18:55 Elmo_ thus, I was leaning towards GlusterFS but yet, I want the best possible solution :-)
18:55 post-factum "best" is pretty flexible definition without certain requirements
18:55 Elmo_ Main requirements are
18:56 JoeJulian imho, if you're afraid of any data loss ever, use gluster. It (normally) stores whole files on disks. If the worst happens, you still have disks with files that can be copied.
18:56 Elmo_ High performance (for runtime reads), high availablility for low failures and ease of use/maintenance for operatores
18:56 Elmo_ *operators
18:57 post-factum JoeJulian++
18:57 glusterbot post-factum: JoeJulian's karma is now 27
18:57 post-factum but
18:57 JoeJulian And if a critical disk fails, you still have the media that can be sent to a clean room for recovery.
18:57 post-factum what is "high performance"?
18:57 Elmo_ post-factum: low latency
18:57 Elmo_ vs other network based storage
18:57 JoeJulian Infiniband RDMA
18:58 Elmo_ like DRBD
18:58 post-factum low latency is the requirement for any network storage and not the consequence
18:58 JoeJulian amen
18:58 post-factum :D
18:59 Elmo_ ok rephraseing that then
18:59 Elmo_ :P
18:59 JoeJulian Engineers start with the product requirements and build a system to meet or exceed them.
18:59 jiffin1 joined #gluster
18:59 post-factum Elmo_: you should consider the type of data you would like to store as well as the required preformance in terms of concurrent access, throughput etc
18:59 Elmo_ Not adding latency because of GlusterFS or Ceph i meant
18:59 post-factum JoeJulian: should i ++ you again or that is enough?
18:59 JoeJulian hehe
19:00 post-factum Elmo_: every intermediator will increase your latency
19:00 JoeJulian post-factum++
19:00 glusterbot JoeJulian: post-factum's karma is now 9
19:00 Elmo_ rather small files (couple megabytes), not much concurrent access rather small thourghput also
19:00 post-factum (yes, with ceph kernel client it is lower, but ceph kernel client is features-weak)
19:00 muneerse joined #gluster
19:01 JoeJulian Not necessarily. If you can program against the api, you can do it all* in userspace to avoid context switches.
19:01 post-factum Elmo_: sounds not so high-load
19:01 Elmo_ post-factum: No not very high load
19:01 JoeJulian * except for the network, unless you can use a userspace network driver.
19:01 Elmo_ post-factum: yet, a single request needs to be very fast
19:02 JoeJulian Then it should be precached in ram.
19:02 post-factum Elmo_: for fast requests you should consider building caching infrastructure
19:02 post-factum hm
19:02 post-factum same
19:02 post-factum ok
19:02 Elmo_ post-factum: real-time usage (well.. not microseconds fast, but can't afford multiple ms delay for a read)
19:02 post-factum Elmo_: what is the type of those data?
19:03 Elmo_ zipped files
19:03 post-factum and why one would want fast access to them?
19:03 Elmo_ real-time data processing
19:03 Elmo_ that needs those zipped files beforehand
19:04 post-factum Elmo_: network introduces the greatest part of general latency, so consider building low-latency backbone first
19:04 post-factum Elmo_: like infiniband as JoeJulian mentioned before
19:05 JoeJulian Sounds like a financial sector application.
19:05 post-factum hadoop then ;)
19:05 Elmo_ nope not financial
19:05 Elmo_ annnnnd I can't say more info ;-)
19:05 Elmo_ haha
19:06 JoeJulian Damn... guess I'll have to keep guessing. ;)
19:06 Elmo_ Hadoop is a bit overkill for my use case
19:06 Elmo_ JoeJulian: haha yes
19:07 Elmo_ so, you're saying that, both GlusterFS and Ceph will suffer from similar "latencies"
19:07 Elmo_ or... not really
19:07 JoeJulian LOL! OH ‘have you considered Nagios?’ (As a baby name) ‘Nagios kept me up all night’ ‘Nagios shit the bed again’
19:07 Elmo_ just that both are
19:07 post-factum JoeJulian: that is how i feel myself when our zabbix setup consumes another additional gigabyte of RAM
19:08 JoeJulian I would say they might be fine. Many potential latencies can be avoided, especially if you have control of the software. IMHO, build a PoC and test.
19:08 post-factum Elmo_: ok, to continue constructive dialogue... the first component is low-latency network
19:09 Elmo_ post-factum: makes sense
19:09 post-factum Elmo_: the second component is the type of storage. do you really need posix storage?
19:10 post-factum Elmo_: i mean, should it be like any other fs, or it could have, saying, S3 interface?
19:10 Elmo_ post-factum: Yeah... kinda need Posix storage
19:10 Elmo_ else, I would need to heavily temper the component itself
19:11 post-factum Elmo_: ok. then a few words about placement. would you like to have your data replicated?
19:11 Elmo_ yes, one of the main reason I would use GlusterFS/Ceph
19:11 Elmo_ :)
19:11 post-factum how many replicas you would like to have?
19:12 PaulCuzner joined #gluster
19:12 Elmo_ At first, I though 2. But then 3 seemed like a better number from the discussions here
19:12 Elmo_ + geo-replication later..
19:12 post-factum between 2 and 3, and georeplication there is a precipice
19:13 Elmo_ post-factum: curious, how come?
19:13 post-factum in glusterfs, each additional replica multiplies your *client* traffic
19:13 Elmo_ thus have an impact on performance, I hear ya
19:14 post-factum also, there are questions regarding high availability (in terms of meeting quorum)
19:14 Elmo_ geo-replication is async though right?
19:14 post-factum yep
19:14 Elmo_ replicas are sync
19:14 post-factum yep
19:14 Elmo_ I see
19:14 Elmo_ OK so 2 replicas 2!!
19:14 Elmo_ :P
19:15 post-factum in order to make things as reliable in gluster as in ceph with replica 2 you need another arbiter replica
19:16 Elmo_ arbiter replica?
19:16 Elmo_ looks fancy
19:16 post-factum aye
19:16 Elmo_ What is it :-)
19:16 post-factum also, pay an attention that cephfs has dedicated metadata server which speeds things up
19:16 post-factum arbiter node keeps metadata without data
19:16 post-factum preventing split-brains
19:17 post-factum @arbiter
19:17 post-factum hmm
19:17 Elmo_ I'll google it ;-)
19:17 Elmo_ IRC bots :P
19:17 post-factum https://gluster.readthedocs.io/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/
19:17 glusterbot Title: Arbiter volumes and quorum options - Gluster Docs (at gluster.readthedocs.io)
19:18 JoeJulian feel free to @learn <keyword> as <content>
19:18 post-factum i thought that is your business ;))
19:18 Elmo_ OK so back to Ceph... any real world usage also like previously mentionned by @JoeJulian
19:18 JoeJulian It's a community bot.
19:19 post-factum Elmo_: i use ceph for rbd only to store VM images
19:19 post-factum also, in ceph you need to maintain the majority of monitors only, and configure replicas independently
19:20 post-factum that is why in ceph you could go with replica 2 and do not fear of split-brain
19:20 post-factum keeping several metadata servers as well
19:20 JoeJulian But they won't support replica 2
19:21 post-factum JoeJulian: what do you mean?
19:21 JoeJulian I mean you cannot purchase a support contract for a pool size < 3.
19:21 post-factum hmmm
19:21 post-factum who needs support contract if there is #gluster
19:22 post-factum or #ceph
19:22 Elmo_ hahaha well
19:23 Elmo_ because post-factum and JoeJulian aren't always there I guess
19:23 Elmo_ ;-)
19:23 post-factum Elmo_: do you really need commercial support?
19:24 JoeJulian Let me turn that around, do you want to ensure you can never get commercial support?
19:24 post-factum :D
19:24 Elmo_ post-factum: nah not really ;-)
19:24 JoeJulian I like keeping my options open.
19:24 Elmo_ JoeJulian: amen
19:26 post-factum Elmo_: otherwise you'd buy nutanix and do not care
19:27 post-factum Elmo_: ok, i guess at this step you need to refine your requirements and possibilities
19:28 level7 joined #gluster
19:29 post-factum Elmo_: including networking and datacenters placement
19:32 johnmilton joined #gluster
19:33 Elmo_ post-factum: meaning?
19:33 Elmo_ post-factum: hardware change for networking?
19:33 post-factum Elmo_: sure
19:34 post-factum Elmo_: or you would go with existing networking? then, what do you have there?
19:34 Elmo_ post-factum: but.. typical 10GE shall handle it OK I guess
19:35 Elmo_ 10GE links between hosts
19:36 post-factum and those ppl talk about low latencies
19:36 post-factum yep, 10GbE is a satisfactory minimum
19:37 Elmo_ post-factum: "satisfactory minimum"? really? lol
19:37 Elmo_ so I think.. network isnt an issue here
19:38 post-factum Elmo_: lets say like this: if i have 2 nodes with 0.5ms RTT between them and add arbiter node with 1ms RTT, my IOPSes decrese with a factor of 2
19:38 post-factum *decrease
19:38 post-factum that is the most fun thing about network storage
19:39 kenansulayman joined #gluster
19:40 post-factum i also tested arbiter with rtt of 5ms
19:40 post-factum do not ask me how it behaved
19:40 Elmo_ post-factum: ...
19:40 Elmo_ post-factum: how did it behave?
19:40 Elmo_ ;-)
19:40 post-factum soooo sloooow
19:40 post-factum but still usable
19:40 post-factum (i'm talking about writing)
19:40 Elmo_ post-factum: that bad?
19:41 nathwill joined #gluster
19:41 post-factum if you need fast reading, that makes little sense to you
19:41 Elmo_ yes, writing isn't such an issue for me
19:41 Elmo_ reading is
19:42 Elmo_ so.. a geo-replicated gluster cluster
19:42 Elmo_ on the passive cluster
19:42 Elmo_ its "read only" I assume?
19:43 post-factum never used georep in fact ;)
19:43 post-factum had no need
19:43 Elmo_ I Gluster brilliant enough to redirect traffic to primary cluster if a write is done?
19:43 Siavash joined #gluster
19:43 Siavash joined #gluster
19:43 post-factum JoeJulian: any georep experience
19:43 Elmo_ OK no problem, this might come later anyway
19:43 post-factum JoeJulian: ?
19:44 JoeJulian geo-rep is unidirectional.
19:45 Elmo_ so.. whats the use case?
19:45 Elmo_ only for backup purpose?
19:45 Elmo_ or.. you may read from the secondary cluster
19:46 Elmo_ aka. GlusterFS being clever enough to discover its geo-based-replicants (if I can call them like that)
19:46 post-factum Elmo_: to mirror things between distant DCs
19:47 post-factum https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Geo%20Replication/
19:47 glusterbot Title: Geo Replication - Gluster Docs (at gluster.readthedocs.io)
19:48 Elmo_ post-factum: oh
19:48 Elmo_ post-factum: I think I was thinking more about, keeping latency down on geodistant locations
19:48 Elmo_ like same component
19:49 Elmo_ 2 sites, far from each
19:49 Elmo_ same storage content
19:50 post-factum i believe those are 2 separate clusters, but geosynchronized
19:50 post-factum with no auto switching
19:50 Elmo_ thats also possible with GlusterFS?
19:52 post-factum i talked about glusterfs...
19:52 Elmo_ geosync != geo-replication no?
19:53 post-factum emmm
19:53 post-factum gluster term is "geo-replication"
19:53 Elmo_ I meant, can someone in the slave cluster
19:53 Elmo_ access the Gluster Volume
19:53 Elmo_ with lower latencies?
19:53 Elmo_ read-only
19:54 Elmo_ I get this is Master/Slave config
19:54 Elmo_ for clusters
19:54 Elmo_ lets say Master is in USA
19:54 Elmo_ Slave in China
19:54 Elmo_ then component in China wants to access the gluster volume
19:54 post-factum i believe, with read-only — yes
19:54 dlambrig joined #gluster
19:54 Elmo_ is Gluster clever enough to fetch files from China bricks?
19:54 post-factum but you would like to cast someone more experienced than me in georep
19:55 Elmo_ np :-)
19:55 post-factum i guess, client shoud explicitly use china cluster
19:55 post-factum no autoswitch there
19:55 Elmo_ is that possible?
19:56 ndevos Elmo_: geo-replication uses a master volume and a slave volume, clients can mount either the master (rw) or the slave (ro)
19:56 Elmo_ Ha!
19:56 Elmo_ ndevos: You are my savior
19:56 Elmo_ That explains it then
19:56 ndevos Elmo_: ah, great!
19:57 Elmo_ ndevos: Soooo.. I would need to mount 2 volumes basically
19:57 Elmo_ for my component
19:57 Elmo_ one for Write purpose
19:57 Elmo_ (might be geographically far)
19:57 Elmo_ and one for read
19:57 Elmo_ (geo-replicated)
19:58 ndevos yes, indeed, and the slave will get updated with some delay
19:58 Elmo_ Makes sense :-)
19:58 ndevos some users setup a master and a slave on both sides, and setup geo-replication both ways
19:59 dlambrig joined #gluster
19:59 Elmo_ I need to think and design around that, but that opens up bunch of ideas in my mind :-)
19:59 ndevos cool, share some results in the mailinglist or a blog post if you can :)
20:11 dlambrig joined #gluster
20:11 DV joined #gluster
20:24 F2Knight joined #gluster
20:26 johnmilton joined #gluster
20:40 hackman joined #gluster
20:43 johnmilton joined #gluster
20:52 guhcampos joined #gluster
20:58 dlambrig joined #gluster
21:00 level7 joined #gluster
22:08 DV joined #gluster
22:13 chirino joined #gluster
22:27 jockek joined #gluster
23:16 bfoster joined #gluster
23:43 nathwill joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary