Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-03-14

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 Merlin_ joined #gluster
00:17 JoeJulian Nothing's exactly chatty on the weekends.
00:18 ovaistariq joined #gluster
00:27 haomaiwa_ joined #gluster
00:35 ninkotech joined #gluster
00:36 ninkotech_ joined #gluster
00:41 ference joined #gluster
00:47 plarsen joined #gluster
01:31 DV joined #gluster
01:37 elitecoder left #gluster
01:39 kevein joined #gluster
01:39 auzty joined #gluster
01:48 ovaistariq joined #gluster
01:55 baojg joined #gluster
02:04 haomaiwa_ joined #gluster
02:10 vmallika joined #gluster
02:27 ovaistariq joined #gluster
02:31 johnmilton joined #gluster
02:47 ilbot3 joined #gluster
02:47 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:49 kdhananjay joined #gluster
03:01 haomaiwa_ joined #gluster
03:02 pgreg joined #gluster
03:24 Gaurav__ joined #gluster
03:26 lord4163 joined #gluster
03:43 itisravi joined #gluster
03:43 nbalacha joined #gluster
03:47 shubhendu joined #gluster
03:49 atalur joined #gluster
03:50 Lee1092 joined #gluster
03:51 atinm joined #gluster
03:54 ramteid joined #gluster
03:58 nbalacha joined #gluster
04:01 haomaiwa_ joined #gluster
04:03 kanagaraj joined #gluster
04:07 nehar joined #gluster
04:20 Gnomethrower joined #gluster
04:20 swebb joined #gluster
04:22 RameshN joined #gluster
04:23 gem joined #gluster
04:25 sakshi joined #gluster
04:31 kshlm joined #gluster
04:32 jiffin joined #gluster
04:33 ppai joined #gluster
04:35 Gnomethrower joined #gluster
04:38 dlambrig_ joined #gluster
04:53 baojg joined #gluster
04:53 Manikandan joined #gluster
04:53 ovaistariq joined #gluster
04:54 atalur joined #gluster
05:01 haomaiwa_ joined #gluster
05:03 ndarshan joined #gluster
05:09 karthikfff joined #gluster
05:16 hgowtham joined #gluster
05:18 haomaiwang joined #gluster
05:19 ashiq joined #gluster
05:19 pur joined #gluster
05:21 poornimag joined #gluster
05:29 Apeksha joined #gluster
05:30 sakshi joined #gluster
05:31 EinstCrazy joined #gluster
05:32 ahino joined #gluster
05:33 rafi joined #gluster
05:36 foster joined #gluster
05:36 skoduri joined #gluster
05:38 atalur joined #gluster
05:45 Bhaskarakiran joined #gluster
05:45 gowtham joined #gluster
05:46 anil joined #gluster
05:46 karnan joined #gluster
05:49 Merlin_ joined #gluster
05:49 overclk joined #gluster
05:51 ramky joined #gluster
05:52 haomaiwa_ joined #gluster
05:52 Merlin_ joined #gluster
05:52 ahino joined #gluster
06:01 R0ok_ joined #gluster
06:01 haomaiwang joined #gluster
06:05 atalur joined #gluster
06:06 Merlin_ joined #gluster
06:09 gowtham joined #gluster
06:10 jiffin1 joined #gluster
06:14 dannyb joined #gluster
06:16 nishanth joined #gluster
06:17 aravindavk joined #gluster
06:23 Merlin_ joined #gluster
06:31 kotreshhr joined #gluster
06:32 Merlin_ joined #gluster
06:34 aravindavk joined #gluster
06:35 atalur joined #gluster
06:35 pgreg joined #gluster
06:36 shruti joined #gluster
06:40 swebb joined #gluster
06:42 prasanth joined #gluster
06:42 Merlin_ joined #gluster
06:42 ggarg joined #gluster
06:42 Saravanakmr joined #gluster
06:43 aravindavk joined #gluster
06:51 Merlin_ joined #gluster
06:54 ovaistariq joined #gluster
07:01 skoduri joined #gluster
07:01 haomaiwang joined #gluster
07:04 Merlin_ joined #gluster
07:05 klaas_ joined #gluster
07:05 shortdudey123_ joined #gluster
07:11 mhulsman joined #gluster
07:14 arcolife joined #gluster
07:14 Champi_ joined #gluster
07:14 marlinc_ joined #gluster
07:15 eryc joined #gluster
07:15 eryc joined #gluster
07:16 sakshi joined #gluster
07:16 gem joined #gluster
07:16 vmallika joined #gluster
07:16 juhaj joined #gluster
07:16 jakobs joined #gluster
07:16 dabukalam joined #gluster
07:16 delhage joined #gluster
07:16 Ramereth joined #gluster
07:16 xMopxShell joined #gluster
07:16 Bardack joined #gluster
07:16 nhayashi joined #gluster
07:16 ndevos joined #gluster
07:17 morse joined #gluster
07:17 lkoranda joined #gluster
07:24 vmallika joined #gluster
07:26 itisravi joined #gluster
07:27 mhulsman joined #gluster
07:27 hchiramm joined #gluster
07:28 jiffin1 joined #gluster
07:30 aravindavk joined #gluster
07:35 itisravi joined #gluster
07:36 robb_nl joined #gluster
07:53 [o__o] joined #gluster
07:54 hchiramm joined #gluster
07:56 [diablo] joined #gluster
08:01 haomaiwa_ joined #gluster
08:02 Wizek joined #gluster
08:05 baojg joined #gluster
08:06 jri joined #gluster
08:08 ivan_rossi joined #gluster
08:09 Merlin_ joined #gluster
08:15 deniszh joined #gluster
08:18 mbukatov joined #gluster
08:18 swebb joined #gluster
08:25 [Enrico] joined #gluster
08:28 ahino joined #gluster
08:28 skoduri joined #gluster
08:35 fsimonce joined #gluster
08:41 nix0ut1aw joined #gluster
08:41 ference joined #gluster
08:43 nishanth joined #gluster
08:44 nix0ut1aw joined #gluster
08:46 msvbhat joined #gluster
08:48 ramky joined #gluster
08:52 atalur joined #gluster
08:53 arcolife joined #gluster
08:54 robb_nl joined #gluster
09:06 hchiramm_ joined #gluster
09:06 haomaiwa_ joined #gluster
09:14 Slashman joined #gluster
09:32 Wizek_ joined #gluster
09:37 ctria joined #gluster
09:42 ctria joined #gluster
09:49 Manikandan joined #gluster
09:53 ference joined #gluster
10:00 itisravi joined #gluster
10:01 haomaiwa_ joined #gluster
10:05 ashiq joined #gluster
10:09 hchiramm_ joined #gluster
10:09 kkeithley1 joined #gluster
10:11 hchiramm joined #gluster
10:19 atalur joined #gluster
10:20 hchiramm_ joined #gluster
10:26 muneerse joined #gluster
10:26 haomaiwa_ joined #gluster
10:27 lkoranda joined #gluster
10:27 lkoranda joined #gluster
10:31 bfoster joined #gluster
10:33 haomaiwa_ joined #gluster
10:39 hgowtham joined #gluster
10:51 johnmilton joined #gluster
10:51 muneerse joined #gluster
10:56 ovaistariq joined #gluster
11:01 haomaiwa_ joined #gluster
11:02 hgowtham joined #gluster
11:05 Manikandan joined #gluster
11:06 Merlin_ joined #gluster
11:09 hchiramm_ joined #gluster
11:12 Merlin_ joined #gluster
11:15 vmallika joined #gluster
11:15 Bhaskarakiran_ joined #gluster
11:29 arcolife joined #gluster
11:32 autoditac joined #gluster
11:39 Merlin_ joined #gluster
11:52 ashiq_ joined #gluster
11:53 Merlin_ joined #gluster
12:00 hgowtham joined #gluster
12:01 haomaiwa_ joined #gluster
12:01 ppai_ joined #gluster
12:06 unlaudable joined #gluster
12:10 Merlin_ joined #gluster
12:16 jiffin1 joined #gluster
12:18 ivan_rossi left #gluster
12:20 autoditac hey, i have a few glusterfs 3.7.8 replica 3 volumes running  and was wondering which ports to allow thorugh my bricks firewall and came up to this list of port ranges (tcp): 49152-49251, 24007-24011, 38465-38470
12:20 sakshi joined #gluster
12:20 autoditac does this look ok to you or am i missing something?
12:21 autoditac or are these too much already?
12:21 Merlin_ joined #gluster
12:23 jiffin1 joined #gluster
12:26 hackman joined #gluster
12:31 Merlin_ joined #gluster
12:32 hgowtham joined #gluster
12:38 post-factum @ports
12:38 glusterbot post-factum: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up. All ports must be reachable by both servers and clients. Additionally it will listen on 38465-38468/tcp for NFS. NFS also depends on rpcbind/portmap ports 111 and 2049.
12:38 post-factum autoditac: ^^
12:39 autoditac post-factum: thanks
12:40 mpietersen joined #gluster
12:40 B21956 joined #gluster
12:41 mpietersen joined #gluster
12:42 Merlin_ joined #gluster
12:42 unclemarc joined #gluster
12:42 kanagaraj joined #gluster
12:43 mpietersen joined #gluster
12:44 mpietersen joined #gluster
12:45 ira joined #gluster
12:45 mpietersen joined #gluster
12:46 ference left #gluster
12:47 mpietersen joined #gluster
12:49 dblack joined #gluster
12:49 mpieters_ joined #gluster
12:51 mpietersen joined #gluster
12:53 mpietersen joined #gluster
12:54 mpietersen joined #gluster
12:55 mpietersen joined #gluster
12:56 glisignoli joined #gluster
12:56 ovaistariq joined #gluster
12:57 Merlin_ joined #gluster
12:58 chirino_m joined #gluster
13:03 plarsen joined #gluster
13:04 EinstCrazy joined #gluster
13:05 haomaiwa_ joined #gluster
13:08 overclk joined #gluster
13:11 Merlin_ joined #gluster
13:13 kanagaraj joined #gluster
13:16 squizzi joined #gluster
13:17 Manikandan_ joined #gluster
13:18 sebamontini joined #gluster
13:21 vmallika joined #gluster
13:22 Manikandan_ joined #gluster
13:24 Merlin_ joined #gluster
13:26 rwheeler joined #gluster
13:30 hamiller joined #gluster
13:30 dgandhi joined #gluster
13:31 dgandhi joined #gluster
13:32 B21956 joined #gluster
13:40 nbalacha joined #gluster
13:47 arcolife joined #gluster
13:50 Merlin_ joined #gluster
13:53 hchiramm_ joined #gluster
13:53 hchiramm joined #gluster
13:55 rafi joined #gluster
13:56 Akee joined #gluster
13:56 ghenry joined #gluster
14:00 hgowtham joined #gluster
14:01 haomaiwa_ joined #gluster
14:05 mancz joined #gluster
14:06 mancz hi
14:06 glusterbot mancz: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:06 mancz root@storage999.int1-dus.dg-m.de:~# gluster vol quota dev enable
14:06 mancz quota command failed : Volume quota failed. The cluster is operating at version 30704. Quota command enable is unavailable in this version.
14:06 mancz anybody has a clue how to fix this?
14:09 Merlin_ joined #gluster
14:19 ekuric joined #gluster
14:22 poornimag joined #gluster
14:25 robb_nl joined #gluster
14:27 Merlin_ joined #gluster
14:30 B21956 joined #gluster
14:39 Merlin_ joined #gluster
14:41 csterling joined #gluster
14:42 ggarg joined #gluster
14:44 wnlx joined #gluster
14:49 daMaestro joined #gluster
14:50 hagarth joined #gluster
14:51 Merlin_ joined #gluster
14:52 post-factum mancz: what gluster version do you run?
14:52 post-factum mancz: probably, you have to update it and even bump opversion manually
14:53 karnan joined #gluster
14:54 mancz glusterfs 3.7.8 built on Feb 12 2016 13:08:20
14:54 mancz bumping the version to 30708 fails
14:54 mancz but from what I read on the internet, quota should work also with 30704..
14:57 ovaistariq joined #gluster
15:00 coredump joined #gluster
15:01 farhorizon joined #gluster
15:01 haomaiwang joined #gluster
15:01 wushudoin joined #gluster
15:03 post-factum try 30707
15:04 mancz ok
15:04 post-factum an inspect logs for possible errors/warnings plz
15:04 mancz root@storage999.int1-dus.dg-m.de:~# gluster vol quota dev enable
15:04 mancz volume quota : success
15:04 mancz argh..
15:04 post-factum umm
15:04 post-factum ok :)
15:04 mancz could have figured that out mysqlf :(
15:04 mancz thanks!
15:05 [diablo] joined #gluster
15:05 post-factum so, 30707 worked?
15:05 mancz yep
15:05 post-factum ok, have fun
15:05 csterling Good morning
15:05 csterling So post-factum: Over the weekend, gluster fixed itself
15:06 csterling I don’t know what was happening
15:06 Merlin_ joined #gluster
15:06 csterling But let’s try this - gluster is now rebalancing - can I safely stop the rebalance with no ill effects?
15:06 csterling Or will it do the same as alast time
15:06 csterling ?
15:06 post-factum csterling: let me remember your issue. was that about distributed volume rebalance?
15:06 csterling yup
15:06 csterling And suddenly, it was just failing
15:07 post-factum csterling: yep, you can stop rebalance, but you must wait for current file to be processed
15:07 skylar joined #gluster
15:07 csterling Alright - I’ll let it run for a little longer and kill him off
15:07 csterling So next question
15:07 csterling (Hopefully my last for a while)
15:08 csterling So I have a distributed replicated system with bricks across 4 servers
15:08 csterling If I take one server down, just with a shutdown
15:08 csterling And grow the (EBS) volume the brick is on
15:08 csterling And restart that node
15:09 wushudoin joined #gluster
15:09 csterling WIll gluster freak out about the new partition size?
15:09 post-factum it will definitely start healing, but in order to rebalance them properly, you have to run rebalance as well
15:10 post-factum so wait for heal to be finished, then start rebalance
15:10 csterling Gotcha
15:10 csterling When I stop a rebalance, does it run an auto-heal?
15:10 csterling Out of curiosity?
15:10 post-factum it does not need to
15:10 post-factum btw, you will have to trigger heal as well
15:10 post-factum clients may trigger that
15:11 post-factum but if you want for process to finish sooner, launch heal manually
15:11 post-factum or mount the volume somewhere and do "find /volume" on it
15:11 Manikandan__ joined #gluster
15:13 post-factum but anyway, heal must go first
15:15 nehar joined #gluster
15:16 anoopcs bt
15:16 post-factum anoopcs: sorry :)?
15:16 post-factum i is not gdb =)
15:16 misc anoopcs: this is not gdb :p
15:16 post-factum misc++
15:16 glusterbot post-factum: misc's karma is now 2
15:17 anoopcs :-)
15:17 jiffin joined #gluster
15:18 JoeJulian csterling: the ebs resize will not cause any problems. I've done the same thing many times live-resizing lvm backed bricks.
15:18 csterling Awesome - that makes me feel a little better
15:20 Merlin_ joined #gluster
15:20 RameshN joined #gluster
15:22 syadnom guys, got a question on gluster native client performance.  In reading various threads online, it's suggested that in some cases, using the gluster nfs export can be faster than the native client.  specifically for small files.   what are 'small files' in this context?
15:23 JoeJulian https://joejulian.name/blog/nfs-mount-for-gluster​fs-gives-better-read-performance-for-small-files/
15:23 glusterbot Title: NFS mount for GlusterFS gives better read performance for small files? (at joejulian.name)
15:24 syadnom yeah, thats one of them.
15:24 unclemarc joined #gluster
15:25 JoeJulian I explain it there. It's not a hard number.
15:25 syadnom I'm wanting to use gluster as the store for a network video recorder.  typical files are motion triggered evens at right around 200KB.
15:26 JoeJulian zm?
15:26 syadnom unifi-video
15:26 syadnom gluster works like a charm for this with lite load.  the point is to spread the IO over a number of boxes though.
15:26 skoduri joined #gluster
15:26 JoeJulian I was just curious. I use zoneminder for mine.
15:27 syadnom one issue is that it's easy enough getting a few TB in one inexpensive box, but getting 8 4TB drives in one has a much higher prerequisite price tag
15:28 JoeJulian You could use nfs. Once written, the files are not going to change. That's my biggest gripe with people blindly using nfs because it's faster. It's also caching metadata so it could be stale.
15:28 syadnom also, they have a small debian based NVR product that only has 1 drive.
15:28 syadnom something in my brain says use the native client lol
15:29 syadnom I'm also seeing IO issues on single drives, and SSD is still too expensive.
15:29 JoeJulian Native is natively HA where if you use nfs, you'll have to plan for a single network connection.
15:30 Merlin_ joined #gluster
15:30 JoeJulian Since you'll be using DHT to spread the load (each filename has a % chance of going to a different distribute subvolume) that should work pretty effectively.
15:30 rafi joined #gluster
15:30 jwrona joined #gluster
15:30 rjoseph joined #gluster
15:31 JoeJulian Does that software create a treed structure so all the filenames are not in one giant directory/
15:31 JoeJulian ?
15:32 post-factum JoeJulian: are files spread among distributed bricks by folders or by filenames?
15:33 Manikandan__ joined #gluster
15:35 syadnom I guess JoeJulian there are in folders
15:35 jwrona Hello, I have a quick question. I know it is OK to have distributed replicated volume (DHT->AFR). In my use case it would be great to have something like replicated distributed (AFR->DHT). First replica to the local brick, second replica to any other brick. Is it possible?
15:38 syadnom jwrona, I put in a feature request for 1/2 of this last week.  To tell DHT to write only locally.  Check the recent feature requests and maybe add a comment
15:38 syadnom JoeJulian, On my office NVR, 1.8M video files (99% are about 200KB, a few are up to 100MB) and 598 directories.
15:39 post-factum jwrona: you may change brick order on volume creation
15:39 post-factum jwrona: and use "force" option, because gluster warns about replica withon one node
15:39 post-factum jwrona: but that is anyway ugly and wrong solution. why do you want that?
15:40 syadnom post-factum, jwrona I think he just wants to write locally first, then replicate in a async manor.
15:40 post-factum umm, geo-rep then
15:41 post-factum gluster replica is sync, afaik
15:42 csterling post-factum: I went to go remove a brick after stopping my rebalance, and it give me back an error message about rebalance still being in process. When I try and look at the status, I get the error messages from before
15:42 csterling [root@gluster01 csterling]# gluster volume remove-brick nmd gluster01.newmediadenver.com:/srv/sdj1/nmd gluster02.newmediadenver.com:/srv/sdj1/nmd start
15:42 csterling volume remove-brick start: failed: Staging failed on gluster03.newmediadenver.com. Error: Rebalance is in progress. Please retry after completion
15:42 csterling [root@gluster01 csterling]# gluster volume rebalance nmd status
15:42 csterling volume rebalance: nmd: failed: Rebalance not started.
15:43 post-factum csterling: meh :). looks like same shit
15:43 csterling It’s the same as before
15:44 post-factum i remember i asked for devs to help. probably, we should try again
15:44 jwrona I don't need async, regular replication is fine. I just need one replica on the local brick. My use case: on every node there is a gluster server and a client. Client is connected to the local server and each file the client creates HAS to be stored on the local node.
15:44 csterling JoeJulian: ^^
15:45 csterling I keep wondering if the rebalance is hung
15:48 post-factum jwrona: afaik, gluster client will prefer local brick over remote because of less latency
15:48 unclemarc joined #gluster
15:48 JoeJulian jwrona: short of eschewing the benefits of the management interface and writing your own volfiles, no.
15:49 JoeJulian csterling: Seems like a reasonably good fit for the fuse client. I would try it. You can always change it later if it's not working out for you.
15:50 csterling JoeJulian: Sorry, I pointed to the wrong help text - let me see if I can summarize
15:50 JoeJulian Oh, for the rebalance thing? ugh
15:51 post-factum JoeJulian: yeah, rebalance
15:51 jwrona JoeJulian: Thanks, I will try to write my own volfile.
15:51 JoeJulian I had that once. It was on a non-critical volume so I stopped it, took the rebalance bits out of the info file for the volume, then started it again.
15:51 jiffin syadnom: u can try out pNFS if u want (confirms local writes from NFS server to brick)
15:51 jiffin refer pNFS in http://gluster.readthedocs.org/en/​latest/Administrator%20Guide/NFS-G​anesha%20GlusterFS%20Intergration/
15:51 glusterbot Title: Configuring NFS-Ganesha server - Gluster Docs (at gluster.readthedocs.org)
15:51 Merlin_ joined #gluster
15:51 csterling JoeJulian: This is a production volume, and it cleaned itself up over the weekend
15:52 csterling I tried a new rebalance today, needed to stop it so we could remove a brick
15:53 shubhendu joined #gluster
15:53 csterling Stopped, and now it’s bricked again, so obviously something cleaned up
15:53 csterling Is there a way of viewing active gluster tasks?
15:53 JoeJulian volume status /should/ do that.
15:54 JoeJulian You can also look at state dumps, but even as long as I've been doing that, I'm never quite sure what I'm looking for in those.
15:54 csterling http://fpaste.org/339692/70867145/
15:54 glusterbot Title: #339692 Fedora Project Pastebin (at fpaste.org)
15:54 csterling Says no active tasks
15:54 JoeJulian If it's fixing itself after a period of time, I'd guess some sort of lock that's stuck. 30 minutes should clear it up.
15:55 csterling gotcha
15:55 csterling Sounds like the perfect time to get coffee :-)
15:55 JoeJulian 30 minutes is "frame-timeout" which is used for things that should never get stuck, but just in case they do...
15:56 post-factum ...one should wait for 30 mins and file bugreport
15:57 JoeJulian +1
16:01 haomaiwa_ joined #gluster
16:03 jwrona I've tried to write my own replicate distribute volfile. It seems to be working fine (tested on one server). Is it OK to have volfile looking like this? http://pastebin.com/NTUWcAUw
16:03 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
16:04 JoeJulian jwrona: I think it should work. Test, test, test. :)
16:05 jwrona JoeJulian: Thanks, I will.
16:05 Iouns joined #gluster
16:05 JoeJulian I suggest burden of testing is on you because it's a graph structure that nobody else is using so there's no preexisting feedback.
16:05 gem joined #gluster
16:06 JoeJulian Try different failures, too.
16:08 Merlin_ joined #gluster
16:19 csterling JoeJulian: I’m still having that rebalance issue, and it’s definitely been over 30 minutes since stopping. Should I keep giving it time?
16:19 csterling What other ops occur automatically that could heal a broken rebalance?
16:20 csterling I’m just trying to reconstruct what happened over the weekend
16:20 JoeJulian I honestly have no idea.
16:20 csterling Gotcha
16:20 csterling Let’s try it this way - is there a way to restart gluster gracefully without afftecting downstream clients?
16:20 JoeJulian You can restart glusterd at any time.
16:20 scones joined #gluster
16:21 csterling OK - I’ll give that a shot on all nodes
16:21 JoeJulian glusterfsd is the brick process.
16:21 scones Is the "Gluster Admin Guide" link throwing 404s for anyone besides me?
16:22 bennyturns joined #gluster
16:22 JoeJulian Yes it is. It's one layer too deep. http://gluster.readthedocs.org/e​n/latest/Administrator%20Guide/
16:22 glusterbot Title: Index - Gluster Docs (at gluster.readthedocs.org)
16:22 JoeJulian misc: ^^
16:23 scones JoeJulian:  Thank ya much!
16:24 misc JoeJulian: mhh ?
16:24 anmol joined #gluster
16:26 csterling JoeJulian: post-factum: So I ssh-ed into each gluster host, and service glusterd restart on each of them - starting the next server the second the previous finished - unblocked me
16:26 csterling Huh
16:26 csterling That’s exciting :-)
16:33 armyriad joined #gluster
16:33 shyam joined #gluster
16:34 squizzi joined #gluster
16:34 Merlin_ joined #gluster
16:35 squizzi joined #gluster
16:40 nishanth joined #gluster
16:40 post-factum csterling: keep going ;)
16:41 nage joined #gluster
16:43 ovaistariq joined #gluster
16:43 csterling Here’s what I’m doing today: Removing 2 bricks and mounting them on 2 new servers, then adding them back in. Then taking 2 old bricks down (one a t a time) rudely, growing their partitions, then adding them back in. Then healing
16:43 csterling It’s gonna be a fun day
16:43 sebamontini joined #gluster
16:43 JoeJulian If it was me...
16:45 JoeJulian I wouldn't remove bricks. I would just kill the brick processes for the two that will be moving, move them, then "replace-brick $old_brick_path $new_brick_path commit force".
16:45 JoeJulian That avoids rebalance.
16:45 csterling Whoops - I already started removing the bricks
16:46 JoeJulian Oh well.
16:46 JoeJulian I just don't like doing that much work. ;)
16:46 csterling I had just rotated those bricks in, so they both only have .8GB used on them
16:46 csterling It’s taking a while to get it all moved off
16:46 csterling But I’m assumning that’s because it’s recomputing the location of each
16:46 csterling s/recomputing/rehashing/
16:46 glusterbot What csterling meant to say was: But I’m assumning that’s because it’s rehashing the location of each
16:47 JoeJulian My guess is that it's walking the entire volume tree.
16:47 JoeJulian The filename hash is extremely fast.
16:47 csterling Gotcha
16:47 csterling I estimated it at about 43mins using the delta inode count
16:48 Merlin_ joined #gluster
16:49 csterling I don’t know if that’s really a good number to use, but I didn’t see a different count
16:50 JoeJulian Huh... well this is new. I'm trying to figure out why tcp frames keep timing out to one of my VMs. The MTU's set to 9k on the servers, the vm interface, the host interface, and the bridge. The switch isn't dropping frames...
16:50 d0nn1e joined #gluster
16:51 karnan_ joined #gluster
16:51 JoeJulian Apparently the new kvm doesn't pass the mtu interface setting back to the host anymore. So even though the interface in the vm had the right mtu, vnet0 did not.
16:51 JoeJulian I've been fighiting this since yesterday afternoon. <sigh>
16:52 JoeJulian post-factum: Something for you to be aware of since you also run arch. ^
16:52 JoeJulian Everyone else will need to be aware in a couple of years.
16:54 post-factum :D
16:54 post-factum hmm
16:54 csterling JoeJulian: Kinda weird, but have you tried running nc on the misbehaving vm just to confirm the entire packet is actually making it through or if maybe some system is strapping on excess metadata to the packet header?
16:54 post-factum i use macvtap trickery with routing
16:55 post-factum and do not use jumbo frames
16:55 JoeJulian True, this is also ovs.
16:55 JoeJulian jumbo for storage is a must.
16:55 post-factum JoeJulian: any measurements taken for storage+jumbo?
16:56 JoeJulian Giving up 5% of your frame to overhead sucks.
16:57 post-factum correct
16:57 post-factum but real numbers would be better
16:57 post-factum especially in terms of latency
16:57 sebamontini joined #gluster
16:57 JoeJulian There's no latency hit. Small frames are still small frames. The only time jumbo comes in to play is for throughput.
16:58 rafi joined #gluster
16:58 post-factum JoeJulian: will find the article on that. they also considered latency to be worse
16:59 post-factum but ovs. is that better than usual bridges? taking into account latency
16:59 JoeJulian no
16:59 syadnom Jumbo frames can reduce latency under load because it reduces the pps, which both switches and routers have limitations on.
16:59 JoeJulian use macvlan/macvtap
17:00 sebamontini joined #gluster
17:00 post-factum i've also switched from bonding to teaming, but still use bridges
17:01 JoeJulian macvlans use less cpu than ovs or bridges.
17:01 post-factum on centos, i mean, on arch i use macvtap
17:01 JoeJulian You just have to get tricky if you want to stack them.
17:01 haomaiwa_ joined #gluster
17:01 post-factum i would like to have multiple vlans on top of teaming
17:02 post-factum and "bridge? those vlans into vms
17:02 post-factum *"bridge"
17:02 post-factum as untagged
17:04 post-factum like team0 → team0.10 → some_bridge_for_vlan_10 → vm interface, untagged
17:04 post-factum also, i would want to have untagged interface in valn 10 on host too
17:05 post-factum now I just attach veth pair into desired bridge
17:05 post-factum s/valn/vlan/
17:05 glusterbot What post-factum meant to say was: also, i would want to have untagged interface in vlan 10 on host too
17:05 JoeJulian I've got that set up with macvtaps for nspawn containers.
17:06 post-factum do that macs support multiqueueing?
17:08 JoeJulian I haven't done it, but it shouldn't be an issue.
17:08 post-factum ok, will check that on some unused hypervisor soon. thanks
17:09 Merlin_ joined #gluster
17:11 topher left #gluster
17:11 Caveat4U joined #gluster
17:11 hchiramm_ joined #gluster
17:12 karnan joined #gluster
17:16 chirino joined #gluster
17:23 Caveat4U JoeJulian: …Could I stop the brick removal safely?
17:23 Caveat4U s/Could I/Can I/
17:23 glusterbot What Caveat4U meant to say was: JoeJulian: …Can I stop the brick removal safely?
17:24 JoeJulian you should be able to remove-brick..stop, yes.
17:25 kovshenin joined #gluster
17:26 calavera joined #gluster
17:29 Merlin_ joined #gluster
17:29 plarsen joined #gluster
17:38 TvL2386 joined #gluster
17:44 post-factum JoeJulian: it seems i can even keep old bridges, and make macvtaps co-exist with them, just binding them to underlying team0.10 interface
17:44 JoeJulian :)
17:44 post-factum u see, it is 50 vms to rework :/
17:47 post-factum it seems i want macvlan in bridged mode
17:48 post-factum here are some benchmarks: http://140.120.15.179/Presentation/20150203/
17:48 glusterbot Title: Virtual networking: TUN/TAP, MacVLAN, and MacVTap (at 140.120.15.179)
17:49 post-factum not seeing huge differenct against ovs, though
17:49 jwrona left #gluster
17:56 Merlin_ joined #gluster
18:01 haomaiwa_ joined #gluster
18:02 shaunm joined #gluster
18:06 tomased joined #gluster
18:07 Caveat4U_ joined #gluster
18:09 Caveat4U Would this group chat happened to be logged somewhere? I lost my back-scroll.
18:09 JoeJulian See /topic
18:10 TvL2386 joined #gluster
18:10 B21956 joined #gluster
18:10 Caveat4U Thanks Joe - sorry for bugging you so much lately
18:10 JoeJulian Not a problem.
18:11 JoeJulian If I'm too busy to answer, I won't. :)
18:11 JoeJulian That's not entirely true, either. I often look in the scrollback after I'm not busy to make sure everything got answered.
18:12 Merlin_ joined #gluster
18:13 mhulsman joined #gluster
18:14 tessier So one half of my two node cluster is way out of date and got data written into it accidentally when disks got mounted in the wrong place and I did a force-start not realizing what the real problem was. If I bring this screwed up node back into the gluster will gluster automatically work it out and delete anything that shouldn't be there etc. or do I need to take some other step?
18:14 lalatenduM joined #gluster
18:15 JoeJulian Hmm, delete, maybe, if the delete queue entry wasn't already satisfied. I would probably run a heal..full just to make sure no replication was missed.
18:16 hagarth joined #gluster
18:19 yosafbridge joined #gluster
18:20 calavera joined #gluster
18:27 tessier # gluster volume heal 9d full
18:27 tessier Commit failed on cffb1749-3c89-47b5-8352-f40dc7c5c74e. Please check log file for details.
18:27 tessier hmm
18:27 tessier "0-management: Lock for vol disk05h not held
18:27 tessier "
18:28 hackman joined #gluster
18:29 tessier There's always so much weird stuff happening in the gluster logs that I can't make any sense of.
18:30 tessier ah-hah... 0-management: Lock owner mismatch. Lock for vol 9d held by 230089fe-021d-4c52-b190-24af095070f
18:30 post-factum @paste
18:30 glusterbot post-factum: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
18:30 post-factum tessier: ^^
18:31 tessier If your gluster brick servers have unrestricted access to the net so as being able to paste to termbin.com you have issues.
18:32 DV joined #gluster
18:37 post-factum tessier: iptables+conntrack, eh?
18:40 kanagaraj joined #gluster
18:41 JoeJulian mmkay....
18:45 ira joined #gluster
18:47 ovaistariq joined #gluster
18:50 sebamontini joined #gluster
18:50 ovaistar_ joined #gluster
18:58 calavera joined #gluster
18:58 JoeJulian imho, air-gap security is generally excessive. Sometime's it's not though.
18:59 tessier I don't have air-gap security. Just a DMZ separating my SAN from the Internet.
18:59 tessier We have various regulatory requirements that prohibit storage machines from being able to talk directly to the net.
19:01 post-factum tessier: that is ok, but solving the problem in another way is not an issue
19:01 JoeJulian Ah yes, the 'ol "we're regulators so since we don't understand something, we just prohibit it" story.
19:01 haomaiwa_ joined #gluster
19:01 post-factum security through obscurity they call it
19:01 tessier Not at all.
19:02 tessier There's generally nothing legit on the net that our storage servers need to talk to. We monitor packets bouncing off the egress rules and have caught intrusions on behalf of customers that way. It is very useful.
19:03 tessier The last time it was a customer's app server scanning MySQL ports on the net when it had no business doing so.
19:03 tessier If we just allowed everything we wouldn't have caught that so soon.
19:03 tessier Firewalls are for blocking bad guys getting in and your data from getting out.
19:03 calavera_ joined #gluster
19:05 JoeJulian PCI DSS, NACHA, HIPAA... they all have rules that are pointless, while auditors don't even generally understand what they're looking at. I'm not suggesting that you don't know why you do what you do, just railing against meaningless regulation.
19:06 JoeJulian btw... if you can ssh into it, you can port-forward through your ssh client.
19:07 JoeJulian That paste command is just as a matter of convenience, of course. You can always just copy/paste to fpaste.org.
19:08 tessier Indeed. Although so far today I've not had to paste anything longer than 1 line. I would never dump anything large in here.
19:08 JoeJulian The one thing that drives me nuts, though, are the people that do have air-gap and post photos of the screen.
19:08 tessier I wonder if force start is the answer now that I really do have everything mounted in the right places...
19:08 tessier Heh.
19:09 tessier oh...force doesn't help with volume ID mismatch.
19:10 JoeJulian Yay!
19:10 tessier Although I wonder how I could have volume ID mismatch now that I'm sure I've got everything in the right place...let me double check that again...
19:10 tessier Isn't the volume ID stored in an attribute or something?
19:10 post-factum show plz "gluster peer status" just in case
19:11 JoeJulian Yes, volume id on the root of the brick is in an ,,(extended attribute)
19:11 glusterbot To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}
19:12 JoeJulian Should match the uuid in the volume info file.
19:13 tessier http://fpaste.org/339825/79827301/ gluster peer status. The ones that say disconnected are old decommissioned test machines. I should figure out how to get them removed from the cluster properly.
19:13 glusterbot Title: #339825 Fedora Project Pastebin (at fpaste.org)
19:13 JoeJulian "gluster peer detach"
19:26 marlinc joined #gluster
19:35 calavera joined #gluster
19:36 calavera joined #gluster
19:37 calavera joined #gluster
19:39 deniszh joined #gluster
19:40 calavera joined #gluster
19:43 mhulsman joined #gluster
19:51 robb_nl joined #gluster
19:57 ju5t joined #gluster
19:58 nathwill joined #gluster
19:58 d0nn1e joined #gluster
20:01 haomaiwa_ joined #gluster
20:18 ovaistariq joined #gluster
20:21 rouven joined #gluster
20:52 timotheus1_ joined #gluster
20:55 chirino_m joined #gluster
20:56 ovaistariq joined #gluster
21:01 haomaiwa_ joined #gluster
21:02 ghenry joined #gluster
21:13 Wizek__ joined #gluster
21:13 DV joined #gluster
21:16 Wizek joined #gluster
21:25 mowntan joined #gluster
21:26 mowntan joined #gluster
21:26 mowntan joined #gluster
21:26 mowntan joined #gluster
21:29 autoditac hey everybody. quick questions: which ports are being used for peer probes?
21:29 JoeJulian @ports
21:29 glusterbot JoeJulian: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up. All ports must be reachable by both servers and clients. Additionally it will listen on 38465-38468/tcp for NFS. NFS also depends on rpcbind/portmap ports 111 and 2049.
21:30 autoditac JoeJulian: so 24007 it is?
21:30 JoeJulian Right, unless you're using rdma.
21:31 JoeJulian Even then, though, I think it's 24007 for peering.
21:32 autoditac hmm, i have 24007-24013 opened on the firewall of both my nodes and still peer probing fails unless i switch of my firewall
21:32 autoditac must be an error in my config
21:33 JoeJulian You should be able to test your firewall with nc or telnet. Either one should at least be able to establish a tcp connection.
21:34 autoditac you' re right
21:34 autoditac doesn't work - strange
21:40 plarsen joined #gluster
21:43 mhulsman joined #gluster
21:51 tessier Sigh. Now I wonder if I've somehow caused some sort of split-brain situation. Each brick server says the volume is online but doesn't list the other brick server as a brick in the volume in the gluster volume status output.
21:52 post-factum tessier: "gluster peer status" again please
21:55 Wojtek joined #gluster
21:56 Wojtek [2016-03-14 21:56:19.762506] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-gv0-client-2: remote operation failed. Path: <gfid:51461d62-f78e-48a6-ad70-dc82abe0ca3a> (51461d62-f78e-48a6-ad70-dc82abe0ca3a) [Permission denied]
21:56 Wojtek [2016-03-14 21:56:19.763222] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-gv0-client-0: remote operation failed. Path: <gfid:a59ac479-93a9-4760-aead-ca2ab661e17f> (a59ac479-93a9-4760-aead-ca2ab661e17f) [Permission denied]
21:56 Wojtek [2016-03-14 21:56:19.765191] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-gv0-client-0: remote operation failed. Path: <gfid:3a931be7-e969-4540-a2d2-d6466f5d3ffc> (3a931be7-e969-4540-a2d2-d6466f5d3ffc) [Permission denied]
21:57 Wojtek I have these messages that loop continuously
21:57 Wojtek What does it not have permission for specifically?
21:59 post-factum @paste
21:59 glusterbot post-factum: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
21:59 post-factum Wojtek: ^^
21:59 JoeJulian Well, it's an open() callback, so I assume it's trying to open something. client-0 and client-2 are the first and third bricks in the volume. If I were to guess, it would be that those gfid files don't exist.
22:00 JoeJulian My other guess would be selinux
22:01 haomaiwa_ joined #gluster
22:03 Wojtek @post-factum Oki. I'll use pastebin next time :)
22:04 Wojtek @JoeJulian I'll check my selinux audit logs, but is there a way to know what file corresponds to the given gfid?
22:04 JoeJulian @gfid lookup
22:04 JoeJulian @gfid
22:04 glusterbot JoeJulian: The gfid is a uuid that's assigned to represent a unique inode that can be identical across replicas. It's stored in extended attributes and used in the .glusterfs tree. See http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/ and http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
22:04 JoeJulian that's not what I'm looking for, glusterbot...
22:05 JoeJulian @gfid resolver
22:05 glusterbot JoeJulian: https://gist.github.com/4392640
22:05 Wojtek ah, marvelous :) Much appreciated!
22:05 JoeJulian Actually... isn't that distributed with gluster now? One sec while I check.
22:06 tessier post-factum: On which machine should I run gluster peer status?
22:06 post-factum tessier: at least on one :)
22:06 tessier I ask because I get different outputs in different places.
22:06 post-factum tessier: show them all, that is weird
22:06 tessier Ok...
22:07 JoeJulian Wojtek: yeah, gfid_to_path.py. On arch that's installed to /usr/lib/glusterfs/glusterfs/gfi​nd_missing_files/gfid_to_path.py
22:10 tessier post-factum: http://fpaste.org/339934/79934451/
22:10 glusterbot Title: #339934 Fedora Project Pastebin (at fpaste.org)
22:11 tessier Something's definitely in a weird state. The client can read/write files but the peers say it is disconnected.
22:12 JoeJulian Check /var/log/glusterfs/etc-glusterfs-glusterd.vol.log. I bet you'll find hash mismatches.
22:12 sebamontini joined #gluster
22:13 post-factum JoeJulian: shouldn't it be Rejected then?
22:13 JoeJulian If so that means that one (or more) of /var/lib/glusterd/vols/*/info doesn't match on .12.
22:13 ovaistariq joined #gluster
22:13 JoeJulian Should be, yeah.
22:13 JoeJulian I've seen this before, too.
22:13 JoeJulian If not, just restart glusterd.
22:14 JoeJulian There's a rejected on 21 and 20.
22:14 post-factum i run several times into volfile meta split-brain also, but it could be fixed for me only by copying volume metadata from one server to another
22:14 JoeJulian Do we have an open bug for that?
22:15 post-factum dunno. it was for 3.7.6. since then i've added third dummy node and forgot about that
22:16 post-factum i guess it was pretty easy to fall into volfiles split-brain on occasional glusterd segfaults, when only one node out of two was operating the cluster
22:18 tessier I don't see the word hash anywhere or anything that looks like a hash mismatch....I do see volume id mismatch and lock owner mismatch from earlier when I had things mismounted etc.
22:18 post-factum tessier: check for smth like "cksum mismatch"
22:19 tessier I see 3 lines like: [2016-03-14 21:32:26.416112] E [MSGID: 106010] [glusterd-utils.c:2620:glus​terd_compare_friend_volume] 0-management: Version of Cksums 9c differ. local cksum = 3151282412, remote cksum = 1929380590 on peer 10.0.1.21
22:20 post-factum voila
22:20 post-factum JoeJulian: could that be fixed by some magic command, or one should just overwrite volfile on one of the nodes?
22:21 ovaistariq joined #gluster
22:23 JoeJulian theoretically "gluster volume sync" should be able to do it.
22:24 JoeJulian I tried it a long time ago and it didn't work, so I've always done it by hand ever since.
22:25 post-factum so, one could ssh into 10.0.1.21 and do "sudo gluster volume sync some.ip all"?
22:27 calavera joined #gluster
22:35 hagarth joined #gluster
23:01 haomaiwa_ joined #gluster
23:02 coredump joined #gluster
23:13 hagarth joined #gluster
23:29 JoeJulian file a bug
23:29 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
23:56 dlambrig_ joined #gluster
23:58 johnmilton joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary