Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-10-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 edwardm61 joined #gluster
00:10 abyss_ joined #gluster
00:23 gildub joined #gluster
00:28 fubada can somene suggest a way to tear down all geo repl
00:28 fubada its all fubared in bad state
00:28 fubada i want to start clean, but i dont want to end up with corrupted master bricks
00:29 fubada is it enough to just wipe /var/lib/gluster/geo-replication/* and restart gluster?
00:29 fubada i want to start fresh
00:29 fubada (with geo repl)
00:54 EinstCrazy joined #gluster
01:09 nangthang joined #gluster
01:12 najib joined #gluster
01:14 vimal joined #gluster
01:15 najib Hi, am testing with 3 nodes gluster HA. currently using 3 bricks ( one brick for each nodes). How can such setup can survive for 2 nodes down? Is two other host should be added to trusted pool?
01:16 shyam joined #gluster
01:28 fubada Arggh
01:29 fubada going nuts over here trying to start with geo-repl from scratch
01:29 fubada what does this eman
01:29 fubada FORCE_BLOCKER|Unable to fetch master volume details. Please check the master cluster and master volume.
01:29 fubada when running push-pem
01:40 CyrilPeponnet @najib if replica of your vol is set to 3 and with proper quorum yes it could survive (hoping one node can hold all the traffic)
01:41 CyrilPeponnet @fubada it means that you have to create the volume on the slave or you mistype the volume name on the master
01:45 fubada CyrilPeponnet: i had some stale volumes on my slaves
01:47 ilbot3 joined #gluster
01:47 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:51 fubada CyrilPeponnet: i created blank volumes on slaves
01:51 fubada same issue
01:51 fubada ;
01:51 fubada ;
01:54 fubada Unable to fetch master volume details. Please check the master cluster and master volume.
01:54 fubada geo-replication command failed
01:56 haomaiwa_ joined #gluster
01:57 side_control joined #gluster
01:57 najib CyrilPeponnet, I need to add 2 other host as additional to 3 nodes, is it right?
02:01 CyrilPeponnet @najib it depends of the usage
02:02 CyrilPeponnet if i's for main read it's fine
02:02 CyrilPeponnet if it's for write you should take a look a distributed replicate or stripe
02:03 CyrilPeponnet @fubada did you check the log files
02:03 fubada yea
02:03 CyrilPeponnet under /var/log/glusterfs/geo-relication/vo_name/
02:04 CyrilPeponnet gtg
02:04 harish joined #gluster
02:04 haomaiwang joined #gluster
02:05 najib I am setting it up for use with ovirt high availability cluster
02:05 CyrilPeponnet so for vms
02:05 CyrilPeponnet I hate overt ;p
02:06 CyrilPeponnet I really got to go
02:11 fubada CP|AFK: looks like [2015-10-09 02:10:37.051487] E [glusterfsd-mgmt.c:1502:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
02:12 fubada [2015-10-09 02:10:37.051524] E [glusterfsd-mgmt.c:1604:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:backups)
02:13 nangthang joined #gluster
02:26 fubada should push-pem create ls /var/lib/glusterd/geo-replic​ation/common_secret.pem.pub on every master node?
02:29 bharata-rao joined #gluster
02:54 gem joined #gluster
03:01 haomaiwa_ joined #gluster
03:07 fubada whats up with geo-replication already running in another machine
03:07 fubada when running push pem
03:07 fubada makes no sense
03:07 fubada only one master node shows up under vol geo-repl status after
03:25 atinm joined #gluster
03:26 dlambrig_ joined #gluster
03:34 shubhendu joined #gluster
03:35 [7] joined #gluster
03:39 stickyboy joined #gluster
03:41 nishanth joined #gluster
03:45 gildub joined #gluster
04:02 haomaiwa_ joined #gluster
04:11 gem joined #gluster
04:13 kdhananjay joined #gluster
04:22 shubhendu joined #gluster
04:24 rafi joined #gluster
04:27 skylar1 joined #gluster
04:30 yazhini joined #gluster
04:31 sakshi joined #gluster
04:44 nbalacha joined #gluster
04:45 ramky joined #gluster
04:48 kshlm joined #gluster
04:48 jiffin joined #gluster
04:53 rjoseph joined #gluster
04:55 ppai joined #gluster
04:55 pppp joined #gluster
05:02 haomaiwa_ joined #gluster
05:02 DV__ joined #gluster
05:04 ndarshan joined #gluster
05:10 arcolife joined #gluster
05:10 archit_ joined #gluster
05:11 ashiq joined #gluster
05:11 kovshenin joined #gluster
05:11 fubada :)
05:11 fubada glad ur back CyrilPeponnet
05:13 Manikandan joined #gluster
05:13 CyrilPeponnet give ne gluster vol info on each cluster and the cmd line you used to create the geo-rep
05:14 CyrilPeponnet did you remove something on your gluster by hand
05:14 fubada i removed the geo-repl dir
05:14 CyrilPeponnet like /var/lib/glustefs/geo-rep
05:14 fubada now when i rerun push pem, only one master can talk to slaves
05:14 fubada the other doesnt showup from 'vol geo status'
05:15 CyrilPeponnet bad idea
05:15 CyrilPeponnet what distro
05:15 fubada rhel
05:15 fubada 6
05:15 fubada using 3.7.4 noq
05:15 fubada now*
05:15 CyrilPeponnet ok reinstall glusterfs-georep package
05:15 CyrilPeponnet you removed some files
05:15 fubada ok
05:16 CyrilPeponnet rpm -qf /var/lib//glusterd/geo-repli​cation/gsyncd_template.conf
05:16 CyrilPeponnet glusterfs-server-3.6.5-1.el7.x86_64
05:16 CyrilPeponnet glusterfs-geo-replication-3.6.5-1.el7.x86_64
05:16 fubada it recreates that file btw
05:16 fubada when you restart glusterd
05:17 fubada but im reinstalling geo-replication rpm now
05:17 CyrilPeponnet not sure as it's part of gluster rpm reinstall
05:17 CyrilPeponnet ;p
05:17 neha_ joined #gluster
05:17 semiautomatic joined #gluster
05:17 fubada im going to run gluster system:: execute gsec_create
05:17 CyrilPeponnet then give me the output of gluster vol info on both cluster (master and slave) and the command you use for geo-rep
05:18 CyrilPeponnet ok good
05:18 fubada okay you want the output first?
05:18 CyrilPeponnet as you want
05:18 CyrilPeponnet not related
05:21 fubada CyrilPeponnet: master https://gist.github.com/ano​nymous/1ec22eb8fdadfbc3c1d1
05:21 glusterbot Title: master · GitHub (at gist.github.com)
05:22 fubada CyrilPeponnet: slave: https://gist.github.com/ano​nymous/5c05ae2a2fc4d1233283
05:22 glusterbot Title: slave · GitHub (at gist.github.com)
05:22 fubada i'm ready to run 'push pem' for those 3 vols on master side
05:22 CyrilPeponnet according to gluster info you still have a geo-rep setting up
05:22 CyrilPeponnet at least the options
05:22 fubada this is where the trouble starts
05:22 fubada okay thats the issue then
05:23 fubada do you know how to disable it
05:23 CyrilPeponnet gluster vol reset volante option
05:23 CyrilPeponnet volname
05:24 CyrilPeponnet do you have anything when issuing gluster vol geo-rep status
05:24 fubada No active geo-replication sessions
05:24 CyrilPeponnet ok so try to reset the options
05:25 CyrilPeponnet then issue the command and if it fails paste the logs
05:25 fubada ok
05:25 semiautomatic1 joined #gluster
05:25 CyrilPeponnet do one vol at a time
05:25 CyrilPeponnet let's say backup
05:25 CyrilPeponnet and paste the cmd
05:25 CyrilPeponnet you are issuing
05:25 fubada volume reset: failed: 'geo-replication.ignore-pid-check' is protected. To reset use 'force'.
05:25 CyrilPeponnet the nforce
05:25 fubada force it?
05:25 fubada k
05:27 maveric_amitc_ joined #gluster
05:28 fubada changelog.changelog: on
05:28 fubada this oen too?
05:28 CyrilPeponnet yep
05:29 fubada https://gist.github.com/ano​nymous/53d0bc78874449a869e5
05:29 glusterbot Title: gist:53d0bc78874449a869e5 · GitHub (at gist.github.com)
05:30 fubada trying to figure out the best log to show
05:31 CyrilPeponnet geo-replication already running in another machine
05:31 CyrilPeponnet did you remove the geo-rep folder everywhere ?
05:31 CyrilPeponnet I think there can be some parts remaining in vol files
05:31 fubada https://gist.github.com/ano​nymous/8cd8298629729406c5f0
05:31 glusterbot Title: gist:8cd8298629729406c5f0 · GitHub (at gist.github.com)
05:32 fubada yah i removed it everywhere
05:32 fubada master and slaves
05:32 fubada but pgls102 is giving me issues
05:32 CyrilPeponnet grep -ri geo /var/lib/glusterfs/vols
05:33 CyrilPeponnet ok so try to delete it
05:33 fubada https://gist.github.com/ano​nymous/96f29f8216d5bf540fe7
05:33 glusterbot Title: gist:96f29f8216d5bf540fe7 · GitHub (at gist.github.com)
05:33 CyrilPeponnet the current session
05:33 Bhaskarakiran joined #gluster
05:33 haomaiwang joined #gluster
05:33 fubada fails
05:33 CyrilPeponnet log ?
05:34 fubada https://gist.github.com/ano​nymous/27dc96cfd6bcdead396a
05:34 glusterbot Title: gist:27dc96cfd6bcdead396a · GitHub (at gist.github.com)
05:36 CyrilPeponnet log from /var/log/glusterfs
05:36 fubada ok
05:36 fubada theres lots in there sorry one sec
05:36 fubada not sure which file
05:37 CyrilPeponnet etc- one
05:38 rafi joined #gluster
05:39 fubada https://gist.github.com/ano​nymous/e34f5fc1d97118560ede
05:39 glusterbot Title: gist:e34f5fc1d97118560ede · GitHub (at gist.github.com)
05:40 CyrilPeponnet ok this is a pain
05:40 CyrilPeponnet on each node of master
05:40 CyrilPeponnet grep -ri geo /var/lib/glusterfs/vols
05:40 CyrilPeponnet it's better if you can stop gluster on 3 nodes at this point
05:41 fubada heres the log from a push pem
05:41 fubada https://gist.github.com/ano​nymous/d14bcfec9aff61f78b9e
05:42 glusterbot Title: gist:d14bcfec9aff61f78b9e · GitHub (at gist.github.com)
05:42 fubada grep yields nothing
05:42 fubada on either master
05:43 kdhananjay joined #gluster
05:44 CyrilPeponnet do you have /var/lib/glusterd/geo-repli​cation/gsyncd_template.conf on every nodes
05:44 CyrilPeponnet master and slave
05:45 CyrilPeponnet and something is wrong you should stop and start your volume again on master
05:46 fubada on slaves, no
05:46 fubada masters, yes
05:47 hagarth joined #gluster
05:47 fubada ouch cant stop volumes
05:47 fubada on mater
05:47 fubada master*
05:47 CyrilPeponnet hmm
05:47 fubada i think this can only be done if I rm -rf the geo-repl dir
05:47 CyrilPeponnet I think when you removed some file in var dir
05:47 CyrilPeponnet it went wrong
05:48 CyrilPeponnet because something is still running somewhere
05:48 fubada https://gist.github.com/ano​nymous/63a6ad2831c99bf83901
05:48 CyrilPeponnet in memory
05:48 glusterbot Title: gist:63a6ad2831c99bf83901 · GitHub (at gist.github.com)
05:48 fubada yah but ive stopped and rebooted the masters twice
05:48 fubada so its somewhere, but not in memory :(
05:48 CyrilPeponnet arg
05:49 CyrilPeponnet pfff maybe 3.7 is buggy on this part
05:49 CyrilPeponnet works fine on 3.6.5
05:49 fubada i had a different set of issues in 3.6.1, which is why i went to 3.7.4
05:49 CyrilPeponnet sorry guy I'm out of clue here. You're best bet is the mailing list
05:50 CyrilPeponnet what kind of issue did you have
05:50 fubada thanks CyrilPeponnet
05:50 fubada i dont recall anymore, but it was basically in Faulty state across the board
05:50 CyrilPeponnet ok
05:50 fubada I only had one brief moment where they all read Active/Passive
05:50 fubada and then Faulty all over
05:50 CyrilPeponnet funny never had this in our setup
05:51 fubada i wish there was a clear documented way on how to completely disable geo repl
05:51 fubada manually
05:51 CyrilPeponnet anyway, good luck with your geo-rep setting, sorry it didn't worked as expected
05:51 fubada cos right now its ina  funky state
05:51 CyrilPeponnet check the ML
05:51 fubada no prob, thanks for your help
05:51 CyrilPeponnet I think there are some thread about
05:51 fubada i did, none of my keywords match anything
05:51 CyrilPeponnet even some of mine ;p
05:52 fubada so maybe i mjissed something when upgrading from 3.6.1 to 3.7.4?
05:52 CyrilPeponnet i think you will have to restart gluster on your master
05:52 fubada is there something that needed to be done?
05:52 CyrilPeponnet I don't know still on 3.6.5
05:53 CyrilPeponnet shout the ML
05:53 CyrilPeponnet sorry to not be able to help you more
05:54 fubada thanks again
05:55 hgowtham joined #gluster
05:56 Philambdo joined #gluster
05:58 kanagaraj joined #gluster
06:02 haomaiwa_ joined #gluster
06:08 XpineX joined #gluster
06:09 fubada CyrilPeponnet: its working
06:09 fubada push pem force is working ;/
06:09 fubada weird shit
06:09 CyrilPeponnet what did you do
06:09 fubada i just did 'push pem force' for each vol
06:10 CyrilPeponnet was not the case before ?
06:10 CyrilPeponnet the force
06:11 CyrilPeponnet well glad it works so
06:11 fubada now I started it
06:11 fubada and it says Faulty
06:11 fubada but atleast im getting there
06:16 dusmant joined #gluster
06:17 kayn joined #gluster
06:17 fubada fixed the Fa
06:17 fubada Faulty
06:17 fubada by disabling meta-volume
06:17 fubada config use_meta_volume false
06:18 fubada CyrilPeponnet: what am i missing out on with the meta vol
06:18 spalai joined #gluster
06:19 mhulsman joined #gluster
06:20 fubada https://gist.githubusercontent.com/anonymo​us/5680285db0fabb248319/raw/5002b1cd2374a8​be958ba4ef1c9aee7c6f637b05/gistfile1.txt
06:20 fubada this looks good now?
06:21 jtux joined #gluster
06:22 CyrilPeponnet yep looks good
06:22 CyrilPeponnet backups is empty ?
06:22 fubada no it has some data
06:22 CyrilPeponnet because all your vol should be in hybrid crawl
06:22 CyrilPeponnet add detail
06:22 CyrilPeponnet gluster vol geo-rep detail
06:22 fubada it was
06:23 fubada not a lot of data there
06:23 fubada can I get rid of the authorized-keys file now on the slaves?
06:23 fubada my config mgmgt will try to over write that
06:24 jwd joined #gluster
06:25 fubada thanks again, gnight
06:32 GB21 joined #gluster
06:32 GB21_ joined #gluster
06:34 arcolife joined #gluster
06:47 anil joined #gluster
06:50 raghu joined #gluster
06:51 sakshi joined #gluster
06:51 vmallika joined #gluster
06:57 spalai joined #gluster
07:02 haomaiwa_ joined #gluster
07:10 LebedevRI joined #gluster
07:14 rafi joined #gluster
07:19 [Enrico] joined #gluster
07:22 haomaiwang joined #gluster
07:29 ivan_rossi left #gluster
07:31 vimal joined #gluster
07:44 rafi1 joined #gluster
07:44 nbalacha joined #gluster
07:49 semiautomatic joined #gluster
07:50 semiautomatic joined #gluster
07:51 ivan_rossi joined #gluster
07:57 pdrakeweb joined #gluster
08:00 Lee1092 joined #gluster
08:00 deniszh joined #gluster
08:01 haomaiwa_ joined #gluster
08:01 kayn joined #gluster
08:03 haomaiwa_ joined #gluster
08:03 ivan_rossi_ joined #gluster
08:03 ivan_rossi_ left #gluster
08:03 RameshN joined #gluster
08:07 ctria joined #gluster
08:15 _shaps_ joined #gluster
08:20 capri joined #gluster
08:25 mhulsman joined #gluster
08:27 kovshenin joined #gluster
08:31 Ray_Tracer joined #gluster
08:33 sakshi joined #gluster
08:37 Slashman joined #gluster
08:51 Ray_Tracer joined #gluster
08:53 chirino joined #gluster
08:56 aravindavk joined #gluster
09:01 haomaiwa_ joined #gluster
09:07 fsimonce joined #gluster
09:08 raghu joined #gluster
09:10 spalai joined #gluster
09:12 TvL2386 joined #gluster
09:14 semiautomatic joined #gluster
09:14 shubhendu joined #gluster
09:22 RayTrace_ joined #gluster
09:27 Saravana_ joined #gluster
09:29 mhulsman joined #gluster
09:39 stickyboy joined #gluster
09:42 vimal joined #gluster
09:49 Saravana_ joined #gluster
09:51 gem joined #gluster
09:56 sc0 joined #gluster
09:57 haomaiwang joined #gluster
10:00 fsimonce` joined #gluster
10:00 fsimonce joined #gluster
10:01 haomaiwang joined #gluster
10:06 kxseven joined #gluster
10:12 yazhini joined #gluster
10:18 mhulsman joined #gluster
10:19 yangfeng joined #gluster
10:19 yangfeng_ joined #gluster
10:20 ramky joined #gluster
10:24 Bhaskarakiran joined #gluster
10:29 gem joined #gluster
10:31 Bhaskarakiran joined #gluster
10:34 semiautomatic joined #gluster
10:38 64MAD0769 joined #gluster
10:41 mhulsman joined #gluster
10:42 Saravana_ joined #gluster
10:52 EinstCrazy joined #gluster
10:56 sc0 joined #gluster
10:59 Raide joined #gluster
11:00 zoldar joined #gluster
11:02 haomaiwa_ joined #gluster
11:02 shubhendu joined #gluster
11:04 zoldar Hello. I'm trying to setup a qourum on my cluster and hit and issue where taking down one node blocks writing on the affected volume. The thing is, I have 3 machines where 2 volumes are setup in a cross-over, like this: [Server1: vol1]<--->[Server2: vol1 vol2]<--->Server3: vol2]. The trusted pool contains 3 servers so AFAIK taking down, for example, Server3 shouldn't take down vol2, but it does with "quorum not met" message in the logs. What
11:05 _shaps_ joined #gluster
11:08 zoldar here's the output of "gluster volume info": http://pastebin.com/LxicMvZr
11:08 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
11:09 zoldar oh, the same paste somewhere else, then: http://fpaste.org/276906/44388935/
11:09 glusterbot Title: #276906 Fedora Project Pastebin (at fpaste.org)
11:20 EinstCrazy joined #gluster
11:23 Norky joined #gluster
11:29 GB21 joined #gluster
11:30 rafi joined #gluster
11:40 Bhaskarakiran joined #gluster
11:48 harish joined #gluster
11:49 spalai joined #gluster
11:56 firemanxbr joined #gluster
12:01 haomaiwa_ joined #gluster
12:02 sakshi joined #gluster
12:11 skylar joined #gluster
12:15 rafi joined #gluster
12:17 lpabon joined #gluster
12:21 armyriad joined #gluster
12:24 bluenemo joined #gluster
12:38 julim joined #gluster
12:40 theron joined #gluster
12:44 RameshN_ joined #gluster
12:45 unclemarc joined #gluster
12:46 shubhendu joined #gluster
12:57 bharata_ joined #gluster
13:10 ivan_rossi @paste
13:10 glusterbot ivan_rossi: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
13:11 bharata__ joined #gluster
13:13 kkeithley @pastebin
13:13 glusterbot kkeithley: I do not know about 'pastebin', but I do know about these similar topics: 'paste', 'pasteinfo'
13:13 kkeithley @pasteinfo
13:13 glusterbot kkeithley: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
13:14 arcolife joined #gluster
13:16 unicky o.o termbin...wow
13:16 clutchk joined #gluster
13:16 rafi glusterbot: you should have known about pastebin :D
13:25 mpietersen joined #gluster
13:27 bennyturns joined #gluster
13:27 mpietersen joined #gluster
13:31 spalai joined #gluster
13:32 shyam joined #gluster
13:35 spalai left #gluster
13:43 harold joined #gluster
13:49 sabansal_ joined #gluster
13:52 ayma joined #gluster
13:55 gmaruzz joined #gluster
13:58 haomaiwa_ joined #gluster
13:58 B21956 joined #gluster
13:59 maserati joined #gluster
13:59 semiautomatic joined #gluster
14:01 haomaiwang joined #gluster
14:03 wushudoin joined #gluster
14:08 skylar joined #gluster
14:08 nbalacha joined #gluster
14:17 baojg joined #gluster
14:26 [Enrico] joined #gluster
14:33 JoeJulian unicky: The thing I like about termbin is that it's the same instructions regardless of distro.
14:34 aravindavk joined #gluster
14:34 Simmo Bye guys... have a nice week-end!
14:35 EinstCrazy joined #gluster
14:37 gmaruzz JoeJulian: ciao Joe, I've read in the IRC archive about you being happy with your gluster experience with FreeSWITCH (particularly voicemail and recordings) with a peak of 100 msgs concurrently written. Also, you were suggesting doing the recording/voicemail on tmp dir, then copying it on gluster as last step. Do you have any other hints, experiences, do and dont's you would like to share?. I want to do the same thing (gluster for FS),
14:44 skoduri joined #gluster
14:50 theron joined #gluster
14:51 dlambrig_ left #gluster
14:53 unicky JoeJulian: Good point
14:54 unicky Pretty clever idea
14:55 plarsen joined #gluster
14:57 nbalacha joined #gluster
15:00 JoeJulian gmaruzz: Nothing comes to mind. It was pretty easy.
15:01 haomaiwa_ joined #gluster
15:02 gmaruzz JoeJulian: thanx! Particular problems? Slowliness (they're small files at the end...) ? Specific config tips?
15:02 gmaruzz You were using FS own standard voicemail? Or your own?
15:02 JoeJulian standard
15:03 gmaruzz kewl!
15:03 JoeJulian Gluster was a default replica 3 volume on 12 bricks.
15:05 gmaruzz why so many bricks? for throughput? I was thinking to use replica 2 on 2 bricks...
15:05 JoeJulian We were already using 3 servers with 4 disks each.
15:05 JoeJulian Oh, wait... no...
15:06 JoeJulian That was on a 2x2.
15:06 gmaruzz ah ok. But you think is a sane idea replica 2 with two bricks? all FS are mounting them as fuse
15:06 gmaruzz super! I understand is a sane idea. Correct?
15:06 JoeJulian I would do it.
15:07 gmaruzz if You say that... :)) thx!
15:08 GB21 joined #gluster
15:09 gmaruzz another question: you modified dialplan to have voicemails first written to temp dir, then copied to right place?
15:09 JoeJulian I didn't need to do that. But it was something I'd considered.
15:10 gmaruzz ok! understood. Maybe if the two replicas are too far one eac h other, to not having latency on the FS write, correct?
15:11 ju5t joined #gluster
15:12 ju5t hello, we're running 3.4.2 and apparently some files can't be healed automatically, one node is showing roughly 1k files that are in heal-failed, what are are best bets to solve this? all files seem to be there, with the same attributes, i'm not seeing anything out of the ordinary. any thoughts?
15:19 JoeJulian gmaruzz: Correct.
15:19 gmaruzz THX!
15:19 JoeJulian ju5t: Best bet... upgrade.
15:20 JoeJulian 3.4's eol and there were some significant bug fixes after 3.4.2 even within the 3.4 version.
15:22 haomaiwa_ joined #gluster
15:28 ju5t_ joined #gluster
15:35 EinstCrazy joined #gluster
15:39 stickyboy joined #gluster
15:43 squizzi joined #gluster
15:46 fubada JoeJulian: hi. What am i missing out on by disabling the meta data volume with georepl?
15:47 hagarth joined #gluster
15:49 JoeJulian fubada: no clue. I've not really used geo-rep much yet.
15:51 cholcombe joined #gluster
15:52 CyrilPeponnet hey @JoeJulian strange issue.. I have volume with one brick, I add a new brick as replica 2. Mountpoints of the volume show it is empty but if I know a file in here I can ls it or cd to folders. just looks like the ls is unable to read the entries
15:53 CyrilPeponnet I had to force a heal to make it appear
15:54 hagarth CyrilPeponnet: possibly the readdir landed on the empty brick
15:55 CyrilPeponnet that what I tought
15:55 CyrilPeponnet it freaking scares me this morning guys !
15:56 hagarth CyrilPeponnet: can understand.. are you on 3.6?
15:56 CyrilPeponnet yep
15:56 CyrilPeponnet 3.6.5
15:56 fubada CyrilPeponnet: do you think its troublesome to use 3.6.1 as "master" and 3.7.4 as "slave" to ingest  avolume
15:56 fubada before switching all clients to the new 3.7.4 slave
15:57 CyrilPeponnet @fubada geo-rep as been updated between those release so I don't know really. It's better to align version, you'll see :)
15:57 hagarth CyrilPeponnet: do you have any of the read child selection options set for afr?
15:58 CyrilPeponnet @hagarth not on this volume AFAIK
15:59 theron_ joined #gluster
15:59 hagarth CyrilPeponnet: the right behavior would be to set the good brick as the source for all reads (till healing is complete)
16:00 CyrilPeponnet I use cluster.read-hash-mode 1
16:00 CyrilPeponnet it's not a big deal as no human will browse files
16:00 CyrilPeponnet (and there is no browsing at all)
16:01 hagarth CyrilPeponnet: ok, what are you using this volume for?
16:01 CyrilPeponnet Maybe it should be added to the documentation
16:01 CyrilPeponnet hosting internal docker-registry and some other stuff like deep mirror
16:01 CyrilPeponnet devpi
16:01 haomaiwa_ joined #gluster
16:01 hagarth CyrilPeponnet: yeah, would you want to send across a patch to the docs repository?
16:01 hagarth CyrilPeponnet: cool stuff, sounds interesting. is the docker-registry volume geo-replicated?
16:02 CyrilPeponnet yep
16:02 CyrilPeponnet well
16:02 CyrilPeponnet not anymore
16:02 CyrilPeponnet but was
16:02 CyrilPeponnet :p
16:02 fubada anyone using setfacl inside their gluster mounts and is there any reason why it wouldnt work?
16:02 CyrilPeponnet 2mb/s lan link, not too effecitient
16:02 CyrilPeponnet (wan)
16:02 hagarth ah ok
16:02 hagarth fubada: have you mounted with mount -o acl?
16:03 fubada no, havent tried yet but that would be a prereq
16:03 fubada do you know if its doable?
16:03 hagarth fubada: yes, it is doable with mount -o acl
16:03 fubada cool thanks
16:03 ivan_rossi left #gluster
16:04 nage joined #gluster
16:04 CyrilPeponnet @hagarth: basically we stopped using geo-rep because of this. We are generating builds (around 40GB per day) and it's impossible to geo-rep them (it never converge). So we wrote custom rsync jobs script to send only what is necessary because geo-rep is sending everything (that is the purpose).
16:04 fubada hagarth: how does this gluster handle uid that do not exist across all clients
16:05 hagarth CyrilPeponnet: would be interested in figuring out why geo-rep did not converge. geo-rep makes use of rsync internally to do transfers.
16:05 CyrilPeponnet as I said 2mb/s
16:05 CyrilPeponnet Changelog are stacking and stacking faster than it can send
16:06 hagarth fubada: it is recommended that you have common uids across clients & servers.
16:06 CyrilPeponnet and while it's stuck sending a 10GB files on a "bucket" we have important files to sync (small ones) that are blocked to the queue of changelog.
16:07 hagarth CyrilPeponnet: did you alter the number of worker threads with geo-rep?
16:08 CyrilPeponnet Yep it doesn't help so much, the best way would be to split our vol in several volumes, one with big files the other with the small files (important ones). But was too complicated to put in place
16:08 ssarah joined #gluster
16:08 CyrilPeponnet basically on a 20TB vol data, adding 40/50GB /per day we only need to replicate 5GB of data.
16:09 CyrilPeponnet (geo-replicate)
16:09 hagarth CyrilPeponnet: very interesting. I think we could consider providing priorities for geo-replication (maybe based on file names, sizes etc.)
16:09 CyrilPeponnet but we are using geo-replication on less used volumes as well but for example it took 25days to send 250GB)
16:09 bowhunter joined #gluster
16:09 hagarth CyrilPeponnet: is this also with 3.6.5?
16:09 CyrilPeponnet yes
16:10 hagarth 3.7 has significant improvements in geo-replication. there have been minor perf. improvements too. In any case, 250GB over 25days does seem low compared to what I have seen.
16:10 CyrilPeponnet I will also put in place the geo-rep between two datacenter with a 2GB link (across the street) we will see how it behave
16:11 CyrilPeponnet it's because our wan sucks
16:11 CyrilPeponnet not because of the geo-rep itself
16:11 fubada guys with geo-repl, what am i missing out on by not enabling metadata volume?
16:11 fubada when i tried to enable, all my repl status were faulty
16:11 fubada disabling it fixed the issue
16:12 CyrilPeponnet I'd rather use 3.6.5 for now as we have a lot of client mounting vol with fuse and nfs.
16:12 hagarth fubada: metadata volume provides better node failover for geo-replication
16:12 fubada i wonder why it didnt work for me thanks hagarth
16:13 hagarth CyrilPeponnet: ok, let us know how it goes. the geo-replication developers are also looking for more real world feedback. If you notice anything of interest, do let us know.
16:14 CyrilPeponnet sure. We are using it to replicate data from west coast to east coast and also to europe
16:14 CyrilPeponnet the west coast europe link sucks
16:14 hagarth CyrilPeponnet: I see, ok.
16:15 CyrilPeponnet I had a lot of trouble with geo-rep in 3.5 and I gave up
16:15 CyrilPeponnet works better in 3.6.5
16:15 CyrilPeponnet at least to setup
16:15 CyrilPeponnet thanks hagarth
16:18 rafi joined #gluster
16:19 ssarah hei guys, can a glusterfs have more than one client writing to it at the same time?
16:20 ssarah I have a setup with a fs in two machines, and i want to mount a client for it in those both machines so i can write to one or another indifferently
16:20 ssarah is that ok
16:20 ssarah ?
16:20 ssarah already have the first client as a result of the quick tutorial
16:20 gmaruzz ssarah: yes
16:21 ssarah cool!
16:21 ssarah :)
16:21 gmaruzz be careful to only write on the mounted filesystem, not directly to the exported directory
16:23 CyrilPeponnet nfs is not handling locking so you better use gluster-fuse for for your mounts
16:24 crashmag joined #gluster
16:26 gmaruzz nfs is handlling locking, but anyway you can have better performances with fuse, and also you gain HA without floatingip-failover
16:29 spcmastertim joined #gluster
16:39 CyrilPeponnet gmaruzz well maybe latest nfs does but I had issue with locking over nfs before
16:41 gmaruzz yep latest has NLM
16:47 crashmag joined #gluster
16:49 haomaiwa_ joined #gluster
16:50 gem joined #gluster
16:51 haomaiwang joined #gluster
16:58 theron joined #gluster
16:59 theron joined #gluster
17:01 haomaiwa_ joined #gluster
17:02 deniszh joined #gluster
17:10 ccoffey does readv on /var/run/702bac7988f3397b4d4c0efdc54afb28.socket failed (Invalid argument) mean much ?
17:13 JoeJulian ccoffey: maybe not. Do you have nfs disabled for all your volumes?
17:14 ParsectiX joined #gluster
17:15 ParsectiX joined #gluster
17:15 rwheeler joined #gluster
17:16 ParsectiX joined #gluster
17:17 ParsectiX joined #gluster
17:19 CyrilPeponnet @JoeJulian @hagarth other topic. a /bin/ls is quite fast (few secs for 400 entries in the folder). a /bin/ls --color take like 15 min. Looks like it take a LOT of time to deal with lstat
17:19 CyrilPeponnet any way to improve this ?
17:19 CyrilPeponnet https://gist.github.com/Cyril​Peponnet/ba6f917fa4bfd26317c5
17:20 glusterbot Title: gist:ba6f917fa4bfd26317c5 · GitHub (at gist.github.com)
17:20 ccoffey @JoeJulian, I don't believe i am using nfs. Just having some inconsistency with the mounted gluster vol showing IO errors on some peers and trying to track it down
17:21 CyrilPeponnet Even the subsequent ls --color are slow (looks like caching is not really working for stat / lstat)
17:22 CyrilPeponnet I use performance.readdir-ahead: on cluster.readdir-optimize: on but it looks that it's even worth than before
17:22 JoeJulian CyrilPeponnet: 15 min for 400 entries?!?! 400 million maybe I could see, but 400?
17:22 CyrilPeponnet @JoeJulian yes really this is a pain in the ass
17:23 CyrilPeponnet our gluster setup was slow ok, but not it's barely usable to list dirs
17:24 CyrilPeponnet en a folder with 3 entries take like 10s if stat /lstat are done
17:24 GB21 joined #gluster
17:24 CyrilPeponnet *even
17:24 ParsectiX joined #gluster
17:25 CyrilPeponnet (that why we migrate our user homedirs to plain nfs which is way faster for those kind of operation)
17:25 CyrilPeponnet So I'll take any suggestion / debug / voodoo
17:25 JoeJulian 100.00    0.029832
17:25 JoeJulian That says it took 2 100ths of a second. What am I missing?
17:26 JoeJulian Well, almost 3.
17:26 CyrilPeponnet I have another running right now strace -Tc
17:26 CyrilPeponnet sure it takes several minutes
17:27 CyrilPeponnet I launched the other strike @10:19:59 still not fone
17:27 CyrilPeponnet done
17:27 CyrilPeponnet *strace (fuc*** autocorrect)
17:27 JoeJulian heh
17:28 JoeJulian Don't you mean "duck autocorrect"?
17:28 CyrilPeponnet :p
17:29 CyrilPeponnet hmm
17:29 CyrilPeponnet I have a strange issue
17:29 JoeJulian Just one?
17:29 CyrilPeponnet looks like it takes ages only on one machine
17:29 CyrilPeponnet I took another and the ls is fast
17:31 CyrilPeponnet ex 1444411852.278053 lstat("/blabla", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 <3.881897>
17:32 CyrilPeponnet 3.8s spend in lstat
17:34 jdossey joined #gluster
17:34 CyrilPeponnet I have a lot of things done to this volume on this client un parallel
17:34 shubhendu joined #gluster
17:34 CyrilPeponnet is there a bottle neck (io / thread) client side ?
17:34 CyrilPeponnet how can I debug this ?
17:40 ccoffey while we are tkaing about ls, takes me 5 minutes to ls 500 dir's http://fpaste.org/277192/14444123/
17:40 glusterbot Title: #277192 Fedora Project Pastebin (at fpaste.org)
17:41 ccoffey anyway, enjoy the weekend. I'll go home now
17:45 CyrilPeponnet So yeah I think it depends on what other operation are done on the client
17:48 mhulsman joined #gluster
17:52 CyrilPeponnet because on other hosts it takes 2s and on this one I stopped it at 8min....
17:53 JoeJulian CyrilPeponnet: There's an option to move io-threads to the client. I forget what it is. Not sure it's safe to change live in production.
17:55 maveric_amitc_ joined #gluster
17:56 fubada Does anyone know whats needed to enable metadata vols with 'config use_meta_volume true'? As soon as I issue this, the gep-rep status is Faulty
18:03 JoeJulian fubada: what version is that? That string is not in master nor release-3.7
18:04 JoeJulian Oh, wait. I misread it twice.
18:06 JoeJulian According to the test sequence, you create the geo-rep session, enable the feature, then start it.
18:07 fubada JoeJulian: as soon as I enable the feature the whole thing breaks
18:07 fubada and until I disable the option
18:10 mufa joined #gluster
18:12 jdossey joined #gluster
18:17 togdon joined #gluster
18:19 semiautomatic joined #gluster
18:25 JoeJulian So you have a not-started geo-rep, enable the feature, start geo-rep and it fails, or you have it already running, enable the feature, and it fails.
18:25 JoeJulian ?
18:28 hagarth fubada: do drop a note on gluster-devel and one of the geo-replication developers should be able to diagnose the problem better.
18:28 togdon joined #gluster
18:32 julim_ joined #gluster
18:41 Slashman joined #gluster
18:42 CyrilPeponnet @JoeJulian that's what I thought
18:49 mblaszczak joined #gluster
19:12 a_ta_ joined #gluster
19:22 a_ta_ left #gluster
19:42 kovsheni_ joined #gluster
20:10 DV joined #gluster
20:14 deniszh joined #gluster
20:37 CyrilPeponnet @hagarth @JoeJulian any idea why healing is taking ages ?
20:43 CyrilPeponnet I know I'm annoying ;p
20:44 primehaxor joined #gluster
20:50 Philambdo joined #gluster
20:51 JoeJulian heh, no, annoying is the people that come in all frustrated and blame all their frustration on something they got for free and put in to production without any testing.
20:51 JoeJulian Then they blame all their problems on other people.
20:51 JoeJulian Asking questions is why I'm here.
20:52 JoeJulian self-heal is in a lower priority queue from your fuse mounts. Perhaps that's why?
20:54 RedW joined #gluster
20:56 CyrilPeponnet hmm it make sense as I have tons of fuse mounts
20:56 CyrilPeponnet clients
20:58 CyrilPeponnet maybe that's why the cmd gluster help vol info is freaking slow
20:58 CyrilPeponnet like 1 entry every minutes
21:01 JoeJulian But that just goes to the local glusterd. There's no way that should be slow.
21:01 JoeJulian Maybe you're in swap.
21:01 CyrilPeponnet 64GB per node I doubt it
21:05 CyrilPeponnet not swapping at all
21:19 julim joined #gluster
21:24 EinstCrazy joined #gluster
21:39 stickyboy joined #gluster
22:01 togdon joined #gluster
22:08 DV joined #gluster
22:27 plarsen joined #gluster
22:34 dgbaley joined #gluster
22:43 dgbaley joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary