Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-02-26

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 JoeJulian Why not just fix the network outages?
00:01 kripper JoeJulian: ping-timeouts because the VM is suspended? gluster is installed on the hosts, not the VM
00:02 kripper JoeJulian: I'm trying to make the whole system more fault-tolerant
00:02 kripper [root@h1 ~]# /etc/init.d/glusterfsd restart
00:02 kripper Stopping glusterfsd:                                       [FAILED]
00:02 JoeJulian iirc, you're using qemu with libgfapi. That means that each qemu is a client. If that client stops talking to the server for more than ping-timeout (42 seconds by default) the server will disconnect that client and, I believe, drop any fds.
00:02 kripper no logs
00:03 JoeJulian And that shouldn't work.
00:03 kripper some hint?
00:03 kripper JoeJulian: oh I see
00:05 kripper JoeJulian: what would be the solution? QEMU reopening the file? or gluster reopening FD's? I guess QEMU shuold reopen the files
00:05 kripper JoeJulian: how can I check what is failing whith glusterfsd?
00:06 JoeJulian seems likely that qemu should reopen the file(s)
00:07 JoeJulian glusterfsd is a brick process that's started by glusterd.
00:07 JoeJulian @processes
00:07 glusterbot JoeJulian: The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal).
00:07 kripper JoeJulian: so is it normal that a restart fails?
00:08 JoeJulian I don't know. In openstack if a VM loses its storage, the vm pretty much just goes read-only and you have to reboot it.
00:11 kripper JoeJulian: in oVirt too
00:12 kripper JoeJulian: sometimes it resumes
00:12 JoeJulian probably if it resumes in less than ping-timeout.
00:12 JoeJulian You can set that longer, up to 29 minutes.
00:19 JoeJulian btw, kripper, FDs are only in memory. There is no metadata server, no writing them to disk. If the servers are down, they will have no record of that fd. I /think/ that as long as one server is up, though, the fd should be retained. Not sure when it's not replicated, though, if that's true.
00:34 kripper JoeJulian: right, I sent an mail to the qemu-devel list to ask if FD's are reopened on a 'cont' (resume) command
00:34 kripper JoeJulian: it would be very usefull when working with images on network mounts (gluster, NFS, etc).
00:34 JoeJulian +1
00:36 T3 joined #gluster
00:36 kripper JoeJulian: btw, I'm a little confused about libfgapi...I know oVirt is supporting it, but how can I check if QEMU is really using it?
00:37 JoeJulian Check ps. If the path for the disk image is gluster:/$server/$volume/$path then it is.
00:37 kripper JoeJulian: I see QEMU is receving an arg  file=<gluster-mount-path>/image
00:37 kripper JoeJulian: well, it isn't:
00:37 kripper -drive file=/rhev/data-center/
00:38 JoeJulian Well then, check /var/log/glusterfs/rhev-data-center*.log
00:39 kripper JoeJulian:  actually, the path is a symbolic link to a fuse.glusterfs mount
00:40 JoeJulian Ok, well you get the idea. Make your problem happen and read the client log.
00:40 kripper any keyword to look for in the logs?
00:41 JoeJulian Not yet. I think you want to figure out, more, what it's actually doing before you can start figuring out how to make it not do it.
00:43 kripper JoeJulian: just wanted to check if libgfapi was really being used. I don't think so
00:43 JoeJulian That was a pretty clear indication.
00:43 kripper http://www.ovirt.org/Featur​es/GlusterFS_Storage_Domain
00:43 kripper have to go
00:44 kripper good night!
00:44 kripper left #gluster
00:44 JoeJulian o/
01:01 plarsen joined #gluster
01:21 MugginsM joined #gluster
01:47 bala joined #gluster
02:02 jbrooks_ joined #gluster
02:02 jbrooks joined #gluster
02:10 harish_ joined #gluster
02:10 bala joined #gluster
02:10 deniszh joined #gluster
02:15 dustint joined #gluster
02:25 rjoseph joined #gluster
02:48 rjoseph joined #gluster
02:54 victori joined #gluster
02:56 Gill joined #gluster
03:07 bharata-rao joined #gluster
03:09 victori joined #gluster
03:20 victori joined #gluster
03:44 nangthang joined #gluster
03:48 RameshN joined #gluster
03:55 prasanth_ joined #gluster
03:56 soumya joined #gluster
03:59 shubhendu joined #gluster
04:01 rafi joined #gluster
04:02 anoopcs joined #gluster
04:04 d-fence joined #gluster
04:10 RameshN joined #gluster
04:13 DV__ joined #gluster
04:14 nishanth joined #gluster
04:16 sprachgenerator joined #gluster
04:16 atinmu joined #gluster
04:27 kanagaraj joined #gluster
04:28 gem joined #gluster
04:30 nbalacha joined #gluster
04:31 nishanth joined #gluster
04:31 Manikandan joined #gluster
04:35 victori joined #gluster
04:35 itpings hi guys
04:45 victori joined #gluster
04:46 kshlm joined #gluster
04:47 deepakcs joined #gluster
04:51 jiffin joined #gluster
04:54 spandit joined #gluster
04:56 victori joined #gluster
04:59 JoeJulian salut
05:01 shubhendu joined #gluster
05:03 smohan joined #gluster
05:04 rjoseph joined #gluster
05:10 schandra joined #gluster
05:16 ndarshan joined #gluster
05:17 aravindavk joined #gluster
05:18 victori joined #gluster
05:22 raghu joined #gluster
05:27 badone_ joined #gluster
05:29 kumar joined #gluster
05:31 meghanam joined #gluster
05:41 coredump joined #gluster
05:45 badone_ joined #gluster
05:46 karnan joined #gluster
05:52 ramteid joined #gluster
05:54 atalur joined #gluster
05:55 bala joined #gluster
06:01 kdhananjay joined #gluster
06:01 ppai joined #gluster
06:03 anrao joined #gluster
06:08 vimal joined #gluster
06:08 badone_ joined #gluster
06:09 vipulnayyar joined #gluster
06:14 _Bryan_ joined #gluster
06:18 Apeksha joined #gluster
06:19 dusmant joined #gluster
06:24 aravindavk joined #gluster
06:27 overclk joined #gluster
06:32 bala1 joined #gluster
06:34 R0ok_ joined #gluster
06:35 Manikandan joined #gluster
06:43 nishanth joined #gluster
06:44 SOLDIERz_ joined #gluster
06:49 itpings has any one proof read my howto ?
06:54 rjoseph joined #gluster
06:57 Apeksha joined #gluster
07:05 hagarth joined #gluster
07:05 ricky-ticky joined #gluster
07:07 lalatenduM joined #gluster
07:12 prasanth_ joined #gluster
07:15 rjoseph joined #gluster
07:16 DV__ joined #gluster
07:21 Andreas-IPO joined #gluster
07:32 vimal joined #gluster
07:34 SOLDIERz_ joined #gluster
07:48 nishanth joined #gluster
07:52 Philambdo joined #gluster
07:56 spot joined #gluster
07:56 bala joined #gluster
08:01 spot joined #gluster
08:04 victori joined #gluster
08:06 spot_ joined #gluster
08:06 victori joined #gluster
08:13 plarsen joined #gluster
08:16 suliba joined #gluster
08:18 kovshenin joined #gluster
08:23 nshaikh joined #gluster
08:35 [Enrico] joined #gluster
08:38 hagarth joined #gluster
08:49 anrao joined #gluster
08:49 ndarshan joined #gluster
08:50 nishanth joined #gluster
08:55 stickyboy joined #gluster
08:55 stickyboy joined #gluster
08:56 SOLDIERz_ joined #gluster
08:57 R0ok_ joined #gluster
08:57 ctria joined #gluster
09:00 jtux joined #gluster
09:03 T0aD joined #gluster
09:04 vipulnayyar joined #gluster
09:07 jtux joined #gluster
09:11 liquidat joined #gluster
09:14 kovshenin joined #gluster
09:18 meghanam joined #gluster
09:18 soumya joined #gluster
09:18 rjoseph joined #gluster
09:19 hagarth joined #gluster
09:25 o5k joined #gluster
09:26 _shaps_ joined #gluster
09:45 Harry joined #gluster
09:47 Harry Hi all, I am a newbie on gluster, just setup a 2 brick (replica) but see gluster processes are top cpu consumer
09:47 Harry Looking for pointers to where to look into
09:47 Harry Its on Centos 6.6 XFS based bricks with isize of 512
09:48 ndarshan joined #gluster
09:48 ndevos Harry: can you ,,(paste) 'gluster volume info' and mention the version you're using?
09:48 glusterbot Harry: For RPM based distros you can yum install fpaste, for debian, ubuntu, and arch it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
09:49 Harry 1 moment ndevos
09:49 Harry [root@yudhister ~]# gluster volume info gvol   Volume Name: gvol Type: Replicate Volume ID: fcafee02-bd12-45b7-a913-3098a268825d Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: yudhister:/dfs/brick Brick2: nakul:/dfs/brick
09:51 ctria joined #gluster
09:51 Harry http://ur1.ca/jt062
09:53 ninkotech joined #gluster
09:54 Harry @ndevos, http://ur1.ca/jt062 has the volume info
09:54 ninkotech_ joined #gluster
09:58 ndevos Harry: okay, and where do you mount the volume?
09:58 Harry ndevos: http://ur1.ca/jt07a
09:59 ndevos Harry: hmm, that looks ok - what is the commandline of the process that causes the high load?
10:00 Slashman joined #gluster
10:01 Harry ndevos http://ur1.ca/jt07q
10:02 Harry glusterfs and glusterfsd
10:02 ndevos Harry: do you have the complete commandline?
10:05 Harry 1 moment
10:07 Harry ndevos: http://ur1.ca/jt08l
10:09 ndevos Harry: hmm, you have two processes for the fuse-mount, that surely looks wrong
10:10 Harry ok, you mean /usr/sbin/glusterfs
10:10 ndevos yes, glusterfs is a client process (used by fuse, nfs, selfheal, ...)
10:11 ndevos and, in your case, the glusterfs processes both are for fuse, and mount /opt
10:11 Harry ok, let me see how it happened
10:12 ndevos Harry: does the load settle again when you unmount /opt completely, and mount it only once?
10:12 Harry Niel, I did not try , wait a sec let me try that
10:19 hagarth joined #gluster
10:21 nbalacha joined #gluster
10:26 zwevans joined #gluster
10:27 Harry ndevos: look it this - http://ur1.ca/jt0by, it was actuall one process, but earlier I posted "htop" to fpaste to ingore old one
10:27 Harry its only 1 process
10:27 Harry ps aux | grep "/usr/sbin/glusterfs" outputs this - http://ur1.ca/jt0by
10:30 Harry ndevos: I rebooted server  and waiting it to start fully, so far looks ok
10:32 o5k joined #gluster
10:33 Harry ndevos: its still bad, I think so
10:35 Harry we have 40 jetty instances in gluster mounted volume and at start glusterfs touching ~70% cpu
10:36 deniszh joined #gluster
10:37 ndevos Harry: what is Jetty?
10:38 Harry its web server for java applications
10:39 Harry as of now glusterfs consuming 110% cpu and glusterfsd 120% cpu
10:39 social joined #gluster
10:40 Harry ndevos: see thsi one http://ur1.ca/jt0es
10:40 ndevos Harry: you might get hit by an issue that is common for ,,(php) too
10:40 glusterbot Harry: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
10:40 glusterbot --fopen-keep-cache
10:42 Harry ok, let me try this
10:46 abyss^ It is possible to ease glusterfs rebalancing? It takes all sys resources :/
10:49 deniszh joined #gluster
10:50 ramteid joined #gluster
10:51 abyss^ joined #gluster
10:57 SOLDIERz_ joined #gluster
10:58 nbalacha joined #gluster
10:58 jdarcy joined #gluster
10:58 badone_ joined #gluster
11:00 Pupeno joined #gluster
11:00 Pupeno joined #gluster
11:00 anrao joined #gluster
11:02 glusterbot News from newglusterbugs: [Bug 1196584] RDMA: [RFE] Cleaner log messages when RDMA volumes fail to mount. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1196584>
11:04 abyss^ joined #gluster
11:08 o5k joined #gluster
11:14 Norky joined #gluster
11:14 deniszh1 joined #gluster
11:15 deniszh joined #gluster
11:19 pcaruana joined #gluster
11:23 deniszh joined #gluster
11:26 Harry how to mount with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
11:26 Harry I dont see any help even 15 min of gogle
11:27 Harry can someone help?
11:27 shubhendu joined #gluster
11:28 liquidat joined #gluster
11:28 Harry one quick question for you all - is glusterfs is good choice when we have lot of web servers which frequenly read write to disk, less size but frequent
11:29 shubhendu joined #gluster
11:33 atalur joined #gluster
11:36 Harry how to mount with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
11:36 Harry ?
11:36 Harry how to mount with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
11:36 Harry ?
11:36 Harry how to mount with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH?
11:40 ndevos Harry: you can execute the whole glusterfs command for the mount (that you gad in 'ps') by hand, and add those options
11:40 * ndevos isnt sure if those options can be passed as mount-options in /etc/fstab
11:41 firemanxbr joined #gluster
11:41 ndevos s/gad/had/
11:41 glusterbot What ndevos meant to say was: An error has occurred and has been logged. Please contact this bot's administrator for more information.
11:41 ndevos JoeJulian: maybe glusterbot could get fixed somehow to not do that?
11:44 Harry ndevos: let me try
11:46 lalatenduM joined #gluster
11:46 Harry glusterbot : /usr/sbin/glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH --fopen-keep-cache --volfile-server=localhost --volfile-id=/gvol /opt
11:46 Harry glusterfs: unknown negative timeout HIGH
11:49 LebedevRI joined #gluster
11:50 Harry ndevos: is glusterfs is good choice when we have lot of web servers which frequenly read write to disk, less size but frequent ?
11:50 ndevos Harry: HIGH in that command should be a number of seconds :)
11:51 dlambrig_ joined #gluster
11:51 Harry ndevos: thanks, all HIGH values should be numbers ? correct?
11:51 ndevos Harry: glusterfs is not high performance for small file workloads, the choise of "good" depends on the expectations that you have
11:52 ndevos yeah, all timeout options should be in seconds
11:53 Harry ndevos: should I try with 300 seconds?
11:53 ndevos Harry: you can, but it might cause delays in updates on other glusterfs mount points
11:54 ndevos as in, other glusterfs mountpoints may not notice when a file was changed
11:54 Harry we are actualy okay with 10 min delayed copy
11:54 ndevos until the timeout expired
11:55 Harry ah, ok
11:56 Harry ndevos,what value is usual?
12:01 ndevos Harry: I normally use the defaults, no idea what others use
12:01 tanuck joined #gluster
12:02 Harry ndevos: can you point me document so I can read in depth for thease options
12:02 ndevos Harry:
12:02 ndevos maybe "glusterfs --help" ?
12:03 glusterbot News from newglusterbugs: [Bug 1196615] [dht]: Failed to rebalance files when a replica-brick-set was removed <https://bugzilla.redhat.co​m/show_bug.cgi?id=1196615>
12:03 ndevos or 'man glusterfs'
12:03 o5k_ joined #gluster
12:04 Debloper joined #gluster
12:04 marbu joined #gluster
12:04 Harry thanks ndevos, let me try to to understand deeper
12:06 ndevos Harry: attribute = attributes from 'stat $FILE', entry = directory entries (new files, deleted ones), negative = (try to) access a non-existing file
12:14 R0ok_ joined #gluster
12:14 shubhendu joined #gluster
12:16 harish_ joined #gluster
12:18 Harry ndevos: is is possible to use gluster foe delayed "sync all", lets say every night not realtime
12:18 stickyboy joined #gluster
12:18 stickyboy joined #gluster
12:19 anrao joined #gluster
12:19 hchiramm joined #gluster
12:20 ndevos Harry: not really, unless you have one master site that has write access and you want to push those changes out asynchronously (that would be called geo-replication)
12:21 Harry I think that will be the choice for us
12:22 Harry ndevos: I was reading http://joejulian.name/blog/glust​erfs-replication-dos-and-donts/
12:23 Harry and joe says that we should not run "servers" on replica
12:24 Harry currenlty I have 2 physical servers from hetzner and I am runing glusterd on both servers
12:24 Harry will this be a problme?
12:27 stickyboy joined #gluster
12:27 stickyboy joined #gluster
12:28 ppai joined #gluster
12:28 ndevos Harry: I do not think there should be any problems, but performance may not be very high
12:31 kanagaraj joined #gluster
12:33 R0ok_ joined #gluster
12:38 jessexoc joined #gluster
12:39 _shaps_ Hi, I'm trying to do a replace-brick, but when doing "gluster peer probe" I get Peer rejected (Connected). I can see this happening in the logs http://pastebin.com/duvA5dS4
12:39 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
12:40 _shaps_ Re: http://fpaste.org/190824/42495441/
12:40 mbukatov joined #gluster
12:40 _shaps_ (I am adding gfs3 to the pool which will replace gfs2 )
12:41 rjoseph joined #gluster
12:42 diegows joined #gluster
12:43 jessexoc joined #gluster
12:44 ira_ joined #gluster
12:47 SOLDIERz_ joined #gluster
12:50 SOLDIERz_ joined #gluster
12:53 shubhendu joined #gluster
12:56 shempe joined #gluster
12:57 anoopcs joined #gluster
12:59 shempe hi everyone
13:02 Harry ~php
13:03 Harry @php
13:03 glusterbot Harry: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
13:03 glusterbot Harry: --fopen-keep-cache
13:03 Harry @logs
13:03 glusterbot Harry: I do not know about 'logs', but I do know about these similar topics: 'chat logs', 'logstash'
13:03 Harry @log
13:03 glusterbot Harry: I do not know about 'log', but I do know about these similar topics: 'Joe's blog', 'chat logs', 'loglevel', 'logstash'
13:03 Harry @logstash
13:03 glusterbot Harry: semiosis' logstash parser for glusterfs logs: https://gist.github.com/1499710
13:03 Harry @java
13:03 glusterbot Harry: https://github.com/semiosis​/glusterfs-java-filesystem
13:04 Slashman_ joined #gluster
13:04 Harry @geo
13:04 glusterbot Harry: I do not know about 'geo', but I do know about these similar topics: 'geo-replication'
13:04 Harry @geo-replication
13:04 glusterbot Harry: See the documentation at http://download.gluster.org/pub/gluste​r/glusterfs/3.2/Documentation/AG/html/​chap-Administration_Guide-Geo_Rep.html
13:07 anil joined #gluster
13:07 shempe @netstorm
13:10 zwevans left #gluster
13:10 mbukatov joined #gluster
13:14 [Enrico] joined #gluster
13:15 Slashman joined #gluster
13:16 hchiramm joined #gluster
13:16 kanagaraj joined #gluster
13:19 anti[Enrico] joined #gluster
13:20 vipulnayyar joined #gluster
13:22 hybrid512 joined #gluster
13:26 shempe short question i have a gluster volume replicated between 2 peers and geo synced to another host all is ok so far and working. There is nearly 0 traffic on the volume. But i get massive Network traffic over the dedicated 20GB channel (72 PiB) ind 2 Days and i have no clue why .
13:32 hybrid512 joined #gluster
13:35 harish_ joined #gluster
13:41 Harry @php
13:41 glusterbot Harry: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
13:41 glusterbot Harry: --fopen-keep-cache
13:41 Harry @java
13:41 glusterbot Harry: https://github.com/semiosis​/glusterfs-java-filesystem
13:47 o5k_ joined #gluster
13:49 bernux joined #gluster
13:50 tberchenbriter_ joined #gluster
13:51 B21956 joined #gluster
13:52 hagarth joined #gluster
13:54 R0ok_ @python
13:54 glusterbot R0ok_: I do not know about 'python', but I do know about these similar topics: 'python-gluster'
13:54 R0ok_ @python-gluster
13:54 glusterbot R0ok_: Check out https://github.com/joejulian/python-gluster
13:54 tberchenbriter_ joined #gluster
13:55 R0ok_ @c++
13:55 glusterbot R0ok_: c's karma is now 1
13:55 R0ok_ @C
13:55 theron joined #gluster
13:56 wkf joined #gluster
13:56 R0ok_ @R0ok++
13:56 glusterbot R0ok_: R0ok's karma is now 1
13:56 R0ok_ @R0ok++
13:56 glusterbot R0ok_: R0ok's karma is now 2
13:56 dooder123 joined #gluster
13:56 tberchenbriter_ joined #gluster
13:57 theron_ joined #gluster
13:59 JustinClift tigert:
13:59 misc JustinClift: the wordpress stuff ?
14:00 misc JustinClift: also, I didn't understood, you removed VM from rackspace, but you added new ones ?
14:00 tberch8 joined #gluster
14:00 JustinClift tigert: What's the right way to get the content from here, into the static site? http://www.gluster.org/community/d​ocumentation/index.php/Who_is_Who
14:00 JustinClift misc: ?
14:01 misc JustinClift: last mail from Humble on gluster infra
14:01 * JustinClift looks
14:01 misc because tigert did tried to fix the issue on wordpress :)
14:01 JustinClift Ahhhh
14:01 misc but you ping is unrelated
14:02 JustinClift yeah, humble jumped into the gluster-meeting channel and got their attention about the issue
14:02 JustinClift It looks like it was caused by tigert doing stuff WP admin stuff on feeds yesterday, with unexpected consquences
14:02 ndarshan joined #gluster
14:02 JustinClift So, learning experience there, and (in theory) it shouldn't be an ongoing issue ;)
14:03 misc we have a list of irc channel we use ?
14:04 JustinClift misc: Looking here, it only lists the main user channel: http://www.gluster.org/community/
14:04 JustinClift And the footer for the page has the same
14:05 JustinClift Interestingly, now that I'm looking at the footer I can barely read it
14:05 JustinClift That's a _very_ dark colour of text
14:05 kshlm joined #gluster
14:05 * JustinClift is using a fairly colour accurate monitor, so it's unlikely to be "just me" ;)
14:05 misc JustinClift: it is hard to read
14:06 misc now, that's quite fun to see that the (c) is already adjusted for 2015
14:06 misc while the last modification is in 2014 :)
14:06 JustinClift :)
14:20 misc JustinClift: on http://www.gluster.org/community/do​cumentation/index.php/Jenkins_setup , there is a step on disable eth1, you have a pointer to the problem ?
14:22 rsf joined #gluster
14:27 squizzi joined #gluster
14:28 georgeh-LT2 joined #gluster
14:28 dgandhi joined #gluster
14:30 JustinClift misc: Gluster only supports using one IP address
14:30 JustinClift It literally can't handle more
14:30 DV__ joined #gluster
14:39 misc JustinClift: yeah, but there is a bug report somewhere, or that's unfixable by design ?
14:42 rsf joined #gluster
14:42 o5k joined #gluster
14:43 deepakcs joined #gluster
14:46 ctria joined #gluster
14:46 _Bryan_ joined #gluster
14:48 JustinClift misc: We're looking at fixing it, but it's non-trivial
14:49 JustinClift misc: We did a bunch of dev work for the 3.6 series, getting a lot of the connection handling into one spot
14:49 JustinClift (apparently is was all over the place before)
14:49 misc well, we have to keep the workaround for a while
14:49 misc I am looking on how we can automate the deployment of jenkins with salt
14:49 JustinClift And Jeff Darcy has a proposal for getting multiple-network handling happening in the next release or two...
14:49 JustinClift misc: Cool
14:50 misc JustinClift: but I have a few question, like
14:50 misc "why use nagins from a non epel repo" :)
14:50 Folken_ JustinClift: in a disperse volume, is it a good idea to run find /mnt -d -exec getfattr -h -n trusted.ec.heal {} \;
14:50 Folken_ JustinClift: via cron each night?
14:50 JustinClift Folken_: I personally have no idea
14:51 JustinClift Folken_: I've become very hands off with GlusterFS itself, doing more stuff like keeping our Jenkins infrastructure running, and other not-GlusterFS-itself Community admin-y tasks :/
14:51 dlambrig_ left #gluster
14:51 Folken_ fair enough ;-p
14:51 JustinClift misc: I have no idea what a nagins is ?
14:51 misc JustinClift: nginx, ouos
14:51 JustinClift nagios ?
14:51 JustinClift Ahhh
14:52 JustinClift Why would we get it from the epel repo, instead of the vendor repo created by them specifically for our OS?
14:52 JustinClift Seems like a weird question
14:53 hagarth joined #gluster
14:54 JustinClift misc: I suppose it's kind of because I think about it the same way I think of GlusterFS EPEL repo...
14:55 JustinClift Gluster provides it's own RPM's for CentOS and similar.  We put time and effort into making sure they work
14:55 JustinClift The Gluster EPEL RPM's... I'm not sure how they're made, or by who. ;)
14:55 JustinClift (could be the same for all I know, I just don't)
14:59 T3 joined #gluster
14:59 o5k_ joined #gluster
14:59 rsf joined #gluster
15:00 misc JustinClift: as a sysadmin, i tend to be wary of 3rd party repo, mostly because I did review of some of them for fedora inclusion
15:00 JustinClift k
15:00 misc JustinClift: that's this day that my alcohol problem started to come again :)
15:00 JustinClift It's for a basic webserver, so I don't think it's super imporant
15:00 swebb joined #gluster
15:00 JustinClift misc: So, pick whichever location you're happy with :)
15:01 johnn joined #gluster
15:06 bennyturns joined #gluster
15:07 o5k_ joined #gluster
15:08 5EXAALOVG joined #gluster
15:11 soumya joined #gluster
15:12 sputnik13 joined #gluster
15:17 rsf joined #gluster
15:17 o5k_ joined #gluster
15:25 corretico joined #gluster
15:25 ninkotech joined #gluster
15:26 ninkotech_ joined #gluster
15:32 harish_ joined #gluster
15:32 deniszh1 joined #gluster
15:34 o5k_ joined #gluster
15:34 rsf joined #gluster
15:37 ricky-ticky1 joined #gluster
15:38 o5k joined #gluster
15:41 plarsen joined #gluster
15:43 ricky-ticky joined #gluster
15:43 luis_silva joined #gluster
15:44 luis_silva Hey all, quick question on brick remove. Does it delete the files on that brick automatically?
15:44 o5k_ joined #gluster
15:45 DV joined #gluster
15:45 o5k__ joined #gluster
15:59 Leildin I don't think it does luis_silva, I've removed a brick and re-attached it and still had my data
15:59 Leildin you mean the remove-brick command right ?
16:00 luis_silva yup
16:00 luis_silva thx for the info
16:01 Leildin if you re-attach the brick you nee to reconstruct the metadata but otherwise no data is lost
16:02 soumya joined #gluster
16:04 sage joined #gluster
16:13 rsf_ joined #gluster
16:21 papamoose joined #gluster
16:47 kkeithley1 joined #gluster
17:04 gem joined #gluster
17:09 ildefonso joined #gluster
17:15 T3 joined #gluster
17:20 PeterA joined #gluster
17:34 glusterbot News from resolvedglusterbugs: [Bug 1196775] [dht]: Failed to rebalance files when a replica-brick-set was removed <https://bugzilla.redhat.co​m/show_bug.cgi?id=1196775>
17:45 neofob joined #gluster
17:49 ira joined #gluster
17:50 jobewan joined #gluster
17:52 Pupeno joined #gluster
17:56 kanagaraj joined #gluster
17:56 Rapture joined #gluster
17:59 karnan joined #gluster
18:02 T0aD joined #gluster
18:05 kbyrne joined #gluster
18:11 victori joined #gluster
18:14 nmbr joined #gluster
18:15 jbrooks_ joined #gluster
18:18 squizzi joined #gluster
18:20 cmtime joined #gluster
18:20 squizzi joined #gluster
18:22 mator_ joined #gluster
18:25 virusuy joined #gluster
18:25 virusuy joined #gluster
18:25 chirino joined #gluster
18:28 coredump joined #gluster
18:28 jbrooks joined #gluster
18:31 jackdpeterson Hey all, got a weird issue that I'd like some help with. We have a gluster mount point that hangs when instances (web heads) come into service. So if one performs a mount -t glusterfs server_ip:/volume /var/www the server just hangs. NFS mounts fine, and systems that already have the mount via Fuse are okay. it's new clients that are hanging. Also note, one of the gluster servers is in a failed state (replica 2)
18:32 jackdpeterson Gluster 3.6.2, CentOS 6 (server side). Gluster 3.6.2 clients, Ubuntu 14.04
18:33 jackdpeterson err, the gluster server doesn't hang... the client attempting the mount does.
18:44 jbrooks joined #gluster
18:45 jbrooks joined #gluster
18:54 jackdpeterson @Reply (self) looks like it has somethign to do with the number of things that need to heal and so forth. had a script that was watching for failed mounts and would kill the mount and attempt a remount before the client could properly get a handle on all of the things that need to heal. first mount takes ~20 + seconds.
18:54 glusterbot jackdpeterson: (self) looks like it has somethign to do with the number of things that need to heal and so forth. had a script that was watching for failed mounts and would kill the mount and attempt a remount before the client could properly get a handle on all of the things that need to heal. first mount takes ~20 + seconds.
18:54 DV joined #gluster
18:56 ekuric joined #gluster
18:57 stickyboy joined #gluster
19:02 T3 joined #gluster
19:02 cmtime I am having trouble to run a rebalance on a now 3 node distributed.  Node 2 of the original 2 node setup shows up as failed about 1 minute into the rebalance.
19:07 rafi joined #gluster
19:30 JoeJulian jackdpeterson: There's a background-self-heal queue. Once that is exceeded, io waits on the new fd until that file is healed.
19:31 T3 joined #gluster
19:31 jackdpeterson so what does that mean with one of the two servers in a failed state (we have one powered off). bringing it back into the rotation cause all mounts to hang w/ epic io errors
19:41 jackdpeterson *caused .... last time we tried to do the restoration process as documented in the RHS 3.0 documentation
19:41 JoeJulian Only if those mounts touch more than background-self-queue files.
19:42 JoeJulian And those files aren't healed in time
19:42 jackdpeterson so if we don't start the volume heal side of things ... then eventually they get consistent? is that the idea
19:43 JoeJulian I think it's a little of that, a little of legacy (there once wasn't a self-heal daemon), a little redundancy...
19:44 JoeJulian My personal preference would be to not block and instead use the clean copy, but perhaps that complicates the rest of the operations that rely on having consistent replicas.
19:47 jackdpeterson At this point, I've already configured a new cluster w/ quorum and we're rsyincing things over. it's just a painful process at this point to try and restore things w/o risking outage. at least w/ a new environment I can cleanly do it w/ loadbalancer failover
19:48 JoeJulian +1
19:51 cmtime I am having trouble to run a rebalance on a now 3 node distributed.  Node 2 of the original 2 node setup shows up as failed about 1 minute into the rebalance.
20:05 squizzi joined #gluster
20:09 deniszh joined #gluster
20:12 coredump|br joined #gluster
20:19 T3 joined #gluster
20:26 cornfed78 joined #gluster
20:27 cornfed78 hi all.. hoping someone might have an insight on this.
20:27 cornfed78 I have a 3-brick replicated volume of ~100GB
20:27 cornfed78 earlier, one of the machines got rebooted, and it seemed to be corrupted
20:27 cornfed78 so I reduced the replica count to 2 bricks, removing the corrupted one, reformatted the brick, and added it back in
20:28 cornfed78 The space used on the other 2 bricks was about 10G
20:28 cornfed78 but after adding the brick in, it stopped syncing after about 6G
20:28 cornfed78 and it appeared that it didn't sync all the data
20:29 cornfed78 so I did a "volume heal <volume> full" based on what I managed to google
20:29 cornfed78 now, however, one of the original bricks is using ~62 GB, while the other two are using 10 GB and 7GB respectively
20:29 cornfed78 not sure what's going on
20:29 cornfed78 but I'm assuming all 3 should be roughly the same size, since it's a 3-brick replica?
20:30 coredump joined #gluster
20:30 cornfed78 how do I force gluster to resync the data so that the 3 bricks look the same on each node?
20:31 cornfed78 that 3rd brick just keeps getting bigger and bigger, too :\
20:34 rotbeard joined #gluster
20:41 skippy joined #gluster
20:41 skippy `gluster volume log <volume> rotate` reports success, but doesn't actually rotate any logs.  Where should I begin troubleshooting?
20:42 rwheeler joined #gluster
20:53 Rapture joined #gluster
21:01 partner skippy: that should rotate the *brick* logs
21:01 partner rest is up to (mostly broken) logrotate, +- your version of platform and packages
21:02 skippy thanks partner.  I just realized the same.
21:03 partner i don't know if the logrotate was fixed already for whatever version you're using
21:09 badone_ joined #gluster
21:12 skippy doesn't appear to have been.  A manual invocation of `logrotate -f` pointing at a custom config did the trick. :(
21:14 andreask joined #gluster
21:25 skippy left #gluster
21:29 social_ joined #gluster
21:29 social joined #gluster
21:30 partner if its debian its not fixed. and rh versions had some issues aswell thought should be addressed already, plenty of bug reports around the topic
21:31 partner oh, left already, too many joins/parts i've ignored them all..
21:39 elico joined #gluster
21:45 MugginsM joined #gluster
21:51 jbrooks joined #gluster
21:52 bennyturns joined #gluster
21:53 MugginsO joined #gluster
21:54 corretico joined #gluster
22:10 kovshenin joined #gluster
22:12 JoeJulian cmtime: Look in your rebalance logs to see why your rebalance is failing.
22:21 Pupeno joined #gluster
22:24 Guest58 joined #gluster
22:24 Guest58 hi everyone
22:24 kenansulayman joined #gluster
22:24 Guest58 I seem to be having this exact issue:
22:24 Guest58 https://bugzilla.redhat.com/show_bug.cgi?id=911361
22:24 glusterbot Bug 911361: high, unspecified, ---, bugs, CLOSED DEFERRED, Bricks grow when other bricks heal
22:25 JoeJulian I'm sure I speak for everyone when I return your greeting.
22:25 Guest58 thanks :)
22:25 Guest58 I'm running glusterfs-3.6.2-1.el6.x86_64
22:25 Guest58 it's really odd
22:26 Guest58 *one* of the bricks is growing exponentially
22:26 Guest58 but I can't figure out why
22:26 Guest58 or what it's doing
22:26 Guest58 this volume is attached to an oVirt cluster
22:26 Guest58 the problem started when I rebooted a node in the cluster
22:26 Guest58 and it seems like the self-heal is running amok
22:27 Guest22028 joined #gluster
22:27 Guest58 There's only about 10G of data on that volume (a 3-way replica).. but right now, the one brick is at 57GB
22:27 Guest58 and growing fast
22:28 Guest58 It happened earlier, too, so I deleted & recreated the volume
22:28 Guest58 then rebooted a machine to see if I could reproduce it..
22:28 Guest58 sure enough, it's going nuts again
22:28 Guest58 @62GB now
22:28 Guest58 so, about a minute since the last check
22:29 Guest58 @65GB
22:30 Guest58 @70GB :)
22:30 JoeJulian Self-heal, without running a heal...full, walks the tree in $brick_root/.glusterfs/indices/xattrop and heals those files against that brick's replica(s). The heal, by default, walks the source file and looks for differences in the sink file. When there are differences, that section is locked, written, then unlocked.
22:31 Guest58 OK.. Why would that procedure cause this kind of growth?
22:31 JoeJulian Right?
22:31 JoeJulian Without writing, how could it?
22:32 Guest58 well, it's 7x the amount of data in the volume
22:32 Guest58 the volume used ~9GB
22:32 JoeJulian So, assuming the impossible isn't possible, something else must be happening. have you checked du?
22:33 JoeJulian I like, "du -ax $brick_root | sort -n | tail -n 50" to get the top 50.
22:33 Guest58 OK, so yeah, that's weird.. 'df' says 76GB used, but 'du' says 50GB
22:33 JoeJulian That would be the difference between allocated and actual.
22:33 Guest58 26214412./brick/.glusterfs/b8
22:33 Guest58 26214412./brick/.glusterfs/b8/78
22:33 Guest58 26214412./brick/.glusterfs/b8/78/b8​78056c-a282-46c4-beaa-c09a954e6997
22:33 Guest58 26442324./brick/.glusterfs/b2
22:33 Guest58 26442324./brick/.glusterfs/b2/44
22:33 Guest58 26442324./brick/.glusterfs/b2/44/b2​442b5f-32c6-4442-a7ac-41001aaea1a8
22:33 Guest58 52712712./brick/.glusterfs
22:33 Guest58 52712720.
22:33 Guest58 52712720./brick
22:33 JoeJulian no!!!!
22:33 JoeJulian @paste
22:33 glusterbot JoeJulian: For RPM based distros you can yum install fpaste, for debian, ubuntu, and arch it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
22:34 Guest58 ah
22:34 Guest58 ok
22:34 Guest58 sorry :)
22:34 JoeJulian I was afraid I was going to see 50 lines. :D
22:34 Guest58 no
22:34 Guest58 heh
22:35 Guest58 fpaste in epel?
22:35 Guest58 cause : Warning: No matches found for: fpaste
22:35 neofob left #gluster
22:36 JoeJulian Focusing on that largest file, that's a GFID hardlink. To find that file, you would have to match that file's inode number to the same inode number elsewhere on the disk.
22:36 JoeJulian @gfid lookup
22:36 JoeJulian @factoid lookup
22:36 JoeJulian @meh
22:36 glusterbot JoeJulian: I'm not happy about it either
22:36 firemanxbr joined #gluster
22:37 JoeJulian @gfid resolver
22:37 glusterbot JoeJulian: https://gist.github.com/4392640
22:37 JoeJulian Guest58: ^ That link is how you would match that gfid file to its filename.
22:38 MugginsM joined #gluster
22:38 Guest58 ah.. i did 'ls -i' on that file, got the inode, then ran 'find -inum ...'
22:39 JoeJulian excellent. :D
22:40 JoeJulian So... should that file be 26gig?
22:40 Guest58 no
22:40 JoeJulian I assume that's a VM image. What's the size that ovirt says it should be?
22:40 Guest58 yeah
22:41 Guest58 well, i sparse-allocate vm images
22:42 Guest58 none should be bigger than about 2GB right now, since I'm just testing
22:42 Guest58 that being said, I do have some defined as 40G
22:42 JoeJulian So, they're unsparsifying somehow.
22:42 * JoeJulian invents words whenever he feels like it.
22:43 Guest58 yeah, the "acutal size" is 2GB
22:43 Guest58 hm
22:43 JoeJulian I wonder if it has to do with locking the file.
22:44 JoeJulian I wish filesystems had a way to deallocate empty inodes in the middle of files easily.
22:44 partner i'm sorry but i do not have an answer to acutal sized unparsifying files issue, have you checked the bugzilla?
22:45 Guest58 i bet that is what's going on
22:45 Guest58 damn.. trying to think of a work around
22:45 JoeJulian You can check du --apparent to see what size it was allocated as, but I'm sure you've got this figured out from here.
22:45 bennyturns joined #gluster
22:46 Guest58 at least I'm not alone :)
22:46 Guest58 http://www.gluster.org/pipermail/glu​ster-users/2014-November/019512.html
22:47 JoeJulian Well, I know when you enable full heal it makes the sink not sparse.
22:47 Guest58 right
22:47 JoeJulian But diff heals should make them more sparse.
22:47 Guest58 i wonder if I change this
22:47 Guest58 cluster.data-self-heal-algorithm
22:47 Guest58 to 'diff'
22:47 JoeJulian And diff is the default.
22:47 Guest58 if that would fix the issue?
22:47 Guest58 damn
22:47 JoeJulian Did you change that?
22:47 Guest58 not intentioanlly
22:47 Guest58 where do I find gluster cluster options?
22:47 JoeJulian Plus, aren't you saying that it's the source that's growing, not the sink?
22:47 Guest58 i know how to get volume options
22:47 T3 joined #gluster
22:48 Guest58 so, yeah, it seems to be the source..
22:48 Guest58 the node I rebooted isn't growing, it's one that's been up the whole time
22:48 JoeJulian "gluster volume info" shows changed options, "gluster volume set help" shows the list of possibilities.
22:49 Guest58 then, no, it should be the default
22:49 Guest58 though, according to this:
22:49 Guest58 http://www.gluster.org/community/documentation/ind​ex.php/Gluster_3.2:_Setting_Volume_Options#cluster.data-self-heal-algorithm
22:49 Guest58 the default is 'reset'
22:50 JoeJulian That's really old.
22:50 squizzi left #gluster
22:50 Slashman joined #gluster
22:51 JoeJulian "If the file does not exist on one of the sinks or empty file exists or if the source file size is about the same as page size the entire file will be read and written i.e "full" algo, otherwise "diff" algo is chosen."
22:51 Guest58 Option: cluster.data-self-heal-algorithm
22:51 Guest58 Default Value: (null)
22:51 Guest58 is what I got from "volume set help"
22:52 Guest58 so I guess that's what's happening?
22:52 Guest58 maybe I should just try setting it to 'diff'
22:52 JoeJulian file a bug report, "self-heal makes sparse source un-sparse" along with your observations.
22:52 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
22:53 Guest58 will do
22:53 Guest58 thanks Joe
22:53 JoeJulian Hopefully this can get fixed in time for 3.6.3
22:57 Guest58 seems like there's a bug similar to this open already
22:57 Guest58 https://bugzilla.redhat.co​m/show_bug.cgi?id=1187547
22:57 glusterbot Bug 1187547: unspecified, unspecified, ---, bugs, ON_QA , self-heal-algorithm with option "full" doesn't heal sparse files correctly
22:57 side_control joined #gluster
22:58 JoeJulian That looks like it's about the sink, not the source.
22:59 monotek joined #gluster
23:02 Pupeno joined #gluster
23:03 Pupeno joined #gluster
23:03 Pupeno joined #gluster
23:13 wkf joined #gluster
23:32 monotek joined #gluster
23:35 glusterbot News from newglusterbugs: [Bug 1196898] nfs: crash with nfs process <https://bugzilla.redhat.co​m/show_bug.cgi?id=1196898>
23:35 glusterbot News from resolvedglusterbugs: [Bug 1010241] nfs: crash with nfs process <https://bugzilla.redhat.co​m/show_bug.cgi?id=1010241>
23:57 gh5046 joined #gluster
23:57 snewpy i had a patch accepted to master for fixing an rdma bug and was wondering what i need to do to have it considered for being backported to stable? https://bugzilla.redhat.co​m/show_bug.cgi?id=1195907
23:57 glusterbot Bug 1195907: low, unspecified, ---, bugs, NEW , RDMA mount fails for unprivileged user without cap_net_bind_service
23:58 gh5046 Could anyone point me to a document/link about the status of ext4 + glusterfs?  I've seen random mail list postings about the problem being resolved, having trouble finding relevant information on gluster.org
23:58 kkeithley_ clone the BZ, change the version to 3.6.2
23:58 kkeithley_ ext4 issue was fixed a long time ago. All current versions work with ext4
23:59 gh5046 Is there some kind of changelog or something indicating as much?
23:59 gh5046 I'd like to learn more about it.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary