Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-11-18

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:19 cjellick joejulian do you know what it is a pre-req in the docs? http://www.gluster.org/community/documentation/index.php/Setting_Up_Clients
00:19 cjellick also,
00:20 cjellick it seems to start after starting the client.
00:24 hgichon joined #gluster
00:28 JoeJulian Geo-replication uses ssh to transfer data to the slave, but there's nothing in the code that starts sshd.
00:29 JoeJulian cjellick: which distro?
00:29 cjellick debian
00:30 cjellick but those docs indicate that its a pre-req for rhel based systems as well
00:30 JoeJulian Does debian use systemd yet?
00:30 cjellick and...this is just the client, so that wouldnt be doing the geo-repl anyway, right? thatd be on the glusterfs 'server' side?
00:31 JoeJulian Right. There's nothing in the source that could start sshd.
00:31 JoeJulian I looked.
00:31 cjellick ok. cool
00:31 cjellick and yes, looks like it
00:31 cjellick re: systemd
00:31 JoeJulian There's a reference to see if sshd is an ancestor process to gsyncd and sshd is mentioned in a few comments.
00:32 JoeJulian have you tried "systemctl disable sshd"
00:32 jatb joined #gluster
00:33 mlhamburg joined #gluster
00:58 suliba joined #gluster
01:06 zhangjn joined #gluster
01:17 plarsen joined #gluster
01:25 Lee1092 joined #gluster
01:32 cjellick joined #gluster
01:56 nangthang joined #gluster
01:59 nangthang joined #gluster
02:16 shyam joined #gluster
02:29 suliba joined #gluster
02:30 haomaiwang joined #gluster
02:31 17SAD258V joined #gluster
02:40 Merlin__ joined #gluster
03:01 haomaiwa_ joined #gluster
03:06 Merlin__ joined #gluster
03:20 trapier joined #gluster
03:21 Vaelatern joined #gluster
03:37 atinm joined #gluster
03:46 atinm joined #gluster
03:47 ashiq joined #gluster
03:51 Manikandan joined #gluster
03:53 nbalacha joined #gluster
03:54 Manikandan joined #gluster
04:00 overclk joined #gluster
04:01 haomaiwa_ joined #gluster
04:03 Vaelatern joined #gluster
04:07 shubhendu joined #gluster
04:13 itisravi joined #gluster
04:13 vimal joined #gluster
04:14 neha_ joined #gluster
04:15 kshlm joined #gluster
04:16 shortdudey123 joined #gluster
04:22 sakshi joined #gluster
04:24 [7] joined #gluster
04:26 ppai joined #gluster
04:27 ramteid joined #gluster
04:31 harish__ joined #gluster
04:36 mlncn joined #gluster
04:38 zhangjn joined #gluster
04:44 PaulCuzner joined #gluster
05:01 haomaiwa_ joined #gluster
05:06 anil joined #gluster
05:08 aravindavk joined #gluster
05:20 kdhananjay joined #gluster
05:22 ndarshan joined #gluster
05:26 zhangjn_ joined #gluster
05:26 Bhaskarakiran joined #gluster
05:27 atalur joined #gluster
05:30 kshlm joined #gluster
05:35 nathwill joined #gluster
05:38 zhangjn joined #gluster
05:42 pppp joined #gluster
05:52 deepakcs joined #gluster
05:56 hgowtham joined #gluster
05:57 jtux joined #gluster
06:00 rafi joined #gluster
06:01 haomaiwa_ joined #gluster
06:07 jiffin joined #gluster
06:12 hgowtham joined #gluster
06:12 rp_ joined #gluster
06:18 rp_ joined #gluster
06:24 raghu joined #gluster
06:25 Manikandan_ joined #gluster
06:26 R0ok_ joined #gluster
06:27 ramky joined #gluster
06:30 atalur joined #gluster
06:37 gem joined #gluster
06:37 nishanth joined #gluster
06:45 mlhamburg1 joined #gluster
06:49 haomaiwa_ joined #gluster
06:54 ppai joined #gluster
07:00 ashiq joined #gluster
07:01 haomaiwa_ joined #gluster
07:03 gildub_ joined #gluster
07:08 Humble joined #gluster
07:09 mhulsman joined #gluster
07:15 ppai joined #gluster
07:26 PaulCuzner left #gluster
07:29 vmallika joined #gluster
07:35 morse joined #gluster
07:41 gildub_ joined #gluster
07:45 arcolife joined #gluster
07:50 RedW joined #gluster
07:52 Merlin__ joined #gluster
08:01 haomaiwa_ joined #gluster
08:03 skoduri joined #gluster
08:14 Manikandan_ joined #gluster
08:16 kotreshhr joined #gluster
08:20 fsimonce joined #gluster
08:21 jwd joined #gluster
08:37 kovshenin joined #gluster
08:42 Merlin__ joined #gluster
08:43 Merlin__ joined #gluster
08:44 kotreshhr left #gluster
08:44 overclk joined #gluster
09:01 haomaiwa_ joined #gluster
09:02 ctria joined #gluster
09:04 ppai joined #gluster
09:06 fsimonce joined #gluster
09:07 ctrianta joined #gluster
09:14 norky joined #gluster
09:17 Saravana_ joined #gluster
09:17 Philambdo joined #gluster
09:20 ahino joined #gluster
09:20 LebedevRI joined #gluster
09:22 cjellick joined #gluster
09:24 obiwan2k joined #gluster
09:25 nathwill joined #gluster
09:25 shubhendu joined #gluster
09:33 kshlm joined #gluster
09:46 mbukatov joined #gluster
09:46 shubhendu joined #gluster
09:46 ivan_rossi joined #gluster
09:46 Pupeno joined #gluster
09:55 klaxa joined #gluster
10:01 haomaiwa_ joined #gluster
10:06 nishanth joined #gluster
10:07 Slashman joined #gluster
10:10 ivan_rossi left #gluster
10:11 spalai joined #gluster
10:13 DV joined #gluster
10:39 Manikandan joined #gluster
10:45 ivan_rossi joined #gluster
10:50 nishanth joined #gluster
10:51 zhangjn joined #gluster
10:53 nbalacha joined #gluster
10:57 mhulsman joined #gluster
11:01 haomaiwa_ joined #gluster
11:11 Manikandan joined #gluster
11:14 bluenemo joined #gluster
11:16 bluenemo hi guys. switched my four webworker nodes from nfs to gluster now, performance didnt really suffer imho. since then no problems.
11:16 Humble joined #gluster
11:17 ziggypup joined #gluster
11:17 ziggypup left #gluster
11:20 rafi1 joined #gluster
11:24 pgreg joined #gluster
11:27 ctria joined #gluster
11:27 Manikandan_ joined #gluster
11:32 ndevos bluenemo: oh, good to hear! well, sort of... I would still like to fix that issue with nfs
11:34 bluenemo i guess i'm happy for now. i will however also setup a dev env for this. if you want i can give you access to hack with me on it.
11:35 bluenemo but yeah i'm also quite happy that it runs - and also performans quite the same
11:35 ndevos bluenemo: I dont have much time to look into it today, we could have a look at it tomorrow
11:36 bluenemo wont make this today as well. i'll hang around ;)
11:38 rjoseph joined #gluster
11:38 jockek joined #gluster
11:43 hos7ein joined #gluster
11:49 mhulsman joined #gluster
11:55 kshlm Weekly Gluster community meeting is starting in 5 minutes in #gluster-meeting
11:57 ashiq joined #gluster
11:57 ctria joined #gluster
12:00 jdarcy joined #gluster
12:02 raghu joined #gluster
12:03 gildub_ joined #gluster
12:04 overclk_ joined #gluster
12:06 neha_ joined #gluster
12:09 spalai joined #gluster
12:09 zhangjn joined #gluster
12:10 zhangjn joined #gluster
12:10 zhangjn joined #gluster
12:10 Pupeno joined #gluster
12:15 hos7ein_ joined #gluster
12:22 rafi joined #gluster
12:25 nllrte joined #gluster
12:26 nllrte left #gluster
12:26 mbukatov joined #gluster
12:31 natarej joined #gluster
12:31 natarej joined #gluster
12:35 SpComb joined #gluster
12:35 ira joined #gluster
12:36 SpComb glusterfs 3.7.3 on Ubuntu wily is connecting to the server using an odd high-numbered TCP source port: 10.107.107.0.65534 > 10.11.11.6.24007
12:36 SpComb that fails with an "Request received from non-privileged port. Failing request" on the glusterd 3.4 server
12:37 SpComb some known change in behavior between 3.4 and 3.7, or a bug?
12:37 SpComb the 65534 port is very odd, it's outside the range of ephemeral ports
12:40 Mr_Psmith joined #gluster
12:46 ashiq joined #gluster
12:49 rastar SpComb: the default behavior changed in 3.7.3
12:50 DV_ joined #gluster
12:50 rastar SpComb: apart from the well known port 24007 all other ports in gluster now use from range 49152-65535
12:51 rastar there are two changes involved.
12:51 rastar SpComb: bind-insecure option if set to on would make all clients to connect to bricks&glusterd over a port >1024
12:52 rastar SpComb: allow-insecure would make bricks allow connection to clients coming from port >1024
12:52 rastar SpComb: rpc-auth-allow-insecure would make glusterd allow connection from bricks and clients coming from port > 1024
12:53 Humble rastar++
12:53 glusterbot Humble: rastar's karma is now 1
12:53 Humble SpComb, better if you can use latest version of Gluster
12:54 rastar SpComb: read upgrade notes section in this mail http://www.gluster.org/pipermail/gluster-users/2015-September/023470.html
12:54 glusterbot Title: [Gluster-users] glusterfs-3.7.4 released (at www.gluster.org)
12:54 Humble SpComb, r u using 3.4 glusterfs server ?
12:55 SpComb I don't particularly want to allow insecure ports, since I don't have SSL authentication
12:55 DV joined #gluster
12:55 SpComb is there some glusterfs (client) option to bind to a non-insecure port?
13:02 overclk joined #gluster
13:07 JoeJulian SpComb: secure ports are a fallacy.
13:11 rastar SpComb: gluster vol set <volname> client.bind-insecure on
13:11 rastar SpComb: gluster vol set <volname> client.bind-insecure off
13:14 SpComb rastar: does the 3.4 glusterfs server support that? :/
13:14 B21956 joined #gluster
13:14 SpComb JoeJulian: good old NFS security, it never failed before :)
13:15 SpComb volume set: failed: option : client.bind-insecure does not exist
13:16 jrm16020 joined #gluster
13:18 SpComb `/usr/sbin/glusterfs --xlator-option '*.client-bind-insecure=no' ...` doesn't seem to help either
13:18 kshlm SpComb, Don't know if 3.4 supports those options.
13:18 SpComb not sure if it's because I'm specifying it wrong (it seems to ignore unknown options?) or it's broken somehow else
13:19 kshlm But 3.7 changed the defaults from requiring to use secure ports below, 1024 to allow unsecure ports.
13:20 sakshi joined #gluster
13:20 Lee1092 joined #gluster
13:21 rjoseph joined #gluster
13:21 atinm joined #gluster
13:26 SpComb ah, seems like that --xlator-option does work for the actual brick/volume connections, but just not the glusterd :24007 part
13:26 SpComb as in http://www.spinics.net/lists/gluster-users/msg24586.html
13:26 glusterbot Title: Re: [Gluster-devel] mixed 3.7 and 3.6 environment Gluster Users (at www.spinics.net)
13:29 eljrax joined #gluster
13:32 mhulsman joined #gluster
13:33 vimal joined #gluster
13:36 SpComb I guess my options are to drop all pretense of host-based access security; setup SSL on the servers; or move to NFS on the clients?
13:40 shyam joined #gluster
13:43 kbyrne joined #gluster
13:50 jrm16020 joined #gluster
13:52 chirino joined #gluster
13:52 klaxa|work joined #gluster
13:55 julim joined #gluster
14:04 ira joined #gluster
14:05 unclemarc joined #gluster
14:16 trapier joined #gluster
14:19 hamiller joined #gluster
14:26 SpComb mounting via NFS seems to be working, it's probably a better option for me than the native fuse client
14:26 rjoseph joined #gluster
14:32 skylar2 joined #gluster
14:33 rjoseph joined #gluster
14:37 mlncn joined #gluster
14:45 spalai left #gluster
14:46 lpabon joined #gluster
14:46 aravindavk joined #gluster
14:51 ToMiles joined #gluster
14:53 ToMiles Upgraded all my bricks from 3.7.2 to 3.7.4 but when I try and do full heal it is unsuccessful, "Commit failed on xxxxxx. Please check log file for details."
14:54 ToMiles and no errors in the log file at the timepoint I ask for the heal
14:56 dusmant joined #gluster
14:57 shubhendu joined #gluster
14:57 amye- joined #gluster
14:59 ToMiles only diff between those that give a commit failed and those that don't is " E [glusterfsd-mgmt.c:619:glusterfs_handle_translator_op] 0-glusterfs: Couldn't get xlator xl-0" at some time before a heal
14:59 mlncn joined #gluster
15:00 uebera|| ToMiles: This doesn't ring a bell, but a quick question--is there a reason why you did not upgrade to v3.7.5/v3.7.6 which are already available? Both releases came with a number of additional fixes...
15:00 glusterbot uebera||: question's karma is now -1
15:01 ToMiles sorry should been 3.7.6 typo
15:01 uebera|| I see.
15:01 ToMiles so 3.7.2 -> 3.7.6
15:03 ToMiles first guess was issue with vol files between diff hosts, but would expect that to be pocked up by the checksum test when bricks start
15:05 vimal joined #gluster
15:05 ira__ joined #gluster
15:06 uebera|| According to https://bugzilla.redhat.com/show_bug.cgi?id=1221935, there's a *client-xlators* package... did you verify that all machines have exactly the same packages?
15:06 glusterbot Bug 1221935: urgent, unspecified, ---, dlambrig, ASSIGNED , Detach tier commit failed on a dist-rep volume
15:07 dgandhi joined #gluster
15:09 ToMiles I'll double check
15:13 ToMiles don't think that package is there on ubuntu, prob glusterfs-client here, and all are same version, upgraded today been no package changes since nov 9
15:13 plarsen joined #gluster
15:13 nishanth joined #gluster
15:17 JPaul joined #gluster
15:21 ahino joined #gluster
15:21 uebera|| Yes, Ubuntu has fewer packages.
15:21 ndevos joined #gluster
15:21 ndevos joined #gluster
15:22 Norky joined #gluster
15:24 plarsen joined #gluster
15:25 wnlx joined #gluster
15:27 uebera|| ToMiles did you try "gluster volume heal <volumeName> info heal-failed"?
15:30 ToMiles dont think heal-failed is supported, it tells me to use info which gives Number of entries: 0 for all bricks
15:30 cjellick joined #gluster
15:32 ToMiles I notice "starting full sweep on subvol" in de log for those with no commit issue, but nothing for problem nodes
15:33 ToMiles strange thing, the 2 nodes that give Commit failed are not the same when I exec the full heal from another brick
15:34 ToMiles some time brick A and B when heal full command on brick D  but when command on brick E its brick A and C that give commit failed
15:35 uebera|| that /could/ be because of existing partitions; depending on where you start, other bricks are contacted first.
15:36 ToMiles the A & B when on D and A & C when on E is consistent
15:37 haomaiwa_ joined #gluster
15:37 ToMiles but that prob becasue of rep/dist pairs
15:38 josh joined #gluster
15:39 bennyturns joined #gluster
15:45 Merlin__ joined #gluster
15:51 bluenemo ndevos, I've got a view errors in my webworker 3 http://paste.debian.net/hidden/4f07a22f/ gluster is still mounted fine.. seems apache died and made its workers zombies somehow
15:51 glusterbot Title: Debian Pastezone (at paste.debian.net)
15:51 cholcombe joined #gluster
15:53 bluenemo maybe the errors could be related? ah i forgot the last line of the log: [2015-11-18 15:50:02.974605] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fbc53110182] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fbc53e4d7c5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7fbc53e4d659] ) 0-: received signum
15:53 glusterbot bluenemo: ('s karma is now -119
15:56 bluenemo another node went just down :(
15:57 bluenemo second node same problem :( ah JoeJulian can I ping you on this one?
16:01 haomaiwa_ joined #gluster
16:05 glafouille joined #gluster
16:15 bluenemo both nodes back up - lookes like gluster was inaccessible for them. apache then became zombie until i could umount -f it. after remount same beavior on one node, not on the other, rebooted one node, seems ok now for some minutes
16:15 rafi joined #gluster
16:16 bluenemo oh, gluster volume heal gfs_fin_web info is packed with info..
16:19 wolsen joined #gluster
16:23 bluenemo the number of entries seems to be rising. i issued a volume heal, so far nothing shown
16:23 m0zes joined #gluster
16:23 bluenemo JoeJulian, number of files rising quite fast now :(
16:24 ToMiles checked exec full heal from one of the failed commit nodes and then it give this error in local log of glustershd: "E [glusterfsd-mgmt.c:619:glusterfs_handle_translator_op] 0-glusterfs: Couldn't get xlator xl-0"
16:25 bluenemo it says  /var/www/foo/img - Possibly undergoing heal
16:25 tomatto joined #gluster
16:27 aravindavk joined #gluster
16:32 thoht__ hello
16:32 glusterbot thoht__: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:32 thoht__ i m matching this bug: https://access.redhat.com/solutions/1225453
16:33 glusterbot Title: Mounting a Red Hat Gluster Storage volume fails on RHEL7 after updating to RHEL7.1, why? - Red Hat Customer Portal (at access.redhat.com)
16:33 thoht__ anyway to fix that ?
16:34 Merlin__ joined #gluster
16:37 thoht__ omg with env -i LC_NUMERIC="en_US.UTF-8" mount blabla
16:37 thoht__ it works
16:37 thoht__ WTF
16:39 bluenemo the number of entries in volume heal info keeps rising, now 447 :/
16:40 bluenemo strangely for the files listed, they are all present on all nodes and all have the same md5 hash
16:47 JonathanD joined #gluster
16:49 bluenemo seems like some php script clearing a cache directory caused it, maybe it wanted to remove directories that contained files another worker was trying to write to - as in what happened by the time my two workers went down
16:51 jiffin joined #gluster
17:04 jockek joined #gluster
17:19 nathwill joined #gluster
17:19 Akee joined #gluster
17:30 RayTrace_ joined #gluster
17:35 Akee joined #gluster
17:38 mlncn joined #gluster
17:42 ivan_rossi left #gluster
17:43 bennyturns joined #gluster
17:55 ToMiles quit
17:55 harish__ joined #gluster
17:55 ToMiles sorry I'll check back in later if I cant fix the heal issue
17:56 shaunm joined #gluster
17:58 bluenemo I now have Number of entries: 884 in my gluster volume foo heal info. I compared some of the files, they are the same on all nodes. whats going on here?
17:58 nishanth joined #gluster
18:00 skoduri joined #gluster
18:15 mhulsman joined #gluster
18:16 RayTrace_ joined #gluster
18:16 Rapture joined #gluster
18:25 rafi joined #gluster
18:26 ahino joined #gluster
18:32 calavera joined #gluster
18:34 rwheeler joined #gluster
18:36 JoeJulian bluenemo: check the self-heal daemon logs /var/log/glusterfs/glustershd.log for clues
18:37 JoeJulian Also check the client logs. If a client can't connect to one of the bricks, you could see that behavior.
18:38 hagarth_ joined #gluster
18:44 bluenemo JoeJulian, /var/log/glusterfs/glustershd.log is empty on both servers
18:46 JoeJulian Well that's odd. How is that possible?
18:46 bluenemo ah sry was just rotated, wait a sec :)
18:47 bluenemo ok thats not really recent http://paste.debian.net/hidden/68093a25/
18:48 glusterbot Title: Debian Pastezone (at paste.debian.net)
18:48 bluenemo also we had no problems at that time
18:49 bluenemo gluster volume heal gfs_fin_web statistics shows some No. of heal failed entries: 120
18:51 bluenemo JoeJulian, volume heal info now shows  Number of entries: 968   so I think I have two problems here - a php script deleting a cache directory in a very strange way, which might or might not cause a gluster mount to fail, and the healing thing rising
18:51 bluenemo btw, the local gluster process died on the client, but the mount was still shown in `mount`
18:52 JoeJulian Should be a log of the crash in the client log
18:52 dlambrig_ joined #gluster
18:53 JoeJulian "Path: (null)" seems broken.
18:57 bluenemo meh to big for pastebin..
18:57 JoeJulian Are client and server on the same version? That's a lot of null path spam failing to mknod.
18:57 JoeJulian just the part after "crash"
18:58 bluenemo JoeJulian, ok here: http://paste.debian.net/333609/
18:58 glusterbot Title: debian Pastezone (at paste.debian.net)
19:02 JoeJulian A crash should look like http://www.gluster.org/pipermail/gluster-users.old/2015-June/022481.html
19:02 glusterbot Title: [Gluster-users] gluster client crash (at www.gluster.org)
19:03 bluenemo JoeJulian, hm strange. I'll keep looking through the logs some more. About the rising heal count - can you give me a hint on that?
19:06 bluenemo A crash should be listed when running a  grep -ri crash /var/log/gluster*  right? Or what could I grep for? Found this word in the link of what crashes should look like
19:06 JoeJulian I would stop glusterd, pkill -f glustershd, start glusterd. If there's something up with a self-heal daemon, that should cure it. If it keeps rising after that, I'd check the client connections to the servers. netstat -t could tell you if it's connecting.
19:06 JoeJulian yes
19:06 RayTrace_ joined #gluster
19:07 JoeJulian If it's not there, then something is sigterm'ing your process.
19:07 JoeJulian er, sigkill
19:08 JoeJulian I think I need a second cup of coffee today.
19:08 bluenemo ;)
19:08 bluenemo stop glusterd on the servers, right?
19:13 amye1 joined #gluster
19:15 JoeJulian Yes, glusterd is the management daemon. It has no functional effect on your volume with the one caveat that if a client is trying to initially mount the volume from that specific server, it will fail to connect and either try the next one in your rrdns or a failover server you specified in your mount options.
19:17 bluenemo ok, so service gluster-server stop, pkill -f glustershd, service gluster-server start?
19:17 JoeJulian yes
19:19 bluenemo I have no such process, my processes: http://paste.debian.net/hidden/9e31f14d/
19:19 glusterbot Title: Debian Pastezone (at paste.debian.net)
19:20 bluenemo this one? /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd
19:20 JoeJulian yes. That's what the -f is for.
19:20 bluenemo i'm not sure which were stopped by service glusterfs-server stop, but i think to remember that some kept running
19:20 bluenemo ah ok
19:21 bluenemo only this process is starting now JoeJulian http://paste.debian.net/hidden/179a3ba7/
19:21 glusterbot Title: Debian Pastezone (at paste.debian.net)
19:21 bluenemo workers dont notice gladly :)
19:22 bluenemo however the number of files dropps now
19:22 bluenemo fast
19:22 JoeJulian Ah, good.
19:23 bluenemo all clients still have connections to both servers in netstat
19:23 bluenemo but why are there no more than the one gluster process?
19:23 bluenemo gluster volume and peer status looks ok
19:24 JoeJulian There should only be one glustershd process.
19:24 JoeJulian If there was more, that was probably the problem.
19:24 bluenemo hm
19:24 bluenemo on alpha there are still lots more
19:24 JoeJulian glustershd. The shd specifically. shd=self heal daemon
19:24 bluenemo both servers: http://paste.debian.net/hidden/b86b13df/
19:24 glusterbot Title: Debian Pastezone (at paste.debian.net)
19:27 RayTrace_ joined #gluster
19:32 whereismyjetpack joined #gluster
19:33 JoeJulian those are two different greps.
19:33 whereismyjetpack hi all. i was planning on using gluster to replicate a webroot amongst many webervers, I got most of this sorted, however i can't find a decent way to decommission a gluster member when the node goes offline. i play on spinning up webserers as needed, and that was simple enough but now when i delete nodes after i don't need them i can't seem to remove the peer
19:35 JoeJulian @replica
19:35 glusterbot JoeJulian: Please see http://joejulian.name/blog/glusterfs-replication-dos-and-donts/ for replication guidelines.
19:35 JoeJulian whereismyjetpack: ^
19:35 mhulsman joined #gluster
19:37 whereismyjetpack thanks joejulian. i was setting the replica count to the number of servers. i guess foolishly i thought having the content distributed would cut down on load on the gluster servers.
19:37 whereismyjetpack i presume we could set replica == 2, and those servers i guess just have to stay online all the time.
19:39 jbrooks joined #gluster
19:41 hagarth_ left #gluster
19:41 bluenemo JoeJulian, would you recommend restarting the gluster on alpha?
19:42 bluenemo the count fell down from ~ 1000 to 180 and then stopped, now slowly rising again
19:43 JoeJulian whereismyjetpack: I prefer replica 3 which allows for my SLA/OLA to be met while still leaving margin for maintenance.
19:44 JoeJulian bluenemo: yes. I would do the same thing we did on the first server to the second.
19:45 bluenemo JoeJulian, in gluster volume heal vol_name info it only shows files for the server i restarted gluster already
19:46 whereismyjetpack JoeJulian, rock and roll. as always, I appreciate your guidance
19:46 gildub_ joined #gluster
19:49 frozengeek joined #gluster
19:57 shyam joined #gluster
19:58 bluenemo oh. and i noticed I send you a wrong paste, the processes present on omega are now after the restart: http://paste.debian.net/hidden/7d69013c/ as no process has similar options, what process ids shouldnt exist?
19:58 glusterbot Title: Debian Pastezone (at paste.debian.net)
19:59 rwheeler joined #gluster
19:59 shaunm joined #gluster
19:59 milkyline joined #gluster
20:01 bluenemo hm ok now i get it - the init script doesnt really have any effect on the processes
20:02 bluenemo i remember there was a gluster sth command that stopps "better" than the init scripts
20:04 bluenemo JoeJulian, if I may ping you on this one once more
20:05 bluenemo although the pkill surely did kill something
20:06 bluenemo I guess the gluster init script started it again when we restarted it before
20:08 bluenemo i did the same procedure on alpha now, no effect to the unhealed files on omega in gluster volume info
20:08 milkyline left #gluster
20:11 bluenemo just restarted omega again, still no change to the remaining 180 files, list is slowly rising again
20:15 bluenemo (restarted as in pkill)
20:16 mhulsman joined #gluster
20:16 jbrooks joined #gluster
20:16 shaunm joined #gluster
20:18 bluenemo number is climbing faster again..
20:18 bluenemo ah i dont think this is going to work..
20:21 DV joined #gluster
20:22 bluenemo and back to 180.. there seems to be an issue with those 180.. i issued a heal full btw
20:23 bluenemo and back to 220. hm.
20:26 bluenemo the volume heal info output now basically only shows gfid's, only one file and one directory, plus /var/www/foo/images possibly undergoing heal - that undergoing heal has been present from the start today
20:32 bluenemo so following this video: https://www.youtube.com/watch?v=AsgtE7Ph2_k files are only healed if they are accessed again? or did i get this wrong?
20:34 mlncn joined #gluster
20:35 hagarth_ joined #gluster
20:36 hagarth_ joined #gluster
20:49 whereismyjetpack is there a good way to get the present replica count #?
20:55 bluenemo so the state is that the number of heal files jumps up and down but never goes below 180
21:16 abyss^ JoeJulian: it is possible to upgrade gluster from 3.3.2 to 3.7.x or better do upgrades from 3.3.2 to 3.4.x then to 3.5 etc?
21:24 cholcombe joined #gluster
21:25 cornfed78 joined #gluster
21:37 bluenemo from what I get it, basically every file that is uploaded through php ends up in the the list output by gluster volume heal info at first
21:38 bluenemo then after some time it gets accessed on the other node and synced i guess.. but that cant be normal can it?
21:38 bluenemo i tried the linux kernel unpacking and removing and so on many times, nothing of the sort happens there
21:56 XpineX joined #gluster
22:05 ira joined #gluster
22:13 JoeJulian abyss^: you can make the jump straight to 3.7. It'll require downtime though.
22:14 JoeJulian bluenemo: glusterd is what starts the self heal daemon, so that was expected.
22:14 XpineX joined #gluster
22:16 JoeJulian gluster creates file entries under $brick/.glusterfs/indices/xattrop with the gfid of the file that needs healed. Occasionally (and I don't know the circumstances that can cause this) those entries get left even though the file was already deleted.
22:16 JoeJulian Perhaps those 180 fit in to that category.
22:17 JoeJulian bluenemo: https://joejulian.name/blog/what-is-this-new-glusterfs-directory-in-33/
22:17 glusterbot Title: What is this new .glusterfs directory in 3.3? (at joejulian.name)
22:18 JoeJulian You can use that information to see if the file still exists.
22:23 Merlin__ joined #gluster
22:27 CP|AFK joined #gluster
22:27 CyrilPeponnet joined #gluster
22:28 CyrilPeponnet joined #gluster
22:47 bluenemo JoeJulian, for the upgrade to 3.7.6 from 3.7.4 - do you recommend to stop one server, upgrade, then start again, and then do the other server? as my init script doesnt seem to be working, what do you recommend for restarting the server? is there a gluster stop or sth command?
22:48 JoeJulian Not on ubuntu. upstart's a ... well it's not very capable.
22:49 JoeJulian I don't keep mounts on my servers, so stopping the service, then "pkill -f gluster", upgrade, start the service
22:49 JoeJulian works for me.
22:54 bluenemo JoeJulian, i've got /var/www mounted on alpha and omega as well (servers), but just for debugging. ok, i will idle some more till load is off the servers and then go with the upgrade
23:01 CP|AFK joined #gluster
23:01 kovshenin joined #gluster
23:10 rwheeler joined #gluster
23:39 ctria joined #gluster
23:40 rafi1 joined #gluster
23:41 kminooie joined #gluster
23:42 Guest68595 joined #gluster
23:44 mlhamburg1 joined #gluster
23:44 ahino joined #gluster
23:44 muneerse joined #gluster
23:47 Mr_Psmith joined #gluster
23:52 kminooie this is not, strictly speaking, a gluster question, but you guys are still my best bet. I am trying to mount gluster over nfs. I can do that on command line with no problem. but when I try to do the same thing with autofs, I get 'Remote I/O error' when I try to access the directory ( ls ), and I get this line in gluster nfs log file:  0-rpc-service: RPC program version not available (req 100003 2) . Now, what I am trying to
23:53 muneerse joined #gluster
23:56 dlambrig_ joined #gluster
23:58 cliluw joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary