Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-11-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 DoubleJ I'm wondering it it's just doing some kind of "clean up" after the version upgrade
00:01 DoubleJ updating xattrs or something like that - I don't know
00:02 DoubleJ otoh, volume status says "There are no active volume tasks"
00:02 vbellur DoubleJ: what is your volume configuration?
00:03 JoeJulian That's what I was wondering. If it is, I suspect it would be part of some "crawl" process so I'm trying to compare functions that use the word "crawl" with previous versions. Nothing stands out so far.
00:03 DoubleJ distribute-replicate, 6 servers, 2 bricks per server
00:03 DoubleJ @vbellur
00:03 vbellur DoubleJ: do you have quota or any other features enabled?
00:03 DoubleJ no
00:03 JoeJulian vbellur: I have both a distribute (6 bricks) and two 2x2 volumes. Also no quota.
00:04 vbellur DoubleJ: would it be possible to strace the brick processes for a short period?
00:05 vbellur DoubleJ: that might provide us some hints
00:05 JoeJulian Would 3.8 crawl to create bit-rot versions?
00:05 JoeJulian That would explain the high cpu load.
00:06 vbellur JoeJulian: let me check
00:06 DoubleJ vbellur: I will see what I can do
00:06 DoubleJ JoeJulian: I have bitrot disabled (or: not enabled), so why would it create checksums?
00:07 JoeJulian Me to, but I notices trusted.bit-rot.version on my files (on the bricks)
00:07 JoeJulian s/notices/noticed/
00:07 glusterbot What JoeJulian meant to say was: Me to, but I noticed trusted.bit-rot.version on my files (on the bricks)
00:08 vbellur JoeJulian: version should not involve a checksum computation
00:09 JoeJulian Ah
00:09 JoeJulian So much for that theory. :)
00:09 vbellur JoeJulian: although I have suspicions about bitd_oneshot_crawl()
00:10 DoubleJ "NOTE: Enabling bitrot spanws the Signer & Scrubber daemon per node. "
00:10 vbellur JoeJulian: that still might be the case, am taking a fresh look at bitrot src :)
00:10 DoubleJ so we should see additional processes for bitrot
00:12 kenansulayman left #gluster
00:16 JoeJulian br_stub_fd_versioning is part of bitrot-stub, a server-side translator in the default graph.
00:17 JoeJulian and it does appear to be part of some crawl.
00:18 JoeJulian DoubleJ: if you could get a core from one of your glusterfsd daemons that's using a lot of cpu, a gdb "thread apply all bt" I think would be able to tell if that's in play.
00:19 DoubleJ vbellur: I was just running strace for like 30 seconds and it grew me a 357 MB logfile
00:19 DoubleJ JoeJulian: ok, let me see...
00:23 DoubleJ JoeJulian: http://paste.ubuntu.com/23483209/ - I don't think that is what you wanted to see...?
00:23 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
00:24 shyam joined #gluster
00:25 Klas joined #gluster
00:26 JoeJulian That looks like a client
00:28 DoubleJ sorry
00:28 Caveat4U joined #gluster
00:32 DoubleJ JoeJulian: http://paste.ubuntu.com/23483241/
00:32 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
00:33 DoubleJ but the glusterfsd are not consuming as much cpu as the glusterfs processes, i.e. most cpu time is consumed by the gfs clients
00:33 JoeJulian Well that's odd.
00:34 JoeJulian They're on the same box, right?
00:34 DoubleJ yes, each server is also a client
00:34 JoeJulian Damn
00:34 JoeJulian Was hoping to be able to capture network traffic.
00:35 DoubleJ the writes are not going to the local brick, though
00:35 DoubleJ I don't have nufa enabled
00:35 DoubleJ so a process on server 1 might write to bricks on server 3/4
00:35 JoeJulian For replicated volumes, the writes go to two replica.
00:36 DoubleJ yes
00:36 Javezim Having an issue with one of our machines, starting glusterfs-server service doesn't start any of the brick processes. Checking glusterd logs - http://paste.ubuntu.com/23483248/
00:36 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
00:36 JoeJulian Which management server do you specify when mounting? localhost?
00:36 JoeJulian Javezim: What if you start more than one?
00:36 Javezim @JoeJulian Sorry how do you mean?
00:37 JoeJulian nevermind. I hadn't opened the paste yet.
00:37 JoeJulian Javezim: Can you please start it with debug logging, glusterd --debug
00:38 DoubleJ JoeJulian: can I PM you a link to a screenshot?
00:38 JoeJulian ok
00:39 DoubleJ that's how it looks like on all servers
00:39 P0w3r3d joined #gluster
00:40 Javezim @JoeJulian - http://paste.ubuntu.com/23483269/ There we go
00:40 Javezim I saw a tonne of forums where people said it was a hostname issue
00:41 JoeJulian DoubleJ: ok, so I'm just going off in the completely wrong direction.
00:41 Javezim but I can't see any issues with the hosts file for all servers part of the volume
00:42 DoubleJ sorry if there was a misunderstanding
00:42 JoeJulian No, it's just one of those things.
00:43 JoeJulian Javezim: Well, first... you should not be getting segfaults regardless of hostnames. That's a bug.
00:44 JoeJulian But it looks like it's segfaulting after it fails anyway, so it may not be a critical bug.
00:50 DoubleJ JoeJulian: do you have any clue what I could be doing next in order to  trace down the  issue? ("the issue" being mainly the frequent disconnects, I don't know if the CPU usage has anything to do with it - just suspecting it)
00:50 JoeJulian Javezim: I don't see anything obvious, but it looks like it's still parsing the volume info files. Perhaps renaming /var/lib/glusterd/vols and copying a known good vols from another server would be worth trying.
00:51 Javezim @JoeJulian I actually did already do that funny enough
00:51 Javezim Because it fixed this a while ago on another server
00:51 Javezim But on this one even with the Vols copied from another machine its still an issue
00:52 JoeJulian DoubleJ: So the high cpu load is at your application layer, and it's not in a wait state, it's actually running. Perhaps something has actually improved write performance such that your application is now running away with cpu when it used to wait for I/O.
00:53 shdeng joined #gluster
00:53 JoeJulian DoubleJ: I would start testing that theory by disabling all the performance xlators. If that solves it, reenable them one at a time until it breaks.
00:53 plarsen joined #gluster
00:55 DoubleJ how can I disable them?
00:55 shdeng joined #gluster
00:55 JoeJulian gluster volume set help
00:55 JoeJulian gluster volume set help | grep performance
00:55 JoeJulian It *should* be fairly straightforward.
00:56 JoeJulian I'm glad nobody's watching me type. It's horrible this evening.
00:56 arif-ali joined #gluster
00:57 DoubleJ Are you talking about "write-behind" "read-ahead" etc.?
00:57 JoeJulian The other thing you could do is profile your application and see what it's spinning on.
00:57 JoeJulian Yes.
00:58 DoubleJ is a restart required for these settings to take effect?
00:58 JoeJulian no
00:58 DoubleJ you mean the "dw.out" process?
00:58 JoeJulian profiling dw.out, yes.
00:58 DoubleJ it's just writing a dozen lines every 10-60 seconds
00:58 JoeJulian And no restart required for volume set changes
00:59 JoeJulian That's a lot of cpu in run state for that.
00:59 DoubleJ It's a GPU compute process, I don't know why it's taking so much CPU
00:59 DoubleJ I think that's a nvidia driver bug
00:59 Javezim @JoeJulian Apart from glusterd, any idea what other logs may assist in finding an issue?
01:00 JoeJulian DoubleJ: so the 99% number is a red herring?
01:00 vinurs joined #gluster
01:00 JoeJulian Javezim: none. It hasn't gotten that far.
01:00 DoubleJ or to be more precise, it's an opencl application - they fixed the "100% cpu usage when running opencl" on windows a while ago, but never on linux...
01:01 JoeJulian Javezim: gdb glusterd then run --debug. When it crashes, thread apply all bt
01:03 DoubleJ "red herring"? :)
01:03 JoeJulian http://www.merriam-webster.com/dictionary/red%20herring definition 2
01:03 glusterbot Title: Red Herring | Definition of Red Herring by Merriam-Webster (at www.merriam-webster.com)
01:03 DoubleJ oh well, then yes, it is.
01:03 JoeJulian :)
01:04 renout_away joined #gluster
01:04 haomaiwang joined #gluster
01:07 DoubleJ oh no
01:08 JoeJulian ???
01:08 DoubleJ after settin write-behind off, I was wondering why load was dropping on some servers
01:09 DoubleJ because the compute jobs are failing now
01:09 DoubleJ "srun: error: giant2: task 0: Bus error (core dumped)"
01:09 JoeJulian Argh!
01:09 DoubleJ all compute jobs on two servers now crashed
01:09 JoeJulian Did the mount crash?
01:10 JoeJulian That shouldn't have happened.
01:10 JoeJulian Never has when I've done that.
01:10 DoubleJ I can still access the gluster mount on those nodes
01:10 JoeJulian weird
01:12 Javezim Hey @JoeJulian Ill try that out, one thing - http://paste.ubuntu.com/23483365/ I see this in the logs. See anything wrong with this? http://paste.ubuntu.com/23483371/
01:12 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
01:13 DoubleJ what's even more weird is that there's nothing in the gluster logs
01:14 DoubleJ at least nothing which would explain the compute jobs crashing
01:14 JoeJulian Javezim: Should be fine. If unsure, copy it from another server.
01:15 JoeJulian So on your top screen shot, the servers are all in either "sleep" or "trace stop" states. The clients are sleeping too, but if we assume that's because the servers are then that backtrace of a server process might still be valuable.
01:15 JoeJulian If this doesn't show us anything.
01:16 DoubleJ they were in the moment I took the screenshot, but the gluster clients are going to 100+% cpu usage for about 1 minute every minute
01:19 DoubleJ but anyway... maybe we are focusing too much on the CPU usage thing? do you think the CPU usage could be the reason for the disconnects?
01:20 JoeJulian That's what I was thinking. If the server app is spinning on something for more than 42 seconds and not responding, that would explain the disconnects. I bet the 100% cpu is every disconnect (or reconnect).
01:20 inodb joined #gluster
01:21 JoeJulian afk for a bit. I need to start some dinner.
01:21 DoubleJ I would need to inspect that further to see if there is a correlation...
01:21 DoubleJ enjoy your meal
01:25 P0w3r3d joined #gluster
01:34 bowhunter joined #gluster
01:41 DoubleJ oh, well, this would explain why the jobs segfaulted
01:41 DoubleJ [2016-11-16 01:08:35.648769] E [MSGID: 108006] [afr-common.c:4317:afr_notify] 14-gv0-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
01:42 DoubleJ this happened after setting write-behind off - and now keeps happening even after setting it back  "on". that's terrible...
02:04 haomaiwang joined #gluster
02:05 P0w3r3d joined #gluster
02:09 ankitraj joined #gluster
02:10 aj__ joined #gluster
02:13 Gambit15 joined #gluster
02:17 P0w3r3d joined #gluster
02:19 nbalacha joined #gluster
02:24 zat joined #gluster
02:30 Javezim @JoeJulian Found issue
02:31 Javezim Basically in /var/lib/glusterd/peers
02:31 Javezim One of the peers had a blank line under it
02:31 Javezim removed the blank line, started working
02:33 kramdoss_ joined #gluster
02:46 bowhunter joined #gluster
02:48 ilbot3 joined #gluster
02:48 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:58 Lee1092 joined #gluster
03:01 dnorman joined #gluster
03:19 JoeJulian Javezim: Can you please file a bug report
03:19 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
03:24 Muthu joined #gluster
03:26 magrawal joined #gluster
03:27 Javezim https://bugzilla.redhat.com/show_bug.cgi?id=1395498 @JoeJulian :) Bloody pain that one!
03:27 glusterbot Bug 1395498: medium, unspecified, ---, bugs, NEW , /var/lib/glusterd/peers File had a blank line, Stopped Glusterd from starting
03:38 nishanth joined #gluster
03:41 Philambdo joined #gluster
03:47 loadtheacc joined #gluster
03:57 atinm joined #gluster
04:05 RameshN joined #gluster
04:09 riyas joined #gluster
04:10 itisravi joined #gluster
04:18 shubhendu joined #gluster
04:24 loadtheacc joined #gluster
04:25 dnorman joined #gluster
04:26 buvanesh_kumar joined #gluster
04:35 jiffin joined #gluster
04:55 ankitraj joined #gluster
05:08 rafi joined #gluster
05:10 buvanesh_kumar_ joined #gluster
05:10 skoduri joined #gluster
05:13 Gnomethrower joined #gluster
05:22 ndarshan joined #gluster
05:23 loadtheacc joined #gluster
05:28 k4n0 joined #gluster
05:29 dnorman joined #gluster
05:30 sbulage joined #gluster
05:32 sbulage joined #gluster
05:34 karthik_us joined #gluster
05:36 kdhananjay joined #gluster
05:39 hgowtham joined #gluster
05:47 apandey joined #gluster
05:47 sanoj joined #gluster
05:57 jkroon joined #gluster
05:58 kramdoss_ joined #gluster
05:58 shubhendu joined #gluster
05:58 Bhaskarakiran joined #gluster
06:00 aravindavk joined #gluster
06:08 prth joined #gluster
06:08 msvbhat joined #gluster
06:16 shubhendu joined #gluster
06:18 jkroon joined #gluster
06:22 ppai joined #gluster
06:23 kotreshhr joined #gluster
06:25 ashiq joined #gluster
06:37 Saravanakmr joined #gluster
06:42 sanoj joined #gluster
06:49 mhulsman joined #gluster
06:51 hchiramm joined #gluster
06:51 sirdancealot joined #gluster
07:03 rastar joined #gluster
07:20 prth joined #gluster
07:24 riyas joined #gluster
07:33 haomaiwang joined #gluster
07:35 mhulsman joined #gluster
07:36 buvanesh_kumar joined #gluster
07:42 Gnomethrower joined #gluster
07:45 [diablo] joined #gluster
07:48 owitsches joined #gluster
07:58 jkroon joined #gluster
08:10 owitsches joined #gluster
08:14 ivan_rossi joined #gluster
08:17 devyani7 joined #gluster
08:31 jri joined #gluster
08:37 riyas joined #gluster
08:39 Javezim When you add an Arbiter brick to a volume, is there any real way of knowing how far the metadata self heal is along?
08:39 Javezim Like to an already existing Cluster
08:40 fsimonce joined #gluster
08:43 owitsches joined #gluster
08:43 itisravi The file count on the arbiter brick would be one way, i.e. checking if the count matches that of the other 2 data bricks.
08:44 itisravi But as long as heal info count doesn't become 0, it means heals are still pending
08:50 kshlm Javezim, Did you figure out what was wrong with you peer files?
08:50 Javezim What I find weird is the Inode count for the Arbiter bricks doesn't seem to be increasing fast at all, its been sitting around 494143761 out of 1171993344 for a day now
09:00 Javezim @kshlm Well I edited them manually and fixed the issue
09:00 Javezim All bricks back online
09:00 Javezim Now just need the damn arbiter to finish
09:00 kshlm Was something wrong in any peer file?
09:01 Javezim Yes, a blank line :/ > https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS   --- I do notice that it's maxing the CPU and Load on the Arbiter server, is there a way to limit the selfhealing process?
09:01 glusterbot Javezim: -'s karma is now -361
09:01 glusterbot Title: Log in to Red Hat Bugzilla (at bugzilla.redhat.com)
09:01 msvbhat joined #gluster
09:02 Javezim Sorry - https://bugzilla.redhat.com/show_bug.cgi?id=139549
09:02 glusterbot Bug 139549: is not accessible.
09:04 R0ok_ joined #gluster
09:11 jiffin1 joined #gluster
09:12 kdhananjay joined #gluster
09:15 sanoj joined #gluster
09:18 prth joined #gluster
09:20 karthik_us joined #gluster
09:32 jiffin1 joined #gluster
09:33 Slashman joined #gluster
09:34 sanoj joined #gluster
09:40 TvL2386 joined #gluster
10:02 kramdoss_ joined #gluster
10:05 sanoj joined #gluster
10:08 kotreshhr left #gluster
10:10 zat joined #gluster
10:15 zat joined #gluster
10:16 devyani7 joined #gluster
10:17 msvbhat joined #gluster
10:18 ws2k3 joined #gluster
10:18 ws2k3 joined #gluster
10:19 ws2k3 joined #gluster
10:19 ws2k3 joined #gluster
10:20 ws2k3 joined #gluster
10:20 mhulsman joined #gluster
10:23 arcolife joined #gluster
10:28 ketarax hi again.  "Multiple bricks of a disperse volume are present on the same server."   is this un-optimal performance-wise, or because it can cause me trouble when something fails / i need to bring nodes down (in this particular case, to fix certain nodes I would actually need to stop the glusterfs for fixing.  I'm okay with that, but interested in performance :-) )
10:29 ketarax this would be a 19/3 disperse set.  I
10:35 witsches joined #gluster
10:36 sanoj keatax, This could result in unavaialability of data when your node goes down. When multiple bricks are on same node they fail simultaneously on brick crash
10:37 sanoj ketarax, i mean node crash not brick crash
10:37 cloph whether it affects performance depends on how the bricks are laid out. - if they don't share i/o bottlenecks, and if there's enough network capacity, it shouldn't affect performance.
10:37 cloph But of course it is kindof pointless to have a dispersed volume rely on a single point of failure, after all you're using disperesed because you cannot afford/don't want to afford full replica
10:38 Gambit15 joined #gluster
10:39 kramdoss_ joined #gluster
10:40 cloph (so don't thing of controlled maintenance, but failure case, where you lose the bricks completely)
10:40 cloph in general, if you're interested in performance, I'd not use dispersed volumes
10:45 ketarax yes.  thanks, that helps.  yeah, with "performance", i'm actually only trying to loose as little as possible, after already adopting something else but RAID0.
10:46 ketarax ie. am not expecting miracles from glusterfs, and sort of trust that it handles this issue as good as possible.
10:48 ketarax another thing -- the order of the bricks listed for 'volume create disperse' -- anything to win there, or will gluster sort it out better than me?  the connections to nodes are in balance, but some nodes have about twice the amount of actual bricks.
10:51 Wizek joined #gluster
10:54 apandey joined #gluster
11:01 sanoj ketarax ordering matters, Gluster expects that they are in round robin order according to your fault tolerance domain.
11:02 sanoj ketarax, it should complain with same message as u mentioned if the ordering was given wrong, unless force option was given
11:03 mhulsman joined #gluster
11:08 skoduri joined #gluster
11:22 aj__ joined #gluster
11:24 devyani7 joined #gluster
11:24 rouven joined #gluster
11:26 rouven hey everybody - i'd love to create a special device (urandom) inside a chrooted document root on one of my gluster volumes. is that possible?
11:27 rouven for now, i receive a permission denied error whenever i try to do so, regardless if it is mounted via nfs or fuse
11:27 rastar joined #gluster
11:29 panina joined #gluster
11:39 ivan_rossi joined #gluster
11:49 msn joined #gluster
11:51 Slashman joined #gluster
11:57 riyas joined #gluster
12:00 csaba joined #gluster
12:01 kshlm Weekly community meeting is starting now in #gluster-meeting
12:02 haomaiwang joined #gluster
12:04 Plam joined #gluster
12:04 Plam hi there!
12:05 Plam tested gluster in a basic replication mode, 2 nodes. Then, created a big file with fallocate, export it in iSCSI with LIO
12:05 Plam it works
12:05 Plam but when I cut a node, when it's back online, the "resync" will use the whole big file size
12:05 Philambdo joined #gluster
12:05 Plam eg if I created a 60GB file size, a resync will transfer 60GB to the node
12:06 rouven joined #gluster
12:08 jdarcy joined #gluster
12:08 om2 joined #gluster
12:14 jkroon joined #gluster
12:16 saltsa joined #gluster
12:16 ndevos joined #gluster
12:16 ndevos joined #gluster
12:16 Plam is it normal?
12:20 mhulsman joined #gluster
12:24 rastar joined #gluster
12:28 panina joined #gluster
12:30 jurrien_ joined #gluster
12:32 P0w3r3d joined #gluster
12:36 Caveat4U joined #gluster
12:38 cloph Plam: depends - there's flag you can set to force full-file heal, but IIRC that is not the default. Also you can have sharding enabled, where a file saved to the brick in separate chunks that are healed individually.
12:39 Plam cloph: hey thanks :) I didn't configure any specific flag but the file was resync entirely (well by measuring the network usage, it was exactly the file size)
12:39 Plam so I think it's not a coincidence
12:39 Plam I'll take a look on sharding
12:40 portante joined #gluster
12:43 nbalacha joined #gluster
12:46 haomaiwang joined #gluster
12:50 k4n0 joined #gluster
12:50 ndarshan joined #gluster
12:53 cheneydeng joined #gluster
12:55 rafi1 joined #gluster
12:56 ctria joined #gluster
13:01 johnmilton joined #gluster
13:05 jiffin1 joined #gluster
13:05 kotreshhr joined #gluster
13:05 kotreshhr left #gluster
13:06 jiffin joined #gluster
13:10 Philambdo joined #gluster
13:17 Saravanakmr joined #gluster
13:34 Slashman joined #gluster
13:36 Muthu joined #gluster
13:42 ndk_ joined #gluster
13:42 d0nn1e joined #gluster
13:47 B21956 joined #gluster
13:56 mhulsman joined #gluster
13:57 unclemarc joined #gluster
13:58 sbulage joined #gluster
13:58 nbalacha joined #gluster
14:03 dlambrig_ joined #gluster
14:10 vbellur joined #gluster
14:14 annettec joined #gluster
14:21 shyam joined #gluster
14:28 skylar joined #gluster
14:30 squizzi joined #gluster
14:31 blues-man joined #gluster
14:32 blues-man hello, do you know if is it possible to reconfigure ganesha setup? I have an invalid pcs setup and I would like to trigger again ganesha setup but cannot figure out how
14:32 prth joined #gluster
14:33 aravindavk joined #gluster
14:36 cloph you can edit the export files manually and just restart it, nothing prevents you from chaning config, adding exported volumes, change permissions for exported volumes.
14:36 kkeithley `gluster nfs-ganesha disable` will tear down the pacemaker setup
14:36 kkeithley fix your config then run `gluster nfs-ganesha enable` again
14:36 jiffin1 joined #gluster
14:37 rouven joined #gluster
14:38 Lee1092 joined #gluster
14:39 blues-man kkeithley: I did it but the enable doesn't trigger any pcs setup, it looks like it check some already created file, is there any way to force running setup?
14:40 kkeithley On Fedora?
14:40 kkeithley Or RHEL/CentOS?
14:41 kkeithley try `rm /etc/corosync/corosync.conf`  then enable `gluster nfs-ganesha enable` again
14:41 blues-man i'm on rhel 7
14:43 blues-man kkeithley: I did it but it tells me "success" immediately and actually doesn't create again /etc/corosync/corosync.conf
14:45 blues-man I wonder if there is a way to purge all ganesha related settings and re-enable it
14:45 rouven joined #gluster
14:46 * kkeithley mumbles about incompatible changes in pacemaker
14:46 kkeithley do `pcs cluster setup --force --name $name $server1.....`
14:46 kkeithley `pcs cluster start --all`
14:47 kkeithley `pcs cluster stop --all`
14:47 kkeithley `pcs cluster destroy`
14:47 kkeithley `rm -rf /etc/cluster/cluster.conf*`
14:47 kkeithley `rm -rf /var/lib/pacemaker/cib/*`
14:48 kkeithley then try `gluster nfs-ganesha enable` again
14:48 blues-man ok, all sequentially?
14:48 kkeithley yes, in that order
14:48 blues-man ok I'm trying thanks
14:49 kkeithley oh, make sure /etc/corosync/corosync.conf is gone before you do `gluster nfs-ganesha enable`
14:49 blues-man /etc/cluster/cluster.conf doesn't exist because it is rhel 7 with only pacemaker pcsd
14:49 kkeithley yup
14:49 blues-man ok
14:50 blues-man /var/lib/pacemaker/cib doesn't exists also
14:51 farhorizon joined #gluster
14:51 blues-man nothing :(
14:52 bowhunter joined #gluster
14:55 rouven joined #gluster
14:57 blues-man kkeithley: removing *ganesha* rpm would help?
14:58 kkeithley try it
15:02 shyam joined #gluster
15:02 shaunm joined #gluster
15:04 farhorizon joined #gluster
15:04 blues-man kkeithley: do you know it's ok to have /etc/ganesha/ganesha.conf with "Clustered=false" ? the documentation doesn't mention it
15:05 kkeithley you can run one ganesha without any HA.
15:05 blues-man ok
15:05 kkeithley If you run multiple ganesha.nfsds you want HA to do things like kick surviving ganeshas into grace so clients can recover locks
15:05 ankitraj joined #gluster
15:07 blues-man I see
15:07 kkeithley if you don't do locking, then you run multiples and not worry about lock recovery
15:09 JoeJulian https://www.youtube.com/watch?v=54y4WkijkoI
15:10 JoeJulian That's kkeithley's talk at Gluster Summit about Ganesha's HA.
15:10 kkeithley you can run multiples
15:11 Plam regarding HA, what's the options for clients which can only use iSCSI or NFS3/v4? (in order to avoid SPOF)
15:11 Plam I'm testing LIO and multipath on client side, it works
15:12 Plam IDK if it's a bad practice or not (fallocate a big file which is exposed in block)
15:15 plarsen joined #gluster
15:16 virusuy joined #gluster
15:18 mhulsman joined #gluster
15:28 hackman joined #gluster
15:29 prth joined #gluster
15:29 squizzi joined #gluster
15:30 rouven joined #gluster
15:42 kkeithley IIRC iSCSI relies on multipath for HA.  For NFSv3 with gnfs use CTDB. For NFSv3/NFSv4/NFSv4+ use pacemaker/corosync.
15:43 kkeithley For NFSv3/NFSv4/NFSv4+ with nfs-ganesha use pacemaker/corosync.
15:44 farhoriz_ joined #gluster
15:44 kkeithley Plam: ^^^
15:46 Jacob843 joined #gluster
15:47 Plam yes so far I use multipath for HA
15:47 Plam but I have data corruption after restarting one of the 2 nodes
15:48 Plam so now I'll try CTDB with Smb
15:48 Plam I tried nfs-ganesha but had catastrophic perfs
15:48 abyss^ JoeJulian: ?;)
15:50 abyss^ JoeJulian: I did getfattr -m .  -d -e hex /ydp/shared5/gfs/nowaera/app/ne-bookshelf-asso.ydp.eu/shared on one of the split-brain dir. And i get only one trusted.afr.$volume-client-9 value: see: https://gist.github.com/anonymous/fdf4b8c7cb6a94629d3f71d1a723b152
15:50 glusterbot Title: split-b · GitHub (at gist.github.com)
15:50 abyss^ this is normal?
15:50 abyss^ Shouldn't I have both client-8 and client-9 ?
15:58 farhorizon joined #gluster
16:04 leafbag JoeJulian: if all the dht.linkto files should be size zero, do you have any idea why am I seeing a number of files with 1000 permissions and a size >0? That's what I'm trying to get to the bottom of.
16:04 leafbag Did you by chance happen to look at my gluster-users post from yesterday?
16:05 Philambdo joined #gluster
16:06 dnorman joined #gluster
16:07 skoduri joined #gluster
16:09 Caveat4U joined #gluster
16:19 witsches joined #gluster
16:28 blues-man kkeithley: also by unistalling rpm I cannor reconfigure ganesha ha cluster, how to deal with this without reinstalling the operating system?
16:28 kkeithley uninstall and then reinstall?
16:29 kkeithley glusterfs-ganesha is where /usr/libexec/ganesha/ganesha-ha.sh comes from. That's what the gluster CLI invokes to do the setup
16:29 kkeithley glusterfs-ganesha.rpm
16:29 blues-man yes, removed related rpm and reinstalled them
16:30 blues-man yes I removed pacemaker* *ganesha* pcs*
16:30 kkeithley you can run /usr/libexec/ganesha/ganesha-ha.sh by hand and get a better idea perhaps of what's not working
16:30 kkeithley `ganesha-ha.sh --setup /etc/ganesha`
16:30 blues-man I did it and it rely on pcs telling 3 times (dunno why) Error: cluster is not currently running on this node
16:30 blues-man also dealed with --add --delete --refresh-config
16:30 kpease_ joined #gluster
16:31 kkeithley is pcsd.service running?
16:31 blues-man yes
16:32 kkeithley and `pcs cluster setup --name $name $node1 $node2 ... ` works? or doesn't?
16:33 kkeithley with or without --force
16:33 kkeithley what is the name of the cluster? Is it 15char or less (it must be < 15char)
16:35 * kkeithley considers adding a check for that
16:35 blues-man yes cluster name is < 15 chars, it is "ha-nfs"
16:35 blues-man I run again --setup now I get
16:36 kkeithley paste your gansha-ha.conf
16:36 kkeithley @paste
16:36 glusterbot kkeithley: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
16:36 hackman joined #gluster
16:38 blues-man ok
16:38 kkeithley you can paste the output of --setup too
16:40 kkeithley lunch, biab
16:41 Caveat4U joined #gluster
16:44 blues-man kkeithley: sorry for delay port 9999 is blocked here :) https://pastebin.mozilla.org/8928812
16:44 jiffin joined #gluster
16:44 shyam joined #gluster
16:45 blues-man it doesn't start cluster ip
16:48 blues-man i had to start nfs-ganesha manually because it didn't find the pid
16:49 blues-man do you know if there are any other operations that the `gluster nfs-ganesha enable` does other than the ganesha.sh --setup ?
17:02 blues-man thanks kkeithley it looks now it is working :) --setup + pcs resource enable.. seem to let it work again
17:13 ashiq joined #gluster
17:18 kkeithley pcs resource enable ???
17:18 kkeithley glad it's working
17:26 elastix joined #gluster
17:33 nishanth joined #gluster
17:33 Caveat4U joined #gluster
17:34 virusuy joined #gluster
17:40 Caveat4U joined #gluster
17:45 nathwill joined #gluster
17:45 nathwill joined #gluster
17:50 ivan_rossi left #gluster
17:50 blues-man kkeithley: cluster ip were stopped, then I had to start it by pcs
17:51 blues-man really don't know why they were stopped after running --setup again but I don't wish reconfiguring ganesha to anyone ever :)
17:51 kkeithley ah, okay
17:51 kkeithley hopefully it gets better, more reliable. :-/
17:52 blues-man yeah, hope so also :) thanks for help
17:53 blues-man at the beginning, I thought it was a gluster hook that start/setuped ganesha, but I've seen the hook only creates the export in /etc/ganesha/exports
17:53 kkeithley yw
17:53 blues-man I wonder what there is between a gluster command such as `gluster nfs-ganesha enable` and the execution of a script like ganesha.sh
17:55 kkeithley very little in between.  `gluster foo bar baz is the traditional way of managing gluster.
17:56 kkeithley hence we have CLIs to also manage the ganesha bits, but they're very thin
17:56 blues-man but is there some tiny plugin system in charge of executing certain commands?
17:56 blues-man how do you logically separate stuff like ganesha from gluster core?
18:00 kkeithley the gluster and glusterd programs are a bowl of spaghetti. Makes my head hurt just thinking about it.
18:01 hchiramm joined #gluster
18:02 kkeithley but it is separate.  `gluster vol ...`  versus `gluster nfs-ganesha {enable,disable}` is part of the answer
18:02 blues-man lol
18:03 blues-man ok I see
18:03 kkeithley and I'm not the one that had to write it, so I'm a bit light on actual details.
18:06 kkeithley NOTICE:  I just changed  the .../LATEST symlink on download.gluster.org. It now points at .../3.9/3.9.0
18:07 kkeithley NOTICE: If you want to stay on 3.8.x, change LATEST in any package repo meta files to LTM-3.8
18:08 vbellur joined #gluster
18:09 jiffin joined #gluster
18:15 blues-man bye
18:18 dnorman joined #gluster
18:51 kpease_ joined #gluster
19:04 dnorman joined #gluster
19:07 jri joined #gluster
19:18 jri joined #gluster
19:31 edong23 joined #gluster
19:39 rouven joined #gluster
19:47 farhoriz_ joined #gluster
19:48 Philambdo joined #gluster
20:05 Caveat4U joined #gluster
20:10 jkroon joined #gluster
20:18 mhulsman joined #gluster
20:21 Caveat4U joined #gluster
20:33 arpu joined #gluster
20:37 dnorman joined #gluster
20:47 farhorizon joined #gluster
20:48 ctria joined #gluster
20:58 hackman joined #gluster
20:59 scuttle|afk joined #gluster
21:01 d4n13L joined #gluster
21:11 hybrid512 joined #gluster
21:12 hybrid512 Hi ! Is there a way to manage a glusterfs volume with bricks one by one when dealing with replicated data and not 2 by 2 with a distributed-replicated volume of replica 2 ?
21:13 hybrid512 I mean, sometime, I need to add or remove only one brick at a time and it is quite annoying to have to this 2 by 2 whgich is not possible all the time
21:13 hybrid512 or maybe I'm just doing it wrong
21:14 jkroon no, you definitely need to add two bricks at a time.
21:14 hybrid512 bad :(
21:14 jkroon not sure about remove - guessing that since you need a multiple of replica bricks the same reasoning would apply.
21:14 hybrid512 I confirm ... and this is very annoying
21:15 jkroon well, let's say you have a distribute=2, replica=2 - and you want to make that distribute=3, replica=2, if you only add a single brick - what would you expect gluster to do?  it can't really use just that 1 brick in any sensible way can it?
21:17 hybrid512 well, I don't know, I just see this is not like this with other network block devices such as Ceph or Sheepdog ... I like the fact that Gluster is some sort of "NFS without SPOF" and that it is very easy to setup but this brick handling is just painful
21:17 jkroon please explain why if you don't mind.
21:18 hybrid512 I tried to workaround this in many ways but didn't find how ... so I thought I was just doing it the wrong way but apparently not
21:18 jkroon i've been using gluster for a while and my biggest complaint was performance ... which turns out was a kernel bug from at least 4.1 until at least 4.7.x (I haven't had the guts to test 4.8 with glusterfs yet)
21:19 hybrid512 Simple : My glusterfs cluster is based on many nodes with just one disk dedicated to gluster. If I want to expand my storage, I must add 2 nodes at a time which I might not alway have
21:19 jkroon ah!
21:19 jkroon the workaround is to have two bricks per server :).  even if those bricks share a disk.
21:19 Caveat4U joined #gluster
21:20 hybrid512 That's the workaround I found too ... but this is highly discouraged and I ended in broken situation quite often doing this way
21:20 jkroon so servers has mounted disk as /mnt/gluster, then with two servers you setup bricks as A:/mnt/gluster/b0 B:/mnt/gluster/b1 and the other pair as B:/mnt/gluster/b0 A:/mnt/gluster/b1
21:20 jkroon and then to expand you follow JoeJulian's guide (the url which escapes me right now)
21:21 hybrid512 just an exemple : I had a failure on 1 node recently, so 2 bricks went down (but those bricks where in fact 2 partitons on the same disk) and just because of that, I ended with a corrupted volume
21:22 jkroon so mirrored/replicated bricks must never be on the same node - you do need to be careful for that.
21:22 hybrid512 okay but how do you ensure that ?
21:22 panina joined #gluster
21:22 jkroon the order in which you add them is critical.
21:23 jkroon (A:b0 B:b1) (B:b0 A:b1)
21:23 jkroon now to expand you replace A:b1 with C:b1, then when data migration is done you add C:b0 A:b1
21:23 hybrid512 humm ... I see
21:23 hybrid512 this is probably where I failed
21:24 hybrid512 anyway, this is a bit tricky and not easy to maintain
21:27 arc0 joined #gluster
21:33 Caveat4U joined #gluster
21:35 aj__ joined #gluster
21:50 jkroon hybrid512, i disagree. for the most part i find glusterfs very intuitive and very easy to use.  when it gets to certain split-brain scenarios is when I start losing it.
22:16 farhorizon joined #gluster
22:28 bhakti joined #gluster
22:42 Caveat4U joined #gluster
22:43 Caveat4U joined #gluster
22:57 bluenemo joined #gluster
22:58 annettec joined #gluster
23:00 JoeJulian abyss^: Unless a heal is necessitates between replicated directories, it's normal for there to be no trusted.afr xattr.
23:01 JoeJulian leafbag: The only way that I know of would be for someone to have created a file through the client mount and set it mode 1000.
23:02 JoeJulian leafbag: If it also has a dht.linkto attribute, it should be 0 sized. If it were me, I would confirm that the file exists somewhere else and delete the dht.linkto file.
23:08 annettec1 joined #gluster
23:10 Caveat4U joined #gluster
23:23 zat joined #gluster
23:33 Caveat4U joined #gluster
23:33 Caveat4U joined #gluster
23:45 Caveat4U joined #gluster
23:45 DoubleJ left #gluster
23:47 Micha2k joined #gluster
23:48 farhorizon joined #gluster
23:48 dnorman joined #gluster
23:48 farhorizon joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary