Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-08-18

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 ZachLanich misc Yea, so here's what it would look like: Gluster Volume Subdir dedicated to a specific container gets bind-mounted into the container at /data/gluster. ACLs get applied to that subdir for R/W using the UID/GID that the LXD is running as and I'll be using GID/UID maps to map root and/or another user inside the container to the host's UID/GID the container is running under so the container can R/W to the subdir. Make sense?
00:02 misc yeah
00:02 misc it sound sane enough
00:02 ZachLanich Excuse my random flipping of GID/UID & UID/GID :P
00:02 misc I would add svirt on top of that and various selinux stuff, but it depend how much control the user have on the wordpress instance
00:03 ZachLanich It seems to me that it wouldn't be nearly as insecure as say a shared hosting environment with files only being separated by users on the same host. Am I correct?
00:03 misc also, that mean you have some kind of shared ldap ?
00:03 ZachLanich I feel like there would have to be a SERIOUS vulnerability in either FUSE or the Kernel for there to be priv escalation going on. Correct?
00:04 misc or a user setting his permission in a rather lax way
00:04 ZachLanich misc I'm not super familiar with LDAP yet and I'm not 100% sure how I'm going to handle that lol. I'll have to do my homework. Do you have advice?
00:04 misc (like one setting 777 on his part of the /data/gluster, because he did read that on a tutorial)
00:05 ZachLanich misc No, I won't be using 777. That's stupid and not necessary if I'm correct.
00:06 ZachLanich If I correctly map the uids inside the container to the uids of the container itself and apply sane permissions to the subdir, I should be able to accomplish R/W without 777.
00:06 misc ZachLanich: you no, but would user have enough access to do that ?
00:08 misc but time to sleep here
00:09 ZachLanich I think the uid maps will accomplish that if I understand them correctly. I'll have to experiment.
00:09 ZachLanich Thanks misc
00:13 chirino_m joined #gluster
00:23 d0nn1e joined #gluster
00:47 wadeholler joined #gluster
00:54 klaas joined #gluster
00:56 shdeng joined #gluster
01:34 kramdoss_ joined #gluster
01:37 Lee1092 joined #gluster
01:41 Alghost joined #gluster
01:58 derjohn_mobi joined #gluster
02:07 wadeholler joined #gluster
02:09 squizzi_1 joined #gluster
02:22 Alghost_ joined #gluster
02:25 plarsen joined #gluster
02:33 DV_ joined #gluster
03:05 julim joined #gluster
03:20 hchiramm joined #gluster
03:24 sanoj joined #gluster
03:35 magrawal joined #gluster
03:41 RameshN joined #gluster
04:01 atinm joined #gluster
04:06 raghug joined #gluster
04:23 om joined #gluster
04:27 shubhendu joined #gluster
04:29 poornimag joined #gluster
04:34 aravindavk joined #gluster
04:36 shubhendu joined #gluster
04:40 kramdoss_ joined #gluster
04:40 nbalacha joined #gluster
04:45 karthik_ joined #gluster
04:47 R0ok_ joined #gluster
05:05 Alghost joined #gluster
05:14 jiffin joined #gluster
05:16 ndarshan joined #gluster
05:17 RameshN joined #gluster
05:18 atinm joined #gluster
05:18 kramdoss_ joined #gluster
05:27 ashiq joined #gluster
05:34 nbalacha joined #gluster
05:37 Saravanakmr joined #gluster
05:37 msvbhat joined #gluster
05:43 rastar joined #gluster
05:44 kdhananjay joined #gluster
05:44 hgowtham joined #gluster
05:46 rafi joined #gluster
05:49 mhulsman joined #gluster
05:57 mhulsman joined #gluster
06:01 Manikandan joined #gluster
06:01 itisravi joined #gluster
06:02 LinkRage joined #gluster
06:05 kshlm joined #gluster
06:08 [diablo] joined #gluster
06:08 ankitraj joined #gluster
06:10 kotreshhr joined #gluster
06:12 aspandey joined #gluster
06:14 arcolife joined #gluster
06:15 nishanth joined #gluster
06:18 mhulsman joined #gluster
06:19 nbalacha joined #gluster
06:19 Bhaskarakiran joined #gluster
06:20 atalur joined #gluster
06:20 anil joined #gluster
06:22 atinm joined #gluster
06:31 Muthu_ joined #gluster
06:31 jtux joined #gluster
06:31 Jacob843 joined #gluster
06:32 Muthu joined #gluster
06:33 jiffin1 joined #gluster
06:39 bkolden joined #gluster
06:41 R0ok_ joined #gluster
06:43 atinm joined #gluster
06:44 jkroon joined #gluster
06:44 aravindavk joined #gluster
06:45 devyani7 joined #gluster
06:48 devyani7 joined #gluster
06:52 jtux joined #gluster
06:58 R0ok_ joined #gluster
06:58 Manikandan joined #gluster
07:05 hackman joined #gluster
07:12 MikeLupe joined #gluster
07:14 jiffin1 joined #gluster
07:15 Sebbo1 joined #gluster
07:17 kxseven joined #gluster
07:20 karnan joined #gluster
07:29 atinm joined #gluster
07:33 telius On two servers of mine
07:33 jkroon hi all, on gluster 3.7.14 (and previously older versions) we're seeing some issues wrt flock.
07:33 hackman joined #gluster
07:33 telius with the hostnames in each others' /etc/hosts file, can ping each other using hostname, ssh, etc
07:33 sacoetzee joined #gluster
07:34 telius I can probe the other using the IPv4 IP, but not the IPv6 IP nor the hostname, why is that?
07:34 telius I get peer probe: failed: Probe returned with Transport endpoint is not connected
07:36 anoopcs jkroon, Can you please explain the issue in detail?
07:36 jkroon we're running php on top of it, and one of our clients is seemingly setting the session path to internal in moodle htdocs folder, and that periodically now results in flock failing and blocking indefinitely.  where we previously encountered the problem we switched to non-blocking flock.
07:36 jkroon anoopcs, what detail would help?  we're seeing php proceses getting stuck on flock()
07:36 Philambdo joined #gluster
07:36 rafa_ joined #gluster
07:37 ndevos jkroon: do you mount over fuse or nfs?
07:37 jkroon fuse ... over nfs flock doesn't work at all (last I tested)
07:39 ndevos ok, good (locking over nfs on storage servers does not work, the nfs-client and nfs-server need their own nlm implementation, only one can be active)
07:39 jkroon i've got a sample "kernel stack" for one of the processes, but I'm not sure it'll be useful.
07:40 jkroon essentially, sys_flock => fuse_setlk => fuse-simple_request => __fuse_request_send => autoremove_wake_function => request_wait_answer => __set_task_blocked => recalc_sig_pending
07:40 jkroon that recalc worries me slightly, not sure if that's indicative of a kernel bug possibly?
07:41 ndevos I very much doubt it'll be a kernel bug
07:41 jkroon i tend to agree, purely based on "other filesystems will be affected too", but that said, i'm no expert and would like to understand your reasoning?
07:42 ndevos have you confirmed that no other process is holding a lock on the file, also not on other servers?
07:42 ndevos I think you can see the locks and inode-nrs in /proc/locks
07:42 kovshenin joined #gluster
07:43 jkroon i can see stuff in there, but I'm not sure how to interpret it.
07:44 Manikandan joined #gluster
07:45 ndevos 1: POSIX  ADVISORY  WRITE 32130 fc:03:1475248 0 EOF
07:45 ndevos #: POSIX ADVISORY <READ/WRITE> <PID> <MAJOR>:<MINOR>:<INODE> <START> <END>
07:46 ndevos well, or something like that
07:48 jkroon ok, but this will be FLOCK instead.
07:48 jkroon those are the locks that's taken?
07:50 hackman joined #gluster
07:50 jkroon ndevos, it seems currently it's flock() on a fd to a deleted file.
07:51 ndevos those are the locks that are taken on this particular client, by that particular PID
07:51 jkroon we need to work back from file (inode) to who has the lock don't we?
07:52 jkroon so since I only have "deleted" files ... that's not all that useful since I am going to struggle to get an inode number ... or is there some trick other than stat on the file itself?
07:52 ndevos if you have multiple systems that have clients taking the lock, you'll need to check those too, for the same inode
07:52 tjikkun joined #gluster
07:52 ndevos yeah... thats a little tricky
07:53 fsimonce joined #gluster
07:54 tjikkun Is there any way to find out how long a self-heal will take, or any way to speed it up? I am on gluster 3.7.3
07:54 ndevos if you're lucky, lsof or /proc/<pid>/fd/ might still give some information
07:55 jkroon stat on /proc/${pid}/fd/${fd} also doesn't seem to be particularly helpful.
07:55 jkroon that gives inode number on /proc/ fs not on the containing filesystem :(
07:55 ndevos and "ls -l" probably gives "(deleted)" as well?
07:56 jkroon https://paste.fedoraproject.org/409997/06994147/
07:56 glusterbot Title: #409997 Fedora Project Pastebin (at paste.fedoraproject.org)
07:56 jkroon perhaps that'll give some insight to what I mean.
07:57 ndevos tjikkun: no, I do not think so, but maube kdhananjay has an idea - but why would you stay on 3.7.3 when 3.7.14 is the stable release for 3.7?
07:57 jkroon inode is specific to a mount point ... can we at least map the mountpoint to the device number (which should also be in /proc/locks)
07:59 jkroon ah crap.  management meeting :(.  sacoetzee will be able to assist with any additional information required/requested (he pointed out the problem to me).
07:59 tjikkun ndevos: Yeah I guess mostly because it did not get the attention it deserved, and "if it ain't broke.. etc." But now I'm thinking updating might be a good idea, but not before self-heal is done
07:59 jkroon later.
07:59 ndevos ttyl jkroon
08:00 sacoetzee *caveat* not as technically strong as jkroon
08:00 tjikkun It is now healing for more then a day, but still the same output from `gluster volume heal VMS info`
08:02 kdhananjay tjikkun: are you using gluster to store vms?
08:02 tjikkun kdhananjay: yes, using ovirt
08:02 kdhananjay tjikkun: are you using sharding as well?
08:02 tjikkun no replica 3
08:03 kdhananjay tjikkun: oops ok. whats the avg vm image size?
08:04 ndevos sacoetzee: I think the devices (major:minor) can be mapped from /proc/devices and /proc/partitions, but I dont know how fuse would be listed there :-/
08:04 tjikkun avg I'd say about 40GB
08:05 kdhananjay tjikkun: ok, you need to disable client side heals first thing.
08:05 itisravi joined #gluster
08:05 kdhananjay tjikkun: otherwise your IO may halt until the file being operated on is healed.
08:06 kdhananjay tjikkun: # for i in data metadata entry; do gluster volume set VOL $i-self-heal off; done
08:06 tjikkun kdhananjay: ok, so that is done
08:07 kdhananjay tjikkun: you could use multi-threaded self-heal which would do parallel heals.
08:07 kdhananjay tjikkun: but it can consume cpu and compete with the normal IO that's happening on the file.
08:07 * sacoetzee looking
08:08 kdhananjay tjikkun: you could try that and see how it works for you.
08:08 kdhananjay tjikkun: this is a short-term workaround. But you'd do well to use sharding in the long run.
08:09 tjikkun ok, but now I set *-self-heal off, how will the self-heal happen? Do I need to trigger it?
08:09 kdhananjay tjikkun: no, we just disabled client side healing (heal that is done from the glusterfs client itself on accessing a file, in the normal data path).
08:09 tjikkun kdhananjay: I think replica 3 was the only option supported in ovirt at the time we configured it
08:09 kdhananjay tjikkun: as long as you have self-heal-daemon on, heals should still run fine.
08:11 kdhananjay tjikkun: i understand, but sharding is something worth considering for your use-case.
08:11 tjikkun kdhananjay: I will read up on it, thanks!
08:12 kdhananjay tjikkun: for multi-threaded self-heal, you need to set the following two options:
08:12 kdhananjay cluster.shd-wait-qlength: 10000
08:12 kdhananjay cluster.shd-max-threads: 4
08:12 tjikkun heh thanks, was just looking for those
08:12 kdhananjay as in #gluster volume set <VOL> cluster.shd-wait-qlength 10000 ...
08:13 tjikkun kdhananjay: It appears I don't have those option, my version might be too old
08:13 kdhananjay tjikkun: One last thing. ;) i hope you have data-self-heal-algorithm set to 'full'.
08:14 kdhananjay tjikkun: which version are you on?
08:14 tjikkun 3.7.3 (I know)
08:14 tjikkun data-self-heal-algorithm is set to (null)
08:15 kdhananjay tjikkun: time to do 3.7.3+10 ;)
08:15 tjikkun +11, right?
08:15 kdhananjay yes, +11 even better
08:16 tjikkun but first the self heal needs to finish
08:16 kdhananjay tjikkun: yeah, now you can't do much, except for waiting.
08:16 tjikkun I don't think it is a good idea to upgrade now, right?
08:16 kdhananjay tjikkun: no, let the heals finish.
08:17 ndevos sacoetzee: there is also a "gluster volume info ... locks" command (not sure how it exactly is spelled), maybe that shows some useful details?
08:17 tjikkun kdhananjay: Thank you very much for the info!
08:17 kdhananjay tjikkun: yw!
08:19 sacoetzee ndevos: thanks will look into that, can't find anything on the fuse in the devices and partitions, as LVM and raid is also in the way
08:20 tjikkun kdhananjay: oh last question: data-self-heal-algorithm (null), what does it use in that case?
08:21 kdhananjay tjikkun: set it to full
08:21 kdhananjay tjikkun: gluster volume set <VOL> data-self-heal-algorithm full
08:22 tjikkun yeah I did that, but when it was set to (null), what did it do then? full, diff or reset?
08:23 jri joined #gluster
08:24 ndevos sacoetzee: yeah, it works fine for normal block devices, but maybe fuse mounts are just not listed at the usual places
08:26 kdhananjay tjikkun: what does `volume info` say>
08:26 kdhananjay ?
08:30 tjikkun kdhananjay: now it says cluster.data-self-heal-algorithm: full, but I don't know what it said before
08:31 kdhananjay tjikkun: ok then it is fine.
08:31 tjikkun And how can I see if self-heal is really doing something?
08:33 deniszh joined #gluster
08:34 kdhananjay tjikkun: gluster volume heal <VOL> info
08:35 tjikkun yeah but that says Possibly undergoing heal
08:35 tjikkun Possibly.. so possibly not?
08:36 kdhananjay tjikkun: no that could be because the file is also witnessing writes.
08:36 tjikkun ah yes that is definetely happening
08:37 Philambdo joined #gluster
08:41 Slashman joined #gluster
08:43 itisravi joined #gluster
08:44 pur joined #gluster
09:01 jiffin1 joined #gluster
09:04 ramky joined #gluster
09:09 Bhaskarakiran joined #gluster
09:09 Philambdo joined #gluster
09:26 rastar joined #gluster
09:26 ansyeblya joined #gluster
09:26 ansyeblya Please specify the full shared storage connection path to use (example: host:/path):
09:26 ansyeblya during ovirt hosted engine install, what should I put here ???
09:26 ansyeblya for glusterFS
09:27 ansyeblya nothing works
09:27 ansyeblya couldnt even google what is considered to be "shared storage connecoitn path" in regards to gluster
09:27 sacoetzee joined #gluster
09:30 jiffin1 joined #gluster
09:40 jri_ joined #gluster
09:43 nbalacha joined #gluster
09:45 derjohn_mobi joined #gluster
10:03 jmpq joined #gluster
10:03 jmpq hi, How can i create a JBR volume? is there any document? what's the maturity of JBR?
10:17 wadeholler joined #gluster
10:18 deniszh joined #gluster
10:22 wadeholler joined #gluster
10:22 msvbhat joined #gluster
10:23 wadeholler joined #gluster
10:28 jri joined #gluster
10:31 [diablo] joined #gluster
10:34 ashiq joined #gluster
10:36 rastar joined #gluster
11:10 ashiq joined #gluster
11:14 rastar joined #gluster
11:15 [diablo] joined #gluster
11:19 B21956 joined #gluster
11:26 Bhaskarakiran joined #gluster
11:26 jri joined #gluster
11:49 kdhananjay joined #gluster
11:51 jkroon ndevos, thanks for the assist with sacoetzee, stat -c%D /mountpoint/ is the most sensible way to find the device ID btw.
11:51 jkroon fuse mounts don't appear in /proc/partitions|devices
11:54 ndevos jkroon: ah. stat! I did check df and similar, but of course it is in the stat outout
11:54 ndevos *output
11:56 jkroon yea, only just thought of that myself now after digging a bit.
11:56 jkroon ok, so the hh:hh:ddddd field in /proc/locks needs to correlate with that somehow, but I'm not sure how the major:minor split works, or whether we're even looking at the right thing.
11:57 jkroon that said, on both servers stat /home shows the device id to be 12 (hex) and I can't find that as either the major or minor in /proc/locks.
12:00 jkroon ndevos, in the statedump - how can I identify locks there?
12:00 jkroon searching for the string lock I find a few but they seem to relate to glusterfs internals, not flock()s.
12:01 ndevos jkroon: hmm, locks are kept on the bricks, but the clients should know about its own locks too, where are you looking?
12:02 rwheeler joined #gluster
12:02 ndevos jkroon: the output of "gluster volume info ... locks" might be easier
12:02 chirino_m joined #gluster
12:03 jkroon ndevos, except no such command exists ... volume clear-locks is the only lock-related command I can find from help.
12:04 overclk joined #gluster
12:05 jkroon and I can't even figure out how to use that ...
12:06 rastar joined #gluster
12:10 jkroon from stat, you get "Device: 902h/2306d" - that 902h is the major:minor, so in this case (for my /) is major "09", minor "02", so in /proc/locks scanning for 09:02:XXXXX gives locks on files on / partition, so give 12h for device id for /home/ we're looking for 00:12:XXXXX in /proc/locks which I can't find on either server so as far as I understand this implies no locks are held (according to the kernel) on any glusterfs files on /home/
12:10 jkroon am i missing something glaringly obvious?  This to me confirms a bug somewhere in the kernel/glusterfs (unsure which yet).
12:13 ndevos hmm, I was convinced there was a command to show locks, but I Cant find that either
12:13 jkroon 10: FLOCK  ADVISORY  WRITE 3637 fd:00:7458 0 EOF
12:13 jkroon 10: -> FLOCK  ADVISORY  WRITE 23487 fd:00:7458 0 EOF
12:13 ndevos thats two PIDs, any common files they have opened?
12:14 jkroon from /proc/locks - that worries me, seeing that neither of those pid numbers can be found on my system.  fd:00 is my /tmp/
12:14 jkroon if they existed in /proc/ I'd check ... ps axf also doesn't show them :)
12:15 jkroon does anybody know if there were changes to the flock implementation recently in the kernel?
12:17 jkroon ok, so that lock is under my control, something I use, but still weird that I can't find those processes.  the file (as located using find /tmp/ -inum) is one that my code uses, and not related to anything wrt glusterfs.
12:17 ndevos well, uh, that really should be an existing pid... maybe namespaces like containers?
12:17 jkroon nope.
12:18 jkroon flock is an interesting beast.  you can open a file, fork(), take the lock in the child and terminate the child, thereafter the parent will have the lock.
12:19 jkroon lockf (posix) locks are different, and in many ways more sane.
12:19 poornimag joined #gluster
12:20 jkroon basically flock() tracks the file descriptor, so any process sharing that fd by implication also has the lock.
12:21 jkroon so the pid there is probably the pid of the process that originally called flock() (and which may in the meantime have terminated)
12:24 rafa_ joined #gluster
12:31 jiffin1 joined #gluster
12:36 Klas currently, for some reason, my gluster lab nodes are disconnected from each other, rebooting and such doesn't seem to help, any idea where I should check?
12:36 hagarth joined #gluster
12:38 jiffin Klas: did u mean gluster peer status show disconnected?
12:38 Klas typ
12:39 Klas and shares are down
12:39 jiffin check the glusterd logs for more clues /var/log/glusterfs/<etc->/log
12:39 Klas s/typ/yup/
12:39 glusterbot What Klas meant to say was: yup
12:41 plarsen joined #gluster
12:42 jiffin also it might be issue related firewalld
12:43 jiffin or iptables
12:49 Klas telnet on 24008 does fail, both on localhost and on peers
12:50 Klas there shouldn't be any firewall issues between them, they are on the same subnet
12:50 Klas and no software firewall
12:50 jkroon Klas, make sure.  iptables -L -v -n will reveal all.
12:51 Klas already checked =)
12:51 jkroon netstat -ntlp | grep gluster ??
12:52 Klas tcp        0      0 0.0.0.0:24007           0.0.0.0:*               LISTEN      4502/glusterd
12:52 Klas there should be 24008, right?
12:52 jkroon not on my systems either.
12:52 Klas oh
12:53 julim joined #gluster
12:55 Klas oh, look, now I did find SSL connect errors...
12:55 ndevos Klas: 24008 is used with rdma, not normally
12:55 DV__ joined #gluster
12:55 ndevos @ports
12:55 glusterbot ndevos: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up. All ports must be reachable by both servers and clients. Additionally it will listen on 38465-38468/tcp for NFS. NFS also depends on rpcbind/portmap ports 111 and 2049.
12:56 Klas huh, /var/lib/glusterd/secure-access was broken
12:58 dlambrig joined #gluster
12:58 nbalacha joined #gluster
12:59 ira joined #gluster
13:00 Klas ok, now I just need to understand why SSL is broken after my update to 3.7.14 and a vacation =P
13:05 Philambdo joined #gluster
13:05 malevolent joined #gluster
13:13 m0zes joined #gluster
13:18 rastar joined #gluster
13:24 derjohn_mobi joined #gluster
13:27 shubhendu joined #gluster
13:33 Sebbo1 I'm not able to delete an empty directory. Also not with "rm -rf directory/". It always returns "rm: cannot remove 'directory/': Directory not empty". I'm able to rename it as well as to move it on the GlusterFS storage. How can I delete it?
13:33 jkroon ls -la ... perhaps some "dot file"?
13:34 Sebbo1 Nope, it's empty. Only "." and ".." is visible
13:34 Sebbo1 "rm -rf directory/" would also delete any content of a directory
13:35 rwheeler joined #gluster
13:35 nbalacha Sebbo1, which version are you running? we fixed this recently
13:35 nbalacha Sebbo1, can you check if the dirs are empty on the backen bricks?
13:36 nbalacha *backend
13:36 Sebbo1 nbalacha: 3.7.14-ubuntu1
13:37 Sebbo1 nbalacha: Yes, it's empty there
13:37 nbalacha Sebbo1, on all bricks?
13:42 Sebbo1 nbalacha: Ah - no. On 1 of 3 bricks are some files.
13:42 nbalacha Sebbo1, that is what is causing the problem. Can you paste the listing somewhere?
13:44 Klas reinstallation at least worked for my lab systems btw, I gave up on finding the issue
13:44 satya4ever joined #gluster
13:46 Sebbo1 nbalacha: Sure, here you go: http://pastebin.com/0G3dew8M
13:46 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
13:51 Sebbo1 nbalacha: In which version has this been fixed?
13:51 nbalacha Sebbo1, I need to look up the exact version.
13:52 nbalacha Sebbo1, this issue looks different from the one we fixed. We used to run into issues where we had empty dirs in the backen that were not listed
13:53 nbalacha Sebbo1, can you please provide some more info? Can we get the xattrs on the dir from the brick and the mount logs?
13:55 baojg joined #gluster
13:55 Sebbo1 nbalacha: Yeah, I've found a new bug! *gg* :D Sure, I'll provide all necessary information, which you require. Do you have a list of commands, which I should execute?
14:01 ppai joined #gluster
14:02 Gnomethrower joined #gluster
14:07 rafi joined #gluster
14:12 Sebbo1 nbalacha: How do I get these xattrs?
14:12 nbalacha Sebbo1, getfattr -e hex -m . -d <path to dir on brick>
14:12 Sebbo1 And do you need all logs of all glusterfs-servers or only of one?
14:13 nbalacha Sorry had stepped away.what type of vol is it? Pure distribute?
14:13 nbalacha Sebbo1, how many clients are accessing the volume?
14:17 Sebbo1 No problem. Yes, it's distributed. Three GlusterFS servers and every GlusterFS server provides one brick with 2 TB disk space. Accessing clients are around 35-40 Linux Ubuntu servers.
14:26 shyam joined #gluster
14:30 Gambit15 Hey, a question on NIC bonding. As long as the kit supports it, isn't mode 4 better than mode 6?
14:46 nishanth joined #gluster
14:56 ankitraj joined #gluster
14:59 atalur joined #gluster
15:20 DV__ joined #gluster
15:22 nishanth joined #gluster
15:30 jwd joined #gluster
15:30 Philambdo joined #gluster
15:44 rastar joined #gluster
15:56 rafi joined #gluster
16:04 Gnomethrower joined #gluster
16:10 poornimag joined #gluster
16:29 ZachLanich joined #gluster
16:47 ZachLanich joined #gluster
16:53 dtrainor_ joined #gluster
16:55 ZachLanich joined #gluster
16:55 deniszh joined #gluster
17:07 pampan joined #gluster
17:30 JoeJulian dammit, kkeithley. Now I look like an idiot. :P When did you figure out how to build them? I thought there was a library missing or something. I'm sure I remember it being unable to be built.
17:31 kkeithley er, dunno. AFAIK I've always been building them.
17:32 JoeJulian No, when I finally got approval to upgrade, they didn't exist and I'm sure it was because some dependency changed.
17:32 johnmilton joined #gluster
17:33 kkeithley cue the scene of Gandalf in Moria saying "I have no memory of this..."
17:33 kkeithley give me a bit, it'll come back to me
17:34 JoeJulian Don't work too hard. If it works, I'm not going to worry about it.
17:36 kkeithley I think we had a dependency for a while on a newer version of URCU than what's in trusty.
17:36 kkeithley Mainly for pkginfo
17:37 kkeithley we rewrote the autoconf to work around missing pkginfo and determined we were able to use the version that's in trusty.
17:37 post-factum joined #gluster
17:37 JoeJulian Aha
17:38 skoduri joined #gluster
17:38 kpease joined #gluster
17:39 neofob joined #gluster
17:39 johnmilton joined #gluster
17:44 ashiq joined #gluster
17:53 JoeJulian Yep, kkeithley, you announced it in here on Nov. 19th. I can't believe it's been that long. https://botbot.me/freenode/gluster/msg/54519088/
17:54 post-factum JoeJulian: and everything undercovered becomes evident
17:56 JoeJulian I blame everything on post-factum. It's all his fault.
17:57 deniszh joined #gluster
17:58 post-factum JoeJulian: sure, don't forget your napalm pack and nose rag
17:59 kkeithley Damnit Jim, I'm a Software Developer, not a Brick Layer.
17:59 kkeithley Wish I could write some software
18:01 Gambit15 joined #gluster
18:01 om joined #gluster
18:02 rwheeler joined #gluster
18:04 jiffin joined #gluster
18:05 arcolife joined #gluster
18:07 DV__ joined #gluster
18:07 rafi joined #gluster
18:09 post-factum if brick layers were writing software, the first bug flying in the city would destroy the whole civilization
18:10 Philambdo joined #gluster
18:21 jiffin joined #gluster
18:30 bowhunter joined #gluster
18:31 kovshenin joined #gluster
18:32 Lee1092 joined #gluster
18:35 acki joined #gluster
18:35 acki hi there
18:36 robb_nl joined #gluster
18:36 acki i have a performance issue with gluster. following setup: all centos 7.2 and gluster 3.7.14, two gluster servers with a replicated volume, one client
18:37 acki disk speed bare: ~280MB/s, disk speed NFS mount ~25MB/s, disk speed glusterfs mount ~3MB/S (tested with dd if=/dev/zero  of=testfile bs=1K count=10240)
18:37 acki anyone have an idea?
18:38 acki it's all local network with internal IPs
18:38 JoeJulian First of all, 1k is less than a TCP packet, so that's wasteful.
18:39 om joined #gluster
18:40 acki @JoeJulian i use the environment for a clustered webserver with WordPress on it - so there would be small files. actually site requests goes up to 5sec when served from gluster
18:41 JoeJulian @php
18:41 glusterbot JoeJulian: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH --fopen-keep-cache
18:41 JoeJulian Also https://joejulian.name/blog/dht-misses-are-expensive/
18:42 glusterbot Title: DHT misses are expensive (at joejulian.name)
18:42 acki i'll give it a try, thx!
18:42 JoeJulian But that's still unreasonably slow. What version?
18:44 acki 3.7.14
18:44 acki centos 7.2
18:44 acki internal network
18:44 acki no firewall
18:45 Gambit15 Hey, I'm trying to find the main pros & cons for striped-replicated & distributed replicated. It's for a VM environment, so there will only be one client per file/VHD, with random R/W I/O weighted towards the 32k-128k region.
18:47 deniszh joined #gluster
18:47 kovshenin joined #gluster
18:49 JoeJulian acki: Any errors in the client log?
18:49 pampan Hi guys. Running 3.5.7 on a replicated two nodes setup. I have files that are on .glusterfs/indices/xattrop/, but aren't being healed despite being exactly the same on both bricks. Am I missing something? If I confirm that those the files are the same on both volumes, can I just delete the files on the .glusterfs/indices/xattrop dir?
18:49 JoeJulian ~stripe | Gambit15
18:49 glusterbot Gambit15: Please see http://joejulian.name/blog/should-i-use-stripe-on-glusterfs/ about stripe volumes.
18:49 JoeJulian Gambit15: plus, stripe is being deprecated
18:49 Gambit15 JoeJulian, yip, just came across that myself
18:49 JoeJulian pampan: I would.
18:50 Gambit15 Ah? I won't bother in that case
18:50 Gambit15 My main thought was that stripe would allow VHDs/file larger than an individual brick, but distributed would probably give better I/O for single clients
18:52 pampan JoeJulian: Thanks!
18:52 acki JoeJulian: no, only information lines
18:53 acki JoeJulian: glusterfs: unknown negative timeout HIGH Mount failed. am i missing something?
18:54 gnulnx joined #gluster
19:00 acki JoeJulian: actual stats: 0.5kB file - 1.6MB/s, 1kB file - 2.6MB/s, 2kB file - 6.0MB/s, 5kB file - 17.2MB/s, 10kB file - 26.0MB/s, 20kB file - 35.3MB/s, 50kB file - 30.0MB/s | tested all with 4096 count and oflag direct
19:02 JoeJulian Gambit15: If you need larger-than-a-brick, I think I should suggest the shard translator. Unfortunately there's no documentation in glusterdocs, and no maintainer listed to harass for those docs. So now I'm not sure if it's supposed to be production or not. I thought it was.
19:02 JoeJulian acki: "HIGH" as in some high amount of time. Use case and SLA specific.
19:02 acki ah sry :D
19:05 Gambit15 JoeJulian, ah yes, I forgot about sharding. No worries at the moment at least, just planning ahead.   WRT dispersed-distributed, I understand that reads are improved, by writes take a hit (in comparison to dist-rep2). Correct?
19:06 JoeJulian I wouldn't expect reads to be improved, but yes, writes take a hit.
19:08 gnulnx I've got about 1.5 million files that I need to update the ownership (user and group) on.  The files are mounted via gluster fuse.  Any better options than a recusrive chown?
19:09 JoeJulian gnulnx: split it up among multiple clients?
19:09 gnulnx Yeah, that's what I"m thinking
19:10 bkolden joined #gluster
19:10 JoeJulian Heck, even splitting it up among multiple instances seems like it would be faster than one single thread.
19:11 gnulnx Well I was giong to run it parallel with either gnu parallel or xargs
19:11 gnulnx Guess I had my question answered already. :)
19:11 JoeJulian But I still would split the job among multiple clients.
19:11 JoeJulian Up to 16 would be more efficient.
19:14 julim joined #gluster
19:28 unclemarc joined #gluster
19:33 hackman joined #gluster
19:41 hagarth joined #gluster
19:50 Jacob843 joined #gluster
19:55 Gambit15 JoeJulian, I understand it's not possible to add an arbiter to a rep2 volume after the volume was created? Is it possible to arbiter with replace-brick?
19:56 derjohn_mobi joined #gluster
19:57 JoeJulian I've never tried it. Try it on a test volume.
20:06 nathwill joined #gluster
20:15 johnmilton joined #gluster
20:16 post-factum Gambit15: it is possible
20:16 post-factum Gambit15: but with 3.8
20:17 post-factum Gambit15: I'm the one who requested this feature
20:17 pampan What's the best approach for restarting the gluster self healing daemon?
20:17 post-factum pampan: pkill+restart glusterd, I think
20:18 Gambit15 3.7 is the current stable, IIRC?
20:18 pampan I don't want to restart glusterd, not in prod... only glustershd
20:18 post-factum Gambit15: 3.8.2
20:18 Gambit15 Oh...hmmm
20:18 pampan if I kill glustershd, will glusterd put it back online?
20:18 post-factum pampan: restarting glusterd is ok, it won't kill your clients
20:18 post-factum pampan: yep
20:19 pampan post-factum: thanks
20:19 post-factum pampan: but kill it with TERM not KILL
20:19 pampan I will, thanks for the advise
20:19 JoeJulian Or
20:19 JoeJulian "gluster volume start $volname force" will restart shd.
20:19 Gambit15 Ah, it's because I'm using the ovirt stable rep. Think I'll keep it as is in which case.
20:20 pampan JoeJulian: despite shd being running, that command will restart it?
20:20 Gambit15 post-factum, btw, are you referring to adding arbiters, or moving them?
20:20 JoeJulian Yeah, I was surprised too.
20:20 pampan lol
20:20 pampan k, thanks
20:20 post-factum Gambit15: moving?
20:20 post-factum Gambit15: wth?
20:20 johnmilton joined #gluster
20:20 post-factum Gambit15: I refer to adding arbiter brick
20:21 post-factum Gambit15: it is possible with 3.8
20:21 Gambit15 Cool
20:21 Gambit15 Moving as in replace-brick to chuck the arbiter on a different server
20:22 post-factum Gambit15: dunno, but I guess it should be possible once arbiter is already present
20:22 Gambit15 Technically, I don't see why it should be different to a normal brick at least...
20:22 post-factum Gambit15: that is what she said
20:22 Gambit15 o0
20:24 Gambit15 Of which, any of you ever had any issues with upgrading glusterd on nodes within a pool? (curiosity's sake)
20:25 post-factum no
20:25 post-factum i test it before upgrading ;)
20:26 JoeJulian I've had a few problems over the years, most of the reasons for those have been fixed.
20:26 Gambit15 nice
20:26 JoeJulian The last reason, upgrading a server before the heals are finished on the replica, that's not fixable.
20:26 JoeJulian currently
20:27 post-factum JoeJulian: Gambit15: http://review.gluster.org/#/c/14126/
20:27 glusterbot Title: Gerrit Code Review (at review.gluster.org)
20:27 Gambit15 post-factum, of course, although there are some services which are *renowned* for being dodgy to upgrade
20:29 JoeJulian not touched since May, huh? Bummer.
20:29 post-factum what? it is merged
20:29 JoeJulian Meh, I keep missing that on this newer gerrit version.
20:30 JoeJulian It used to be right up at the top.
20:30 post-factum disorientation happens
20:31 Gambit15 Ah, it was Ravi's posts I found in the mailing list when looking it up
20:33 JoeJulian That PR only exists in master. It has not been ported to any release version.
20:33 post-factum 3.8 was branched from master after that
20:33 JoeJulian git tag --contains 61c1b2cee973b11897a37d508910012e616033bc is empty
20:35 post-factum JoeJulian: https://github.com/gluster/glusterfs/blob/release-3.8/cli/src/cli-cmd-volume.c#L2590
20:35 glusterbot Title: glusterfs/cli-cmd-volume.c at release-3.8 · gluster/glusterfs · GitHub (at github.com)
20:36 post-factum JoeJulian: it is *there*
20:36 JoeJulian Ah, they're separate PRs.
20:37 post-factum i guess one would like to grep by change-id
20:37 JoeJulian and different bug ids
20:38 JoeJulian different change-ids too.
20:39 JoeJulian So basically there's nothing consistent but the change text, and that's only true if the submitter leaves them consistent.
20:39 post-factum JoeJulian: different change-id is a no-no and should be considered a bug
20:39 rastar joined #gluster
20:40 post-factum JoeJulian: poornima recently rejected my suggestion to merge two commits into one because "devs need tracking changes across branches"
20:40 post-factum JoeJulian: ...by consistent change-id
20:40 Gambit15 Given sharding was only added in 3.7, is it still considered experimental? Or stable as it's in the stable branch?
20:41 post-factum Gambit15: should be stable unless you face a bug
20:41 Gambit15 lol
20:42 JoeJulian Ask Krutika Dhananjay <kdhananj@redhat.com>. They seem to have the most commits to that xlator so it seems like they should be the maintainer.
20:43 Gambit15 Intrigued in the supposed improvement it offers in healing
20:51 kkeithley shard is not experimental AFAIK; also AFAIK stripe is deprecated as of 3.8, use shard instead.
20:52 kkeithley weasel words. Someone else may come after me and say something different.
20:52 Gambit15 JJ said the same
20:53 Gambit15 TBH, I'd already read that too, I'd just forgotten
20:59 Philambdo joined #gluster
21:01 Gambit15 Um, on the bugzilla tracker, does "Fixed In Version:" reflect how far back a fix has been backported to, or just where it was merged?
21:01 Gambit15 https://bugzilla.redhat.com/show_bug.cgi?id=1324004
21:01 glusterbot Bug 1324004: unspecified, unspecified, ---, ravishankar, CLOSED CURRENTRELEASE, arbiter volume write performance is bad.
21:02 Gambit15 It says 3.8 there, but I saw in the mailing list that it was to be backported to 3.7
21:03 derjohn_mob joined #gluster
21:14 JoeJulian That bug is just a clone bug 1324809 against the master branch. The original is for 3.7
21:14 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1324809 unspecified, unspecified, ---, ravishankar, CLOSED CURRENTRELEASE, arbiter volume write performance is bad.
21:15 ankitraj joined #gluster
21:17 * Gambit15 celebrates
21:18 Gambit15 Cheers
21:28 Gambit15 JoeJulian, I can't find any explanation for how the arbiter is defined in the syntax. Any ideas? I can only find examples of the command in configuring 3 nodes with "replica 3 arbiter 1". In my case, I'll be using more nodes but only need 1 arbiter
21:28 JoeJulian how many replica?
21:29 JoeJulian "replica 3 arbiter 1" means you have a replica 3 but one of those 3 is only an arbiter.
21:29 Gambit15 In this case, 4 nodes with distributed replica 2. The arbiter should sit on one of the other pair
21:30 JoeJulian So it'll still be a replica 3 arbiter 1
21:31 Gambit15 But which one in the other pair will be the arbiter?
21:31 JoeJulian That command doesn't affect distribute, it's only adjusting replicate.
21:31 Gambit15 In my case, I'll be defining 4 nodes...
21:31 JoeJulian Whichever one you specify.
21:32 JoeJulian Put your current volume creation string in a gist.
21:32 Gambit15 Could you give me an example of a 4 node dist-rep*2?
21:32 JoeJulian Actually, I changed my mind. Paste it in channel.
21:32 JoeJulian I want this preserved for posterity.
21:33 Gambit15 gluster volume create VOL replica 3 arbiter 1 v0:brick v1:brick v2:brick v3:brick
21:34 JoeJulian So your *current* volume creation is actually: gluster volume create VOL replica 2 v0:brick v1:brick v2:brick v3:brick
21:34 Gambit15 Yeah. So that'd be v0 & v1 mirrored, distributed to v2 & v3
21:34 siel joined #gluster
21:34 Gambit15 How'd I make v2 the arbiter in this case?
21:34 JoeJulian To create the same structure but add an arbiter: gluster volume create VOL replica 3 arbiter 1 v0:brick v1:brick v2:arbiter_brick v2:brick v3:brick v0:arbiterbrick
21:35 JoeJulian This is where you probably slap your forehead and go "Of course!!!"
21:35 Gambit15 Heh - doesn't seem to be well documented, so wasn't sure
21:36 Gambit15 Cheers!
21:37 Gambit15 The docs explain how the arbiter works & then just give an example on how to setup a 3 node volume, but don't actually explain the syntax
21:37 Gambit15 ...or I'm being a bit dense
21:37 JoeJulian Probably not dense. Most documentation was written by developers. By the time they write it it all seems obvious.
21:38 Gambit15 Yeah, I know the feeling
21:38 cloph arbiter has info "file should have this or that attribute, is modified at x, has that checksum", so even when the data itself is not there, you know which of the bricks has the "good" file
21:40 Gambit15 Yeah, the docs explain well enough what it *does*, but skip over breaking down the syntax for cases outside of their example
21:53 om joined #gluster
22:10 JoeJulian Gambit15: Is this any better? https://github.com/joejulian/glusterdocs/blob/arbiter_improvement/Administrator%20Guide/arbiter-volumes-and-quorum.md
22:10 glusterbot Title: glusterdocs/arbiter-volumes-and-quorum.md at arbiter_improvement · joejulian/glusterdocs · GitHub (at github.com)
22:10 JoeJulian (I also moved usage to the top, since that's what an administrator is most interested in)
22:11 hagarth joined #gluster
22:11 JoeJulian There's a lot of redundant text in that document, too. I should probably run through it and eliminate that.
22:17 Philambdo joined #gluster
22:19 julim joined #gluster
22:31 chirino_m joined #gluster
22:37 chirino joined #gluster
22:39 john51 joined #gluster
22:59 Gambit15 JoeJulian, your explanation, alongside the volume info output, makes it much clearer!
23:01 JoeJulian cool
23:09 Gambit15 Part of my confusion with the readthedocs description was that they don't explain whether the arbiter is the last defined brick in the replica set, the last defined brick(s), etc, etc. Perhaps if you're already well accustomed to managing bricks & volumes, but it's something important enough that it should be part of the consideration for people's *first* volumes.
23:09 Gambit15 (me in this case...)
23:11 Gambit15 When putting together complex infrastructure, I tend to spend a *lot* of time reading all of the available docs, recommendations & use cases, before I enter even the first command
23:12 plarsen joined #gluster
23:35 JoeJulian Gambit15: Can I get a +1? https://github.com/gluster/glusterdocs/pull/147
23:35 glusterbot Title: Arbiter documentation improvement by joejulian · Pull Request #147 · gluster/glusterdocs · GitHub (at github.com)
23:36 JoeJulian Hey! The Atom plugin put the gluster commands back in the TOC after I removed them. Argh.
23:37 JoeJulian Ok, *now* can I get a +1.
23:39 Gambit15 Sure...I'm logged in, where's the + button?
23:40 JoeJulian Just add a comment.
23:40 JoeJulian +1 or :+1:
23:40 Gambit15 Ah, "reaction". Done!
23:42 JoeJulian thanks
23:52 om joined #gluster
23:59 Philambdo joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary