Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-05-27

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 haomaiwang joined #gluster
00:02 unforgiven512 joined #gluster
00:03 unforgiven512 joined #gluster
00:04 unforgiven512 joined #gluster
00:04 unforgiven512 joined #gluster
00:05 unforgiven512 joined #gluster
00:35 davpenguin joined #gluster
00:35 davpenguin hi all
00:35 davpenguin anyone know to change the op-version?
00:36 davpenguin this change i should to do on the file /var/lib/glusterd/glusterd.info
00:36 davpenguin or set parameter with the gluster command on the volume
00:36 davpenguin ?
00:37 davpenguin gluster volume set all cluster.op-version
01:17 amye joined #gluster
01:32 lossanarch joined #gluster
01:33 lossanarch hi all, can anyone tell me why I'm getting unrecognized word errors for gluster volume geo-replication?
01:35 haomaiwang joined #gluster
01:37 lossanarch never mind, i just found it in irclog.perlgeek.de/gluster/2014-01-29/text
01:37 lossanarch there's a separate package.
01:37 lossanarch thanks
01:37 lossanarch left #gluster
01:41 EinstCrazy joined #gluster
01:42 Lee1092 joined #gluster
01:48 ilbot3 joined #gluster
01:48 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:01 haomaiwang joined #gluster
02:04 newdave joined #gluster
02:23 newdave joined #gluster
02:26 daMaestro joined #gluster
02:53 ahino joined #gluster
02:57 sakshi joined #gluster
03:01 haomaiwang joined #gluster
03:05 JesperA joined #gluster
03:33 fortpedro joined #gluster
03:34 Gnomethrower joined #gluster
03:41 arcolife joined #gluster
03:46 atinm joined #gluster
03:49 RameshN joined #gluster
03:53 muneerse2 joined #gluster
04:01 haomaiwang joined #gluster
04:08 hackman joined #gluster
04:15 Siavash___ joined #gluster
04:22 nbalacha joined #gluster
04:22 nehar joined #gluster
04:30 nbalacha joined #gluster
04:33 Gnomethrower joined #gluster
04:34 Apeksha joined #gluster
04:39 nehar joined #gluster
04:42 shubhendu joined #gluster
04:43 kdhananjay joined #gluster
04:52 nishanth joined #gluster
04:54 ndarshan joined #gluster
04:57 ppai joined #gluster
05:01 haomaiwang joined #gluster
05:03 Siavash___ joined #gluster
05:06 daMaestro joined #gluster
05:07 JoeJulian @op-version
05:07 glusterbot JoeJulian: The operating version represents the RPC and translator capabilities required to accommodate the volume settings ( http://gluster.org/community/docume​ntation/index.php/OperatingVersions ). To allow older version clients to connect to newer servers, reset any volume options that require the newer op-version.
05:08 Bhaskarakiran joined #gluster
05:10 aravindavk joined #gluster
05:11 sakshi joined #gluster
05:11 JoeJulian @forget op-version
05:11 glusterbot JoeJulian: The operation succeeded.
05:13 JoeJulian @learn op-version as The operating version represents the RPC and translator capabilities required to accomidate volume settings.
05:13 glusterbot JoeJulian: The operation succeeded.
05:13 JoeJulian @learn op-version as If you cannot connect your older client to your newer server, reset any volume options that require the newer op-version.
05:13 glusterbot JoeJulian: The operation succeeded.
05:14 JoeJulian @learn op-version as To manually set the op-version of your cluster: gluster volume set all cluster.op-version $version
05:14 glusterbot JoeJulian: The operation succeeded.
05:14 JoeJulian @op-version
05:14 glusterbot JoeJulian: (#1) The operating version represents the RPC and translator capabilities required to accomidate volume settings., or (#2) If you cannot connect your older client to your newer server, reset any volume options that require the newer op-version., or (#3) To manually set the op-version of your cluster: gluster volume set all cluster.op-version Supybot 0.83.4.1+limnoria 2015.05.23
05:14 JoeJulian hehe
05:15 daMaestro JoeJulian, ltns (my absence)
05:15 daMaestro supybot ... glad we have one gluster focused now
05:15 JoeJulian @factoids change op-version 3 's/op-version .*/\$version'
05:15 glusterbot JoeJulian: Error: "'s/op-version .*/\\Supybot 0.83.4.1+limnoria 2015.05.23'" is not a valid regular expression.
05:15 JoeJulian @factoids change op-version 3 's/op-version .*/\$version/'
05:15 glusterbot JoeJulian: Error: "'s/op-version .*/\\Supybot 0.83.4.1+limnoria 2015.05.23/'" is not a valid regular expression.
05:16 JoeJulian gah
05:16 daMaestro zodbot's been lonely
05:16 JoeJulian Hiya daMaestro! How's it been going?
05:16 daMaestro good, how are things with you?
05:16 JoeJulian @factoids change op-version 3 's/op-version .*/$$version/'
05:16 glusterbot JoeJulian: Error: "'s/op-version .*/$version/'" is not a valid regular expression.
05:16 daMaestro been seeing a bunch of awesome gluster progress in the recent past
05:17 karthik___ joined #gluster
05:17 JoeJulian @factoids change op-version 3 "s/op-version .*/$$version/"
05:17 glusterbot JoeJulian: The operation succeeded.
05:17 JoeJulian Yeah, those guys have been busy.
05:17 JoeJulian And the users are submitting really good bug reports.
05:17 JoeJulian And it's 10 years old!
05:18 daMaestro it looks like we've finally figured out how the translators should be stacked correctly
05:18 harish_ joined #gluster
05:18 JoeJulian Or at least have a consistent definition. Whether it's optimal or not... I'm not sure that was ever performance tested. It was just decided.
05:18 daMaestro i'm also really glad that kkeithley has been rocking the package updates ;-)
05:19 daMaestro ah, .... yeah performance is too close to the workload so unless we build an adaptive system... it's gonna be how you configure it :-D
05:19 JoeJulian There's been a lot of volunteers, mostly in Bangalore, that have been managing the packages each release. Lots of good experience growing from that.
05:20 ramky joined #gluster
05:20 daMaestro i know there has been a huge push for it, but has the usecase for virt layers stabilized?
05:20 JoeJulian I hate to say it, but I only run Fedora on my laptop any more. I'm mostly running Arch.
05:21 JoeJulian virt layers?
05:21 daMaestro ah distro doesn't matter. contribution the community matters more
05:21 daMaestro libvirt storage backend based on gluster
05:21 daMaestro i've seen it work for some workloads, but not all
05:21 JoeJulian Oh, yeah. That's old news these days.
05:22 daMaestro being it's been fixed?
05:22 JoeJulian Only one I've seen a problem with was heat related with lots of attaches and detaches.
05:22 daMaestro very good to hear, maybe i need to look at it again
05:23 JoeJulian I moved to ceph then back again, so yeah. Maybe.
05:23 daMaestro ceph caught my attention after i stopped maintaining the packages, but that was only because a good chunk of the openstack community was using it
05:23 daMaestro ha, right...
05:23 gowtham_ joined #gluster
05:23 daMaestro block storage was not great back in the 2.x, 3.0... days
05:24 JoeJulian I have about 400Tb sitting in a ceph cluster that I can't get to right now. Lots of unhappy people about that one.
05:24 daMaestro oh, i'm really sorry to hear that
05:24 JoeJulian At least if it was gluster, I could fix it manually if I had to.
05:24 JoeJulian I'
05:25 JoeJulian I'm not blaming ceph though. The hardware all flaked on us.
05:25 JoeJulian And the vendors are finger pointing.
05:25 daMaestro when i had a major gluster failure (the reason i took over the maintenance of the packages) we had a 300TB cluster that one of the distribute volumes using xfs from EL5 (out of band kmods) would not repair and everything failed in that inode path
05:26 daMaestro it *was* an underlying xfs filesystem issue, but it was that gluster had written *something* funky to the xattrs
05:26 daMaestro and it broken the entire path to resolve a directory structure
05:27 daMaestro luckily, as you mention, the data was on disk so we were able to get the data and remove the bricks and then write back the data
05:27 daMaestro but this was back in... i think like 3.0.8 transition period
05:27 JoeJulian 180 disks across 6 servers where, if the load gets to high, the drives start dropping off. I cannot fix it unless someone physically disconnects the Wiwynn Knox2U from power and plugs it back in. Wiwynn's trying to blame the compute.
05:28 daMaestro ah, 3.0.7 looking at the changelog
05:28 JoeJulian But there's no way it's the compute. If I power cycle the compute and it was the compute, the problem should go away - at least for a while. But no. The disks and the sas expander are gone on boot.
05:29 daMaestro ah, yeah that is a really hard thing to recover from
05:30 daMaestro we hit peta scale and i just could not sleep at night running it via gluster
05:30 JoeJulian And everyone's trying to run it all through the vendor instead of talking to me. I'm one of the decision makers for what we install in 200+ data centers and they're dicking me around.
05:30 daMaestro so we switched over to isilon, pre emc acquisition
05:30 daMaestro now we have multiple PB and i don't worry about much other than running out of space
05:31 JoeJulian Meh, I know people that work there. I'm not a huge fan.
05:31 ashiq joined #gluster
05:31 daMaestro pre-emc is an important distinction. it's cool tech, but i've seen some not-so-amazing changes since then
05:31 JoeJulian I interviewed there and they presented me with a problem to solve. I did and they told me I was wrong. I wasn't.
05:32 JoeJulian And yes, that was pre-emc.
05:32 daMaestro we (dayjob) looked under the hood at the hardware back then
05:32 JoeJulian They had like 4 really knowledgeable guys and they lived attached to phones. I wouldn't work there.
05:33 daMaestro you could put more paired ram and another paired CPU in it and ... you could take their "NL" performance to another level
05:33 daMaestro all based on the BSD kernel
05:33 JoeJulian bsd fork.
05:33 daMaestro one of the things i didn't really like was likewise... it failed a lot
05:33 daMaestro yeah, fork. to be clear
05:34 daMaestro compiling kernel modules for stuff like ataoe was a challenge
05:34 daMaestro using an open iscsi mod was also a challenge
05:35 daMaestro now at least they have a mature iscsi stack (okay i don't use it, so zero trust of it)
05:35 daMaestro we are 100% NFS from those things
05:35 harish_ joined #gluster
05:35 daMaestro and they do well for our bulk storage, served via http to customers via web heads
05:36 daMaestro they did some good stuff making nfsv4 do HA (with nasty, nasty hacks)
05:36 JoeJulian heh, yeah
05:37 daMaestro if your network was not okay with MAC portability via rapid convergence or other protocols, you'd still have packets in flight that fail
05:37 JoeJulian I'm also liking ganesha-nfs for nfsv4 lately.
05:37 daMaestro gonna look at that now, have not heard of it
05:38 JoeJulian It's all userspace.
05:38 daMaestro hmm, yeah just read that. pretty awesome
05:38 daMaestro that was something i thought we had on-track with the fuse layer... but it turned out to be very unstable
05:38 JoeJulian Welllll, I'm off to bed.
05:38 daMaestro nfs4.1 needs more work
05:38 JoeJulian davpenguin: ,,(op-version)
05:38 glusterbot davpenguin: (#1) The operating version represents the RPC and translator capabilities required to accomidate volume settings., or (#2) If you cannot connect your older client to your newer server, reset any volume options that require the newer op-version., or (#3) To manually set the Supybot 0.83.4.1+limnoria 2015.05.23
05:38 daMaestro JoeJulian, well g'night
05:39 JoeJulian Goodnight. See ya tomorrow. ;)
05:39 prasanth joined #gluster
05:39 davpenguin finally i found the option to configure a new op-version
05:39 JoeJulian Son of a... It kept the expanded string
05:39 davpenguin thanks
05:39 davpenguin i know taht it do
05:40 JoeJulian @factoids change op-version 3 "s/op-version .*/$$$$version/"
05:40 glusterbot JoeJulian: The operation succeeded.
05:40 JoeJulian @op-version
05:40 glusterbot JoeJulian: (#1) The operating version represents the RPC and translator capabilities required to accomidate volume settings., or (#2) If you cannot connect your older client to your newer server, reset any volume options that require the newer op-version., or (#3) To manually set the $version
05:40 JoeJulian <sigh>
05:40 JoeJulian @forget op-version 3
05:40 glusterbot JoeJulian: The operation succeeded.
05:41 davpenguin gluster volume set all cluster.op-version 30710
05:41 JoeJulian @learn op-version as To manually set the op-version of your cluster: gluster volume set all cluster.op-version $$$$version
05:41 glusterbot JoeJulian: The operation succeeded.
05:41 davpenguin for me it's well
05:41 JoeJulian @op-versoin
05:41 glusterbot JoeJulian: I do not know about 'op-versoin', but I do know about these similar topics: 'op-version'
05:41 JoeJulian @op-version
05:41 glusterbot JoeJulian: (#1) The operating version represents the RPC and translator capabilities required to accomidate volume settings., or (#2) If you cannot connect your older client to your newer server, reset any volume options that require the newer op-version., or (#3) To manually set the op-version of your cluster: gluster volume set all cluster.op-version $$version
05:41 davpenguin cat /var/lib/glusterd/glusterd.info
05:41 davpenguin operating-version=30710
05:41 davpenguin thanks
05:42 JoeJulian omg... I give up for the night.
05:42 JoeJulian @forget op-version 3
05:42 glusterbot JoeJulian: The operation succeeded.
05:42 JoeJulian @learn op-version as To manually set the op-version of your cluster: gluster volume set all cluster.op-version $$version
05:42 glusterbot JoeJulian: The operation succeeded.
05:42 JoeJulian @op-version
05:42 glusterbot JoeJulian: (#1) The operating version represents the RPC and translator capabilities required to accomidate volume settings., or (#2) If you cannot connect your older client to your newer server, reset any volume options that require the newer op-version., or (#3) To manually set the op-version of your cluster: gluster volume set all cluster.op-version $version
05:42 JoeJulian out
05:43 davpenguin ok good night
05:44 ramky_ joined #gluster
05:50 Manikandan joined #gluster
05:55 hackman joined #gluster
05:56 overclk joined #gluster
05:56 tom[] joined #gluster
06:01 hgowtham joined #gluster
06:01 haomaiwang joined #gluster
06:09 jtux joined #gluster
06:17 atalur joined #gluster
06:18 Wizek joined #gluster
06:21 ramky joined #gluster
06:31 [Enrico] joined #gluster
06:34 MikeLupe joined #gluster
06:36 hchiramm joined #gluster
06:37 pkoro joined #gluster
06:39 karnan joined #gluster
06:39 arcolife joined #gluster
06:43 amye joined #gluster
06:43 nbalacha joined #gluster
06:48 ramky joined #gluster
06:51 Wizek joined #gluster
06:55 [Enrico] joined #gluster
07:01 haomaiwang joined #gluster
07:02 ramky_ joined #gluster
07:02 deniszh joined #gluster
07:03 pur__ joined #gluster
07:06 ramky__ joined #gluster
07:08 tessier_ [2016-05-27 07:08:40.175079] E [rpcsvc.c:638:rpcsvc_handle_rpc_call] 0-rpc-service: Request received from non-privileged port. Failing request
07:09 hchiramm joined #gluster
07:09 tessier_ Why are my brick logs constantly streaming with this message?
07:16 Mmike joined #gluster
07:16 ramky_ joined #gluster
07:17 mowntan joined #gluster
07:18 kovshenin joined #gluster
07:26 tdasilva joined #gluster
07:38 atinm joined #gluster
07:39 ahino joined #gluster
07:42 ctria joined #gluster
07:43 anil joined #gluster
07:48 jri joined #gluster
07:50 Gnomethrower joined #gluster
07:53 Gnomethrower joined #gluster
07:54 atalur joined #gluster
08:01 haomaiwang joined #gluster
08:14 ahino joined #gluster
08:16 Slashman joined #gluster
08:21 ivan_rossi joined #gluster
08:23 ivan_rossi left #gluster
08:34 karthik___ joined #gluster
08:39 jiffin joined #gluster
08:49 karnan joined #gluster
08:50 Bhaskarakiran joined #gluster
08:56 jiffin joined #gluster
08:59 kovshenin joined #gluster
08:59 paul98 joined #gluster
09:00 paul98 am i right in saying, I need to have the volume mounted on the gluster fs
09:01 haomaiwang joined #gluster
09:01 atalur joined #gluster
09:01 paul98 or can i just do "gluster volume data server1:/path/to/folder/
09:03 gem joined #gluster
09:03 jiffin paul98: sorry i didn't get the context
09:04 jiffin do u want to create a glusterfs volume and mount in a client machine?
09:04 paul98 sorry, so i created a lvm, do i need ot have that mount as a local share on the server
09:04 paul98 or can i just create gluster fs volume
09:04 jiffin u need to mount it as share
09:04 paul98 ok
09:05 jiffin and create volume using
09:05 jiffin gluster v create <volname> dataserver1/<mnt path>/directory
09:05 paul98 ta
09:06 paul98 tohught as much
09:14 paul98 jiffin: mounting the mount with in /etc/fstab i've used /dev/vol_grp/data/storage/dataxfs defaults1 2
09:16 paul98 is that corect
09:16 paul98 as i get wrong fs type
09:19 jiffin i am getting the same
09:19 jiffin paul98: /dev/mapper/vg1-thinp1 on /mnt/bricks/b1 type xfs
09:19 paul98 being a lvm is it not xfs
09:19 paul98 it's bit new to me lol
09:20 jiffin paul98: i am not sure
09:24 karnan joined #gluster
09:26 XpineX joined #gluster
09:27 poornimag joined #gluster
09:29 paul98 ah i hadn't formatted the new lvm's lol
09:32 post-factum ndevos:
09:32 post-factum ndevos: https://build.gluster.org/job/rackspace-r​egression-2GB-triggered/21179/consoleFull
09:32 post-factum ndevos: could you explain me that?
09:32 post-factum ndevos: loop: can't delete device /dev/loop9: Device or resource busy
09:34 ndevos post-factum: sounds like a problem on the jenkins slave :-/
09:35 ndevos post-factum: maybe previous regression tests failed in a similar way? https://build.gluster.org/computer​/slave26.cloud.gluster.org/builds
09:35 atalur joined #gluster
09:36 ndevos post-factum: oh, no those failed du to different things
09:36 post-factum ndevos: my bad?
09:37 ndevos post-factum: I dont think so, I would not know why
09:37 post-factum ndevos: who should i poke to get some help ;)?
09:41 paul98 jiffin: do i need to specify what type of replication i need when i create it
09:41 ndevos post-factum: I do not think the loop-device error caused a problem for your test
09:41 ndevos post-factum: the "not ok 17 , LINENUM:39" suggests that the umount failed, which seems correct, because mounting was not expected to work
09:42 post-factum ndevos: ah, im idiot
09:42 post-factum ndevos: the mount will fail in case of invalid arg, so no need to invoke umount
09:43 ndevos post-factum: no need to unmount, just drop those ones, if something fails, "cleanup" will unmount any stale mountpoints
09:43 jiffin paul98: yes
09:43 jiffin if  u can want replication 2, then mention
09:44 jiffin gluster v create <> replica 2 <server1:/> <server2> ...
10:00 deniszh joined #gluster
10:01 haomaiwang joined #gluster
10:11 paul98 what at the othervoptions ?
10:20 paul98 nvm sorted :D
10:22 Gnomethrower joined #gluster
10:24 jiffin paul98: http://gluster.readthedocs.io/en/latest/Adm​inistrator%20Guide/Setting%20Up%20Volumes/
10:25 glusterbot Title: Setting Up Volumes - Gluster Docs (at gluster.readthedocs.io)
10:25 jiffin paul98: striped is not maintained any more
10:25 jiffin paul98: also we are started to support shard instead of stripe
10:35 karnan joined #gluster
10:46 muneerse joined #gluster
11:01 kdhananjay Ulrar: ping
11:01 glusterbot kdhananjay: Please don't naked ping. http://blogs.gnome.org/mark​mc/2014/02/20/naked-pings/
11:01 Ulrar kdhananjay: pong
11:01 haomaiwang joined #gluster
11:02 kdhananjay Ulrar: regarding your vm pause issues, did you see any 'possible split-brain' errors in the logs when you saw the vm pause?
11:03 kdhananjay Ulrar: i saw your latest mail said you're able to recreate the pause even without ever doing addition/removal of bricks.
11:03 kdhananjay Ulrar: i brought up my setup too (finally) with 3.7.11 running on a three-node cluster with 1x3 volume.
11:04 kdhananjay Ulrar: while performing io, i hit vm pause issues.
11:04 kdhananjay Ulrar: turns out these are all mostly root-caused and fixed by now by pranithk.
11:04 kdhananjay Ulrar: should be available in 3.7.12
11:04 Ulrar kdhananjay: I wasn't clear I think, I have VM pauses with 3.7.6 during heals, and indeed there is possible split-brain in the heal infos. With 3.7.11 I don't have that problem during heals, but the VM is showing I/O errors as seen in the screen I posted even without heals
11:05 kdhananjay Ulrar: i was referring to the 3.7.11 problems.
11:05 Ulrar kdhananjay: For 3.7.11 my only problem are those I/O errors then
11:05 Ulrar The VM ends up going read only after a few hours
11:06 kdhananjay Ulrar: i hit that too. the one where `heal info` doesn't report any split-brain but logs report 'possible split-brain with input/output error' correct?
11:06 Ulrar kdhananjay: Mh I don't remember seeing that
11:06 Ulrar I don't actually play around with healing anymore, I just setup the cluster, install the VM and see it get corrupted
11:07 Ulrar Looks like it works fine as long as I don't reboot it, but after a reboot it starts having I/O errors
11:07 Ulrar Nothing visible from the glusterFS side, as far as I can tell
11:07 Ulrar Just that suspicious inode still active when the VM powers off
11:08 Ulrar kdhananjay: Did you have a chance to go over the libgfapi logs I sent the other day by any chance ?
11:08 gvandeweyer hi, I'm having a split-brain situation, which doesn't seem to reflect the docs.  Normally, I should have "trusted.afr.glustervolume-client-X for the split-brain entry whith X present for two replicated bricks. In my case, the split-brain is for a directory, one brick holds two entries, while the replicate brick only holds one.
11:09 kdhananjay1 joined #gluster
11:09 gvandeweyer Can I assume the one with a single entry (only local subvolume idx), is the faulty one, delete the folder on the physical brick and the corresponding .glusterfs/<gfid> path ?
11:09 hi11111 joined #gluster
11:09 gvandeweyer and then trigger a heal?
11:10 kdhananjay1 Ulrar: oh okay. then was it the one where you saw 'quorum not met' even though quorum number of bricks were up?
11:11 Ulrar Mh I don't think that was me no, I don't recall that
11:11 kdhananjay1 Ulrar: oops. then could you share the steps with me so i can try it out?
11:12 kdhananjay1 Ulrar: until now we were debugging lindsay's issues in-house.
11:12 Ulrar kdhananjay1: Sure
11:12 kdhananjay1 Ulrar: now that that's mostly done (hopefully), we can try out the steps that got you into a problem
11:12 harish_ joined #gluster
11:13 Ulrar kdhananjay1: I have a 3.7.11 volume, you can find the config in my corruption thread. It's a replica 3, and on this I create a debian VM on proxmox with virtio disk, directsync cache (that setting doesn't seem to matter). On that VM I use the disk a lot, and it works fine
11:13 Ulrar If i reboot it or live migrate it, it starts showing I/O errors as seen in the screen I sent to the ML
11:13 Ulrar I tried with libgfapi and with NFS, same thing
11:13 johnmilton joined #gluster
11:14 Ulrar But I tried with the same VM on local storage, not gluster, and it works fine
11:14 kdhananjay Ulrar: what io were you performing inside the vm when you rebooted it?
11:14 Ulrar kdhananjay: I do database imports, but I don't reboot while doing that, I wait for it to finish
11:15 Ulrar I have other VMs on the same cluster that aren't showing any I/O problems as long as I don't use them
11:16 Ulrar So basically I didn't find any way to have VMs work on gluster 3.7.11 yet, whatever I do I get those I/O errors
11:17 Ulrar Lindsay mentionned she was using writethrough for her VMs, but I tried that and still the same problem
11:18 kdhananjay Ulrar: ok, so did you try FUSE as well by any chance?
11:19 Manikandan joined #gluster
11:20 Ulrar kdhananjay: Oh, you mean adding it as a directory storage in proxmox on put the VM on there ? Didn't think of that
11:20 Ulrar I will !
11:20 kdhananjay joined #gluster
11:21 Ulrar kdhananjay: Oh, you mean adding it as a directory storage in proxmox on put the VM on there ? Didn't think of that
11:21 Ulrar I will !
11:22 kdhananjay Ulrar: i meant, mounting your gluster volume using fuse and then trying the same test.
11:22 Ulrar Yeah, but I need to add it to proxmox
11:23 Ulrar but as directory it should work
11:23 kdhananjay Ulrar: this is just to eliminate the mount protocol as the possible cause of the issue.
11:23 Ulrar kdhananjay: btw, always wondered, is it normal it's mentionning glusterFS 3.3 everywhere in the logs and not 3.7.11 ?
11:24 kdhananjay Ulrar: in lindsay's case, he (yes apparently lindsay's not a lady), was seeing problems only with libgfapi, never with FUSE.
11:24 kdhananjay Ulrar: ah, you can safely ignore that. :)
11:24 Ulrar Allright
11:25 Ulrar I will try with fuse then, but I had the problems with nfs too so I'm not so sure
11:25 kdhananjay Ulrar: not too sure about nfs.
11:26 Ulrar well, worth a try anyway
11:27 Ulrar I'll try to do that right now, my client isn't answering for now, I have a bit of time
11:28 kdhananjay Ulrar: sure, just let me know what you find. and if you successfully hit the issue with fuse, then even better, because we can get easy access to the client logs and investigate. they'll be under /var/log/glusterfs/<hyphen​ated-mount-point-path>.log
11:29 Ulrar kdhananjay: I did send the libgfapi logs the other day, but maybe there isn't as much in them
11:30 kdhananjay Ulrar: i saw a message about quorum loss
11:31 kdhananjay Ulrar: nothing other than that
11:31 Ulrar I didn't notice this, might have missed it
11:31 Ulrar I'm speaking about my mail in "VM disks corruption on 3.7.11" on  Wed, 25 May 2016 17:58:47 +0200
11:31 B21956 joined #gluster
11:34 kdhananjay Ulrar: yes, smae thing. the problem with gfapi i figured is that unlike fuse it doesn't log when a fop fails
11:34 kdhananjay Ulrar: with fuse, it logs the path/gfid/error code etc.
11:36 Ulrar kdhananjay: I see. I'm trying it now, I should have an answer quickly
11:36 kdhananjay Ulrar: Cool :)
11:39 paul98 any one setup a iscsi target using the gluster docs
11:39 AppStore If my gluster nodes has a floating IP moving between them (In addition to static IP addresses), would this be a likely cause of problems?
11:41 Ulrar kdhananjay: Oh wow, indeed
11:41 Ulrar [2016-05-27 11:38:44.276984] W [MSGID: 101159] [inode.c:1206:__inode_unlink] 0-inode: be318638-e8a0-4c6d-977d-7a937aa84806/e07​0f40b-d1d7-4cd5-8510-d9359a461127.1568: dentry not found in 034b55a1-2717-4de7-9ccd-037809172bc
11:41 Ulrar I have those a lot
11:41 luizcpg joined #gluster
11:41 Ulrar [2016-05-27 11:40:31.981118] W [fuse-bridge.c:1287:fuse_err_cbk] 0-glusterfs-fuse: 599252: FSYNC() ERR => -1 (Invalid argument)
11:42 kdhananjay Ulrar: any other fuse errors?
11:42 kdhananjay Ulrar: grep -i 'fuse_' <mount-log>
11:43 Ulrar That's the only one from today
11:43 Ulrar That's actually the only error, the rest of the lines from grep are just init stuff
11:43 kdhananjay Ulrar: oh okay.
11:44 kdhananjay Ulrar: could you in any case send or fpaste the log and send the link, so i can take a look?
11:44 Ulrar sure
11:45 Ulrar kdhananjay: The whole logfile ? The mountpoint wasn't really used before, it's just a helper from proxmox
11:46 kdhananjay Ulrar: how big is it? :)
11:46 Ulrar 78K
11:48 Ulrar kdhananjay: https://dpaste.de/h1Jt
11:48 glusterbot Title: dpaste.de: Snippet #365354 (at dpaste.de)
11:48 Ulrar Here is at least the log from my test just now
11:49 Ulrar The mv at the start is just me "moving" the disk in proxmox from storage
11:51 kdhananjay Ulrar: hmm ok.
11:52 kdhananjay Ulrar: anything in the brick logs?
11:53 Ulrar kdhananjay: [2016-05-27 11:45:32.000507] I [dict.c:473:dict_get] (-->/usr/lib/x86_64-linux-gnu/libglus​terfs.so.0(default_getxattr_cbk+0xab) [0x75adce00d39b] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xla​tor/features/marker.so(marker_getxattr_cbk+0xa7) [0x75adc200e8e7] -->/usr/lib/x86_64-linux-gnu/li​bglusterfs.so.0(dict_get+0x93) [0x75adcdffe0f3] ) 0-dict: !this || key=() [Invalid argument]
11:53 glusterbot Ulrar: ('s karma is now -136
11:53 kdhananjay Ulrar: that should be harmless.
11:53 Ulrar Poor (
11:53 kdhananjay ;)
11:53 Ulrar Nothing else
11:54 kdhananjay strange
11:56 kdhananjay Ulrar: i'll need to read the code a bit.
11:56 Ulrar Sure
11:56 Ulrar Thanks !
11:56 kdhananjay Ulrar: we had fixed similar pause issues with readv, writev etc.
11:57 kdhananjay Ulrar: but that was shard + afr interaction
11:57 kdhananjay Ulrar: this looks different
11:57 Ulrar The volume is sharded
11:57 kdhananjay Ulrar: same error ==> Invalid argument
11:57 kdhananjay Ulrar: yeah, but fsync in shard doesnt do inode link.
11:57 kdhananjay Ulrar: so this is something else
11:57 Ulrar Allright, well thanks for the help anyway :)
11:58 Ulrar I'm always on here, don't hesitate if you need more tests
11:58 kdhananjay Ulrar: i will get back to you on this, just give me some time. i'll need to do a git-grep across the code base to figure out who's sending an EINVAL ;P
11:59 kdhananjay Ulrar: could you try just one more test:
11:59 kdhananjay Ulrar: disable write-behind
12:00 kdhananjay Ulrar: # gluster volume set <VOL> write-behind off
12:00 ehermes joined #gluster
12:00 kdhananjay Ulrar: and then run the same test case
12:01 ramky joined #gluster
12:03 Ulrar Sure, give me a minute
12:06 Ulrar kdhananjay: Same thing
12:06 Ulrar Well no
12:06 Ulrar Got a lot of [2016-05-27 12:06:02.198064] W [fuse-bridge.c:2302:fuse_writev_cbk] 0-glusterfs-fuse: 704145: WRITE => -1 (Invalid argument)
12:07 Ulrar But same thing in the VM, I meant :)
12:07 kdhananjay Ulrar: got it.
12:07 kdhananjay Ulrar: the issue you saw now is due to the afr shard interaction :)
12:07 Ulrar Ha !
12:07 Ulrar So 3.7.12 solves this ?
12:07 kdhananjay Ulrar: yes :(
12:08 Ulrar Do you know when that should be out ?
12:08 kdhananjay hagarth: when are we doing 3.7.12? :)
12:09 kdhananjay hagarth: quite a few issues have been found in vm store use cases, the worst part is most of them don't have workarounds. :(
12:10 Ulrar kdhananjay: Well bonus question, would a previous version have both the fix for heal freez and not that bug by any chance ?
12:10 Ulrar Maybe 3.7.10 or 9 ?
12:10 kdhananjay Ulrar: no. it's been there for a while now. :(
12:11 Ulrar Too bad
12:11 Ulrar Well, we'll wait then
12:11 kdhananjay Ulrar: wait.
12:11 kdhananjay Ulrar: you use debian right?
12:11 Ulrar Yep
12:12 kdhananjay Ulrar: if i give you the patches to be applied (or maybe a tar ball of the source with all these patches applied), would it be possible for you to build your own packages? :)
12:13 Ulrar Oh yeah, should be
12:13 kdhananjay Ulrar: awesome, i will send the tar ball to you and to Lindsay over mail. give me some time.
12:14 Ulrar kdhananjay: Great, thanks a lot ! Might do that only monday though, I have some urgent thing this afternoon, not sure how much time that'll take
12:14 Ulrar But I'll give that a try next week for sure
12:14 kdhananjay Ulrar: sure, np!
12:18 kdhananjay Ulrar: sure.
12:26 hagarth kdhananjay: need to schedule 3.7.12 after a discussion in maintainers list.. will propose a date soon
12:26 kdhananjay hagarth: ok.
12:27 Ulrar Any idea if that'll be in the weeks to come, or more in the months to come ?
12:28 kdhananjay hagarth: ^^
12:29 wnlx joined #gluster
12:30 edong23 joined #gluster
12:42 chirino joined #gluster
12:56 unclemarc joined #gluster
12:59 stormi Hi there. Is there a way from a given node to know other peers when the peers/ directory has been wiped?
13:05 plarsen joined #gluster
13:08 Manikandan joined #gluster
13:15 amye joined #gluster
13:15 gem joined #gluster
13:23 jiffin joined #gluster
13:28 shyam joined #gluster
13:30 haomaiwang joined #gluster
13:37 Siavash___ joined #gluster
13:39 atinm joined #gluster
13:40 TvL2386 joined #gluster
13:40 rwheeler joined #gluster
13:46 MikeLupe joined #gluster
13:47 MikeLupe Hi - I have this given situation: 3 nodes, 1 additional data disk for volume replica 3 arbiter 1
13:47 MikeLupe I would like to add storage without adding 3 additional nodes. So please correct me if I'm wrong: The only way is to add to each node an additional disk, create a new brick on each node and extend the volume.
13:52 post-factum MikeLupe: yep
13:52 MikeLupe post-factum: Thanks!
13:53 post-factum MikeLupe: you could show volume info if you need further assistance
13:53 rafi joined #gluster
13:53 MikeLupe One more question: For the 3rd arbiter node - do I have to also add a full additional disk to the arbiter node, or may I use a folder on the given device?
13:54 MikeLupe well "full" as "another small disk"
13:55 post-factum MikeLupe: no brick including arbiter brick need separate disk. it may be the same disk, but the brick should be located under different folder
13:55 post-factum MikeLupe: arbiter node might have only, saying, 1 SSD with multiple "arbiter" folders on it
13:56 MikeLupe post-factum: Everything makes sense and is clear now. thanks again
13:56 post-factum MikeLupe: np
13:57 MikeLupe post-factum: I hope you noted I haven't misspelled your nick this time :)
13:57 post-factum MikeLupe: oh, that was you!
13:58 MikeLupe post-factum: For sure the only one ever ;)
13:58 post-factum MikeLupe: no need to type my nick each time, use TAB to autocomplete it
13:58 MikeLupe post-factum
13:58 MikeLupe wth
13:58 stormi yeah
13:58 stormi magic
13:58 post-factum :D
13:59 MikeLupe post-factum tab&space not tab&enter ;)
13:59 MikeLupe kind of
13:59 post-factum MikeLupe: it depends on client, i guess. Konversation inserts ": " automatically for me, irssi did the same, afaik
14:00 stormi it does
14:00 MikeLupe post-factum but you get a ping, when no ":"?
14:00 stormi depends on your client, but usually yes
14:00 post-factum MikeLupe: i get it even without :
14:00 MikeLupe oky thx
14:00 post-factum MikeLupe: Konversation pings me if my nick is mentioned in chat
14:00 gowtham_ joined #gluster
14:01 Manikandan joined #gluster
14:01 haomaiwang joined #gluster
14:01 MikeLupe you spoiled an neo-irc-virgin who hasn't used irc a lot for about 10 years
14:02 stormi I guess you could use tab 18 years ago already :)
14:02 post-factum MikeLupe: same here. started using irc for chatting with gluster and ceph devs, and the occasionally opened znc and bitlbee, and attached skype/xmpp to it
14:02 post-factum haha stormi
14:02 stormi Oh god, I've been using IRC for 18 years
14:02 stormi I'm old
14:02 post-factum you are so old
14:02 MikeLupe stormi for sure :)
14:03 MikeLupe don't tell me
14:04 jiffin1 joined #gluster
14:04 MikeLupe post-factum how did you attach xmpp to it? I managed to attach Jitsi to a chatroom - is there any other way?
14:04 skylar joined #gluster
14:05 post-factum MikeLupe: bitlbee does the right job
14:05 MikeLupe ok
14:06 wushudoin joined #gluster
14:07 atinm joined #gluster
14:07 wushudoin joined #gluster
14:14 MikeLupe2 joined #gluster
14:16 swebb joined #gluster
14:19 squizzi joined #gluster
14:20 MrAbaddon joined #gluster
14:40 squizzi joined #gluster
14:46 hagarth joined #gluster
14:50 Wizek joined #gluster
15:00 rafi joined #gluster
15:01 haomaiwang joined #gluster
15:05 kpease joined #gluster
15:05 hagarth joined #gluster
15:07 jiffin joined #gluster
15:14 pembo13 joined #gluster
15:15 pembo13 I would like to setup a bidirectional directory sync between two machines, is GlusterFS a good fit for this?
15:19 EinstCrazy joined #gluster
15:20 rafi1 joined #gluster
15:47 ahino joined #gluster
15:50 sobersabre do people use this channel or is all the action in the glusterfs-users ?
15:51 sobersabre Is there a way to have 0 downtime when one of the bricks goes down, if I'm using gluster client?
15:51 sobersabre I mean no hickups for healing, seamless write/read operation
15:52 sobersabre no 20sec stop for organizing things...
15:57 shubhendu joined #gluster
16:01 haomaiwang joined #gluster
16:10 ndevos pembo13: yeah, gluster is a distributed+replicated filesystem, you can mount it on two or more servers and contents written to it will get replicated to the "bricks
16:10 ndevos "
16:11 pembo13 ok thanks, i've since done a lot of reading, and looks like glusterfs is what i need
16:11 pembo13 i was worried that the file system wasn't directly readable on the server side, or needed to have a client to mount the system locally
16:11 ndevos sobersabre: 0 downtime is not really possible, the timeout (by default 42 seconds) is used to detect if a brick is down/unresponsive
16:12 squizzi joined #gluster
16:12 ndevos pembo13: depending on the volume type, the files will just be normal files on the bricks, but you should always read/write through a glusterfs client (like the fuse mount)
16:13 ndevos pembo13: if you read/write directly to the files on the bricks, gluster will not be aware of those actions and can not do the replication, consistency checks and such
16:13 pembo13 ndevos: so i shouldn't create the brick at /var/www for example? I should create it somewhere else and fuse mount it at /var/www ?
16:13 pembo13 i see
16:14 pembo13 thanks, i hadn't realised that
16:14 ndevos pembo13: yes, correct, I'd create /bricks/www and create a volume called www with the /bricks/www/data directory, then mount the www volume on /var/www
16:14 wushudoin joined #gluster
16:15 pembo13 ndevos: thanks
16:15 ndevos pembo13: note that gluster is a network filesystem, and small files (like webpages) can be a pretty painful workload, you might want to check ,,(php) for some notes
16:15 glusterbot pembo13: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH --fopen-keep-cache
16:16 pembo13 is there any "shadow copy" like tech that works with glusterfs?
16:16 pembo13 ndevos: eventually i may want to use with a web server, but initial use is as a storage/file server thankfully
16:18 Siavash___ joined #gluster
16:18 ndevos pembo13: there is no "shadow copy" available yet, we're thinking about that - https://github.com/gluster/glusterfs-specs/bl​ob/master/under_review/reflink-based-fsnap.md
16:18 glusterbot Title: glusterfs-specs/reflink-based-fsnap.md at master · gluster/glusterfs-specs · GitHub (at github.com)
16:19 pembo13 ndevos: thanks, "shadow copy" is the one thing my windows server guy keeps holding over my head
16:19 pembo13 ndevos: one more question, the tutorials all seem to use XFS, but i haven't seen an explanation for that
16:20 ndevos pembo13: yes, I think "shadow copy" is the Windows term, we'd call it something like "file base snapshots"
16:20 ndevos s/base/based
16:20 ndevos s/base/based/
16:20 glusterbot What ndevos meant to say was: pembo13: yes, I think "shadow copy" is the Windows term, we'd call it something like "file based snapshots"
16:20 pembo13 ndevos: yah, i just saw that term in the link you provided
16:21 ndevos pembo13: because XFS performance for big filesystems is very good, and parallel access too
16:21 pembo13 understood
16:21 pembo13 the only thing i need to figure out is how best to provide point-in-time file recovery
16:21 ndevos pembo13: it is what Red Hat uses for testing the Red Hat Gluster Storage product, other filesystems see fewer testing
16:22 pembo13 makes sense
16:23 ndevos pembo13: we have a volume snapshot functionality, it uses LVM in the background, depending on the configuration, users can access the older versions by themselves without sysadmin assistance
16:23 pembo13 ndevos: ok, thanks, i'll readup on that
16:24 shaunm joined #gluster
16:25 pembo13 ndevos: i'm guessing this copy-on-right type snapshot, so it doesn't require double the diskspace for each snapshot
16:26 ndevos pembo13: yes, indeed, LVM snapshots with LVM-thin-provisioning
16:26 pembo13 ah, so i would definitely need to create on not of LVM, and not mkfs.xfs directly to disk partition
16:27 ndevos pembo13: yes, check the docs for how to setup lvm-thinp
16:28 pembo13 thanks
16:29 ndevos amye: do you have the power to update the /topic and add a link to the docs?
16:30 pembo13 ndevos: thanks for the topics for me to research
16:30 pembo13 have a good day
16:31 ndevos pembo13: you're welcome, enjoy the weekend!
16:31 * amye is not a mod here
16:31 amye try hagarth
16:32 F2Knight joined #gluster
16:32 nehar joined #gluster
16:33 ndevos hagarth: do you have the power to update the /topic and add a link to the docs?
16:34 skylar joined #gluster
16:59 ahino joined #gluster
17:01 haomaiwang joined #gluster
17:02 dnunez joined #gluster
17:15 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
17:16 JoeJulian Now the search just needs to get fixed.
17:16 JoeJulian Because that's been a steaming pile of poo for 6 months.
17:17 DV_ joined #gluster
17:22 sobersabre ndevos: what if I set the brick down timeout to say 10 sec. how does this affect on 40-60 bricks fs ?
17:22 JoeJulian @ping-timeout
17:23 glusterbot JoeJulian: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. With an average MTBF of 45000 hours for a server, even just a replica 2 would result in a 42 second MTTR every 2.6 years, or 6 nines of uptime.
17:23 sobersabre :) cool
17:24 sobersabre Another thing: performance. How many "buttons/potentiometers" exist for tuning performance? (metrics interesting to me are: read throughput, write throughput)
17:25 JoeJulian gluster volume set help
17:25 sobersabre JoeJulian: thanks.
17:25 JoeJulian There's a big list but I've never seen any good analysis of them.
17:26 sobersabre when the brick is discovered dead, is the "smart" client able to push the data that's been attempted to be written to the remaining bricks?
17:29 sobersabre if someone is using plain stupid app using sync io, i.e. write(), (via fprintf), would the app see the downtime of the brick in any way ?
17:30 sobersabre would the write() wait for I/O? (so we'd need to tune the write waits)
17:37 DV_ joined #gluster
17:41 shyam joined #gluster
17:58 hagarth anoopcs: ping, around?
17:59 shubhendu joined #gluster
18:01 haomaiwang joined #gluster
18:03 JoeJulian sobersabre: Sorry, had to run an errand. The client manages replication synchronously*, so if a server fails, the data is safe.
18:04 JoeJulian If the TCP connection closes, there is no noticable wait. If the TCP connection does not close, but the server ceases responding, the write will wait ping-timeout.
18:12 amye joined #gluster
18:15 chirino joined #gluster
18:28 Siavash___ joined #gluster
18:56 shyam left #gluster
18:59 hagarth joined #gluster
19:01 haomaiwang joined #gluster
19:02 post-factum sobersabre: JoeJulian: you both believe 42 to be scientific value :)?
19:18 JoeJulian It's the answer to life, the universe, and everything. :)
19:18 JoeJulian My point, however, is that it's not an unreasonable value. It could be 60 and still provide a reasonable MTTR.
19:20 JoeJulian I kind-of like the "however long it takes to reboot a switch" metric.
19:25 robb_nl joined #gluster
19:28 kovshenin joined #gluster
19:40 kpease joined #gluster
19:58 rwheeler joined #gluster
20:01 haomaiwang joined #gluster
20:05 julim joined #gluster
20:08 jermudgeon joined #gluster
20:24 arcolife joined #gluster
20:34 MrAbaddon joined #gluster
21:01 haomaiwang joined #gluster
21:05 ira joined #gluster
21:50 hackman joined #gluster
22:01 haomaiwang joined #gluster
22:48 plarsen joined #gluster
22:57 johnmilton joined #gluster
23:00 alanh joined #gluster
23:01 haomaiwang joined #gluster
23:02 alanh Hi all. Not sure if this where I can get help with this but thought I'd give it a shot :-) I've got glusterfs and ganesha-nfs setup. All used to work well  I had a server crash. Well, got everything back up and running but now when connecting via the nfs client, only the files in the root of the brick are listed. Subdirectories are empty and all ownerships are set to 4294967294. I can't seem to fix this. Any thoughts?
23:02 alanh Accessing via the fuse client works well.

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary