Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-02-08

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 haomaiwa_ joined #gluster
00:19 gildub joined #gluster
00:30 siel_ joined #gluster
00:50 gildub joined #gluster
01:01 haomaiwa_ joined #gluster
01:06 haomai___ joined #gluster
01:15 gildub joined #gluster
01:17 CyrilPeponnet joined #gluster
01:50 EinstCrazy joined #gluster
01:58 nishanth joined #gluster
02:01 haomaiwa_ joined #gluster
02:03 muneerse joined #gluster
02:32 unlaudable joined #gluster
02:38 bharata-rao joined #gluster
02:47 ilbot3 joined #gluster
02:47 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:00 gem joined #gluster
03:01 gem_ joined #gluster
03:01 haomaiwa_ joined #gluster
03:30 atinm joined #gluster
03:31 vmallika joined #gluster
03:31 kanagaraj joined #gluster
03:32 kdhananjay joined #gluster
03:36 shyam joined #gluster
03:46 EinstCrazy joined #gluster
03:51 mhorstman joined #gluster
03:54 finbit1 joined #gluster
03:56 finbit1 joined #gluster
03:58 gildub joined #gluster
03:59 itisravi joined #gluster
04:00 shubhendu joined #gluster
04:01 haomaiwa_ joined #gluster
04:04 kotreshhr joined #gluster
04:08 nbalacha joined #gluster
04:09 sakshi joined #gluster
04:14 ramteid joined #gluster
04:24 nbalacha joined #gluster
04:28 coredump joined #gluster
04:39 telmich_ joined #gluster
04:39 rtlaur joined #gluster
04:39 Nuxr0 joined #gluster
04:39 atinm joined #gluster
04:42 skoduri joined #gluster
04:50 karthikfff joined #gluster
04:53 nehar joined #gluster
05:01 haomaiwa_ joined #gluster
05:02 gowtham joined #gluster
05:06 pppp joined #gluster
05:09 poornimag joined #gluster
05:10 ndarshan joined #gluster
05:13 aravindavk joined #gluster
05:20 rafi joined #gluster
05:27 ramteid joined #gluster
05:27 Apeksha joined #gluster
05:31 anil joined #gluster
05:31 hgowtham joined #gluster
05:40 kdhananjay joined #gluster
05:42 atalur joined #gluster
05:46 Bhaskarakiran joined #gluster
05:49 itisravi joined #gluster
05:49 Manikandan joined #gluster
05:52 nishanth joined #gluster
05:53 jiffin joined #gluster
06:00 dusmantkp_ joined #gluster
06:00 Skumar joined #gluster
06:00 vmallika joined #gluster
06:01 haomaiwang joined #gluster
06:02 Bhaskarakiran joined #gluster
06:02 vimal joined #gluster
06:06 Saravannakmr joined #gluster
06:10 skoduri joined #gluster
06:11 karnan joined #gluster
06:26 dlambrig_ joined #gluster
06:26 dlambrig_ left #gluster
06:44 itisravi joined #gluster
06:45 itisravi joined #gluster
06:46 glafouille joined #gluster
06:52 unlaudable joined #gluster
06:56 EinstCrazy joined #gluster
07:01 haomaiwa_ joined #gluster
07:03 [Enrico] joined #gluster
07:06 dlambrig_ joined #gluster
07:06 kovshenin joined #gluster
07:07 mbukatov joined #gluster
07:08 bharata-rao joined #gluster
07:22 ovaistariq joined #gluster
07:23 jtux joined #gluster
07:24 mhulsman joined #gluster
07:38 mhulsman joined #gluster
07:44 [diablo] joined #gluster
07:44 [diablo] joined #gluster
07:52 inodb joined #gluster
08:01 haomaiwa_ joined #gluster
08:08 mbukatov joined #gluster
08:10 haomaiwang joined #gluster
08:32 karnan joined #gluster
08:34 sakshi joined #gluster
08:38 fsimonce joined #gluster
08:44 ramky joined #gluster
08:53 b0p joined #gluster
08:53 karnan joined #gluster
08:59 ahino joined #gluster
09:01 haomaiwang joined #gluster
09:01 EinstCrazy joined #gluster
09:04 shubhendu joined #gluster
09:15 itisravi_ joined #gluster
09:15 ctria joined #gluster
09:17 itisravi joined #gluster
09:17 ekuric joined #gluster
09:27 toshywoshy joined #gluster
09:33 fulcrum joined #gluster
09:33 Slashman joined #gluster
09:39 nishanth joined #gluster
09:40 dusmantkp_ joined #gluster
09:44 mbukatov joined #gluster
10:01 haomaiwa_ joined #gluster
10:16 hgowtham joined #gluster
10:19 fulcrum joined #gluster
10:28 gildub joined #gluster
10:33 nishanth joined #gluster
10:44 dusmantkp_ joined #gluster
10:46 atinm joined #gluster
10:52 hgowtham joined #gluster
11:01 7YUAAU9E2 joined #gluster
11:02 SOLDIERz joined #gluster
11:08 fulcrum joined #gluster
11:19 aravindavk joined #gluster
11:20 bluenemo joined #gluster
11:21 ivan_rossi joined #gluster
11:30 aravindavk joined #gluster
11:31 Wizek joined #gluster
11:50 ndarshan joined #gluster
12:01 haomaiwa_ joined #gluster
12:05 anil joined #gluster
12:14 itisravi_ joined #gluster
12:16 shyam joined #gluster
12:21 karnan joined #gluster
12:25 monotek joined #gluster
12:26 b0p joined #gluster
12:31 atinm joined #gluster
12:35 dusmantkp_ joined #gluster
12:35 skoduri joined #gluster
12:43 mdavidson joined #gluster
12:50 kdhananjay joined #gluster
12:54 [diablo] Good afternoon #gluster ... I've got a 2 x node cluster, and a mount in fstab to mount via gluster the machines ip:/vol ... however it fails to mount on boot...
12:55 [diablo] the glusterd fails on boot. However I can manually start the daemon, and once started I can mount -a , and works fine
12:55 ira joined #gluster
12:55 [diablo] anyone have any ideas as to what could be wrong please?
12:56 bhuddah what error message do you have in the logs?
12:56 bhuddah and in dmesg?
12:56 [diablo] dmesg yields nothing
12:56 [diablo] and hi bhuddah
12:56 bhuddah i imagine it might be starting before the network is up... but that is just a wild guess.
12:57 [diablo] I'll pastebin the log
12:57 [diablo] oh btw, it's RHGS
12:57 [diablo] on RHEL7
13:00 [diablo] http://pastebin.com/raw/UDDVSzgU
13:00 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
13:01 bhuddah are you running in the cloud or on premises?
13:01 [diablo] http://fpaste.org/319832/14549364/raw/
13:01 [diablo] on premises
13:01 [diablo] OK, so from what I can see it's trying to connect to itself
13:01 [diablo] and failing
13:01 haomaiwa_ joined #gluster
13:01 bhuddah that's not a second host there?
13:02 DV joined #gluster
13:02 [diablo] nope thats the IP of the box
13:02 [diablo] the 2nd machine is on 192.168.18.11
13:02 bhuddah i have a vague idea... can you try and mount the fs with _netdev in fstab?
13:03 [diablo] ah it is already
13:03 [diablo] _netdev,defaults,transport=tcp,xlat​or-option=*client*.ping-timeout=10
13:03 bhuddah ah
13:04 [diablo] RHGS actually creates the fstab line automagically
13:04 [diablo] via their scripts
13:04 bhuddah very strange.
13:04 [diablo] it's a pita
13:04 [diablo] believe me
13:05 bhuddah i've had issues like that with nfs volumes and spanning tree. because that sometimes takes ages to bring up a port. but that shouldn't have local impact.
13:05 EinstCrazy joined #gluster
13:06 [diablo] hmmm ok
13:07 bhuddah sorry, i cannot provide further insights atm.
13:07 [diablo] hey, no problem
13:07 [diablo] appreciate the input
13:07 bhuddah good luck
13:07 [diablo] sadly it's a little quiet in here
13:16 poornimag joined #gluster
13:17 theron joined #gluster
13:21 shyam joined #gluster
13:33 unclemarc joined #gluster
13:33 B21956 joined #gluster
13:42 gowtham joined #gluster
13:43 unclemarc joined #gluster
13:54 burn joined #gluster
13:57 fulcrum joined #gluster
14:01 haomaiwa_ joined #gluster
14:03 csaba joined #gluster
14:16 theron joined #gluster
14:20 theron_ joined #gluster
14:23 rwheeler joined #gluster
14:28 post-factum [diablo]: consider using systemd automount for gluster share
14:29 plarsen joined #gluster
14:29 [diablo] hi post-factum
14:30 [diablo] sorry just seen ur message
14:30 [diablo] well, now we're facing another issue... glusterd still fails to start even when the fstab line is commented out
14:30 [diablo] [2016-02-08 14:09:42.350319] E [name.c:242:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host 192.168.18.11
14:31 post-factum ah, so you have dns resolving issue
14:32 post-factum you may apply dirty hack like manual glusterd starting from /etc/rc.local
14:32 post-factum but that look very dirty
14:32 post-factum s/look/looks/
14:32 glusterbot What post-factum meant to say was: but that looks very dirty
14:33 [diablo] gotta luv that bot
14:33 mhulsman joined #gluster
14:35 gowtham joined #gluster
14:37 [diablo] http://pastie.org/pastes/1071360​0/text?key=bg2p7iqz19vie23rqxilw
14:37 glusterbot Title: Private Paste - Pastie (at pastie.org)
14:37 [diablo] thats the current failed boot
14:43 [diablo] gonna try this https://www.gluster.org/pipermail/glu​ster-users/2015-November/024330.html
14:43 glusterbot Title: [Gluster-users] [ovirt-users] Centos 7.1 failed to start glusterd after upgrading to ovirt 3.6 (at www.gluster.org)
14:45 hamiller joined #gluster
14:49 ahino joined #gluster
14:52 jiffin joined #gluster
14:57 EinstCrazy joined #gluster
15:01 haomaiwang joined #gluster
15:04 neofob joined #gluster
15:13 skylar joined #gluster
15:13 skylar left #gluster
15:13 coredump joined #gluster
15:13 skylar joined #gluster
15:21 bennyturns joined #gluster
15:32 Akee joined #gluster
15:32 nishanth joined #gluster
15:34 dlambrig_ joined #gluster
15:35 rafi joined #gluster
15:36 wushudoin joined #gluster
15:36 rafi joined #gluster
15:39 rafi joined #gluster
15:45 ro_ joined #gluster
15:47 rafi joined #gluster
15:47 ro_ Hey guys - we're running a 40 node distributed/replicated cluster that has been working great for 3-4 months now. Suddenly over the last week we've been experiencing issues when the volume hangs. I'm seeing a lot of "saved_frames_unwind" errors in the logs. There is one node that's down, but I don't think that should take the entire cluster out. Any ideas what I could be looking into?
15:48 ro_ if I do a "df" it hangs, if a navigate to the volume and try an 'ls' it hangs
15:52 nbalacha joined #gluster
15:57 raghu joined #gluster
16:00 Akee joined #gluster
16:00 farhoriz_ joined #gluster
16:01 haomaiwa_ joined #gluster
16:14 mhulsman joined #gluster
16:16 mhulsman joined #gluster
16:18 theron joined #gluster
16:26 rcampbel3 joined #gluster
16:32 hagarth joined #gluster
16:35 farhoriz_ joined #gluster
16:38 hamiller joined #gluster
16:42 skoduri joined #gluster
16:43 robb_nl joined #gluster
16:56 Akee joined #gluster
17:01 haomaiwa_ joined #gluster
17:04 turkleton joined #gluster
17:04 jmarley joined #gluster
17:05 turkleton I'm seeing an odd issue after recovering a node. There are certain directories that are accessible via FUSE, but if you access them over NFS, you get the error: "Operation not permitted"
17:05 turkleton Any thoughts or ideas? I'm not seeing anything specific in the logs
17:09 bennyturns joined #gluster
17:14 vimal joined #gluster
17:17 gem joined #gluster
17:19 rcampbel3 joined #gluster
17:20 ivan_rossi left #gluster
17:32 cliluw joined #gluster
17:35 turkleton The nfs.log is not super helpful either: http://termbin.com/5tzx :(
17:41 calavera joined #gluster
17:46 ovaistariq joined #gluster
17:57 mhulsman joined #gluster
17:57 JoeJulian ro_: ,,(paste) a clean client log. Maybe up the log-level to debug if there's nothing obvious in it.
17:57 glusterbot ro_: For a simple way to paste output, install netcat (if it's not already) and pipe your output like: | nc termbin.com 9999
17:58 JoeJulian ro_: "saved_frames_unwind" happens at the end of the failure (it's a good thing). We're more interested in why the frame needs to be unwound.
17:59 ahino joined #gluster
17:59 tswartz joined #gluster
17:59 JoeJulian turkleton: 0-storage-client-0 is the first brick on the "storage" volume. Check the brick log to see of the nfs client fault matches anything on the brick.
18:01 theron joined #gluster
18:01 turkleton Oh holy shit JoeJulian, that log file has a ton of entries
18:01 haomaiwa_ joined #gluster
18:02 turkleton http://termbin.com/1i1m
18:02 turkleton Tons of operation not permitted
18:02 turkleton What's interesting is that if you mount it via FUSE, it works perfectly fine
18:02 turkleton It seems the issue is happening here:
18:02 turkleton getdents(3, 0x22aff30, 32768)           = -1 EPERM (Operation not permitted)
18:03 turkleton That syscall works fine over FUSE but not NFS
18:05 virusuy Hi guys, silly question here. Let´s asume that i have two bricks with a folder called /brick, which have the same data (copied through rsync) .. can i create a replicated volume using those folder ( /brick) with data already in there ?
18:05 turkleton This is the ls over NFS http://termbin.com/8125
18:05 turkleton and this is over FUSE http://termbin.com/vhl4
18:06 farhori__ joined #gluster
18:09 tswartz joined #gluster
18:12 turkleton Recursively deleting a directory that's failing over NFS and re-creating it allows it to work, but that's not really sustainable
18:13 gem joined #gluster
18:15 tswartz joined #gluster
18:18 jwang joined #gluster
18:21 turkleton virusuy: I wouldn't do that
18:22 virusuy turkleton: doing some testing using vagrant, yeah, seems the worst idea ever
18:22 JoeJulian virusuy: if you wipe one of them you should be ok.
18:22 turkleton You should start with two bricks that are clean slates, and for the best results, you'll want them to have unique names such as /brick/brick1 and /brick/brick2
18:22 virusuy JoeJulian: yeah, doing replication by glusterfs and not by other tool,
18:22 turkleton you don't want the directories to be the same on both nodes
18:23 turkleton and you don't want it to be the root of the block device either
18:23 JoeJulian I always have the directories be the same on both.
18:23 turkleton You do?
18:23 JoeJulian Sure
18:23 JoeJulian It keeps things sane.
18:23 turkleton Interesting. Doesn't the gluster command throw a warning when you do that?
18:23 JoeJulian nope
18:23 turkleton My structure is /gbrick/<UUID>
18:23 virusuy turkleton: about the root block, it´s just a test, obviously on prod we have those brick on separated FS
18:23 turkleton :P
18:23 turkleton JoeJulian: Any thoughts on the NFS weirdness I'm seeing?
18:24 JoeJulian turkleton: did you perhaps upgrade something recently?
18:24 turkleton I don't believe so. They're both 3.6.6.
18:25 JoeJulian I'm afraid I have no idea. You might see if "gluster volume start storage force" has any affect.
18:26 JoeJulian ndevos might be a better resource for nfs related wierdness, but he's in GMT+0 so already gone for the day.
18:26 turkleton I am seeing this: "/var/log/glusterfs/glfshe​al-storage.log:[2016-02-08 17:52:46.893021] I [client-handshake.c:1210:client_setvolume_cbk] 0-storage-client-0: Server and Client lk-version numbers are not same, reopening the fds"
18:26 glusterbot turkleton: This is normal behavior and can safely be ignored.
18:26 dgandhi joined #gluster
18:27 turkleton What's really odd about this is that some directories work whereas others do not and throw that operation not permitted error :(
18:27 JoeJulian selinux?
18:28 JoeJulian If it was selinux, though, I would expect fuse to fail too.
18:28 JoeJulian Unless, maybe, they're choosing different read subvolumes and one has a selinux restriction that the other doesn't.
18:28 JoeJulian s/the/an/
18:28 glusterbot What JoeJulian meant to say was: Unless, maybe, any're choosing different read subvolumes and one has a selinux restriction that the other doesn't.
18:28 turkleton It's an Ubuntu box, and we are running apparmor
18:28 turkleton Hrm, lemme stop apparmor
18:28 JoeJulian I know nothing about apparmor
18:29 turkleton It's basically Debian's selinux
18:29 turkleton Even with apparmor torn down, we got an operation not permitted error on getdents
18:29 JoeJulian Except selinux is in-kernel.
18:29 turkleton apparmor is a kernel module, so somewhat in kernel
18:29 turkleton just not mainstream
18:30 ovaistariq joined #gluster
18:31 JoeJulian ext4?
18:32 JoeJulian shouldn't matter, just curious.
18:34 JoeJulian turkleton: Try setting "use-readdirp=off"
18:35 JoeJulian Then you can tell me you're using 32bit nfs clients.
18:38 turkleton It is ext4
18:38 turkleton alright, will try that
18:41 turkleton with use-readdirp set to yes and no, FUSE still works and NFS doesn't
18:41 turkleton It doesn't appear to be a valid NFS mount option
18:42 JoeJulian damn
18:43 dlambrig_ joined #gluster
18:43 JoeJulian nfs mount option is "nordirplus"
18:44 calavera joined #gluster
18:45 turkleton :( No dice
18:46 JoeJulian Show me a brick log entry.
18:46 JoeJulian (Just one will due)
18:49 turkleton http://termbin.com/1i1m
18:50 turkleton Hrm, interesting
18:51 turkleton 9597ff0d-c113-4d49-b6a6-f2874fde6531 is a broken symlink
18:51 turkleton http://termbin.com/emmh
18:52 turkleton and it's a broken symlink on both
18:54 JoeJulian Ah, I have an idea but I have to leave right this moment for a dr appt.
18:54 turkleton Ok
18:54 turkleton Thanks for your help :)
18:54 turkleton I'm excited to hear your idea when you get back
18:56 turkleton Think I figured it out maybe! https://bugzilla.redhat.com/show_bug.cgi?id=961668
18:56 glusterbot Bug 961668: unspecified, unspecified, ---, bugs, CLOSED EOL, gfid links inside .glusterfs are not recreated when missing, even after a heal
18:56 turkleton maybe
19:00 skylar joined #gluster
19:01 haomaiwang joined #gluster
19:02 theron joined #gluster
19:04 EinstCrazy joined #gluster
19:05 natarej_ joined #gluster
19:13 b0p joined #gluster
19:15 post-factum guys, anyone use cache tier here? is it considered stable to enable for existing volumes?
19:20 hagarth post-factum: I would recommend testing out tiering with 3.7.8 or so. I have not heard enough feedback to recommend using it for existing volumes.
19:21 post-factum hagarth: got that, thanks
19:31 farhoriz_ joined #gluster
19:33 mhulsman joined #gluster
19:35 xavih joined #gluster
19:35 malevolent joined #gluster
19:42 mrrrgn left #gluster
19:46 ro_ how should I go about fixing a split brain issue when it's basically every single file on one of the nodes?
19:49 theron joined #gluster
20:01 haomaiwa_ joined #gluster
20:03 farhoriz_ joined #gluster
20:14 gildub joined #gluster
20:14 robb_nl joined #gluster
20:19 nickage_ joined #gluster
20:21 a_ta joined #gluster
20:21 turkleton joined #gluster
20:22 cliluw joined #gluster
20:30 JoeJulian ~ split-brain | ro
20:30 glusterbot ro: To heal split-brains, see https://gluster.readthedocs.org/en/release-3.7.0​/Features/heal-info-and-split-brain-resolution/ For additional information, see this older article https://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/ Also see splitmount https://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/
20:44 turkleton JoeJulian: I figured it out
20:45 turkleton We moved a bunch of things around as part of a package install in the postinst script, and it looks like the GFIDs weren't fixed. They still pointed to the old paths. I have no idea how the glusterfs FUSE mount worked and NFS didn't, but I was able to update the paths, and the self-healing fixed it
20:51 cy8aer joined #gluster
20:55 papamoose2 joined #gluster
20:59 turkleton We fixed it by fixing the paths under .glusterfs
20:59 turkleton to update the GFID symlink to the correct directory
21:00 turkleton s/the GFID/the corresponding GFID/
21:00 glusterbot What turkleton meant to say was: We moved a bunch of things around as part of a package install in the postinst script, and it looks like the corresponding GFIDs weren't fixed. They still pointed to the old paths. I have no idea how the glusterfs FUSE mount worked and NFS didn't, but I was able to update the paths, and the self-healing fixed it
21:01 haomaiwa_ joined #gluster
21:02 turkleton Due to GlusterFS' self-healing nature, it also replicated those changes to the other node and fixed NFS on both sides
21:03 turkleton I don't know if I'm truly comfortable with what I did as I'm uncertain as to whether or not we just made something worse... but it seems to have worked
21:07 johnmilton joined #gluster
21:11 JoeJulian Nice, sounds pretty good to me.
21:11 JoeJulian And that's what I thought happened.
21:12 a_ta left #gluster
21:17 calavera joined #gluster
21:36 cuqa joined #gluster
21:40 cuqa hello, we run a 3 node gluster with 1,8TB of storage in a single volume. The volume contains a www directory, logs and some config files
21:41 cuqa there is not really a lot going on, but the cpu load is pretty high due to glusterfsd
21:41 cuqa any hints what might be wrong here? glusterfs 3.6.8-1 is running on all systems with debian 7
21:55 xavih joined #gluster
21:55 malevolent joined #gluster
22:01 farhoriz_ joined #gluster
22:01 haomaiwa_ joined #gluster
22:04 JoeJulian The tools for answering that question are "gluster volume status" "gluster volume heal $vol info" and looking at the log files for the process that you're concerned with. Take a look yourself and if you need a second opinion, use a paste service and share the link.
22:49 ovaistariq joined #gluster
22:55 theron joined #gluster
22:58 EinstCrazy joined #gluster
23:01 haomaiwa_ joined #gluster
23:40 liewegas joined #gluster
23:48 dlambrig_ joined #gluster
23:50 theron joined #gluster
23:51 dlambrig_ left #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary