Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-05-05

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 haomaiwang joined #gluster
00:04 RobertTuples Hello. I wanted to see how badly latency between two gluster 3.7.11 replica nodes affects performance. the abysmal result tells me all writes to AFR servers are serialized/single-threaded. is there a volume configuration option I'm missing? or maybe a client mount option? Here is a gist containing a Vagrantfile with my exact environment and test commands: https://gist.github.com/anonymous/d8
00:04 RobertTuples bb4cf1b67d299593b347e94df6a8e1
00:15 hagarth joined #gluster
00:42 chirino_m joined #gluster
00:57 plarsen joined #gluster
01:06 RameshN joined #gluster
01:11 haomaiwang joined #gluster
01:12 bennyturns joined #gluster
01:15 cholcombe joined #gluster
01:19 haomaiwang joined #gluster
01:19 atrius joined #gluster
01:24 DV joined #gluster
01:27 EinstCrazy joined #gluster
01:28 Lee1092 joined #gluster
01:33 MugginsM joined #gluster
01:37 harish_ joined #gluster
01:46 luizcpg joined #gluster
01:46 julim joined #gluster
02:01 haomaiwang joined #gluster
02:51 theron joined #gluster
02:59 MugginsM joined #gluster
03:01 haomaiwang joined #gluster
03:09 sakshi joined #gluster
03:15 overclk joined #gluster
03:16 rafi joined #gluster
03:17 RameshN joined #gluster
03:18 natarej_ joined #gluster
03:22 russoisraeli joined #gluster
03:22 F2Knight joined #gluster
03:48 kpease joined #gluster
03:51 RobertTuples joined #gluster
03:52 level7 joined #gluster
03:53 nbalacha joined #gluster
03:53 itisravi joined #gluster
03:54 aravindavk joined #gluster
03:56 atinm joined #gluster
04:01 haomaiwang joined #gluster
04:10 RobertTuples joined #gluster
04:14 nehar joined #gluster
04:16 gem joined #gluster
04:16 Javezim joined #gluster
04:17 Javezim Hey All, Last night our system auto updated to Samba v4.3.9-Ubuntu. Now our Samba VFS has stopped with the following error:
04:17 lanning RobertTuples: they are not serialized.  They are synchronous. All writes must return from all replicas, unless a replica is down, then there are other code paths taken (for quarum, arbiter and the likes.)
04:17 Javezim [2016/05/05 11:19:39.483520,  0] ../lib/util/modules.c:48(load_module)
04:17 Javezim Error loading module '/usr/lib/x86_64-linux-gnu​/samba/vfs/glusterfs.so': /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0: version `SAMBA_4.3.8_UBUNTU' not found (required by /usr/lib/x86_64-linux-gnu/samba/vfs/glusterfs.so)
04:18 Javezim anyone know how to get around this? Or does anyone have the glusterfs.so file for Samba VFS 4.3.9 they could provide?
04:20 aspandey joined #gluster
04:20 MugginsM joined #gluster
04:24 kdhananjay joined #gluster
04:28 MugginsM joined #gluster
04:30 nehar joined #gluster
04:35 shubhendu joined #gluster
04:37 skoduri joined #gluster
04:38 RobertTuples joined #gluster
04:46 mowntan joined #gluster
04:50 rafi joined #gluster
04:52 kshlm joined #gluster
05:00 kotreshhr joined #gluster
05:01 haomaiwang joined #gluster
05:09 ramky joined #gluster
05:12 ndarshan joined #gluster
05:14 MugginsM joined #gluster
05:16 hgowtham joined #gluster
05:19 jiffin joined #gluster
05:21 RobertTuples lanning: I understand what synchronous means. I don't understand why gluster is only writing one file at a time even though I have 100 threads writing at once.
05:25 Apeksha joined #gluster
05:27 nishanth joined #gluster
05:30 JesperA joined #gluster
05:33 shubhendu joined #gluster
05:34 prasanth joined #gluster
05:34 Bhaskarakiran joined #gluster
05:46 rafi joined #gluster
05:49 gowtham joined #gluster
05:50 RameshN joined #gluster
05:54 Siavash_ joined #gluster
05:54 karthik___ joined #gluster
05:56 aravindavk joined #gluster
05:56 Javezim Anyone had their Samba upgraded to 4.3.9 and still got the Samba VFS Working? Or does anyone know how to rollback to 4.3.8 Samba?
05:56 hchiramm joined #gluster
05:57 atrius_ joined #gluster
06:00 robb_nl joined #gluster
06:01 haomaiwang joined #gluster
06:02 anoopcs Javezim, Are you using https://launchpad.net/~monotek/+arc​hive/ubuntu/samba-vfs-glusterfs-3.7 ?
06:02 glusterbot Title: samba-vfs-glusterfs-3.7 : André Bauer (at launchpad.net)
06:03 skoduri joined #gluster
06:04 spalai joined #gluster
06:04 Javezim Yes
06:05 anoopcs Javezim, May be samba-4.3.9 have not been built with glusterfs module.
06:05 anoopcs for ubuntu.
06:05 poornimag joined #gluster
06:05 Javezim Hmm yeah, Any idea how to rollback to 4.3.8?
06:06 anoopcs Javezim, I am not that familiar with apt-get.
06:06 anoopcs Don't we have a downgrade option with apt-get?
06:07 Javezim I know there is a Yum downgrade
06:07 Javezim But I dont think Ubuntu has apt-get downgrade
06:08 anoopcs apt-cache showpkg samba lists 4.3.8, right?
06:10 anoopcs A quick search says that we just need to specify the version with apt-get during install as follows:
06:10 anoopcs apt-get install packagename=version
06:10 anoopcs Javezim, I don't know how well it behaves with dependencies.
06:11 anoopcs But it's worth a try.
06:12 Javezim Okay we run it but it doesn't seem to be doing anything. Goes to next line as if it completes but still shows 4.3.9
06:14 anoopcs Javezim, for which command?
06:15 karnan joined #gluster
06:15 Javezim Ill paste bin what happens
06:15 Javezim 1 second
06:15 anil joined #gluster
06:16 nbalacha joined #gluster
06:18 Javezim http://pastebin.com/PaiUbhDh @anoopcs
06:18 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
06:19 Javezim http://paste.ubuntu.com/16231208/
06:19 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
06:19 anoopcs Javezim, Can you please share the output of apt-cache showpkg samba
06:20 Javezim @anoopcs No problem - http://paste.ubuntu.com/16231215/
06:21 Manikandan joined #gluster
06:23 DV joined #gluster
06:23 anoopcs Javezim, What was the previous version of Samba? 4.3.8?
06:24 Javezim Yeah
06:26 Javezim & It was working fine. Then Auto-Security-Update last night screwed us over. Will need to make sure security updates aren't installed automatically going forward.
06:26 anoopcs Javezim, I know.
06:27 * anoopcs is still wondering why apt-get install is not doing the downgrade.
06:35 skoduri joined #gluster
06:36 anoopcs Javezim, May be something to do with version number.
06:36 anoopcs I don't know.
06:36 Manikandan joined #gluster
06:39 anoopcs Javezim, What does dpkg -l | grep samba tell ?
06:41 Javezim http://paste.ubuntu.com/16231346/
06:41 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
06:44 anoopcs Javezim, Hm. It's the same.
06:46 EinstCrazy joined #gluster
06:48 Javezim Yeah, so it upgraded on 4 of our Nodes. Tried the - apt-get install samba=2:4.3.8+dfsg-0ubuntu0.1​4.04.2glusterfs3.7.11trusty1 - on all 4, no luck :(
07:00 pur joined #gluster
07:01 haomaiwang joined #gluster
07:03 anoopcs Javezim, Anyway you can file a bug under the component packaging.
07:03 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
07:05 anoopcs Meanwhile you can ask for help in #ubuntu channel regarding general downgrading guidelines.
07:06 anoopcs Or else wait here for someone with more knowledge on debian packages notices this conversation
07:06 mhulsman joined #gluster
07:06 anoopcs especially ubuntu.
07:06 Javezim Thanks Anoopcs :) I appreciate the help
07:10 ppai joined #gluster
07:13 Javezim Okay fingers crossed :)      https://bugzilla.redhat.co​m/show_bug.cgi?id=1333264
07:13 glusterbot Bug 1333264: high, unspecified, ---, bugs, NEW , 4.3.9 Samba not working with Gluster 3.7.10
07:18 fsimonce joined #gluster
07:23 chirino joined #gluster
07:23 DV joined #gluster
07:27 anoopcs Javezim, One quick question: Why wasn't glusterfs updated?
07:28 anoopcs I see 3.7.11 from https://launchpad.net/~gluster​/+archive/ubuntu/glusterfs-3.7 ?
07:28 glusterbot Title: glusterfs-3.7 : “Gluster” team (at launchpad.net)
07:28 Javezim We don't really want to take the Gluster Volume offline for this upgrade
07:29 Javezim 4.3.8 & 3.7.10 were working amazingly up until Samba decided it wanted to upgrade
07:30 anoopcs Javezim, Ok.
07:31 Javezim We're now using Samba to share the Folders we need directly, it's slow, but at least it's up
07:31 Javezim But we really need samba-vfs back
07:32 Javezim Good news is I think I found how to stop future Samba upgrades
07:32 Javezim Using apt
07:32 Javezim You can hold a package using
07:32 Javezim sudo apt-mark hold package_name
07:32 Javezim Source: http://askubuntu.com/questions/18654/how-​to-prevent-updating-of-a-specific-package
07:32 glusterbot Title: updates - How to prevent updating of a specific package? - Ask Ubuntu (at askubuntu.com)
07:34 F2Knight joined #gluster
07:36 Saravanakmr joined #gluster
07:41 ivan_rossi joined #gluster
07:48 level7_ joined #gluster
07:54 arcolife joined #gluster
07:59 arcolife joined #gluster
08:00 arcolife joined #gluster
08:01 haomaiwang joined #gluster
08:02 kovshenin joined #gluster
08:07 skoduri joined #gluster
08:09 karthik___ joined #gluster
08:20 nbalacha joined #gluster
08:21 muneerse joined #gluster
08:38 harish_ joined #gluster
08:44 muneerse2 joined #gluster
08:54 nishanth joined #gluster
08:57 rastar joined #gluster
08:57 raghug joined #gluster
08:58 raghug xavih: ping
08:58 glusterbot raghug: Please don't naked ping. http://blogs.gnome.org/mark​mc/2014/02/20/naked-pings/
08:58 raghug xavih: you there?
08:59 JesperA joined #gluster
09:00 anoopcs Javezim, I hope you got the answers.
09:00 gem joined #gluster
09:00 Javezim @anoopcs Yeah I did, all systems functional again :) Thanks Anoopcs!
09:01 haomaiwang joined #gluster
09:01 anoopcs Javezim, yw. Andre builds packages for ubuntu. That's why I assigned the bug to him.
09:06 level7 joined #gluster
09:12 muneerse joined #gluster
09:15 arcolife joined #gluster
09:20 nishanth joined #gluster
09:36 Gnomethrower joined #gluster
10:01 haomaiwang joined #gluster
10:10 sakshi joined #gluster
10:10 morse joined #gluster
10:14 morse_ joined #gluster
10:16 kkeithley joined #gluster
10:20 klaas joined #gluster
10:20 morse joined #gluster
10:23 chirino_m joined #gluster
10:25 mhulsman joined #gluster
10:27 morse joined #gluster
10:30 qwebirc83604 joined #gluster
10:32 klaas joined #gluster
10:36 marbu joined #gluster
10:37 morse_ joined #gluster
10:39 karnan joined #gluster
10:44 morse joined #gluster
10:48 nishanth joined #gluster
10:53 arcolife joined #gluster
10:54 gem joined #gluster
10:55 hchiramm joined #gluster
10:57 hi11111 joined #gluster
11:01 haomaiwang joined #gluster
11:13 nehar joined #gluster
11:17 arcolife joined #gluster
11:17 ghenry joined #gluster
11:18 prasanth joined #gluster
11:19 johnmilton joined #gluster
11:24 shubhendu joined #gluster
11:26 B21956 joined #gluster
11:31 ira_ joined #gluster
11:32 spalai left #gluster
11:38 B21956 joined #gluster
11:44 chirino_m joined #gluster
11:49 sakshi joined #gluster
11:52 marbu joined #gluster
11:55 Debloper joined #gluster
11:58 level7 joined #gluster
12:03 level7 joined #gluster
12:10 nishanth joined #gluster
12:12 overclk joined #gluster
12:13 nehar joined #gluster
12:14 Apeksha joined #gluster
12:15 prasanth joined #gluster
12:22 chirino_m joined #gluster
12:22 skoduri_ joined #gluster
12:24 russoisraeli joined #gluster
12:24 kotreshhr left #gluster
12:30 haomaiwang joined #gluster
12:30 haomaiwang joined #gluster
12:35 russoisraeli joined #gluster
12:38 overclk joined #gluster
12:40 Siavash__ joined #gluster
12:42 bowhunter joined #gluster
12:49 unclemarc joined #gluster
12:52 shyam joined #gluster
12:55 kdhananjay joined #gluster
13:01 Manikandan joined #gluster
13:01 haomaiwang joined #gluster
13:01 jiffin1 joined #gluster
13:11 mpietersen joined #gluster
13:12 overclk joined #gluster
13:16 jri joined #gluster
13:18 Siavash___ joined #gluster
13:25 amye joined #gluster
13:40 luizcpg joined #gluster
13:41 atinm joined #gluster
13:42 gem joined #gluster
13:45 overclk joined #gluster
13:48 d0nn1e joined #gluster
13:49 EinstCrazy joined #gluster
13:51 karnan joined #gluster
13:54 nehar joined #gluster
13:56 shaunm joined #gluster
13:56 nbalacha joined #gluster
13:57 mhulsman joined #gluster
13:58 kdhananjay joined #gluster
13:59 plarsen joined #gluster
14:00 dlambrig_ joined #gluster
14:00 MikeLupe joined #gluster
14:01 ashiq joined #gluster
14:01 haomaiwang joined #gluster
14:03 bennyturns joined #gluster
14:07 anil joined #gluster
14:08 jiffin1 joined #gluster
14:11 nbalacha joined #gluster
14:13 cholcombe joined #gluster
14:16 level7 joined #gluster
14:18 shubhendu joined #gluster
14:26 wushudoin joined #gluster
14:38 gowtham joined #gluster
14:41 skylar joined #gluster
14:44 Bhaskarakiran joined #gluster
14:47 dlambrig_ joined #gluster
14:49 brettnem JoeJulian: hey I'm trying out the lookup-unhashed option, but I'm not real sure where to put it. I originally thought glusterd.vol but my brick configuration isn't there :/
14:59 shyam joined #gluster
15:01 dlambrig_ joined #gluster
15:01 haomaiwang joined #gluster
15:02 anil joined #gluster
15:02 squizzi joined #gluster
15:10 jiffin joined #gluster
15:24 julim joined #gluster
15:25 jiffin joined #gluster
15:26 lanning joined #gluster
15:30 edong23 joined #gluster
15:33 kpease joined #gluster
15:36 DV joined #gluster
15:37 jiffin joined #gluster
15:38 vh joined #gluster
15:43 jiffin1 joined #gluster
15:45 vh Hello - I asked this question last week with no response yet. We’ve noticed that the .glusterfs directory is larger than the contents of the volume (double in size). Our application only has access through the client so I don’t suspect anything was deleted on the brick. How could we have come into this state? Is there a way to find what is orphaned?
15:45 vh We've already checked heal info and it looked normal. When I added a second node and performed a full heal, the .glusterfs directory size on the second node is the same as the volume content size which is that we expected.
15:46 pavel__ joined #gluster
15:50 pavel__ Hey guys, I want to ask a question about replication. We have volume on single brick and want to replicate it to the other brick. Volume contains some VM disk etc. so I/O is expected during replication. If we start 'volume ... add-brick replica 2 ...', clients having this volume mounted suddenly report file sizes of 0 (gradually increasing) and VMs report I/O fail, remount FS as read-only and everything slowly dies... (cont)
15:50 pavel__ Is there a way of doing replication online?
15:51 foster joined #gluster
15:51 pavel__ We run Gluster 3.7.11 on Ubuntu 16.04, XFS bricks
15:57 farhorizon joined #gluster
16:01 haomaiwang joined #gluster
16:11 Philambdo joined #gluster
16:11 atinm joined #gluster
16:21 mhulsman joined #gluster
16:23 hgowtham joined #gluster
16:32 ivan_rossi left #gluster
16:32 B21956 joined #gluster
16:46 atrius joined #gluster
16:47 rafi joined #gluster
16:56 skoduri joined #gluster
16:57 muneerse2 joined #gluster
17:01 harish_ joined #gluster
17:01 haomaiwang joined #gluster
17:05 JoeJulian brettnem: gluster volume set $vol cluster.lookup-unhashed off
17:07 JoeJulian vh: find $brick/.glusterfs/[0-9a-f][0-9a-f] -type d -links 1
17:07 JoeJulian s/type d/type f/
17:07 glusterbot What JoeJulian meant to say was: vh: find $brick/.glusterfs/[0-9a-f][0-9a-f] -type f -links 1
17:08 JoeJulian Not sure why I said 'd' there...
17:09 JoeJulian vh: so you're finding gfid files with only 1 link. Those are inodes that have somehow lost their filename. You can do with them what you wish, perhaps delete them, perhaps move them offline in case they're important.
17:14 amye joined #gluster
17:21 anil joined #gluster
17:38 post-factum JoeJulian: what does cluster.lookup-unhashed do?
17:39 post-factum JoeJulian: does that mean in case of unbalanced layout gluster server will return something like ENOENT on unbalanced file?
17:42 nottc joined #gluster
17:58 dlambrig_ joined #gluster
18:01 haomaiwang joined #gluster
18:04 Manikandan joined #gluster
18:10 jiffin1 joined #gluster
18:15 vh JoeJulian: Thanks for the reply! I had tried this previously and was not able to find any orphaned files. The command you provided did not return any results... =\
18:21 m0zes joined #gluster
18:28 F2Knight joined #gluster
18:29 nickage_ <JoeJulian: is there a way to log changed files ?
18:37 JoeJulian post-factum: correct. When lookup-unhashed is on, when the filename is not found by hash, a new query is done against all the bricks to see if it's misplaced. With that off, if it's not on the correct brick it doesn't exist.
18:40 JoeJulian vh: I can't think of a failure model that could produce the results you're describing. The only thing I can think to next test would be to get a list of gfids using getfattr (actually, I'd be using pyattr because I'd be scripting this in python) and comparing that list with the list of files in .glusterfs.
18:40 post-factum JoeJulian: thanks
18:41 JoeJulian nickage_: not really, no. I know they're working on an inotify-like tool, but it doesn't exist yet.
18:43 JoeJulian nickage_: Oh, one idea that comes to mind. Compare ls -lR between the replica that has what you expect, and the replica that does not.
18:46 jiffin joined #gluster
18:46 mhulsman joined #gluster
18:48 shubhendu_ joined #gluster
18:51 jiffin joined #gluster
18:52 ahino joined #gluster
18:53 shubhendu joined #gluster
18:56 vh JoeJulian: ok, i'll see what I can find. thanks again!
19:01 cholcombe joined #gluster
19:01 haomaiwang joined #gluster
19:03 jiffin joined #gluster
19:41 cholcombe joined #gluster
19:46 wushudoin joined #gluster
19:49 mhulsman joined #gluster
19:57 plarsen joined #gluster
19:59 level7 joined #gluster
20:01 haomaiwang joined #gluster
20:01 dlambrig_ joined #gluster
20:11 Guest3901 is there a way to change the backing store location of a volume? i've tried googling, but i just found info on replacing a brick, which i don't really want to do
20:11 DV joined #gluster
20:19 gem joined #gluster
20:20 mhulsman joined #gluster
20:24 JoeJulian Guest3901: "gluster volume replace-brick"
20:25 RobertTuples joined #gluster
20:34 Guest3901 JoeJulian: yeah, i figured it out, i was confused by the man page not mentioning commit force
20:35 Guest3901 this helped https://www.gluster.org/pipermail/gl​uster-users/2012-October/011502.html
20:35 glusterbot Title: [Gluster-users] 'replace-brick' - why we plan to deprecate (at www.gluster.org)
20:35 Guest3901 interestingly i can see the data moving, but volume status doesn't say there is an operation in progress
20:39 dlambrig_ joined #gluster
20:49 shyam joined #gluster
20:54 cholcombe joined #gluster
20:54 adamaN joined #gluster
20:58 amye joined #gluster
21:00 B21956 joined #gluster
21:01 haomaiwang joined #gluster
21:05 jri joined #gluster
21:08 post-factum joined #gluster
21:15 ws2k3 joined #gluster
21:37 cholcombe joined #gluster
21:45 Siavash___ joined #gluster
22:01 haomaiwang joined #gluster
22:07 DV joined #gluster
22:18 nathwill joined #gluster
22:20 MikeLupe2 joined #gluster
22:29 luizcpg joined #gluster
22:41 nickage joined #gluster
22:49 cholcombe joined #gluster
22:59 russoisraeli joined #gluster
23:01 haomaiwang joined #gluster
23:17 dlambrig_ joined #gluster
23:32 marbu joined #gluster
23:36 dlambrig_ joined #gluster
23:48 johnmilton joined #gluster
23:49 renout_away joined #gluster
23:50 crashmag joined #gluster
23:53 russoisraeli joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary