Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-12-18

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:05 _Bryan_ joined #gluster
00:16 Rydekull joined #gluster
00:23 JoeJulian l0uis: Nothing there looks ominous. No errors. Are they any actual " E " errors in the client log when you fail to open?
00:25 JoeJulian sysadmin-di2e: I've never had success with a rebalance. Always seems to break one thing or another. If it were me, I'd start by "pkill glusterd; pkill -f nfs.log; pkill -f glustershd" then I'd restart glusterd (or glusterfs-server if you're on ubuntu).
00:26 JoeJulian sysadmin-di2e: I doubt that would fix it, but if it does, count it as a win and move on.
00:26 JoeJulian sysadmin-di2e: Next, I would try to make sure that all the files that are on that brick pair are healed, then I would reboot one of them and wait for the heal to complete again, then reboot the other.
00:38 systemonkey joined #gluster
00:49 systemonkey joined #gluster
00:52 marcoceppi joined #gluster
01:01 nishanth joined #gluster
01:06 leo__ joined #gluster
01:09 l0uis JoeJulian: No, no E errors... but for example this just happened in one of the jobs: mkdir: cannot create directory `/srv/gv0/scratch/f3a2b930': Input/output error
01:09 l0uis and the command was: mkdir -p /srv/gv0/scratch/f3a2b930
01:10 l0uis and again in the logs: http://fpaste.org/160830/86500814/
01:11 l0uis I don't see any errors related to open at all, which is strange. The basic flow is, mkdir, chdir, open ...
01:11 l0uis I would expect the chdir to fail if the directory wasn't there, but it doesnt... but the open is failing and returning -1 and all the open is doing is opening a file for writing
01:21 JoeJulian l0uis: Make sure your client is connecting to all the bricks. This almost sounds like it's not and it has to wait for that self-heal your log showed.
01:24 l0uis JoeJulian: How can I check that?
01:25 JoeJulian netstat?
01:25 JoeJulian I just blame networking, then I make them prove I'm wrong.
01:27 l0uis :)
01:27 l0uis I should see 1 tcp conx per brick?
01:28 JoeJulian 1 connection to 24007, and 1 to the brick (you can see the port with gluster volume status)
01:30 kalzz joined #gluster
01:34 l0uis JoeJulian: It looks right to me ...  http://fpaste.org/160834/88664911/
01:35 l0uis JoeJulian: the only variant between compute nodes is which gluster node they chose to mount from.. i presume the 24007 connection is the "control" or "mount" connection ?
01:37 JoeJulian Yes, 24007 is glusterd.
01:46 l0uis Hrm..
01:47 l0uis what happens if two nodes simultaneously do 'mkdir -p x' will they both attepmt to make the dir?
01:47 l0uis I would expect internally to gluster there is locking etc to ensure that doesn't happen...
01:48 JoeJulian I don't think the gnu mkdir does any sort of locking, so I suppose they would.
01:49 l0uis so if node a and b both attempt to make the dir, i guess one gets it and one doesn't, the one that doesn't has ot go through a self-heal to get the right state?
01:49 JoeJulian I'm not even familiar enough with the posix spec to know if you can lock a directory while you create entries for it.
01:49 JoeJulian possibly.
01:49 MacWinner joined #gluster
01:49 JoeJulian Can't see how though...
01:50 l0uis Perhaps the moral of the story is to not try to do distributed directory creation :)
01:50 JoeJulian That's a good question though. If you could file a bug on that, I suspect you're right.
01:50 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
01:50 l0uis I guess I'll add a setup step the job to ensure only one thing is making that dir before it gets all parallel
01:51 l0uis I'll write up a report, for sure.
01:52 tpetersh joined #gluster
01:53 tpetersh hi, does anyone here have a mirror of Jeff Darcy's blog?
01:53 tpetersh The one here is down http://gluster.org/documentat​ion/architecture/Translators/
01:53 l0uis JoeJulian: which component should I select?
01:55 edong23 joined #gluster
02:09 l0uis JoeJulian: opened: https://bugzilla.redhat.co​m/show_bug.cgi?id=1175551
02:09 glusterbot Bug 1175551: unspecified, unspecified, ---, bugs, NEW , Intermittent open() failure after creating a directory
02:10 l0uis JoeJulian: thanks for the help
02:11 haomaiwa_ joined #gluster
02:21 glusterbot News from newglusterbugs: [Bug 905747] [FEAT] Tier support for Volumes <https://bugzilla.redhat.com/show_bug.cgi?id=905747>
02:21 glusterbot News from newglusterbugs: [Bug 1175551] Intermittent open() failure after creating a directory <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175551>
02:51 prasanth_ joined #gluster
02:55 DV joined #gluster
03:29 julim joined #gluster
03:41 RameshN joined #gluster
03:50 coredump joined #gluster
03:54 _pol joined #gluster
03:56 spandit joined #gluster
03:57 _pol_ joined #gluster
03:59 Telsin joined #gluster
03:59 vimal joined #gluster
04:03 nishanth joined #gluster
04:04 shubhendu joined #gluster
04:05 kanagaraj joined #gluster
04:14 itisravi joined #gluster
04:16 harish joined #gluster
04:24 DJClean joined #gluster
04:33 lalatenduM joined #gluster
04:34 nbalacha joined #gluster
04:36 anoopcs joined #gluster
04:39 gildub joined #gluster
04:43 coredump joined #gluster
04:43 jiffin joined #gluster
04:44 rafi joined #gluster
04:53 smohan joined #gluster
05:03 rjoseph joined #gluster
05:26 hagarth joined #gluster
05:32 soumya joined #gluster
05:36 sahina joined #gluster
05:36 an joined #gluster
05:37 Debloper joined #gluster
05:40 kumar joined #gluster
05:43 sputnik13 joined #gluster
05:44 nbalacha joined #gluster
05:44 anil joined #gluster
05:47 nishanth joined #gluster
05:48 ramteid joined #gluster
05:50 atalur joined #gluster
05:51 kdhananjay joined #gluster
05:53 sputnik13 joined #gluster
05:56 ppai joined #gluster
05:57 TvL2386 joined #gluster
06:00 SOLDIERz joined #gluster
06:01 soumya joined #gluster
06:13 overclk joined #gluster
06:18 anil joined #gluster
06:22 fubada hi purpleidea master of the gluster puppet module is failing
06:22 fubada missing gluster::versions
06:22 sputnik13 joined #gluster
06:23 fubada and theres a specific include for it in gluster::host line 101
06:23 fubada https://github.com/purpleidea/puppet-​gluster/blob/master/manifests/host.pp
06:24 fubada just fyi, thanks for an awesome module mang
06:39 nbalacha joined #gluster
06:43 sputnik13 joined #gluster
06:50 atalur joined #gluster
06:53 bala joined #gluster
06:58 meghanam joined #gluster
06:59 meghanam joined #gluster
07:00 meghanam joined #gluster
07:04 ctria joined #gluster
07:06 meghanam joined #gluster
07:08 soumya joined #gluster
07:09 soumya joined #gluster
07:09 lalatenduM joined #gluster
07:10 nishanth joined #gluster
07:12 kshlm joined #gluster
07:14 atalur joined #gluster
07:21 kovshenin joined #gluster
07:22 side_control joined #gluster
07:25 jtux joined #gluster
07:30 meghanam joined #gluster
07:33 sputnik13 joined #gluster
07:39 Fen2 joined #gluster
07:42 rgustafs joined #gluster
07:44 cultavix joined #gluster
07:45 DV joined #gluster
07:48 [Enrico] joined #gluster
07:56 pcaruana joined #gluster
08:02 _pol joined #gluster
08:19 rgustafs joined #gluster
08:19 LebedevRI joined #gluster
08:22 glusterbot News from newglusterbugs: [Bug 1175617] Glusterd gets killed by oom-killer because of memory consumption <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175617>
08:24 anoopcs joined #gluster
08:24 ricky-ti1 joined #gluster
08:24 saurabh joined #gluster
08:29 DV joined #gluster
08:41 jiffin joined #gluster
08:48 an joined #gluster
08:48 deniszh joined #gluster
08:53 Philambdo joined #gluster
09:01 Norky joined #gluster
09:17 Philambdo1 joined #gluster
09:21 Slashman joined #gluster
09:23 glusterbot News from newglusterbugs: [Bug 1175641] mount.glusterfs fails <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175641>
09:24 Fen2 joined #gluster
09:28 _shaps_ joined #gluster
09:29 liquidat joined #gluster
09:30 liquidat joined #gluster
09:38 sac`away joined #gluster
09:45 necrogami joined #gluster
09:45 T3 joined #gluster
09:46 jiffin joined #gluster
09:46 necrogami joined #gluster
09:46 necrogami joined #gluster
09:51 _pol joined #gluster
09:53 glusterbot News from newglusterbugs: [Bug 1175645] [USS]: Typo error in the description for USS under "gluster volume set help" <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175645>
09:53 tvb joined #gluster
09:53 tvb hi guys
09:53 warci joined #gluster
09:53 tvb is there a client version of 3.6.1 available for ubuntu 12.04?
09:54 warci howdy, i've updated gluster 3.4 to 3.5.3, but now suddenly our windows hosts see all shares as 'empty' through the nfs client
09:54 ghenry joined #gluster
09:54 ghenry joined #gluster
09:54 warci did something change in the implementation of nfs between these two versions?
09:55 warci i have -one- windows machine which still works though, but all the others see just empty shares without providing any errors
09:56 rgustafs joined #gluster
09:56 sac_ joined #gluster
09:59 prasanth_ joined #gluster
10:01 tvb is there a client version of 3.6.1 available for ubuntu 12.04?
10:10 an joined #gluster
10:16 _pol joined #gluster
10:34 DV joined #gluster
10:40 an joined #gluster
10:43 ricky-ticky1 joined #gluster
10:52 Pupeno joined #gluster
10:54 jiffin joined #gluster
10:55 anil joined #gluster
10:57 warci i've got this error in my nfs.log: [nfs3.c:4752:nfs3svc_fsinfo] 0-nfs-nfsv3: Error decoding arguments
10:57 meghanam joined #gluster
10:58 warci [rpcsvc.c:547:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
10:58 warci and the client gets an empty directory listing
11:05 warci basically it's the same as this old bug:
11:05 warci https://bugzilla.redhat.com/show_bug.cgi?id=763398
11:05 glusterbot Bug 763398: medium, low, 3.1.1, shehjart, CLOSED CURRENTRELEASE, GlustNFS is incompatible with Windows 7 NFS client.
11:09 tvb is there a client version of 3.6.1 available for ubuntu 12.04?
11:10 T3 joined #gluster
11:13 atalur joined #gluster
11:16 capri joined #gluster
11:21 kkeithley1 joined #gluster
11:24 meghanam joined #gluster
11:34 ricky-ticky joined #gluster
11:39 saurabh joined #gluster
11:40 soumya_ joined #gluster
11:40 the-me joined #gluster
11:49 elico joined #gluster
11:53 raghu joined #gluster
12:01 meghanam joined #gluster
12:01 kanagaraj joined #gluster
12:04 DV joined #gluster
12:08 Fen1 joined #gluster
12:11 fuzzy_id joined #gluster
12:11 fuzzy_id i'm trying to set the cache-timeout option of io-cache
12:12 fuzzy_id but gluster returns
12:12 fuzzy_id volume set: failed: option : io-cache-cache-timeout does not exist
12:12 fuzzy_id Did you mean md-cache-timeout?
12:12 fuzzy_id i'm using version 3.4.5
12:19 DV joined #gluster
12:20 tvb sigh I compiled glusterfs 3.6.1 on ubuntu 12.04 without issues.
12:20 tvb checking the version
12:20 diegows joined #gluster
12:20 tvb -bash: /usr/sbin/glusterfs: No such file or directory
12:20 tvb grr
12:21 tvb to be sure, I logged out and back in
12:21 tvb rechecked
12:21 tvb glusterfs: error while loading shared libraries: libglusterfs.so.0: cannot open shared object file: No such file or directory
12:23 glusterbot News from newglusterbugs: [Bug 1175692] [SNAPSHOT]: snapshoted volume is read only but it shows rw attributes in mount <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175692>
12:26 tvb hmm echo "%{_prefix}/lib64" > /etc/ld.so.conf.d/glusterfs.conf and 'ldconfig' seems to fixed it
12:29 soumya_ joined #gluster
12:32 hagarth1 joined #gluster
12:37 edward1 joined #gluster
12:47 calum_ joined #gluster
12:53 rolfb joined #gluster
12:54 glusterbot News from newglusterbugs: [Bug 1175711] os.walk() vs scandir.walk() performance <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175711>
12:58 rgustafs joined #gluster
13:06 harish joined #gluster
13:07 lpabon joined #gluster
13:10 ctria joined #gluster
13:14 ppai joined #gluster
13:17 B21956 joined #gluster
13:18 calisto joined #gluster
13:20 meghanam joined #gluster
13:24 fandi joined #gluster
13:31 theron joined #gluster
13:35 meghanam joined #gluster
13:41 ctria joined #gluster
13:42 kshlm joined #gluster
13:47 ppai joined #gluster
13:51 fandi joined #gluster
13:52 sysadmin-di2e @JoeJulian: what else would you try?  We rebooted both nodes and the wa: percentages in top are still hovering around 40-50%.
13:54 glusterbot News from newglusterbugs: [Bug 1175728] [USS]: All uss related logs are reported under /var/log/glusterfs, it makes sense to move it into subfolder <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175728>
13:54 glusterbot News from newglusterbugs: [Bug 1175730] [USS]: creating file/directories under .snaps shows wrong error message <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175730>
13:54 glusterbot News from newglusterbugs: [Bug 1175732] [SNAPSHOT]: nouuid is appended for every snapshoted brick which causes duplication if the original brick has already nouuid <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175732>
13:54 glusterbot News from newglusterbugs: [Bug 1175733] [USS]: If the snap name is same as snap-directory than cd to virtual snap directory fails <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175733>
13:54 glusterbot News from newglusterbugs: [Bug 1175735] [USS]: snapd process is not killed once the glusterd comes back <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175735>
13:54 glusterbot News from newglusterbugs: [Bug 1175736] [USS]:After deactivating a snapshot trying to access the remaining activated snapshots from NFS mount gives 'Invalid argument' error <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175736>
13:54 glusterbot News from newglusterbugs: [Bug 1175738] [USS]: data unavailability for a period of time when USS is enabled/disabled <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175738>
13:54 glusterbot News from newglusterbugs: [Bug 1175739] [USS]: Non root user who has no access to a directory, from NFS mount, is able to access the files under .snaps under that directory <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175739>
13:54 glusterbot News from newglusterbugs: [Bug 1175742] [USS]: browsing .snaps directory with CIFS fails with "Invalid argument" <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175742>
13:58 nbalacha joined #gluster
14:00 leo__ joined #gluster
14:00 sputnik13 joined #gluster
14:03 bene2 joined #gluster
14:07 loki27_ joined #gluster
14:07 loki_ joined #gluster
14:08 virusuy joined #gluster
14:08 virusuy joined #gluster
14:13 anoopcs joined #gluster
14:18 coredump|br joined #gluster
14:19 rwheeler joined #gluster
14:20 warci joined #gluster
14:21 bennyturns joined #gluster
14:23 warci joined #gluster
14:24 glusterbot News from newglusterbugs: [Bug 1165938] Fix regression test spurious failures <https://bugzilla.redhat.co​m/show_bug.cgi?id=1165938>
14:24 glusterbot News from newglusterbugs: [Bug 1175744] [USS]: Unable to access .snaps after snapshot restore after directories were deleted and recreated <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175744>
14:24 glusterbot News from newglusterbugs: [Bug 1175745] AFR + Snapshot : Read operation on  file in split-brain is successful in USS <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175745>
14:24 glusterbot News from newglusterbugs: [Bug 1175749] glusterfs client crashed while migrating the fds <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175749>
14:24 glusterbot News from newglusterbugs: [Bug 1175751] use different names for getting volfiles <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175751>
14:24 glusterbot News from newglusterbugs: [Bug 1175752] [USS]: On a successful lookup, snapd logs are filled with Warnings "dict OR key (entry-point) is NULL" <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175752>
14:24 glusterbot News from newglusterbugs: [Bug 1175753] [readdir-ahead]: indicate EOF for readdirp <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175753>
14:24 glusterbot News from newglusterbugs: [Bug 1175754] [SNAPSHOT]: before the snap is marked to be deleted if the node goes down than the snaps are propagated on other nodes and glusterd hungs <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175754>
14:24 glusterbot News from newglusterbugs: [Bug 1175755] SNAPSHOT[USS]:gluster volume set for uss doesnot check any boundaries <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175755>
14:24 glusterbot News from newglusterbugs: [Bug 1175756] [USS] : Snapd crashed while trying to access the snapshots under .snaps directory <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175756>
14:24 glusterbot News from newglusterbugs: [Bug 1175758] [USS] : Rebalance process tries to connect to snapd and in case when snapd crashes it might affect rebalance process <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175758>
14:24 glusterbot News from newglusterbugs: [Bug 1175765] [USS]: When snapd is crashed gluster volume stop/delete operation fails making the cluster in inconsistent state <https://bugzilla.redhat.co​m/show_bug.cgi?id=1175765>
14:24 glusterbot News from resolvedglusterbugs: [Bug 1104714] [SNAPSHOT]: before the snap is marked to be deleted if the node goes down than the snaps are propagated on other nodes and glusterd hungs <https://bugzilla.redhat.co​m/show_bug.cgi?id=1104714>
14:24 glusterbot News from resolvedglusterbugs: [Bug 1101463] [SNAPSHOT]: Stale entries of snapshots needs to be cleaned from /var/run/gluster/snaps/ <https://bugzilla.redhat.co​m/show_bug.cgi?id=1101463>
14:25 Fen1 joined #gluster
14:30 bala joined #gluster
14:38 warci quick other question,
14:38 warci i'm trying to upgrade my 3.4 -> 3.5 geo replication
14:38 warci so i run the script mentioned in the doc
14:38 warci bash generate-gfid-file.sh localhost:dvdimages $PWD/get-gfid.sh    /tmp/master_gfid_file.txt
14:39 warci as an example
14:39 warci but i get: find: `/usr/share/glusterfs/scripts/get-gfid.sh': Permission denied
14:40 sysadmin-di2e what are the permissions on that file?
14:41 warci -rw-r--r-- 1 root root  254 18. Dez 15:24 get-gfid.sh
14:41 glusterbot warci: -rw-r--r's karma is now -17
14:41 sysadmin-di2e what user are you running the scripts as?
14:41 warci haha silly glusterbot :D
14:41 warci root
14:42 sysadmin-di2e do a chmod u+x get-gfid.sh
14:42 warci i added a set -xv to both scripts, but i'm unsure what it's actually trying to do?
14:43 warci damn... it was that easy.. :$
14:43 warci thanks a bunch sysadmin-di2e
14:43 sysadmin-di2e :)
14:46 daxatlas joined #gluster
14:46 tdasilva joined #gluster
14:49 vimal joined #gluster
14:56 Jampy joined #gluster
14:56 Jampy Hi. Some gluster expert around here?
15:01 sac_ joined #gluster
15:07 mator each directory is repeated twice
15:08 mator can someone look http://fpaste.org/160974/14189152/
15:08 mator ? thanks
15:09 mator it's distributed volume mounted under /mnt/glusterfs
15:11 uebera|| joined #gluster
15:13 ctria joined #gluster
15:17 sysadmin-di2e I'm having the same error mator
15:18 mator sysadmin-di2e, i'm going to stop cluster, remove .glusterfs subdir and run it again
15:18 mator from each of the servers, bricks
15:18 calisto joined #gluster
15:19 mator lets see how it will come out
15:19 sysadmin-di2e mator: let me know how that turns out
15:20 mator i have a constant crashed on underlying fs (xfs) on the bricks... i've no idea whom to blame (hardware or OS)
15:21 mator so, xfs_check / xfs_repair "fixes" some files on xfs, and after that I have all kinds of filesystem wizardry like duplicated names =)
15:21 sysadmin-di2e mator: how long has that been happening?
15:22 mator sysadmin-di2e, flashed latest BIOS to HP Proliant servers today, flashed latest firmware to storage controllers a week earlier
15:23 sysadmin-di2e mator: are you getting a kernel panic?  If so, what message are you receiving?
15:23 mator i believe introduction of .glusterfs subdir to bricks maybe a good idea, but if it has some inconsistencies, like after fsck/xfs_repair, it will cause problems
15:23 mator sysadmin-di2e, kernel panic as well, before flashing latest bios to server
15:24 mator since I've written a new BIOS today, it's too short will it fix kernel panics or not
15:24 mator too short to tell
15:24 glusterbot News from newglusterbugs: [Bug 1170548] [USS] : don't display the snapshots which are not activated <https://bugzilla.redhat.co​m/show_bug.cgi?id=1170548>
15:24 mator ok, .glusterfs removed from all the servers... lets start
15:25 RameshN joined #gluster
15:28 jobewan joined #gluster
15:30 mator sysadmin-di2e, yeah, it fixed double listing
15:31 kshlm joined #gluster
15:31 kovshenin joined #gluster
15:31 mator sysadmin-di2e, the following page http://www.gluster.org/community/​documentation/index.php/Projects mentions gfsck tool, but I don't know anything about it, more than it written on this page
15:32 sysadmin-di2e mator: did it mess anything up that you know of?
15:33 mator sysadmin-di2e, brickN/.glusterfs is new starting from 3.3 version ... a week ago I was running 3.2.x hapilly, but outdated
15:33 mator http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
15:35 sysadmin-di2e mator: what version are you running?
15:35 mator sysadmin-di2e, current stable 3.5.3
15:35 wushudoin joined #gluster
15:36 sysadmin-di2e mator: Also, on which node did you remove the glusterfs file?  Did you stop glusterd and glusterfsd before removing it
15:36 mator besides, we only actually use distributed volume
15:36 sysadmin-di2e oh okay
15:36 mator sysadmin-di2e, stop glusterfs cluster (service glusterd stop; service glusterfs stop; pkill glusterfs)
15:37 mator for i in $servers; do ssh $i "find /exports -name .glusterfs | xargs rm -rf"; done
15:37 sysadmin-di2e oh okay.
15:37 mator start glusterfs cluster (service glusterd start; mount /mnt/glusterfs)
15:37 sysadmin-di2e and gluster created a new file for you?
15:37 mator a new .glusterfs subdir on each brick, yes
15:37 shubhendu joined #gluster
15:37 sysadmin-di2e thanks
15:38 mator sysadmin-di2e, http://fpaste.org/160986/91709414/
15:39 hagarth joined #gluster
15:39 mator sysadmin-di2e, http://fpaste.org/160988/89171471/
15:45 mator sysadmin-di2e, having 2 virtual machines as test cluster (for example with fedora 21) help a lot. You can do whatever you want :)
15:45 sysadmin-di2e mator: looks good. I wonder if you ran it on the bricks where the duplicate files were, would that correct the issue as well?
15:48 mator sysadmin-di2e, since it was directories, and not files, it was on each of the servers
15:49 mator in case of files, i usually stop cluster, decide what file should be left intact, remove duplicate, and start cluster again
15:49 mator probably not a proper way to handle it, but at least it fixes problem
15:50 mator it has official name as "split brain" but only for replicated volumes
15:50 mator and self-heal, afaik works only with replicated volumes
15:52 _pol joined #gluster
15:52 T3 joined #gluster
15:54 fubada purpleidea: hi!
15:55 fubada Im wondering if you had plans to add gluster::version to master, as it is included from gluster::host
15:55 fubada and currently failing
16:00 theron joined #gluster
16:00 nishanth joined #gluster
16:02 soumya_ joined #gluster
16:03 Norky joined #gluster
16:05 meghanam joined #gluster
16:06 _pol joined #gluster
16:13 coredump joined #gluster
16:15 plarsen joined #gluster
16:15 plarsen joined #gluster
16:15 plarsen joined #gluster
16:20 theron joined #gluster
16:25 theron_ joined #gluster
16:33 lmickh joined #gluster
16:33 Norky joined #gluster
16:34 daMaestro joined #gluster
16:41 theron joined #gluster
16:42 Jampy is it normal that booting a CT from a NFS storage (glusterfs at localhost) is extreeemely slow? I startet the CT 10 minutes ago and it's still walking slowly through the init scripts...
16:45 Jampy could it be that you folks don't see my messages?
16:45 JoeJulian Jampy: http://en.wikipedia.org/wiki/CT I can't figure out which one should apply.
16:45 purpleidea fubada: i don't understand, can you elaborate?
16:46 Jampy JoeJulian: OpenVZ container (just like in Proxmox it says "create CT")
16:46 Jampy whoops, sorry, my fault
16:46 JoeJulian And also everyone here has their own job to do. It's not uncommon for answers to take 10s of minutes or even more than an hour for someone to even read your question. Please be patient. Nobody here gets paid.
16:46 JoeJulian Well, at least not for being here.
16:47 purpleidea JoeJulian: exactly what i was going to say... man you're fast
16:47 JoeJulian and I haven't even had my coffee yet.
16:47 Jampy JoeJulian: no offense. sorry. it's the first time I'm using IRC with this client and I was wondering if the chat itself was accepting my messages.
16:47 purpleidea A lot of us get paid, just not necessarily for being in here... we do it because we like good freedom storage ;)
16:47 JoeJulian no worries.
16:48 sysadmin-di2e JoeJulian: I have tried the suggestions that you made yesterday about our 2 brick pairs that were performing very slow.  We restarted both of them and did a self heal.  No change in performance.
16:49 fubada purpleidea: gluster::version seems to be missing from master, you do an include on this class on line 101 of host.pp
16:49 Jampy OK, let my try again.... ;-)  when mounting a gluster volume (2 nodes, redundant setup) via glusterfs/fuse, directory listings are very slow. however, when mounting via NFS, things are much more faster (0.5 secs instead of 20) when reading directory contents. is that normal?
16:49 sysadmin-di2e What else would you suggest?
16:49 fubada purpleidea: which results in this https://gist.github.com/aa​merik/1faac84307ea92ab9983
16:50 fandi hi all
16:50 fandi i configure new gluster server
16:50 JoeJulian Jampy: Unless that directory has 10k+ items, that's really slow. Are you using at least a semi-recent kernel version? What version of glusterfs?
16:51 purpleidea fubada: ahhhh! you're right, sorry I forgot to git add it. Done and pushed to master. thanks!
16:51 fandi why always like http://fpaste.org/161014/89214801/
16:51 fubada thank you sir!
16:51 fubada purpleidea:
16:51 purpleidea fubada: yw, but thank you!
16:51 fandi this is my first error part of volume
16:52 fandi hope any better suggestion here :) thansk
16:52 JoeJulian sysadmin-di2e: Anything different about either side of those two pairs? Load, io throughput, hardware, network, environment?
16:52 fubada purpleidea: any idea when the recent work will end up on puppet forge?
16:52 fubada Ideally i like to avoid using master
16:52 purpleidea fubada: if you test git master and confirm it works, i'll roll a new release when you ACK it.
16:52 fubada one sec
16:53 Jampy JoeJulian: gluster 3.5. yes, about 20.000 files in ~1000 dirs. it's not high-end server hardware (just doing some tests). Using Kernel 2.6.32-34-pve (proxmox)
16:53 sysadmin-di2e @JoeJulian: It is the same as the other bricks.  We just added two more pairs to bring the total bricks up to 20.
16:54 fubada purpleidea: still see an issue
16:54 sysadmin-di2e The wait io on the vm is high at about 50%.  The Hypervisor and gluster node are very low at 0-1%
16:54 purpleidea fubada: which is...
16:54 fubada https://gist.github.com/aa​merik/d55b9e4c4473703d2c13
16:54 fubada are you using .split?
16:55 fubada future parser?
16:55 JoeJulian What's wrong with .split? I just suggested he use that this morning.
16:55 purpleidea fubada: whar ruby version are you using?
16:56 purpleidea fubada: this isn't an issue for me... it's ruby, in the template, not puppet
16:56 fandi hi all, it's ok to running gluster and webserver in the same hardware thanks
16:56 fubada purpleidea: 1.9
16:56 purpleidea JoeJulian: you suggested split to me ??
16:56 JoeJulian gluster-devel
16:56 purpleidea fubada: oh! wait
16:56 purpleidea fubada: this is a new bug... what puppet version are you using?
16:57 fubada purpleidea: puppetserver 0.4.0
16:57 purpleidea fubada: that's a version?
16:57 fubada its compliant with 3.7
16:57 fubada 3.7.3
16:57 purpleidea fubada: run facter -p | grep interfaces # to prove my hunch
16:58 TrDS joined #gluster
16:58 fubada interfaces => eth5,lo
16:58 purpleidea fubada: run facter -p | grep interfaces | fpaste # to prove my hunch
16:59 purpleidea fubada: so i'm running split in the template on @interfaces, which traditionally is a string, separated by commas... but i'm guessing if you see this error, either 1) you've overloaded @interfaces with a new value (that's an array) or two: puppet labs broke this backwards incompatibility, and now it's an array
16:59 fubada yea Im not messing with interfaces
17:00 fubada you could add some case statement based on validate_array/validate_string
17:00 fubada ?
17:00 purpleidea fubada: no, first please run:
17:01 purpleidea notice(inline_template('interfaces: <%= @interfaces.inspect %>'))
17:01 fubada one second
17:01 purpleidea fubada: on your puppet server in some file... and show me the output from /var/log/messages on master
17:02 Jampy JoeJulian: any idea?
17:03 JoeJulian conf call... bbiab
17:04 fubada purpleidea: https://gist.github.com/aa​merik/013b505c6431019de821
17:04 DV joined #gluster
17:07 purpleidea fubada: confirmed
17:07 kumar joined #gluster
17:08 purpleidea fubada: two issues: 1. interestingly 'lo' wasn't in that list, 2. they've changed it, thus breaking any existing piece of code. so IOW, it's puppetlabs fault. I'd report this as a critical regression, although I'm sure someone else has...
17:09 semiosis tvb: uploading to glusterfs-3.6 ppa now
17:09 semiosis for trusty
17:09 semiosis other releases to follow soon
17:09 tvb semiosis: nice
17:09 elico joined #gluster
17:10 tvb thanks
17:10 tvb reintalled with 3.5 tho
17:10 semiosis yw
17:12 fubada purpleidea: any temp fix suggestions or workarounds?
17:13 hagarth semiosis++ :)
17:13 glusterbot hagarth: semiosis's karma is now 2000007
17:14 purpleidea fubada: yeah, roll back to a stable version of puppet
17:14 hagarth wow, the karma looks like some op-version :D
17:15 fubada purpleidea: unfortunately not an option :( btw puppetserver is waht they ship with as a default under enterprise
17:16 fandi joined #gluster
17:17 diegows joined #gluster
17:19 purpleidea fubada: oh wow, seriously... well you're getting some unstable junk, imo. i figure this should cause a regression in * a lot * of code out there... it will require patching... you can send me the patch, or maybe i can look at it later, but there are many other places that might have similar issues too
17:20 purpleidea JoeJulian: ^^^ can you believe this shit, puppetlabs is changing the types of built in facts between point releases i guess...
17:28 JoeJulian Yes, I can.
17:29 semiosis i was horrified when i looked into the gory details of puppet
17:29 semiosis documented features that just plain didn't work, with bugs open for them sitting untouched for a long time, devs saying "meh" just dont use that feature
17:32 Slashman joined #gluster
17:32 _pol joined #gluster
17:34 necrogami joined #gluster
17:45 l0uis semiosis: any idea on eta of getting 3.5.3 into the ppa ?
17:45 purpleidea semiosis: lol... i've actually had puppet features dissapear, because they apparently we're "never supported" and suddenly got killed
17:46 skippy the new versions of Facter support "complex" facts.  i.e. facts are no longer constrained to strings.
17:48 skippy https://docs.puppetlabs.com/facter/2.3/fa​ct_overview.html#writing-structured-facts
17:51 purpleidea skippy: indeed i know, but they shouldn't just arbitrarily change the old strings to arrays... they were going to have "new" key names for the structured ones pre 4.0
17:52 purpleidea semiosis: btw, here's a patch i had to write because they "closed a loophole" read the commit message, lol https://github.com/purpleidea/puppet-gluster/c​ommit/06af205a562d543bbeb7c4d5c55143ade3bdb4e6
17:59 purpleidea fubada: so iow, want to write the patch? if not, at least open a bug or find the bug report about this issue please
18:01 fubada purpleidea: okay ill dig around, thank you
18:14 purpleidea fubada: yw
18:20 ron-slc joined #gluster
18:25 MacWinner joined #gluster
18:36 sputnik13 joined #gluster
18:54 coredump joined #gluster
19:05 ramon_dl joined #gluster
19:05 ramon_dl left #gluster
19:13 tom[] joined #gluster
19:29 ira joined #gluster
19:47 semiosis l0uis: what ubuntu release for 3.5.3?  i think trusty is already there
19:49 ricky-ti1 joined #gluster
19:51 Pupeno joined #gluster
19:56 Philambdo joined #gluster
19:56 calisto joined #gluster
19:56 Philambdo joined #gluster
20:00 jaank joined #gluster
20:01 sysadmin-di2e JoeJulian:  So we have isolated the problem to one brick.  We stop glusterd and glusterfsd on the node we suspected to have to problem  We ran top and noticed that the read and write times were below the other nodes.  Once we stopped the node, the other node, its pair started to perform normally.  It was performing slow as well until we disconnected it pair.  Do bricks read and write to the slowest brick within the pair?
20:02 sysadmin-di2e We will are provisioning another brick to take its place.
20:03 JoeJulian writes are synchronous so yes, the slowest.
20:03 sysadmin-di2e @JoeJulian.  What do you think could cause this?  Everything is the same between the servers.  There are no amber lights or warnings on the slow server
20:04 JoeJulian reads are (basically) first to respond by default, but in reality it's usually the leftmost (as listed in volume-create) brick that gets asked first and thus responds first.
20:05 sysadmin-di2e @JoeJulian, when we run the remove and replace brick.  Will Gluster know to self heal its files over to the new brick?
20:06 JoeJulian That read behavior is tunable setting cluster.read-hash-mode
20:06 JoeJulian Don't remove.
20:07 JoeJulian replace-brick blah blah commit force
20:07 JoeJulian and yes, it will start the heal right away.
20:07 sysadmin-di2e @JoeJulian:  Thank you.  You have been a great help
20:07 JoeJulian What could cause it? Network, disk, memory, vibration, power... those are all things I've personally experienced.
20:13 [o__o] joined #gluster
20:15 n-st joined #gluster
20:15 M28 is it okay to have a cluster of storage servers in a datacenter, and a cluster of client servers in a datacenter about ~15ms away?
20:16 _Bryan_ joined #gluster
20:17 JoeJulian purpleidea: No, no, no... I'm saying that the NOTE should tell people to announce any changes to op-version in gluster-devel. I think it's unfair to expect contributors to find you specifically to inform you. Just configure a filter in gluster-devel to make sure you notice it.
20:18 chirino joined #gluster
20:19 purpleidea JoeJulian: oh
20:19 purpleidea JoeJulian: any issues with putting my email address in there and saying please email me?
20:20 purpleidea JoeJulian: tbh, i don't think that's the worst thing in the world
20:20 JoeJulian What about any other people that affects? Should they all get specific emails? I'm just sayin'...
20:21 purpleidea JoeJulian: i mean, if that becomes the case, then yeah, but it's not so hard to send the same email to N people. email clients are magic these days :)
20:21 JoeJulian ... and I probably should have been talking about this in gluster-dev... taking it there.
20:21 purpleidea ya
20:28 chirino joined #gluster
20:36 chirino_m joined #gluster
20:47 fubada purpleidea: come to puppet for a sec?
20:50 DougBishop joined #gluster
20:51 calum_ joined #gluster
20:54 fubada purpleidea: mybad i thought there was a good chance to get some answers
20:54 purpleidea fubada: i'm filing a bug
21:00 l0uis semiosis: precise
21:00 l0uis semiosis: I dont see any 3.5.3 versions on the ppa
21:03 semiosis right
21:13 chirino joined #gluster
21:15 chirino joined #gluster
21:21 calum_ joined #gluster
21:31 sputnik13 joined #gluster
22:10 cyberbootje joined #gluster
22:12 cyberbootje joined #gluster
22:16 bene2 joined #gluster
22:34 calisto joined #gluster
22:38 gildub joined #gluster
22:43 Paul-C joined #gluster
23:00 ninkotech__ joined #gluster
23:00 the-me_ joined #gluster
23:06 DJClean joined #gluster
23:06 tom[] joined #gluster
23:07 ron-slc joined #gluster
23:07 jobewan joined #gluster
23:07 aulait joined #gluster
23:12 _pol joined #gluster
23:13 _pol_ joined #gluster
23:18 primusinterpares joined #gluster
23:46 sputnik13 joined #gluster
23:51 T3 joined #gluster
23:58 _pol joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary