Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-10-24

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 freemanbrandon joined #gluster
00:07 Zordrak joined #gluster
00:07 al joined #gluster
00:10 diegows joined #gluster
00:31 Rydekull joined #gluster
00:54 meghanam joined #gluster
00:54 meghanam_ joined #gluster
01:05 jbrooks joined #gluster
01:23 bennyturns joined #gluster
01:28 topshare joined #gluster
01:29 msmith joined #gluster
01:32 topshare_ joined #gluster
01:40 haomaiwa_ joined #gluster
01:48 sithik joined #gluster
01:50 21WAABCFG joined #gluster
01:52 sithik Hey guys, trying to wrap my head around what exactly I'm looking to do and how to go about doing it. Essentially I'm looking to have two webservers using the same data. Am I essentially just needing to have both machines be gluster clients AND servers?
01:53 bala joined #gluster
01:53 sithik I mean, technically I can have one gluster server, and two gluster clients and just have the webservers read from the gluster mount but I'm fairly certain having webservers reading and compiling files over a network mountpoint would be fairly silly?
01:54 harish joined #gluster
02:23 topshare joined #gluster
02:25 msmith joined #gluster
02:32 topshare joined #gluster
02:35 mojibake joined #gluster
02:40 topshare joined #gluster
02:49 David_H__ joined #gluster
02:51 topshare joined #gluster
03:05 haomaiw__ joined #gluster
03:06 meghanam joined #gluster
03:06 meghanam_ joined #gluster
03:10 sputnik13 joined #gluster
03:14 topshare joined #gluster
03:15 haomaiwa_ joined #gluster
03:16 mkasa joined #gluster
03:18 haomai___ joined #gluster
03:18 David_H_Smith joined #gluster
03:21 DV_ joined #gluster
03:24 Gugge joined #gluster
04:06 rjoseph joined #gluster
04:10 bennyturns joined #gluster
04:11 jwillis joined #gluster
04:13 sputnik13 joined #gluster
04:14 jwillis has anyone installed glusterfs server via semiosis package for ubuntu 14
04:16 JoeJulian jwillis: Lots of people. Why? What's up?
04:17 jwillis I can't figure out why glusterd does not show up in my /etc/init.d
04:18 jwillis there is glusterd in /usr/sbin, but i don't think this is the daemon...
04:21 calisto joined #gluster
04:23 jwillis JoeJulian: ?
04:29 davemc joined #gluster
04:34 JoeJulian jwillis: because ubuntu uses upstart. Startup configs are in /etc/init
04:39 mkasa I have a persisting self-heal problem on CentOS 6.5 with GlusterFS 3.5.2 (w/ replicated & distributed setting), which I do not know how to resolve.
04:39 mkasa I usually resolve files with self-heal failure or split-brain issue by removing the corresponding files in the underlying bricks (usually after saving the right one).
04:39 mkasa However, this time I couldn't find the corresponding file. "gluster v heal vol1 info heal-failed" says the following file has been unable to fix. (the file name and the gfid are modified a bit to protect real info.)
04:39 mkasa 2014-10-24 04:03:51 <gfid:0cc5bfa1-d764-4856-b​4cc-5e71e2c2b116>/ch7.txt
04:40 mkasa It is reported every ten minutes for over a month, even after rebooting, so I believe there must be something in the brick,
04:40 mkasa but the following commands returned nothing: "find /gluster_fuse_mount_point -name 'ch7.txt' returns nothing", "sudo ls -alR /path/to/brick | grep ch7.txt" (on every host), "sudo ls -alR /path/to/brick/.glusterfs/0c/c5/0cc​5bfa1-d764-4856-b4cc-5e71e2c2b116" (on every host)
04:40 overclk joined #gluster
04:42 mkasa I'd like to know how to resolve this and also how this gfid come to the log even though there is no such file and there is no parent dir with the shown gfid...
04:44 JoeJulian Interesting.
04:45 JoeJulian mkasa: Since it doesn't seem to exist on any of your bricks, that would probably be why it's failing to heal. The key that the self-heal daemon triggers from is in .glusterfs/indices/xattrop.
04:46 JoeJulian Theoretically if you remove the gfid file, it should stop trying.
04:46 mkasa Last resort I can think of might be attaching a debugger to the self-heal daemon and see what's happening, but this is a production filesystem so I'd like to avoid this if possible.
04:48 ramteid joined #gluster
04:49 kdhananjay joined #gluster
04:50 mkasa JoeJulian: I have no idea what .glusterfs/indices/xattrop does, but I'll take a look.
04:51 JoeJulian That's what it does.
04:51 JoeJulian Holds the indicators for files to automatically self-heal.
04:52 hagarth joined #gluster
04:52 mkasa I looked in the directory (.glusterfs/indices/xattrop) on the very host I saw a self-heal complaint in the log.
04:53 mkasa There are two files with zero bytes whose name looks like gfid, and another three whose name looks like 'xattrop-gfid'.
04:54 JoeJulian You want one that has the gfid being logged as a failed heal.
04:55 JoeJulian Any reference to that missing gfid can be removed.
04:55 mkasa Hmm, but how to find the gfid? The log says <gfid:0cc5bfa1-d764-4856-b​4cc-5e71e2c2b116>/ch7.txt, so I only know the gfid of the parent dir, but not the file itself. In addition, there is no such dir with that gfid..
04:56 JoeJulian Oh, I misread taht.
04:57 JoeJulian Next thing I would do would be to restart the self-heal daemon. pkill -f glustershd; then restart glusterd
04:58 JoeJulian ... which is perfectly safe in production.
04:58 ndarshan joined #gluster
04:59 mkasa Is there any difference between restarting the whole server and restarting only glustershd? I tried the former already.
04:59 JoeJulian wow
05:01 JoeJulian when you say you restarted the whole server, did you reboot or just stop and start the service?
05:01 mkasa Can I put the directory listing of .glusterfs/indices/xattrop here? (less than 10 lines)
05:02 JoeJulian no point
05:02 JoeJulian If you do want to share, though, please use fpaste.org
05:03 mkasa OK. Then, I don't put the list here until it is important.
05:05 JoeJulian just a sec... I'm testing a theory
05:06 mkasa What I did was, (1) unmounted all GlusterFS(via FUSE) mount points in all client hosts, (2) stopped the gluster daemons on the problematic host, (3) restarted the problematic host, (4) started the gluster daemons manually on the problematic host.
05:07 mkasa The volume isn't mounted to anywhere now, but it still complains every (approx.) 10 minutes. xattrop has 5 files, 2 of which look like gfid, the other 3 look like xattrop-<gfid>.
05:08 JoeJulian Ah, ok. They share data so the data's probably cached elsewhere.
05:09 mkasa I searched .glusterfs/xx/xx/gfid in all bricks on all hosts, and found that there 2, 1, 0, 0, 0 corresponding gfid files on the bricks.
05:10 JoeJulian Oh, so the gfid *does* exist.
05:10 lalatenduM joined #gluster
05:10 JoeJulian That's probably more useful.
05:11 JoeJulian ... how can it be in one twice? That doesn't make sense.
05:11 mkasa What I did not understand is that there are 5 gfids (w/ or wo/ "xattrop-" prefix) in .glusterfs/indices/xattrop but only one file is reported to be self-heal failed.
05:11 JoeJulian The other may be in a changing state.
05:11 JoeJulian Nothing to worry about.
05:12 mkasa The four files in xattrop are more than months old, so you mean it may take a year to change a state?
05:12 JoeJulian Good point
05:14 JoeJulian I would probably check the ,,(extended attributes) on the corresponding gfid file under .glusterfs/XX/YY/XXYY* to compare the trusted.afr attributes...
05:14 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
05:15 JoeJulian If that doesn't explain anything, I'd next look at the stat to look for differences.
05:15 JoeJulian The glustershd.log /should/ tell you why the heal failed.
05:16 JoeJulian Somewhere along the line it /may/ be useful to do a ,,(gfid lookup) to find what filename is associated.
05:16 glusterbot JoeJulian: Error: No factoid matches that key.
05:16 JoeJulian ... damn, wrong factoid
05:16 JoeJulian @gfid
05:16 glusterbot JoeJulian: The gfid is a uuid that's assigned to represent a unique inode that can be identical across replicas. It's stored in extended attributes and used in the .glusterfs tree. See http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/ and http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
05:17 JoeJulian @gfid resolver
05:17 glusterbot JoeJulian: https://gist.github.com/4392640
05:18 JoeJulian And I'm sorry, but my wife is calling me to the other room. Good luck and good night. I'll be back in the morning.
05:20 mkasa Thank you Joe! I've already read all your Gluster-related articles, so probably I'll find the answer in an hour. Good night. (here it's 2pm though)
05:25 nbalachandran joined #gluster
05:27 doekia joined #gluster
05:28 _zerick_ joined #gluster
05:38 kdhananjay joined #gluster
05:40 davidhadas_ joined #gluster
05:43 overclk joined #gluster
06:06 mkasa OK. Resolved!
06:06 mkasa Let me wrap up the problem. (for others who have similar problems)
06:07 rgustafs joined #gluster
06:08 Slydder joined #gluster
06:09 mkasa (1) I wish to know the gfid of a "phantom" file, which is repeatedly reported as a "heal-failed" file by glusterd but there is no such file from the viewpoint of users.
06:13 mkasa (2) The phantom file actually had a gfid, but its file name was presumably cached to somewhere, so the log file shows only the file name that does not exist any longer.
06:14 sputnik13 joined #gluster
06:17 mkasa (3) Files in .glusterfs/indices/xattrop triggers self-heal, so we can look there to identify which files are being healed.
06:20 mkasa (4) I looked all the underlying bricks, searching for the files w/ the gfids in .glusterfs/indices/xattrop, and found the culprit.
06:23 mkasa (5) I removed the file with that gfid, and the self-heal error disappeared. Done.
06:30 Philambdo joined #gluster
06:44 ndarshan joined #gluster
06:45 Fen1 joined #gluster
06:52 nbalachandran joined #gluster
06:53 kdhananjay joined #gluster
06:58 ctria joined #gluster
07:00 side_control joined #gluster
07:00 brettnem joined #gluster
07:00 portante joined #gluster
07:00 ndk joined #gluster
07:00 Humble joined #gluster
07:00 Yossarianuk joined #gluster
07:00 tomased joined #gluster
07:00 hchiramm_ joined #gluster
07:00 cmtime joined #gluster
07:00 suliba_ joined #gluster
07:00 SpeeR joined #gluster
07:00 samsaffron___ joined #gluster
07:00 churnd joined #gluster
07:00 _NiC joined #gluster
07:00 marcoceppi joined #gluster
07:00 eryc joined #gluster
07:00 the-me joined #gluster
07:00 eshy joined #gluster
07:00 sauce joined #gluster
07:00 nage joined #gluster
07:00 Slasheri joined #gluster
07:00 NuxRo joined #gluster
07:00 sage joined #gluster
07:00 hflai joined #gluster
07:00 atrius` joined #gluster
07:00 Gib_adm joined #gluster
07:00 vincent_vdk joined #gluster
07:00 kke joined #gluster
07:01 ricky-ti1 joined #gluster
07:08 hchiramm joined #gluster
07:10 bennyturns joined #gluster
07:10 MrAbaddon joined #gluster
07:10 kedmison joined #gluster
07:10 ccha joined #gluster
07:10 johnmark joined #gluster
07:10 eightyeight joined #gluster
07:10 abyss^^_ joined #gluster
07:10 ur joined #gluster
07:10 charta joined #gluster
07:10 kkeithley joined #gluster
07:10 Norky joined #gluster
07:10 SteveCooling joined #gluster
07:10 edong23 joined #gluster
07:10 semiosis joined #gluster
07:10 jaroug joined #gluster
07:10 georgeh joined #gluster
07:10 prasanth|brb joined #gluster
07:10 guntha_ joined #gluster
07:10 msvbhat joined #gluster
07:10 kodapa joined #gluster
07:10 foster joined #gluster
07:10 T0aD joined #gluster
07:10 partner joined #gluster
07:10 jiqiren joined #gluster
07:11 sputnik1_ joined #gluster
07:16 samsaffron___ joined #gluster
07:16 eshy joined #gluster
07:16 cjanbanan joined #gluster
07:16 7GHAAFGTM joined #gluster
07:32 7GHAAFGTM joined #gluster
07:32 kke joined #gluster
07:32 sage joined #gluster
07:32 nage joined #gluster
07:32 churnd joined #gluster
07:32 SpeeR joined #gluster
07:32 suliba_ joined #gluster
07:32 cmtime joined #gluster
07:32 tomased joined #gluster
07:32 Yossarianuk joined #gluster
07:32 portante joined #gluster
07:32 bennyturns joined #gluster
07:32 MrAbaddon joined #gluster
07:32 kedmison joined #gluster
07:32 ccha joined #gluster
07:32 johnmark joined #gluster
07:32 eightyeight joined #gluster
07:32 abyss^^_ joined #gluster
07:32 ur joined #gluster
07:32 charta joined #gluster
07:32 kkeithley joined #gluster
07:32 Norky joined #gluster
07:32 SteveCooling joined #gluster
07:32 edong23 joined #gluster
07:32 semiosis joined #gluster
07:32 jaroug joined #gluster
07:32 georgeh joined #gluster
07:32 prasanth|brb joined #gluster
07:32 guntha_ joined #gluster
07:32 msvbhat joined #gluster
07:32 kodapa joined #gluster
07:32 foster joined #gluster
07:32 T0aD joined #gluster
07:32 partner joined #gluster
07:32 jiqiren joined #gluster
07:35 fsimonce joined #gluster
07:36 Fen1 Hi all :)
07:37 side_control joined #gluster
07:37 brettnem joined #gluster
07:37 ndk joined #gluster
07:37 hchiramm_ joined #gluster
07:37 _NiC joined #gluster
07:37 marcoceppi joined #gluster
07:37 eryc joined #gluster
07:37 the-me joined #gluster
07:37 sauce joined #gluster
07:37 Slasheri joined #gluster
07:37 NuxRo joined #gluster
07:37 hflai joined #gluster
07:37 atrius` joined #gluster
07:37 Gib_adm joined #gluster
07:37 vincent_vdk joined #gluster
07:37 JordanHackworth joined #gluster
07:40 portante joined #gluster
07:40 Yossarianuk joined #gluster
07:40 tomased joined #gluster
07:40 cmtime joined #gluster
07:40 suliba_ joined #gluster
07:40 SpeeR joined #gluster
07:40 churnd joined #gluster
07:40 nage joined #gluster
07:40 sage joined #gluster
07:40 kke joined #gluster
07:40 cjanbanan joined #gluster
07:40 lalatenduM joined #gluster
07:44 mkasa pkill
07:45 mkasa Oops! Typed in a wrong window.
07:50 samsaffron___ joined #gluster
07:50 eshy joined #gluster
07:56 ndarshan joined #gluster
08:03 rgustafs joined #gluster
08:05 liquidat joined #gluster
08:16 hchiramm joined #gluster
08:18 ricky-ticky1 joined #gluster
08:33 haomaiwang joined #gluster
08:41 ricky-ticky joined #gluster
08:46 davidhadas joined #gluster
08:51 rafi1 joined #gluster
08:52 haomaiwa_ joined #gluster
08:55 ivok joined #gluster
08:57 haomai___ joined #gluster
09:14 vimal joined #gluster
09:20 elico joined #gluster
09:29 ndarshan joined #gluster
09:31 tryggvil joined #gluster
09:36 badone joined #gluster
09:36 elico left #gluster
09:46 ndarshan joined #gluster
09:47 ackjewt joined #gluster
09:49 rgustafs joined #gluster
09:53 bjornar joined #gluster
09:53 dockbram joined #gluster
09:53 apscomp joined #gluster
09:53 AaronGr joined #gluster
09:53 clutchk joined #gluster
09:53 nixpanic joined #gluster
09:53 Intensity joined #gluster
09:56 hchiramm joined #gluster
09:56 portante joined #gluster
09:56 Yossarianuk joined #gluster
09:56 tomased joined #gluster
09:56 cmtime joined #gluster
09:56 suliba_ joined #gluster
09:56 SpeeR joined #gluster
09:56 churnd joined #gluster
09:56 nage joined #gluster
09:56 sage joined #gluster
09:56 kke joined #gluster
09:58 rafi1 joined #gluster
09:58 and` joined #gluster
09:58 Thilam joined #gluster
09:58 UnwashedMeme1 joined #gluster
09:58 russoisraeli joined #gluster
09:58 1JTAAQ5CI joined #gluster
09:58 dblack joined #gluster
09:58 wgao joined #gluster
09:58 jvandewege joined #gluster
09:58 Telsin joined #gluster
09:58 purpleidea joined #gluster
09:58 JustinClift joined #gluster
09:58 sman joined #gluster
09:58 codex joined #gluster
09:58 ttk joined #gluster
10:03 Fen1 joined #gluster
10:11 ndarshan joined #gluster
10:16 Slashman joined #gluster
10:34 davidhadas joined #gluster
10:35 XpineX__ joined #gluster
10:38 XpineX_ joined #gluster
10:54 edward1 joined #gluster
11:03 calisto joined #gluster
11:05 mariusp joined #gluster
11:12 mariusp joined #gluster
11:26 tryggvil joined #gluster
11:38 calisto joined #gluster
11:38 LebedevRI joined #gluster
11:41 mojibake joined #gluster
11:44 julim joined #gluster
11:50 diegows joined #gluster
11:57 B21956 joined #gluster
11:59 B21956 joined #gluster
12:00 Fen1 joined #gluster
12:00 haomaiwa_ joined #gluster
12:01 Slashman_ joined #gluster
12:11 haomaiwa_ joined #gluster
12:11 elico joined #gluster
12:15 virusuy joined #gluster
12:15 calisto joined #gluster
12:21 glusterbot New news from newglusterbugs: [Bug 1156404] geo-replication fails with dispersed volumes <https://bugzilla.redhat.co​m/show_bug.cgi?id=1156404> || [Bug 1156405] geo-replication fails with dispersed volumes <https://bugzilla.redhat.co​m/show_bug.cgi?id=1156405>
12:21 Norky joined #gluster
12:32 bene joined #gluster
12:51 sithik joined #gluster
12:52 sithik Hey guys if I first peer probe'd a secondary server, then ran      gluster volume create volume1 replica 2 transport tcp 10.132.1.1:/gluster-storage 10.132.1.2:/gluster-storage force    and then started the volume on one of the servers, both servers should replicate each other, correct?
12:53 sithik while both servers read the same directory structure, show the same files, etc when one of them is offline the other seems to hang when trying to ls the /gluster-storage directory? Shouldn't then "work" independantly of each other?
12:54 rjoseph joined #gluster
12:57 MrAbaddon joined #gluster
13:01 Philambdo joined #gluster
13:20 sputnik13 joined #gluster
13:22 kkeithley sithik: servers don't replicate to each other on a 'replica 2' volume.  When a client mounts (fuse, a.k.a. gluster native) the client writes to each replica.  If one of the servers is off-line, the client will hang for a bit (42 seconds is the default) the first time a write occurs. After that client writes won't hang.
13:24 sithik Hmm, I don't think I installed fuse at all with this config? I think literally both are running as bricks?
13:25 kkeithley how and where are you mounting the volume?
13:25 kkeithley on a third machine, I presume.
13:25 sithik Nope?
13:26 theron joined #gluster
13:26 kkeithley so.... how and where are you mounting the volume?
13:26 sithik Let me explain, on the secondary server I literally only yum installed glusterfs-server, and then put localhost:/volume1   /storage-pool   glusterfs defaults,_netdev 0 0 in /etc/fstab
13:27 kkeithley then you are using gluster native for the mount.
13:28 sithik On the "primary" (for sake of easy explaining, using the word primary) I did: gluster peer probe 10.132.1.2 then gluster volume create volume1 replica 2 transport tcp 10.132.1.1:/gluster-storage 10.132.1.2:/gluster-storage force and then gluster volume start volume1
13:28 sithik restarted both, and both are showing the same files/directory structure in /storage-pool
13:28 kkeithley so far so good
13:30 kkeithley and as I indicated previously, if one server goes off line, the client will hang for 42 seconds the first time you try I/O after it goes off line. Once the client has figured out the server is down it won't hang after that
13:31 kkeithley That 42 seconds is the default TCP timeout
13:31 kkeithley you can shorten it, but you don't want it to be too short
13:33 sithik Sorry, phone call... but yea, everything "works", but with the way it's setup now...
13:34 michaellotz kkeithley: http://linux.die.net/man/7/tcp says 60 sec. timeout
13:34 glusterbot Title: tcp(7): TCP protocol - Linux man page (at linux.die.net)
13:34 sithik Ultimately here's what I'm looking to do. I have multiple nginx/php-fpm servers. php-fpm needs to have files residing locally so I'm just looking to make sure all files are sync'ed at all times, and even with a 5 second timeout, there'd still be a 5 second "hang"/no-availability of that site if ANY(?) of the servers went offline?
13:35 sithik I guess REALLY what I'm curious of is the best way to achieve a sync'ed filestate, with 0 down-time (if possible?) if any of the nginx/php-fpm/gluster servers went offline?
13:36 Humble joined #gluster
13:36 sithik Oh, and when I say nginx/php-fpm/gluster I don't mean "hey I need help load-balancing". I already got that covered. I said it because all three of those services appear to need to be ran on every webserver.
13:41 kumar joined #gluster
13:45 mariusp joined #gluster
13:51 kkeithley client writes to a 'replica N' volume are synchronous.  Whatever is doing the write, the write(2) syscall does not return until it has been sent to all the servers participating in the replica volume. Gluster has similar semantics to POSIX though, i.e. applications may need to fsync() or use O_SYNC on file open(2) or creat(2) if they need guarantees that the data has actually reached the disk at any point.
13:53 sithik Would you recommend against having a master (gluster-server) and then all clients (fuse) just mount the gluster-server filesystem? Not actually having any of the files stored locally at all. That, to me is the best solution, but I'm unsure just how bad read performance would be. :-/
13:56 haomaiw__ joined #gluster
13:59 msmith joined #gluster
14:00 theron joined #gluster
14:02 kkeithley I'm not sure what you're asking. You'll have to benchmark your particular workload to know what the performance will be and tune accordingly. For lots of small files, many people recommend using gluster's NFS because the kernel (on the client) caches more aggressively than gluster native.
14:04 CP|AFK joined #gluster
14:08 theron_ joined #gluster
14:10 gehaxelt joined #gluster
14:10 chirino joined #gluster
14:10 tdasilva joined #gluster
14:10 _dist joined #gluster
14:16 plarsen joined #gluster
14:23 lmickh joined #gluster
14:36 jobewan joined #gluster
14:42 _Bryan_ joined #gluster
14:49 tdasilva joined #gluster
14:55 mariusp joined #gluster
14:59 haomaiwa_ joined #gluster
15:01 bluefoxxx joined #gluster
15:02 bluefoxxx If I run 'ls' on a subfolder in my volume, ls hangs
15:02 burn420 joined #gluster
15:02 bluefoxxx gluster> volume heal wowza info
15:02 bluefoxxx operation failed
15:02 bluefoxxx ^^^ this also happens
15:02 bluefoxxx How do I track this issue down?
15:03 bluefoxxx it's version 3.3.1 on CentOS 6
15:03 bluefoxxx Rebooting doesn't fix it.
15:04 _dist read logs in /var/log/glusterfs
15:04 _dist I honestly don't remember which ones will be important in 3.3.1, but probably the brick log, and glusterd.vol.log
15:05 bluefoxxx [2014-10-24 10:57:24.316896] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: 1443649e-e782-447e-970f-b208ecca4714, lock held by: 1443649e-e782-447e-970f-b208ecca4714
15:06 bluefoxxx ... even with rebooting of all nodes in the cluster, apparently something is holding a cluster lock
15:07 haomai___ joined #gluster
15:09 bluefoxxx ok I have no idea how to fix this
15:12 _dist bluefoxxx: I had that problem before
15:12 _dist bluefoxxx: mine was because a client didn't have access to all the bricks due to a DNS issue
15:13 bluefoxxx ALL clients?
15:13 bluefoxxx they're all hard-coded into the hosts file.
15:13 _dist bluefoxxx: no I mean, a client didn't have access to all bricks and it was writing files and things got messed cause it couldn't fix stuff (I'm sure there are other reasons locks get stuck)
15:13 bluefoxxx okay, how do I *fix* it?
15:14 _dist bluefoxxx: But in the end you've got a file that the volume thinks someone is holding, you can find out who by that uuid "held by uuid"
15:14 _dist bluefoxxx: I'm not a glusterfs expert, but when I fixed it I had to do a statedump to get the ID for the file to perform a clear lock command and it fixed it
15:14 _dist I was on version 3.4.1 but it shouldn't be that much different
15:15 bluefoxxx can you give me a guide to doing that?
15:15 _dist well I can't be 100% since I haven't used 3.3.1 in forever. type "gluster volume help" and it'll give you a list of all the gluster commands
15:15 _dist one of them should be statedump
15:16 _dist however, you'll likely first have to change the statedump directory (or location), it'll be a gluster volume set option "gluster volume set help | grep state" should show it
15:16 _dist https://access.redhat.com/documenta​tion/en-US/Red_Hat_Storage/2.0/html​/Administration_Guide/ch21s02.html
15:16 glusterbot Title: 21.2. Troubleshooting File Locks (at access.redhat.com)
15:16 bluefoxxx ah
15:16 bluefoxxx volume statedump just gives empty set.
15:17 _dist is that what it says? I think in mine I had to tell it where to go, are you getting an empty file?
15:17 _dist I'm not sure if that redhat article is exactly what I followed, but it looks very close
15:18 bluefoxxx thanks, this looks helpful
15:18 _dist bluefoxxx: I know it's scary when stuff like this happens, did you have some kind of event that caused the problem? I'm curious in case it happens to me
15:19 bluefoxxx no, it just happened
15:19 bluefoxxx We only use gluster in a limited capacity because it always does stuff like this.
15:19 _dist hmm, well something must have broke but it wasn't obvious
15:19 rwheeler joined #gluster
15:20 _dist bluefoxxx: I suspect that might be because of your version, we're running 3.5.2 as backend storage for 800k+ files and 40 live VMs 24/7.
15:20 _dist btw, despite what the tutorials tell you, you can't add/remove/rebalance bricks while live for replication (we use a 3-way replica)
15:20 _dist that's supposed to be fixed in 3.5.3 I've been told
15:22 bluefoxxx Volume clear-locks unsuccessful
15:22 harish joined #gluster
15:22 bluefoxxx I wonder what would happen if I just upgraded gluster
15:22 bluefoxxx destroy all data?
15:23 _dist bluefoxxx: someone in here yesterday did a 3.3.1 --> 3.5.2 upgrade. As long as your volume is offline it should be ok. Personally I would rebuild a new volume but he did say it worked
15:23 bluefoxxx offline?
15:23 _dist right, "gluster volume stop volname" before the upgrade
15:23 bluefoxxx ah
15:23 bluefoxxx _dist,  yes but I mean would it balk at the broken state atm
15:24 _dist right, oh I'm pretty sure you have to clear the locks offline as well, but I didn't re-read that tutorial
15:24 bluefoxxx oh.
15:24 _dist the peer uuid is a present in your cluster still right?
15:24 coredump joined #gluster
15:24 bluefoxxx yeah
15:28 _dist bluefoxxx: also something different is we run our bricks on debian & ubuntu, but I since RH does the dev I can only imagine cent/fedora/rhel being bettter
15:28 _dist bluefoxxx: and our bricks on our ZFS, but again those complications should make our setup _less_ stable not more
15:29 davemc 53 responses to the gluster survey at https://t.co/Cosm63qWY6 i so far. If you haven't responded yet, could you  take a couple of minutes to do so?
15:29 glusterbot Title: GlusterFS use survey (at t.co)
15:29 bluefoxxx stopping and starting the volume fixed it
15:29 bluefoxxx I think.
15:29 bluefoxxx I don't friggin' know
15:31 clutchk1 joined #gluster
15:31 _dist bluefoxxx: haha, yes I understand that frustration but at least it did fix it
15:31 bluefoxxx no, you don't get it.  Do you know what historical information is?
15:32 bluefoxxx Work performance information, lessons learned, etc.
15:32 bluefoxxx what worked, what didn't work, how we fixed the stuff that didn't work, what we'd do different in the future
15:32 bluefoxxx stuff that magically unbreaks itself while I'm trying to grope for a handhold can't be categorized in that way :|
15:32 bluefoxxx so when it happens again, I am going to be like, oh crap.
15:33 tdasilva joined #gluster
15:33 bluefoxxx it's like vampire romance novels
15:33 bluefoxxx the vampire has no pulse, how the heck can there be any "romance" as such?
15:33 bluefoxxx you know you need bloodflow to get those parts to work
15:33 bluefoxxx and you just sit there confused
15:34 bluefoxxx that's how I feel right now
15:34 bluefoxxx this is why Sanderson is so great:  he doesn't write vampire novels.  Everything he writes makes sense.
15:34 * bluefoxxx ramble
15:35 nixpanic joined #gluster
15:36 _dist bluefoxxx: I do get it :)
15:38 bluefoxxx heh :)
15:38 bluefoxxx _dist,  I am studying project management and obsessive about a few things.
15:39 _dist bluefoxxx: that's good, maybe not the obsessive part but you've gotta have drive to work and learn simultaneously. Nothing make me more angry than when a problem "dissapears"
15:41 bluefoxxx I'm actually hedging.
15:41 bluefoxxx I'm studying project management and politics.  I want to run for State Representative.
15:41 bluefoxxx so replace "good" or "obsessive" with "crazy" or "sociopath" or "a road to no human contact" or something I don't know
15:42 freemanbrandon joined #gluster
15:44 calisto joined #gluster
15:58 freemanbrandon left #gluster
15:58 theron joined #gluster
16:02 dockbram joined #gluster
16:02 ron-slc joined #gluster
16:02 lanning joined #gluster
16:02 davemc joined #gluster
16:03 asku joined #gluster
16:04 RioS2 joined #gluster
16:14 rotbeard joined #gluster
16:15 kanagaraj joined #gluster
16:21 Humble joined #gluster
16:36 sputnik13 joined #gluster
16:38 JoeJulian bluefoxxx: I'll help you here as long as you push my agenda... Public entities should only use public licensed software. If a software package the state/county/city needs doesn't exist, they should have a department that works with communities to help make it exist, in the open-source way. ;)
16:39 davemc okay, the web site is deliberately hiding things from me.  Where are the 3.6.0 beta downloads?
16:42 JoeJulian packaged, or source?
16:42 davemc at this point either
16:43 JoeJulian @beta
16:43 glusterbot JoeJulian: I do not know about 'beta', but I do know about these similar topics: 'beta-yum', 'yum-beta'
16:43 JoeJulian @yum-beta
16:43 glusterbot JoeJulian: The official community glusterfs packges for RHEL 6 (including CentOS, SL, etc.), Fedora 17-19 (i386, x86_64, arm, armhfp), and Pidora are available at http://goo.gl/LGV5s
16:43 JoeJulian wow, that's an old factoid...
16:44 hagarth davemc: http://download.gluster.org/pub/glus​ter/glusterfs/nightly/glusterfs-3.6/
16:44 JoeJulian still valid though
16:44 glusterbot Title: Index of /pub/gluster/glusterfs/nightly/glusterfs-3.6 (at download.gluster.org)
16:44 JoeJulian nightly??? That's new.
16:44 hagarth davemc: this one is more apt - http://download.gluster.org/pub​/gluster/glusterfs/qa-releases/
16:44 glusterbot Title: Index of /pub/gluster/glusterfs/qa-releases (at download.gluster.org)
16:44 bluefoxxx JoeJulian, are you asking if we pay for glusterfs support here?
16:44 bluefoxxx we don't
16:44 JoeJulian No....
16:45 hagarth JoeJulian: right .. qa-releases is the location where we host most/all pre-release versions
16:45 bluefoxxx my employer has steak tastes and hotdog money.
16:45 davemc thanks. I never would have found it.
16:45 JoeJulian I'm just throwing out my own political agenda to you since you said you want to run for office.
16:46 * davemc thinks a good hot dog sounds really nice right now
16:46 JoeJulian You'll get help either way. The winking emoticon suggested that I was being facetious about not helping.
16:47 JoeJulian Man... I wish I knew why this one glusterfsd was using 1500% cpu.
16:48 cultav1x joined #gluster
16:48 bluefoxxx JoeJulian,  oh psh.  I'm working on social policy.  It's very simple and logical:  I've solved poverty, so the next step is to actually erase that blight from our society.
16:49 JoeJulian lol
16:49 JoeJulian solve poverty
16:50 bluefoxxx It's simple.
16:50 bluefoxxx Humans are greedy.  You create a market condition where you can profit from the poor in the only logical way:  by providing them food and shelter.
16:50 JoeJulian They'll still be poor.
16:50 bluefoxxx Our welfare system is inefficient as all hell, and costs $1.6 trillion here.  Working out the numbers, you get just about enough.
16:50 bluefoxxx oF course they'll still be poor.
16:50 torbjorn__ joined #gluster
16:51 bluefoxxx The bottom is always the bottom, no matter how high you raise it; and market forces cause raising the bottom to do nothing once you've defined *A* bottom.
16:51 JoeJulian Wow... a politician that actually will say that.. amazing.. :)
16:51 bluefoxxx But they won't be in danger of losing their homes, of starvation, of losing our haughty definition of the basic human right to life.
16:52 bluefoxxx If humans have a basic right to life in our society, why don't we ensure they don't die as a function of our society?
16:52 bluefoxxx That sounds all well-and-good, but you have to have an actual plan and the money to put it through, without creating a fiscal disaster.
16:52 bluefoxxx I've found both.
16:53 JoeJulian Have you published this anywhere?
16:53 rotbeard joined #gluster
16:53 bluefoxxx It didn't exist before.  Wealth grows over time:  we find more efficient ways to do things while expending less energy, so the real cost of things goes down, and the same effort creates more.
16:53 bluefoxxx Wealth has grown to feasibility.  It's time.
16:53 Intensity joined #gluster
16:54 bluefoxxx JoeJulian, no.  Barely.  Rough napkin sketches of the numbers.  Nothing coherent yet.
16:54 JoeJulian darn. I'd like to read it when it's written up.
16:54 bluefoxxx JoeJulian,  it's basically the expansion of social security, with grandfathering periods and strange taxation schemes.
16:55 semiosis davemc: ,,(qa releases)
16:55 glusterbot davemc: QA releases are now available here: http://goo.gl/c1f0UD -- the old QA release site is here: http://goo.gl/V288t3
16:55 bluefoxxx Get rid of the OASDI 6.2% (it's higher now) tax that funds social security retirement, as well as the proportion of income taxes that pay for a specific variety of welfare services that amount to paying people for unemployment, houses, and food if they're poor
16:56 bluefoxxx put in a 14.5% tax on ALL INCOME instead.  Reap $1.72T, divide among all natural-born American citizens who have attained the age of 18.  In 2013 numbers, that's $593/mo.
16:56 bluefoxxx The cost of renting a 750sqft apartment is $725/mo, under $1/sqft.  A single person can live in a well-designed 224sqft apartment; assuming $300/mo, there's enough money left for all other basic needs.
16:57 bluefoxxx And it grows with inflation.  It's never taken away--all welfare traps evaporate.
16:57 glusterbot bluefoxxx: away's karma is now -1
16:57 bluefoxxx Very simple.
16:57 semiosis sounds like foxconn
16:58 bluefoxxx The major imperative is that you examine the actual cost of supplying services, and estimate risks, and control them, so that services can be supplied in the given range.
16:58 bluefoxxx If you can't actually supply livable space to someone with such little income, it doesn't work.
16:59 bluefoxxx There are a numbero f considerations there.  Besides just raw cost, there's also the matter of wage garnering:  this dividend is legally exempt from garnering, because it is your right to live.
16:59 bluefoxxx There is also the consideration that a person may arrange to have part of his dividend diverted to a third party, removing risk from low-income housing landlords:  they WILL get paid FIRST.
16:59 bluefoxxx these reduce risk, and thus cost, and thus make it possible to provide those goods cheaper.
17:00 bluefoxxx anyway
17:00 bluefoxxx long, incoherent, rough overview
17:00 bluefoxxx and this isn't #politics
17:00 bluefoxxx you have sidetracked me; I was intending to go get a cup of hot cocoa.
17:01 bluefoxxx JoeJulian, it's basically a Citizen's Dividend pegged to a percentage of the economy, rather than some arbitrary land tax.
17:02 bluefoxxx Unemployed labour is important to an economy:  0% unemployment would seize up business, with no labour to hire with changes in demand or new opportunities
17:02 semiosis bluefoxxx: may i suggest a blog post for this?
17:02 bluefoxxx semiosis,  I have my own web site; I've just been lazy too much
17:03 pdrakeweb joined #gluster
17:03 bluefoxxx and either nobody eles is talking, or i've flooded them out :|
17:03 bluefoxxx hot cocoa time
17:19 17WAACAR9 joined #gluster
17:19 7F1ABO2FF joined #gluster
17:28 _zerick_ joined #gluster
17:31 PeterA joined #gluster
17:32 n-st_ joined #gluster
17:35 Philambdo joined #gluster
17:43 lpabon joined #gluster
17:49 _dist does anyone know when libgfapi repaths? I'm certain I have an example where it hasn't for weeks and it's doing something silly. When does qemu do that?
17:50 JoeJulian _dist: I'm not uncertain that there's a problem in that somewhere, based on some of the inconsistent bugs I've heard reported.
17:51 _dist how do I tell it to do what I want :)
17:52 pdrakewe_ joined #gluster
17:52 ricky-ticky joined #gluster
17:55 _dist JoeJulian: basically, after I do a compute migration the running qemu still paths to the same storage server as always. If I take that storage server down libgfapi waits 30 seconds or so to time out. This puts hosts in RW etc, this was NOT a problem until an upgrade of qemu to 2.1.2 from 2.1.0 so I have to assume it's either qemu's implmentation of libgfapi or a bug in the libgfapi qemu was compiled over
17:56 JoeJulian Is that true even if you stop glusterfsd with kill? Sounds like the client isn't getting the RST.
17:57 _dist JoeJulian: it is
17:57 _dist JoeJulian: though the evidence of that is anecdotal, but I'm 90% certain. It wasn't last week that I realized what was going on for certain
17:57 JoeJulian It's the same client translator so it should operate the same way.
17:58 _dist JoeJulian: right now I've got VMs that were migrated 6km away (over a 10gbe link) to our DR site. However, qemu is writing to a brick that is here (not its' localhost) and then gluster is writing back to it, it's kind of hilarious but all the same it means I can't take down any computer/storage host
17:59 JoeJulian Well the server that's serving reads won't change until the file is opened again.
17:59 _dist so what I want is libgfapi to remap to localhost brick, and then that brick is the one that writes "out" to the other 2 replica bricks
17:59 JoeJulian And, of course, bricks don't write to anything.
18:00 JoeJulian The client is the one that writes to the replica bricks.
18:02 _dist JoeJulian: that's a game changer
18:02 _dist JoeJulian: that must mean, that my compute qemu's libgfapi is running on a different compute node. How is that even possible
18:04 Philambdo joined #gluster
18:04 _dist JoeJulian: isn't the qemu process and the libgfapi client connection one in the same? or am I wrong on that
18:12 MrAbaddon joined #gluster
18:13 _dist JoeJulian: I have servers a b & c. All three servers are compute & storage nodes. All compute operates over libgfapi, all three servers have 1 brick that is part of a 3-way replica.
18:13 _dist JoeJulian: When compute VM 1 is migrated from server c to server a, all writes done in VM 1 are still sent "out" from server c. This is not what I want to happen :)
18:15 _dist JoeJulian: I must be mistaken, my scenario does not seem possible or logical. I'll do some more digging before I bug this channel again on it
18:15 JoeJulian yeah, not possible.
18:15 JoeJulian but all reads will still come from server c
18:15 JoeJulian What you must be seeing are the reads from qemu.
18:16 sputnik13 joined #gluster
18:25 firemanxbr joined #gluster
18:26 _dist JoeJulian: I agree with you, the traffic I thought I was seeing was reversed and the problem is much simpler. It appears that pending writes from libgfapi are waiting for a downed brick and that is what causes the issue
18:30 _dist JoeJulian: is there anywhere I can read about what specific ip traffic a server going offline should send so I can look for it and run tests?
18:33 ricky-ticky joined #gluster
18:33 semiosis _dist: TCP FIN or RST most likely
18:34 semiosis http://www.tcpipguide.com/free/t​_TCPConnectionTermination-2.htm
18:34 glusterbot Title: The TCP/IP Guide - TCP Connection Termination (at www.tcpipguide.com)
18:34 semiosis seems as good a reference as any
18:34 _dist hmm, proxmox added a firewall, I wonder if they have weird default iptable commands that are causing trouble
18:47 ndevos a little related, but different: bug 1129787
18:47 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1129787 high, high, ---, ndevos, POST , file locks are not released within an acceptable time when a fuse-client uncleanly disconnects
18:48 ndevos aaand that still needs some work...
18:48 JoeJulian Even clean disconnects are very poorly tested. Hopefully that new testing infrastructure will help with that.
18:52 semiosis _dist: furthermore, http://en.wikipedia.org/wiki/Transmission​_Control_Protocol#Connection_termination -- has the bit about the RST
18:52 glusterbot Title: Transmission Control Protocol - Wikipedia, the free encyclopedia (at en.wikipedia.org)
18:56 _dist semiosis: thanks!
18:56 semiosis yw
18:56 _dist ndevos: libgfapi wouldn't be fuse, but it might be the same type of problem
18:58 ndevos _dist: yeah, the issue was found with the fuse client, but it's in the communication layer that is also used with libgfapi
19:01 _dist ndevos: how big is the penalty for lowering the timeout? JoeJulian I think you spoke about this as well in the past
19:11 ndevos _dist: I'm not sure what timeout you mean?
19:11 ndevos maybe ,,(ping-timeout) ?
19:11 glusterbot The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
19:12 _dist even if eager.lock is turned on?
19:13 ndevos I'm not sure if that makes a difference
19:15 _dist I wonder if the penalty even matters in my case (large VM images) where one process would have a lock 99% of the time anyway
19:23 JoeJulian _dist: Just don't get in to a race condition where the re-establishing of FDs causes a ping-timeout.
19:24 _dist JoeJulian: well I think 42 seconds is extreme, perhaps something like 5 seconds but I'll have to play with it. I expect most virtual controllers will drop a disk after 7 seconds
19:25 JoeJulian Which filesystem?
19:26 _dist virtio scsi on ext4 ?
19:27 JoeJulian Oh, cool. You can avoid the disk going read-only with ext4.
19:27 _dist JoeJulian: I'm not saying 42 seconds is always extreme, just for large VM files
19:27 calisto joined #gluster
19:27 JoeJulian Perhaps
19:27 _dist JoeJulian: that's cool, but I'm not sure I'd want to? NTFS is obviously the other major one we'd have trouble with
19:27 JoeJulian I just prefer avoiding ping-timeout.
19:29 JoeJulian Just a reminder in case there's anybody reading that hasn't done it yet: "We'd like to find out how you're using GlusterFS.  A short  survey is up at https://t.co/Cosm63qWY6 i. If you could take a could of minutes to run through it, that would be great."
19:29 glusterbot Title: GlusterFS use survey (at t.co)
19:31 m0zes joined #gluster
19:31 _dist JoeJulian: How does one access glusterfs via ZFS ?
19:33 _dist survey completed
19:34 semiosis davemc: ZFS should probably be removed from that question
19:34 semiosis at least, i'd be surprised if anyone chose it
19:35 _dist semiosis: I don't think it's possible to choose it in the framework of that question
19:35 semiosis agree
19:35 failshell joined #gluster
19:35 _dist for clarity sake that would be like accessing glusterfs via XFS
19:35 _dist perhaps the survey author intended to ask what backing FS was being used but missed the question
19:36 davemc semiosis, roughly 25% of future use mention zfs
19:36 davemc 0% current users
19:36 semiosis the question is just worded strangely
19:36 hagarth yes, even I found the question confusing
19:36 semiosis zfs is a backend storage medium
19:37 semiosis fuse, libgfapi, samba, etc are front end consumers/clients of glusterfs
19:46 JoeJulian I must not have actually read the question...
19:47 tryggvil joined #gluster
20:12 rshott joined #gluster
20:17 neofob joined #gluster
20:33 calisto joined #gluster
20:43 theron joined #gluster
21:09 balacafalata joined #gluster
21:10 rotbeard joined #gluster
21:11 theron joined #gluster
21:11 rotbeard joined #gluster
21:14 plarsen joined #gluster
22:04 AaronGr joined #gluster
22:07 davemc semiosis, too late to remove now without mucking the validity of the survey.
22:07 semiosis fair enough
22:07 davemc will discount it in analysis
22:17 tryggvil joined #gluster
22:20 tryggvil joined #gluster
22:22 cfeller joined #gluster
22:22 calisto joined #gluster
22:43 RioS2 joined #gluster
22:44 apscomp joined #gluster
22:44 bjornar joined #gluster
22:55 plarsen joined #gluster
23:00 badone joined #gluster
23:01 _Bryan_ joined #gluster
23:36 rjoseph joined #gluster
23:38 al0 joined #gluster
23:39 bennyturns joined #gluster
23:41 allgood joined #gluster
23:41 allgood joined #gluster
23:42 allgood hi folks
23:42 allgood i am trying to make a fs image inside a gluster share
23:42 allgood i am getting short read and writes when issuing the mkfs.ext4 command
23:43 allgood after this, the gluster client disconnects and i need to umount and remount the volume
23:43 allgood can anybody help me?
23:43 JoeJulian what version
23:43 allgood 3.2.7
23:43 JoeJulian there's your problem.
23:43 allgood debian wheezy
23:43 JoeJulian @ppa
23:43 glusterbot JoeJulian: The official glusterfs packages for Ubuntu are available here: STABLE: 3.4: http://goo.gl/M9CXF8 3.5: http://goo.gl/6HBwKh -- QA: 3.4: http://goo.gl/B2x59y 3.5: http://goo.gl/RJgJvV 3.6: http://goo.gl/ncyln5 -- QEMU with GlusterFS support: http://goo.gl/e8IHnQ (3.4) & http://goo.gl/tIJziO (3.5)
23:43 JoeJulian I'd recommend 3.5
23:44 allgood it is strange because i already did this on another setup, with the same versions!
23:44 JoeJulian Yeah, it's a known race condition.
23:44 allgood is there  a specific bug report on this issue?
23:44 JoeJulian Probably, but it's so old...
23:45 JoeJulian I know I was CC'd on it at the time...
23:46 allgood the most strange is that both setups are the same
23:46 allgood is there a way to detect what is triggering it?
23:47 JoeJulian wireshark
23:47 allgood :-D
23:47 JoeJulian iirc, that's how I figured out what to report.
23:47 allgood trying to find 3.5 packages for debian
23:48 JoeJulian semiosis: ^
23:48 JoeJulian I'm sure I remember him saying something about building those
23:49 allgood only on jessie / testing
23:50 JoeJulian Apparently nobody uses debian. ;)
23:50 allgood yes, they use
23:50 allgood the testing one!
23:50 JoeJulian I'm just kidding.
23:50 allgood im too!
23:50 JoeJulian We just don't get many people in here asking for packages.
23:51 allgood i like packages!
23:51 JoeJulian That's what she said...

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary