Camelia, the Perl 6 bug

IRC log for #gluster, 2013-07-02

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 aknapp no that's not it
00:01 aknapp we are writing to the fuse mounts
00:02 brosner joined #gluster
00:03 the-me joined #gluster
00:10 aknapp we have moutned the volume on each of the bricks
00:10 aknapp and trying to write to ti
00:10 aknapp it*
00:25 duerF joined #gluster
00:38 joelwallis joined #gluster
00:46 vpshastry joined #gluster
00:47 aknapp we are using our Gluster bricks as clients as well
00:47 aknapp so 2 machines
00:47 aknapp each fuse mount to localhost
00:47 aknapp to see the cluster
00:47 aknapp and trying to write to one and see it show up on both
00:48 aknapp does it make sense to use Gluster this way?
00:48 aknapp basically using it to sync our primary/secondary ftp servers
00:48 aknapp so we designate one ftp as our primary
00:48 aknapp if it fails, we use ucarp to move a VIP over and make the secondary act as master
00:49 aknapp all while using Gluster to sync the storage portion
00:56 msacks joined #gluster
00:57 tc00per repo
00:57 tc00per ?repo
00:57 tc00per Where is the repository?
00:58 msacks Hi all. I am having some trouble expanding a volume. i read up on https://access.redhat.com/site/document​ation/en-US/Red_Hat_Storage_Software_Ap​pliance/3.2/html/User_Guide/sect-User_G​uide-Managing_Volumes-Expanding.html but the amount of information in there isn't enough to get me going. I
00:58 glusterbot <http://goo.gl/bMtGf> (at access.redhat.com)
00:58 msacks I keep getting this annoying error:
00:59 msacks https://gist.github.com/ms​acks/b8b465238466f8c3a831
00:59 glusterbot <http://goo.gl/vNzMT> (at gist.github.com)
00:59 msacks any tips?
01:08 tc00per left #gluster
01:10 semiosis back
01:10 semiosis oops wrong window
01:10 * nightwalk wouldn't have thought that was inappropriate in ANY window...
01:12 nightwalk msacks: you probably need to erase the volume, destroy the .glusterfs dir in the top level of it, and strip the files of the glusterfs xattrs
01:12 bala joined #gluster
01:13 msacks hmm
01:13 msacks the volume is working though
01:13 msacks I'm just trying to add nodes and increase the size
01:14 nightwalk here's the script I use. It was in a forum thread, and I believe it was written by JoeJulian: http://paste.ubuntu.com/5834806/
01:14 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
01:14 nightwalk msacks: yes, that's what I'm telling you. the new brick you're trying to add must have been in a brick previously. new versions of gluster don't sanitize removed bricks for you iirc
01:15 msacks hmm, i spun up two completely new vms
01:15 msacks not sure how those bricks could have been added before
01:15 nightwalk if you run that script on the new brick's directory that you're trying to add, it'll clear anything that's gumming up the add
01:16 msacks neat script
01:17 nightwalk indeed. I'd actually written my own before I found that though :/
01:17 msacks does that script need to be run on targets as well as the node you are running from?
01:18 nightwalk no, you wouldn't run it on a gluster mount. you'd run it directly on the node on the directory that's going to be the backing store
01:18 nightwalk so for example if you add by host:/.dir, then you'd log into that host and run 'clear_xattrs.sh /.dir'
01:20 nightwalk msacks: it looks like this might have been the original mail thread: http://comments.gmane.org/gmane.co​mp.file-systems.gluster.user/9096
01:20 glusterbot <http://goo.gl/Ra69l> (at comments.gmane.org)
01:20 nightwalk but if you're curious, googling 'clear_xattrs.sh' turns up a lot of stuff
01:25 msacks reading
01:26 msacks if i have a production volume serving images for my website, and i start getting that error, do i have to rebuild the entire volume?
01:31 aknapp so i'm getting a lot of these in my logs
01:31 aknapp [2013-07-01 21:31:17.721617] W [client-handshake.c:275:client_ping_cbk] 0-ingest-client-1: timer must have expired
01:32 aknapp the mount hangs ... for a while, then i see this in the logs
01:32 aknapp any idea what this means?
01:36 nightwalk msacks: you shouldn't "start getting" that error. *if* it happens, it should happen on brick addition and error out like it did. Once you've successfully added a brick, gluster will add back the relevant xattrs and so forth and it'll recognize the brick as belonging to the volume
01:37 msacks thanks nightwalk
01:37 msacks k
01:37 kevein joined #gluster
01:37 nightwalk msacks: if you should ever destroy the volume and recreate it for any reason, you'd have to run that script on each brick's backing stores after the removal and prior to the new creation
01:37 msacks ill play around with my environment a bit
01:37 msacks got it
01:38 msacks thanks again
01:38 nightwalk but other than a case of split brain here and there, it should be all good
01:38 nightwalk welcome :)
01:39 nightwalk aknapp: You might have to wait for the regulars to get back in 8-12 hours or so. The only suggestion I can offer is to make sure that you have the time synced on all nodes and clients
01:39 nightwalk (via ntpd, ntpdate, etc)
01:39 aknapp yeah.. we're running ntp
01:39 aknapp so regulars are usually in here during normal business hours?
01:39 aknapp it's a wierd issue
01:40 aknapp very strange
01:40 aknapp the mount hangs from time to time
01:40 aknapp and the file will appear on both but on one it's correct
01:40 aknapp and the other node it shows up but if you cat the file it's empty
01:40 nightwalk I think they're in the americas, so EST or PST or somewhere between and normal business hours, yes
01:40 aknapp yeah americas... ok well thanks for the suggestion nightwalk
01:40 nightwalk though last time I said that, one of them peaked in about a half an hour after, so you might get lucky :)
01:40 aknapp haha
01:45 aknapp just nticed this remote operation failed: Stale NFS file handle.
01:46 aknapp which is odd b/c I don't have it mounted via nfs
01:46 aknapp using fuse
01:46 semiosis that message is overused
01:48 vpshastry joined #gluster
01:50 semiosis aknapp: what version of glusterfs?  what distro?
01:51 aknapp gluserfs v 3.3.
01:51 aknapp v3.3.1
01:51 aknapp and running on Debian Wheezy
01:53 semiosis aknapp: looks like you're having network issues... how is your client connected to servers?  all on the same switch?
01:53 aknapp yeah, same switch, same vlan
01:53 aknapp or wait that's not ture
01:53 aknapp true*
01:54 aknapp 2 nodes, each in a different rack
01:54 aknapp so they are connected to different switches
01:54 aknapp traffic goes through our core router i think in this case
01:54 aknapp so we are running clients on the servers
01:54 aknapp each node is a server and a client
01:55 guigui1 joined #gluster
01:56 semiosis can you pastie the log surrounding the timer expired line?
01:57 aknapp sure
01:57 aknapp let me grab it
01:57 semiosis also, would you say the client hangs for 42 seconds?
01:57 aknapp 42 is the default network ping timeout right?
01:57 semiosis yes
01:58 semiosis as in client_ping_cbk, i'm guessing
01:58 aknapp [2013-07-01 19:52:11.739740] E [socket.c:1715:socket_connect_finish] 0-ingest-client-0: connection to 10.10.115.27:24012 failed (Connection refused)
01:59 semiosis is that the right ip for the other server?
02:00 semiosis possible you have an IP address conflict?
02:00 aknapp that is correct but that might be when the server was down
02:00 aknapp [2013-07-01 21:14:37.795894] C [client-handshake.c:126:rpc​_client_ping_timer_expired] 0-ingest-client-0: server 10.10.115.27:24013 has not responded in the last 42 seconds, disconnecting.
02:01 aknapp and im actually able to telnet to that port
02:03 aknapp telnet 10.10.115.27 24013
02:03 aknapp Trying 10.10.115.27...
02:03 aknapp Connected to 10.10.115.27.
02:03 semiosis please use pastie.org or similar for multiline pastes
02:03 aknapp ah sorry
02:04 aknapp will do
02:06 aknapp so when i write data to node1
02:06 aknapp i see this on node2 logs
02:06 aknapp [2013-07-01 22:05:34.531677] I [server3_1-fops.c:203:server_lk_cbk] 0-ingest-server: 190: LK -2 (--) ==> -1 (No such file or directory)
02:08 semiosis weird
02:09 brosner joined #gluster
02:10 aknapp ok so
02:10 aknapp here are the log lines that are generated after the hang times out
02:10 aknapp http://pastie.org/8101936
02:10 glusterbot Title: #8101936 - Pastie (at pastie.org)
02:12 semiosis ok, so this is in fact a ping timeout.  99% of the time that means there really is an underlying network connectivity issue
02:14 harish joined #gluster
02:15 aknapp ok
02:15 aknapp I'm checking with my network guys now to see if there's anything going on there
02:15 aknapp thanks for your time semiosis
02:15 semiosis yw, good luck
02:28 brosner joined #gluster
02:44 vshankar joined #gluster
02:52 brosner joined #gluster
03:02 vpshastry joined #gluster
03:05 brosner joined #gluster
03:06 bharata-rao joined #gluster
03:16 aknapp semiosis you were right
03:16 aknapp it was a newworking issue
03:16 semiosis :)
03:16 semiosis glad i could help
03:16 aknapp we had a bonded ports and had errors on some of the ports
03:16 aknapp took 'em out, everything works!
03:17 semiosis it's always the bonded ports!
03:17 semiosis no one ever comes here and says "hey i have these bonded ethernet ports and they're so awesome"
03:19 nightwalk joined #gluster
03:21 aknapp hahaha
03:45 mohankumar joined #gluster
03:45 vpshastry joined #gluster
03:48 bharata-rao joined #gluster
03:51 sgowda joined #gluster
03:53 brosner joined #gluster
04:05 CheRi joined #gluster
04:06 andreask joined #gluster
04:15 shylesh joined #gluster
04:27 vpshastry joined #gluster
04:39 anands joined #gluster
04:40 ngoswami joined #gluster
04:44 psharma joined #gluster
04:51 bala joined #gluster
04:59 Guest41201 joined #gluster
04:59 bala joined #gluster
05:01 hagarth joined #gluster
05:02 pkoro joined #gluster
05:05 atrius_ joined #gluster
05:06 raghu joined #gluster
05:08 vpshastry joined #gluster
05:10 brosner joined #gluster
05:13 JoeJulian Hey, semiosis, I have bonded ethernet ports and they are, indeed, awesome.
05:14 samppah :O
05:14 JoeJulian :O
05:19 ricky-ticky joined #gluster
05:22 bharata-rao joined #gluster
05:30 ricky-ticky joined #gluster
05:40 satheesh joined #gluster
05:46 bala joined #gluster
05:49 rastar joined #gluster
05:51 jimlin joined #gluster
05:51 Cenbe joined #gluster
05:52 anands joined #gluster
05:53 hagarth joined #gluster
05:58 rgustafs joined #gluster
06:01 JoeJulian joined #gluster
06:04 harish joined #gluster
06:04 aknapp joined #gluster
06:15 jtux joined #gluster
06:18 shireesh joined #gluster
06:39 mooperd joined #gluster
06:44 vimal joined #gluster
06:53 ctria joined #gluster
06:54 deepakcs joined #gluster
06:56 FilipeMaia joined #gluster
07:00 ramkrsna joined #gluster
07:04 jiku joined #gluster
07:05 saurabh joined #gluster
07:09 raghu joined #gluster
07:13 bharata-rao joined #gluster
07:14 ngoswami joined #gluster
07:16 kanagaraj joined #gluster
07:18 Cenbe joined #gluster
07:21 ricky-ticky joined #gluster
07:24 harish joined #gluster
07:26 ujjain joined #gluster
07:33 satheesh joined #gluster
07:38 vpshastry joined #gluster
07:45 rgustafs joined #gluster
07:48 pkoro hi everyone, I am seeing the following error within a log file of a brick process that has failed: "transport.address-family not specified. Could not guess default value from (remote-host:(null) or transport.unix.connect-path:(null)) options" . has anyone seen anything similar before?
07:49 pkoro the error comes in from : name.c:141 http://fossies.org/linux/privat/glus​terfs-3.3.1.tar.gz:a/glusterfs-3.3.1​/rpc/rpc-transport/socket/src/name.c
07:49 glusterbot <http://goo.gl/jnSUT> (at fossies.org)
07:51 anands joined #gluster
07:51 hagarth joined #gluster
07:52 dobber joined #gluster
08:07 mooperd joined #gluster
08:19 jord-eye2 Hi. Is it possible to mount a gluster volume read-only? I get this message when I introduce 'ro' flag in fstab line and try to mount: "Mount failed. Please check the log file for more details."
08:19 jord-eye2 Using the command '/usr/sbin/glusterfs --read-only --direct-io-mode=on --volfile-id=/volume --volfile-server=gluster.server /mnt' fails silently
08:20 jord-eye2 and if I use read-only option in fstab it says the option is unknown
08:21 bulde joined #gluster
08:24 FilipeMaia joined #gluster
08:37 atrius_ joined #gluster
08:37 harish joined #gluster
08:50 bharata-rao joined #gluster
08:53 pkoro jord-eye2: we have had the same issue on our setup (3.3.1 community rpms) and the workaround we have implemented is mount ro via nfs
08:56 FilipeMaia joined #gluster
08:56 jord-eye2 pkoro: really? :(
08:56 jord-eye2 I have the same version as you
08:57 jord-eye2 I don't want to use NFS, but what really bugs me is that the option of read-only is there, and is documented. But it does not work at all
08:59 ndevos bug 853895
08:59 glusterbot Bug http://goo.gl/xCkfr medium, medium, ---, csaba, ON_QA , CLI: read only glusterfs mount fails
08:59 pkoro well neither did we want to use NFS but it was the only option at the time. Actually I remember one of the first things I tries after upgrading to 3.3.1 (from 3.3.0) was to check if that ro mount would work but it didn't
09:02 ProT-0-TypE joined #gluster
09:02 jord-eye2 thanks nvedos. I see last update is from Jan 2013... and is not solved
09:03 jord-eye2 well, it is, but in 3.4.0qa3
09:03 jord-eye2 don't understand why if the problem appears in 3.3.0, it is not fixed in 3.3.1
09:03 jord-eye2 but int 3.4.0
09:15 jord-eye2 ndevos: I've seen you commited the patch: http://review.gluster.org/#/c/4163/ . Is it in 3.3.1? (I assume not, but I have to ask, sorry)
09:15 glusterbot Title: Gerrit Code Review (at review.gluster.org)
09:22 harish joined #gluster
09:23 piotrektt joined #gluster
09:37 ndevos jord-eye2: no, I dont think it is included in 3.3.x
09:45 mooperd joined #gluster
09:48 mooperd joined #gluster
09:52 bharata-rao joined #gluster
10:25 saurabh joined #gluster
10:28 hagarth joined #gluster
10:28 manik joined #gluster
10:32 edward1 joined #gluster
10:34 anands joined #gluster
10:39 FilipeMaia_ joined #gluster
10:40 FilipeMaia joined #gluster
10:52 kkeithley1 joined #gluster
11:10 anands joined #gluster
11:17 CheRi joined #gluster
11:18 GLHMarmot joined #gluster
11:24 hagarth joined #gluster
11:40 ngoswami joined #gluster
11:47 manik joined #gluster
11:49 CheRi joined #gluster
11:54 bulde joined #gluster
12:03 spider_fingers joined #gluster
12:04 vpshastry1 joined #gluster
12:09 hagarth joined #gluster
12:09 venkat joined #gluster
12:10 manik joined #gluster
12:12 vpshastry1 joined #gluster
12:19 vpshastry1 left #gluster
12:19 vpshastry1 joined #gluster
12:20 vpshastry1 left #gluster
12:20 vpshastry1 joined #gluster
12:28 bulde joined #gluster
12:31 balunasj joined #gluster
12:41 Debolaz Hmm... Is there any way I could speed up access to a file, if I don't need the file, if present locally, to not necessarily be the most up to date one?
12:42 Debolaz I'm currently using the native client, but I'm reading that accessing through NFS has big performance benefits for read-mostly storage.
12:45 hagarth joined #gluster
12:48 recidive joined #gluster
12:51 rcheleguini joined #gluster
12:57 deepakcs joined #gluster
13:00 vpshastry1 joined #gluster
13:01 robo joined #gluster
13:02 Debolaz Hmm.... Using NFS did not decrease latency.
13:03 manik joined #gluster
13:03 edward1 joined #gluster
13:16 awheeler_ joined #gluster
13:19 spider_fingers joined #gluster
13:24 semiosis Debolaz: nfs caches some attributes but it won't speed up data operations
13:24 bugs_ joined #gluster
13:24 semiosis awheeler_: ,,(qa-releases)
13:24 glusterbot awheeler_: I do not know about 'qa-releases', but I do know about these similar topics: 'qa releases'
13:24 semiosis awheeler_: ,,(qa releases)
13:24 glusterbot awheeler_: The QA releases are available at http://bits.gluster.com/pub/gluster/glusterfs/ -- RPMs in the version folders and source archives for all versions under src/
13:28 Debolaz The main issue I'm having is latency. A request that usually takes about 60ms (WordPress index.php) goes up to about 1s when done on the server serving from GlusterFS. Although GlusterFS is "only" responsible for about 500ms of this, MariaDB also seems to play a factor.
13:28 semiosis see ,,(php) for ways to optimize php
13:28 glusterbot php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details.
13:29 semiosis ideally you would remove all require/include statements and use autoloading, but not all php software can do that
13:29 semiosis for those, you can use APC to cache files in the apache process
13:29 semiosis or php process if not apache
13:29 plarsen joined #gluster
13:30 semiosis however changes to files cached in that manner will not be noticed until the process is restarted (like apache restart, etc)
13:30 semiosis possibly reload
13:30 semiosis the first step you should take is to optimize your include_path so that the most common place where files will be found comes first
13:30 semiosis that should improve latency a bit
13:31 Debolaz Being required to restart apache to re-read files is a limitation I have no problems living with. :)
13:31 semiosis then you should be very happy with APC
13:32 semiosis gotta run, bbl
13:32 * semiosis afk
13:35 failshell joined #gluster
13:35 alinux joined #gluster
13:40 rwheeler joined #gluster
13:44 pkoro joined #gluster
13:46 lpabon joined #gluster
13:50 jclift joined #gluster
13:51 hagarth joined #gluster
13:52 aliguori joined #gluster
13:57 alinux joined #gluster
13:58 zaitcev joined #gluster
14:04 bennyturns joined #gluster
14:10 cenit joined #gluster
14:13 puebele joined #gluster
14:20 jthorne joined #gluster
14:25 msacks joined #gluster
14:25 vpshastry2 joined #gluster
14:27 Technicool joined #gluster
14:29 msacks joined #gluster
14:31 andreask joined #gluster
14:32 alinux joined #gluster
14:35 andreask joined #gluster
14:37 bsaggy joined #gluster
14:40 chirino joined #gluster
14:43 semiosis this morning i achieved the first end-to-end test of a java7 filesystem provider backed by libgfapi
14:44 semiosis really excited about this
14:44 dewey joined #gluster
14:45 kkeithley_ kewl
14:48 semiosis FileSystems.newFileSystem(new URI("gluster://127.0.2.1:foo/"), null) actually connects to the volume!
14:49 semiosis using volume name in place of port
14:51 ndevos nice, but why use a different URI format as qemu? Or do they abuse the :port too?
14:51 semiosis the projects, in case anyone wants to see code: https://github.com/semiosis​/glusterfs-java-filesystem which is backed by https://github.com/semiosis/libgfapi-jni
14:51 glusterbot <http://goo.gl/KNsBZ> (at github.com)
14:51 semiosis ndevos: just came up with it on the fly this morning.  didn't know qemu had one.  where can I find an example?
14:51 * ndevos is more a python/c guy
14:52 semiosis ndevos: these should work with jython, in theory
14:53 ndevos semiosis: http://raobharata.wordpress.com/2012/10​/29/qemu-glusterfs-native-integration/ contains the url-scheme
14:53 glusterbot <http://goo.gl/f2MhH> (at raobharata.wordpress.com)
14:53 ndevos semiosis: well, gfapi.py exists already in the main source tree under api/examples
14:54 andreask If i create a directory on a distributed-replicated volume when are the trusted.afr xattributes added?
14:54 semiosis ndevos: i wonder what their motivation was for that url scheme.  imo removing confusing over volname vs. directory name was worth not allowing a custom port
14:54 semiosis glusterd on a port other than 24007 is just too weird
14:55 semiosis s/confusing/confusion/
14:55 glusterbot What semiosis meant to say was: ndevos: i wonder what their motivation was for that url scheme.  imo removing confusion over volname vs. directory name was worth not allowing a custom port
14:55 andreask on gluster 3.3.1 btw
14:55 semiosis also looks familiar to anyone who's made a normal fuse mount... server:volname
14:55 ndevos semiosis: yes, I do not know why they chose that, I assume :port is pretty standard use and an int
14:56 semiosis agreed however the Java URI object allows for scheme://authority/path
14:56 mohankumar joined #gluster
14:56 semiosis and if authority is string:int then it further parses host & port, otherwise it just provides the whole string as "authority"
14:57 ndevos uh... yes? you're loosing me
14:57 semiosis i mean, if the int is missing, things still work
14:57 semiosis that was all
14:57 ndevos ah, okay, but in your gluster:// the port is not optional, is it?
14:57 semiosis port is not allowed
14:57 semiosis it's always going to be 24007
14:58 ndevos s/port/volume/
14:58 glusterbot What ndevos meant to say was: ah, okay, but in your gluster:// the volume is not optional, is it?
14:58 semiosis volume is not optional
14:59 semiosis bbl, interviewing someone
14:59 ndevos sure
15:02 semiosis actually he's not here yet
15:07 andreask hmm .. strange thing that directory afr xattributes
15:12 andreask anyone an idea where I can read about that in more detail?
15:13 FilipeMaia joined #gluster
15:14 daMaestro joined #gluster
15:16 shireesh joined #gluster
15:17 Debolaz semiosis: APC cut the request time by about half. apache still seems to love doing a lot of stats requests, so I guess the best solution to this problem is to go with varnish.
15:17 semiosis varnish \o/
15:18 semiosis apache's own mod_cache is a bit easier to get started with than varnish if you're not already familiar with varnish
15:18 semiosis you might want to try that first
15:19 vpshastry joined #gluster
15:21 alinux joined #gluster
15:30 bala joined #gluster
15:31 manik joined #gluster
15:37 daMaestro joined #gluster
15:47 msacks joined #gluster
15:48 _Bryan_ joined #gluster
15:48 _Bryan_ Cliff: Can you check to see if the corpvpn server thinks I am connected from somewhere
15:48 failshell using -l with ls is bad with gluster as it causes a stats(), is there some gotchas like that with du?
15:48 _Bryan_ errr.wrong channel
15:52 umarillian1 joined #gluster
15:55 Savaticus joined #gluster
15:56 rwheeler joined #gluster
16:05 edward1 joined #gluster
16:06 andreask anyone an idea why I don't see trusted.afr attributes on a directory?
16:06 semiosis ,,(extended attributes)
16:06 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
16:06 semiosis using that command exactly?
16:07 andreask yes
16:08 andreask # getfattr -m . -d -e hex  testdir
16:08 andreask # file: testdir
16:08 andreask trusted.gfid=0x8d2df3b8dbd84948996fb5df1645f049
16:08 andreask thats all ... simple replicated setup
16:08 \_pol joined #gluster
16:13 andreask I expected the trusted.afr attributes are set once I create files in the directory, doesn't look like
16:17 andreask hmm ... looks like they are only created "on-demand" with one brick missing the attributes are created on e.g. a file creation in the dir
16:17 Mo__ joined #gluster
16:27 FilipeMaia joined #gluster
16:30 glusterbot New news from newglusterbugs: [Bug 980541] GlusterFS testing framework needs rdma support added <http://goo.gl/zZkUL>
16:38 msacks joined #gluster
16:39 jclift Hmmm, this "testing framework" is not very well documented.
16:42 hagarth joined #gluster
16:52 tw joined #gluster
16:54 _Bryan_ joined #gluster
17:06 joelwallis joined #gluster
17:11 tw In a distributed volume, does a brick receive its proportionate size in files or are they spread evenly by absolute usage (ie, brick1 1G, brick2 5G; of 6x 1MB files, will it be spread 1 & 5 or 3 & 3)? Is that behavior configurable or should it be emulated by allocating multiple same-size bricks on hosts where I want more files to be placed?
17:15 ThatGraemeGuy joined #gluster
17:16 vpshastry joined #gluster
17:28 robos joined #gluster
17:34 \_pol joined #gluster
17:37 * Debolaz has managed to get GlusterFS down to "only" adding 250ms to his web request.
17:57 rwheeler_ joined #gluster
17:58 rwheeler joined #gluster
18:00 msacks joined #gluster
18:09 FilipeMaia joined #gluster
18:20 hateya joined #gluster
18:25 dbruhn joined #gluster
18:43 sprachgenerator joined #gluster
18:51 Koma joined #gluster
19:16 semiosis tw: in a distributed volume, files are evenly distributed among the bricks
19:17 semiosis by name
19:17 vpshastry joined #gluster
19:22 manik joined #gluster
19:24 Debolaz semiosis: Status so far: A 250ms request on a non-glusterfs becomes a 500ms request on glusterfs. (This used to be 1s though, so thats certainly an improvement from earlier today) That is understandable considering all the horrible things WordPress does under the hood, I understand why it happens. But it still leaves me with a problem.
19:25 Debolaz semiosis: Do you think it would be better to avoid GlusterFS in this situation?
19:26 dbruhn Does word press use php?
19:27 semiosis i have a wordpress install on a glusterfs volume, and I don't even use APC to disable stat
19:27 Debolaz dbruhn: Yes.
19:27 dbruhn turn on PHP Caching?
19:27 semiosis Debolaz: so i can't say what's good for your setup
19:27 Debolaz semiosis: Apparently, APC doesn't completely disable stats, only in some known-to-not-break situations.
19:27 semiosis interesting
19:27 Debolaz dbruhn: APC is enabled yes, and it helped.
19:28 Debolaz dbruhn: Pre-APC was 1s. A 50% reduction of latency is very good, but still not enough.
19:28 semiosis Debolaz: did you optimize your include_path?
19:29 bstr does gluster play nicely with bind mounts based off a replicated local fuse mount point?
19:30 robos joined #gluster
19:33 tw semiosis: thank you, from the documentation I've read, I assume it is not configurable without meddling in the transport's code.
19:34 Debolaz semiosis: I haven't really done anything with it.
19:34 semiosis tw: there's a feature called "custom layouts" that should allow you to manage where files are placed without meddling in code, but I dont know how ready that is for prime time.
19:34 bstr more in-depth ; i configured a two node cluster with two bricks, one geo-replicated, and one replicated between the two nodes. A co-worker of mine began using this cluster, adding his applications and bind mounts, but now for some reason when i reboot one of the nodes the fuse mounts are not added automaticlly, although i can mount -a and they come up. looking in logs, i see the following: glusterfsd[1605]: [2013-07-02 14:29:36.872947] C [glusterfsd.c:1220
19:34 bstr erfs: ERROR: parsing the volfile failed (No such file or directory)
19:39 plarsen joined #gluster
19:43 dewey joined #gluster
19:45 Debolaz semiosis: Is there no feature in glusterfs that will let me say "reads can be faulty, just ensure writes are ok"?
19:45 Debolaz Ie, relying exclusively on local storage for read operations.
19:45 semiosis not that i know of
19:45 Debolaz Or at least delay the self heal check.
19:49 tw Considered setting performance.cache-refresh-timeout higher? That sounds like it should cache stat() longer before recheck.
20:11 \_pol joined #gluster
20:15 Debolaz semiosis: Managed to push it down to 100ms now with some application-specific things. Combined with using nginx or varnish for caching static files, I think this might be sufficient.
20:20 rwheeler joined #gluster
20:24 edward1 joined #gluster
20:33 semiosis Debolaz: that's pretty good
20:42 FilipeMaia joined #gluster
20:54 \_pol joined #gluster
21:00 badone joined #gluster
21:06 mooperd joined #gluster
21:07 recidive joined #gluster
21:11 badone joined #gluster
21:20 Savaticus joined #gluster
21:26 tc00per joined #gluster
21:28 tc00per @JoeJulian Recall your brief suggestion for moving brick(s) to subdirs of the mount-point yesterday... In the end I ended up blowing away the gluster volume, rebuilding from scratch and restoring (in work) from backup is what worked more easily for me (not much data). Though with more data I would want to try to avoid that.
21:28 AnthonyFrancis joined #gluster
21:29 tc00per It seems that may how-to documents illustrate using the mount points as the bricks. Might be good if at least the 'gold master' how-to on gluster.org didn't do the same (which I believe it does).
21:30 tc00per For me the failure (possibly user error) occurred when I tried to replace-brick after killing the brick processes... The message "Source brick server-00:/srv/glusterfs/bricks/gvol.01 is not online." was returned.
21:31 tc00per Perhaps you could add the detailed instructions to your long list of HOW-TO's... :)
21:31 tc00per I will have to test mine on a 'new' development system.
21:39 Savaticus joined #gluster
21:47 realdannys1 joined #gluster
21:48 realdannys1 Ok guys - installed FFmpeg on the server with Gluster FS Server on, breaks Gluster - anybody know why? I need the two to live with each other if possible. I used this guide to install on CentOS can anyone see anything installed there that might be causing the issue? http://wiki.razuna.com/display/ecp/FFM​peg+Installation+on+CentOS+and+RedHat
21:48 glusterbot <http://goo.gl/wdxRl> (at wiki.razuna.com)
21:50 aknapp glusterbot: ?
21:51 plarsen joined #gluster
21:55 semiosis realdannys1: "breaks gluster" ?!
21:55 semiosis realdannys1: can you say more please?  an error message, a log?
21:58 * jclift just quickly looked over that doc
21:58 jclift The first potential problem is this:
21:59 jclift rpm -Uhv http://packages.sw.be/rpmforge-release/rp​mforge-release-0.5.2-2.el6.rf.x86_64.rpm, along with then:
21:59 jclift yum -y update
21:59 glusterbot <http://goo.gl/jR7QT> (at packages.sw.be)
21:59 jclift realdannys1: Did you see if any of the CentOS system provided packages were being "upgraded" ?
21:59 Savaticus joined #gluster
22:00 realdannys1 Hold on - I've just restarted my Ec2 back to pre-ffmpeg install, and I can't get gluster working at all again now…bear with me
22:00 jclift np
22:01 NuxRo realdannys1: i recommend you do not use repoforge/rpmforge, you could use the static builds from here if you like http://ffmpeg.gusari.org/static/
22:01 glusterbot Title: Index of /static (at ffmpeg.gusari.org)
22:02 realdannys1 arghh I'm back to not being able to mount bricks again now on my client :/
22:04 jclift realdannys1: If for some reason the static approach (mentioned by NuxRo) doesn't work, you could try compiling Gluster rpms yourself after you've done all of the repoforge installation.
22:04 jclift realdannys1: That way, if rpmforge has upgraded any of the system rpms, the new gluster you build would work with them (most likely).
22:05 jclift realdannys1: Note, compiling your own gluster rpms on CentOS 6 is very easy.  Nearly trivial.  We have cut-n-pasteable instructions online. :D
22:07 realdannys1 yep, it'd be ok if I could get back to the work instance I had before first :(
22:09 jclift realdannys1: Have you tried this combo out in a local vm or something first, just so you don't have to debug initial problems remotely in EC2 and pay for it at the same time? :D
22:09 realdannys1 nah, its quicker to do it direct in EC2 to be honest
22:09 jclift No worries :D
22:10 realdannys1 hmm no route to host…yet again
22:10 realdannys1 I had 8 days of this when I first tried gluster
22:10 realdannys1 but I've just booted my instance that did work fine, originally
22:11 jclift You don't have a GATEWAY entry setup in your network config or something?
22:11 * jclift keeps out of it
22:11 jclift I've got other stuff to do, and I haven't used EC2 in ages :D
22:12 msacks joined #gluster
22:12 realdannys1 no idea - its all setup the way it was originally as far as I can see
22:15 realdannys1 Hmmm, the glusterfs server was rebooted on the 28th after making an AMI backup, an fit looks like it never connected to the client again
22:16 realdannys1 glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libpthread.so.0() [0x3e8f607851] (-->/usr/lib64/libglusterf​s.so.0(gf_timer_proc+0xd0) [0x325e82a$
22:16 realdannys1 [2013-06-28 11:29:35.913489] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
22:16 realdannys1 [2013-06-28 11:29:35.913570] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host ec2-23-21-168-59.compute-1.amazonaws.com
22:16 realdannys1 [2013-06-28 11:29:35.913611] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
22:16 jclift Yeah, that makes sense for the case where the box is having routing issues
22:16 jclift But, the routing issue is causing the gluster problem, not the other way around
22:17 realdannys1 can't understand how a simple reboot has broken it all though
22:17 jclift That "getaddrinfo failed (Name or service not known)" is where isn't doing an address lookup (probably of a DNS name), and basically the system is responding back to it.
22:18 jclift realdannys1: Is this the first reboot after the yum upgrade?
22:18 jclift s/upgrade/update/
22:18 glusterbot What jclift meant to say was: realdannys1: Is this the first reboot after the yum update?
22:18 realdannys1 jclift it will be yes
22:18 realdannys1 both the client machine and the server with gluster installed will have both regularly been "yum updated"
22:19 jclift k.  It sounds like something actually important to the system's operation has been screwed up by the yum update **from the repoforge repo** (most likely)
22:20 jclift realdannys1: Have you check if AWS is having any network layer problems at the moment or something, just in case that's the problem instead?
22:20 jclift realdannys1: They have an outage page somewhere don't they?
22:20 realdannys1 yeah its all up at the min
22:20 brosner joined #gluster
22:21 realdannys1 why do you think the reforge repo?
22:21 jclift Well, the normal CentOS repo has updates that are tested with each other, and known to work fairly well.
22:21 realdannys1 are you thinking FFmpeg was installed at this time? Because it wasn't - the gluster server has just glisterfs running on a clean CentOS with nothing else
22:22 jclift Ok, that's weird then
22:23 jclift What's the output from /etc/resolv.conf on your box, and also from "netstat -rn" ?
22:23 jclift You'll probably need to fpaste them or something (fpaste.org)
22:23 realdannys1 on the gluster-fs server instance?
22:24 jclift On whichever box did this: "[2013-06-28 11:29:35.913489] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)"
22:24 realdannys1 oh thats the client instance
22:24 jclift Oh, I just noticed the "6" in "gf_resolve_ip6"
22:24 jclift Is the Gluster server doing ipv6 only or something?
22:25 realdannys1 ummm, not that I know of?
22:25 jclift Cool
22:25 realdannys1 One thing
22:25 jclift ?
22:25 realdannys1 I can't peer probe the client anymore from the server - I know I don't need to it as its the client, however if I can't it obviously shows they're not talking anymore for some unknown reason
22:25 realdannys1 gluster peer probe 54.235.173.127
22:25 realdannys1 Probe unsuccessful
22:25 realdannys1 Probe returned with unknown errno 107
22:26 jclift Can they ping each other at all?
22:26 tc00per left #gluster
22:27 duerF joined #gluster
22:28 realdannys1 hmmm
22:28 realdannys1 no
22:29 jclift You're probably looking at a problem with the default route not being set correctly on either one or the other of the boxes
22:30 jclift realdannys1: (or something else similar causing comm's issue between them)
22:30 realdannys1 let me try rebooting the gluster server again, I set the elastic ip just on it (swapping it form the other machine)
22:30 realdannys1 but then again it does seem to have been broken before this
22:30 jclift realdannys1: Hmmm... any chance there's a security groups problem here?
22:30 jclift eg gluster ports getting blocked between them or something?
22:30 realdannys1 is there anything else that could be broken after a reboot you can think of?
22:30 realdannys1 shouldn't be - same security group as before
22:30 realdannys1 all the ports are wide open
22:31 jclift k
22:31 jclift There's not _generally_ a lot that should be able to block this stuff
22:31 jclift However, unexpected things out of no-where is kind of a given with computers :/
22:32 jclift Once the box comes back up, check if they can ping each other
22:32 jclift If that can, good sign.  If not, then it could be routing problems, or potentially ICMP being blocked between them
22:33 jclift If they can't ping each other, try sshing from one to the other instead, in case it's just ICMP being blocked.  ssh should get through in that situation
22:34 realdannys1 yes hopefully, when I had the 8 days or so of this it wasn't  this because I could ping, ssh and peer probe between instances
22:34 realdannys1 in the end I had to just use the clean official centos to get gluster to work
22:35 jclift That's definitely weird
22:41 realdannys1 really weird - I've just gone into security groups and added the security groups name itself to access ALL tcp, icmp and udp ports so both instances should be able to access each other as if directly connected…still can't ping
22:43 jclift Well, I have no idea then.  Sorry dude. :(
22:43 realdannys1 no worries, cheers for the help - I need to figure out why they won't ping each other...
22:45 Technicool realdannys1, you can't telnet to the gluster port either?
22:45 Technicool assuming server to server, or from the client to the server
22:46 realdannys1 can't even ping between the two at the minute :/
22:46 Technicool i never trust ping since anything could block it
22:46 Technicool typically i will `telnet <host> 24007` as the acid test
22:48 \_pol joined #gluster
22:48 realdannys1 nope, no route to host - something has majorly broke
22:48 realdannys1 this is all from my main client
22:49 realdannys1 I wonder if a yum update has broken something
22:49 realdannys1 but I don't see how it has just broken it talking to another Ec2 instance and nothing else, seems really odd
23:00 realdannys1 well I thought id just cracked it then
23:00 realdannys1 forgot that iptables had to be off - its the only thing thats changed, seemed reasonable enough
23:00 realdannys1 so i turned it off
23:00 realdannys1 but still problems
23:00 rcoup joined #gluster
23:01 realdannys1 TELL A LIE! It worked when I mounted, TFFT!
23:01 realdannys1 (still can't telnet though for some odd reason?! but the drive mounts)
23:02 realdannys1 so I need to disable this pesky IPtables on boot now, incidentally none of the installation instructions say to start the gluster demon at boot, whats the best way to do that?
23:02 jclift chkconfig glusterd on
23:02 realdannys1 cheers thought so
23:03 realdannys1 that should be in the tutorials really shouldn't it
23:03 realdannys1 ?
23:03 jclift If you've got access to add that, it's probably worth doing yeah. :)
23:08 realdannys1 f**king IPtables
23:13 sprachgenerator joined #gluster
23:17 sprachgenerator_ joined #gluster
23:21 realdannys1 if I delete files on the gluster mounted volume via sFTP on the client, should I see this instantly updated on the terminal when checking directory contents of the brick on the server, because I'm not?
23:26 realdannys1 quicker question - how can I unmount a volume on the client?
23:29 realdannys1 "umount" of course
23:29 realdannys1 done
23:29 realdannys1 working thankfully with ffmpeg which is hopefully working ok
23:30 jclift Is there a way you can verify that ffmpeg is working properly?
23:30 realdannys1 I can type "ffmpeg"
23:30 jclift Just from the "better to check and know than be sorry" perspective. :)
23:30 realdannys1 when I rebooted it complained about a shared library
23:30 realdannys1 but I appear to have fixed that
23:30 realdannys1 I need to stick a file on and test really
23:32 fidevo joined #gluster
23:34 realdannys1 hmmm
23:34 realdannys1 seems to be having all kinds of errors with shared libraries, typical
23:36 jclift realdannys1: You know how to use ldd ?
23:36 realdannys1 nope
23:37 jclift (it's been a while, but...)
23:37 jclift ldd /path/to/executable
23:37 jclift That should print out the list of libraries it's trying to find, and the path it's looking for, for each one.
23:37 jclift It should also show which ones it can find, and which ones it can't
23:38 jclift eg damn helpful in the "missing shared libraries" situation
23:38 realdannys1 good thinking
23:39 realdannys1 "error while loading shared libraries: libvpx.so.0: cannot map zero-fill pages: Cannot allocate memory"
23:40 realdannys1 perhaps not enough ram on my micro Ec2 instance?
23:42 jclift Dunno.  Google time? :D
23:50 realdannys1 three results all to do with puppy linux!

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary