Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-01-10

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 diegows joined #gluster
00:04 mattappe_ joined #gluster
00:15 wildfire left #gluster
00:17 KaZeR joined #gluster
00:56 jag3773 joined #gluster
01:05 diegows joined #gluster
01:17 ira joined #gluster
01:18 KaZeR joined #gluster
01:21 _pol joined #gluster
01:26 _pol_ joined #gluster
01:27 shyam joined #gluster
01:28 s2r2 joined #gluster
01:34 harish joined #gluster
01:45 shyam joined #gluster
01:54 dusmant joined #gluster
02:13 TonySplitBrain joined #gluster
02:26 harish joined #gluster
02:29 dusmant joined #gluster
02:42 bharata-rao joined #gluster
02:48 theron joined #gluster
03:17 johnbot11 joined #gluster
03:18 KaZeR_ joined #gluster
03:20 aravindavk joined #gluster
03:30 dylan_ joined #gluster
03:41 davinder joined #gluster
03:42 _pol joined #gluster
03:43 primechuck joined #gluster
03:46 mohankumar__ joined #gluster
03:47 johnbot11 joined #gluster
03:50 satheesh joined #gluster
04:07 neofob left #gluster
04:07 wushudoin joined #gluster
04:19 KaZeR_ joined #gluster
04:27 zapotah joined #gluster
04:27 zapotah joined #gluster
04:32 aravindavk joined #gluster
04:38 pk joined #gluster
04:52 MiteshShah joined #gluster
05:04 nshaikh joined #gluster
05:11 kdhananjay joined #gluster
05:19 KaZeR_ joined #gluster
05:20 psharma joined #gluster
05:27 mattappe_ joined #gluster
05:28 blook joined #gluster
05:32 theron joined #gluster
05:44 MiteshShah joined #gluster
05:48 davinder joined #gluster
06:01 benjamin__ joined #gluster
06:12 ngoswami joined #gluster
06:20 KaZeR joined #gluster
06:24 bulde joined #gluster
06:32 mohankumar__ joined #gluster
06:40 davinder2 joined #gluster
06:48 vimal joined #gluster
06:51 Philambdo joined #gluster
07:00 tor joined #gluster
07:02 shri joined #gluster
07:14 bulde joined #gluster
07:17 KORG|2 joined #gluster
07:18 KORG|2 joined #gluster
07:19 blook joined #gluster
07:20 KaZeR joined #gluster
07:22 jtux joined #gluster
07:23 sbabbaro_ joined #gluster
07:24 XATRIX joined #gluster
07:31 theron joined #gluster
07:46 TDJACR joined #gluster
07:47 ekuric joined #gluster
07:48 TonySplitBrain joined #gluster
07:57 eseyman joined #gluster
08:04 ctria joined #gluster
08:13 blook joined #gluster
08:21 KaZeR_ joined #gluster
08:38 Philambdo joined #gluster
08:41 benjamin__ joined #gluster
08:45 andreask joined #gluster
08:58 mgebbe_ joined #gluster
09:00 _pol joined #gluster
09:02 KaZeR_ joined #gluster
09:16 GabrieleV joined #gluster
09:19 KaZeR joined #gluster
09:29 CheRi joined #gluster
09:38 dneary joined #gluster
09:39 jclift joined #gluster
10:03 calum_ joined #gluster
10:11 khushildep joined #gluster
10:19 pk left #gluster
10:21 harish joined #gluster
10:25 tryggvil joined #gluster
10:33 MiteshShah joined #gluster
10:34 KaZeR joined #gluster
10:46 reqsd joined #gluster
10:48 reqsd Hi all! After upgrade from 3.3.2 to 3.4.2 have high load average and cpu load on servers and clients. glustershd.log
10:48 reqsd [2014-01-10 10:42:38.736198] W [socket.c:514:__socket_rwv] 3-smb-client-1: readv failed (No data available) - every seconds see this. Can anybody help?
10:50 jclift left #gluster
10:56 KaZeR_ joined #gluster
11:01 zapotah joined #gluster
11:01 _pol joined #gluster
11:01 s2r2 Hi there.  I'm in the process of exploring/evaluating gluster for our needs… one thing I wonder about: when I create a volume (gluster volume create …) I specify mount points….these would, in a real deployment, be mount points of HDDs, right?
11:02 s2r2 so, given $x drives per host, I'd list all drives' mount points from all hosts
11:02 s2r2 and they'd be used for this volume, exclusively
11:07 reqsd Hi all! After upgrade from 3.3.2 to 3.4.2 have high load average and cpu load on servers and clients. glustershd.log [2014-01-10 10:42:38.736198] W [socket.c:514:__socket_rwv] 3-smb-client-1: readv failed (No data available) - every seconds see this. Can anybody help?
11:15 samppah reqsd: are all servers up?
11:18 reqsd in one server glusterd and glusterfsd daemons is off ( in my reasons). It's can solve the problem?
11:19 reqsd in my configuration i have 3 nodes
11:22 samppah okay, it sounds like heal daemon is trying to connect to that node and that's the cause of messages in log files
11:23 samppah not sure if it's causing load also
11:28 reqsd thanks, it solve error messages !! But, in 'gluster volume status' i haven't running self-heal daemon on 2 servers in 4 . It's behavior i see after update. How i can run this daemon?
11:28 samppah it should run automatically when gluster daemon starts
11:29 edward1 joined #gluster
11:29 s2r2 from skimming the mailing list, I got the impression that to survive a HDD failure, you would depend on some kind of hw/sw raid to "self-heal", i.e. on a storage-level.  Is that correct?
11:29 s2r2 I had the prior expectation that you'd have one brick per hdd of a server and when it fails, it's just gone….
11:30 s2r2 ah, it's one glusterfsd per brick.
11:37 harish joined #gluster
12:09 getup- joined #gluster
12:13 diegows joined #gluster
12:20 psyl0n joined #gluster
12:21 andreask joined #gluster
12:24 jclift joined #gluster
12:34 TonySplitBrain s2r2: brick is directory, not block device; you can use any directory as brick; brick dir could be on filesystem on HDD/HDD partition/(software/hardware) RAID/NAS/SAN/etc.
12:35 herdani joined #gluster
12:35 psyl0n joined #gluster
12:38 herdani Hello. Just wondering: I've 2 nodes to serve web static files. I'd like to setup Nginx to serve them on each node. Is it a good practice then to make them act as clients as well ? like mounting on 127.0.0.1 ?
12:39 TonySplitBrain s2r2: if you use replicated volume with each brick on different HDD, it will survive HDD failure; it's like RAID1, but on higher level; but for sure it's better to use some RAID for bricks
12:40 herdani I know that the first reaction is to mount the client on the application web server (with failover), but I don't want the static traffic to surcharge my application web server, I'd like to have some static.domain.ltd pointing directly on the gluster cluster
12:41 TonySplitBrain herdani: GlusterFS know nothing about HTTP
12:41 herdani TonySplitBrain: yes, Nginx would handle the http protocol
12:43 TonySplitBrain herdani: you say there are 2 nodes.. are you plan to run gluster servers and Nginx on them, or there are some another servers?
12:44 herdani TonySplitBrain: I plan to run Nginx on them directly
12:45 herdani TonySplitBrain: Because I already have varnish load balancers in front, so 2 more servers with only Nginx would be somehow overkill
12:46 ira joined #gluster
12:48 TonySplitBrain herdani: how do you want to use gluster? where will be servers, bricks and clients?
12:50 TonySplitBrain herdani: i got some clues, but please explain _your_ plan
12:51 herdani TonySplitBrain: I would have 2 servers, with 1 replica 2 volume, mounted on /data on each server. Then I install Nginx on each of them for serving http://static.domain.ltd. But I cannot make the nginx look into the "/data" folder since I could have sync/healing issues, instead nginx should serve a mount client folder. So I plan to mount /mnt/datastore as a glusterfs-client on each of them, and Nginx would then point
12:51 herdani to /mnt/datastore instead of /data.
12:51 herdani I hope this is clear :)
12:52 TonySplitBrain herdani: sounds like good plan
12:53 herdani TonySplitBrain: nice! thanks :)
12:55 TonySplitBrain herdani: i thinl there will be no sync/healing issues if you will _read_ files right from the brick dir, but that is not recommended setup
13:02 _pol joined #gluster
13:07 ekuric joined #gluster
13:07 dylan_ joined #gluster
13:09 bulde joined #gluster
13:10 morse joined #gluster
13:11 benjamin__ joined #gluster
13:11 bennyturns joined #gluster
13:21 DV_ joined #gluster
13:22 s2r2 TonySplitBrain: thanks for the input.  What does an inconsequential brick failure (i.e. with replicated data) feel like from the client side?
13:24 TonySplitBrain s2r2: if data is fully replicated, it's transparent; client will read from another brick
13:28 dylan_ joined #gluster
13:41 stickyboy joined #gluster
13:57 bulde joined #gluster
14:05 B21956 joined #gluster
14:07 mattappe_ joined #gluster
14:16 satheesh joined #gluster
14:17 sroy_ joined #gluster
14:18 sroy joined #gluster
14:18 dbruhn joined #gluster
14:21 sroy joined #gluster
14:22 primechuck joined #gluster
14:22 marcoceppi joined #gluster
14:24 dbruhn herdani, I run my servers as clients for a similar setup, not Nginx, though. It works well, there are a couple caveats here and there, mainly if for some reason you need to down a node from the server perspective you are downing a client node too.
14:26 jag3773 joined #gluster
14:31 aixsyd joined #gluster
14:31 aixsyd jclift: you around buddy?
14:34 jobewan joined #gluster
14:40 japuzzo joined #gluster
14:43 lpabon_ joined #gluster
14:47 dbruhn aixsyd, what are you running into this morning?
14:48 neofob joined #gluster
14:49 aixsyd dbruhn: i spent hours troubleshooting this infiniband problem
14:49 aixsyd and im now pretty sure its the server hardware itself
14:49 dbruhn agh man, that sucks
14:49 blook joined #gluster
14:50 aixsyd a 10gig card getting sometime 690mbits/s
14:50 aixsyd is thats not abysmal, i dunno what is
14:50 aixsyd same card, different server 7.5gbits
14:50 dbruhn have you updated the bios firmwares and raid firmwares on everything?
14:51 aixsyd updated the cards? no - theyre all on the same firmware - so i know its not the firmware - or else the problem would be everywhere
14:51 aixsyd AFAIK, bios is the same on all the servers
14:51 dbruhn raid controllers?
14:52 aixsyd iperf doesnt even touch the disks
14:53 aixsyd so i was testing 4 servers - i believe if even one sever was having say, bus issues with the card - it would affect a pair.
14:53 dbruhn makes sense
14:53 sroy joined #gluster
14:53 aixsyd so i got a 5th server to throw into the mix - figure out which server is causing the issue
14:53 aixsyd i was ready for some shotgun mouthwash yesterday. i spent 18 hours on this issue
14:54 dbruhn I do not envy days like that, I've had plenty myself.
14:54 dbruhn Did you get it nailed down?
14:54 dbruhn Assuming this is used/grey market hardware?
14:54 aixsyd with speeds ranging frmo 7.5, to 6.5, to 6, to 5.5, 5, 1.5, 1, and sub-1gbps - theres a major issue here somewhere
14:54 aixsyd dbruhn: just got the 5th server up today, so im testing as we speak
14:56 aixsyd jclift: spent a while with me in PM's eliminating the issue at the IB level. but I stumped him, too.
14:58 semiosis ANNOUNCEMENT: we'll be doing a live hangout starting in one hour, 1600 UTC / 1100 EST.  we'll take your questions in #gluster-meeting -- http://www.gluster.org/2014/01/hangout-with-semiosis-louis-z-today/
14:58 dbruhn That's no good, well at least you've got an idea of where the issue is. That should help, sucks that you have to go through the leg work to figure out whats bad still. From your description I would say you might have a problem with two of the servers
14:58 tor joined #gluster
14:58 dbruhn semiosis, do you have the video feed link yet?
14:59 semiosis i do not
14:59 dbruhn kk
15:01 aixsyd dbruhn: heres an interesting thing to note that i found out about iperf - theres a way to test if your NIC is working at full speeds - just iperf to yourself on the same server via the NIC's outbound IP - and if my IB cards show up as 10GB/s - I know its good. and so far, its been right
15:03 rshade98 joined #gluster
15:03 rshade98 Hey Guys, anyone use monit to monitor the client side of gluster and if the volume is down remount
15:04 kaptk2 joined #gluster
15:05 dbruhn aixsyd did you rule out all of the cabling? seems like that test wouldn't go out the cable just test the nic to the transceiver?
15:05 aixsyd dbruhn: exactly - i have some nics self-testing at 10+gbits and some self-testing at 5, for example. if i just did a server 1 to server 2 iperf - itd show up as 5 on both - no idea which ones giving me the issue
15:06 dbruhn rshade98, I haven't. I did at one time have a nagios system set up to run a umount, and then remount the mount but that was a long time ago.
15:08 aixsyd dbruhn: right now, i think ive narrowed the issue down to server 4 - all because of that self-test. frankly, if i can get 4 out of 5 servers working - thats what I need - no need bothering looking for config differences, or bios versions, etc. 5th (bad) server can become parts :P
15:09 dbruhn hahah, yes they can
15:11 chirino_m joined #gluster
15:12 diegows joined #gluster
15:12 johnmark anyone who wants to participate in the hangout today and ask semiosis touch questions, let me know
15:12 JMWbot johnmark: @3 purpleidea reminded you to: thank purpleidea for an awesome JMWbot (please report any bugs) [2233890 sec(s) ago]
15:12 JMWbot johnmark: @5 purpleidea reminded you to: remind purpleidea to implement a @harass action for JMWbot  [2162654 sec(s) ago]
15:12 JMWbot johnmark: @6 purpleidea reminded you to: get semiosis article updated from irc.gnu.org to freenode [2067184 sec(s) ago]
15:12 JMWbot johnmark: @8 purpleidea reminded you to: git.gluster.org does not have a valid https certificate [138491 sec(s) ago]
15:12 JMWbot johnmark: Use: JMWbot: @done <id> to set task as done.
15:13 TonySplitBrain aixsyd: ..and some day someone will use some part from that pile, and there will be some new mistery & interesting troubleshooting times...
15:14 TonySplitBrain aixsyd: ;-)
15:15 jclift dbruhn: With aixsyd's stuff, we tested the IB layer (it's older Infinihost III Ex) cards, but they're working fine.
15:16 jclift They're all in PCIx8 mode, 5GT/s.  No issues at the electrical level.  OpenSM is seeing everything too.
15:16 jclift The thing that's throwing me, because I have no experience with it, is aixsyd is running each of the two IB links on the adapters in IPoIB bonding mode.
15:17 jclift I have no idea how it interacts with IB.  Sounds like "badly" in some cases, but I don't have the skillz to diagnose (or even investigate really).
15:17 rshade98 @dbruhn do you find that is a waste of time? I am worried about intermittent cloud hiccups
15:18 neofob left #gluster
15:18 dbruhn jclift, same here, I am running qdr just to avoid all of the mess of bonding and what not.
15:18 dneary joined #gluster
15:19 dbruhn rshade98, it allowed me to get a little more sleep here and there, but I am not running it now. Granted I can lose multiple servers on my systems and keep running without any issues.
15:19 jclift Hmmmm.....
15:20 jclift Suddenly I remember that IB can be bonded itself...
15:20 lpabon joined #gluster
15:20 jclift So I wonder if it'd just be easier to bond the IB bonded adapter, and not do the individual IPoIB links?
15:20 dbruhn instead of doing it on the IP layer?
15:20 jclift Yeah
15:20 jclift IB bonding is apparently very good
15:20 jclift Again, haven't used it tho
15:21 jclift (another thing on the "I'd like to look at that sometime" list)
15:21 kl4m joined #gluster
15:23 dbruhn jclift, that list is a mile long
15:23 dbruhn almost as long as the list of books I want to read
15:24 doc|holliday joined #gluster
15:25 rshade98 So i ran into a redundancy/scalability idea the other day. We use scalable arrays for app servers, each has about 500GB of disk. Do you think it would be possible to add every app server to the gluster group? it would theoretically make it faster, but would the sync kill it,every time it scales
15:26 spechal joined #gluster
15:27 KaZeR joined #gluster
15:28 kl4m joined #gluster
15:29 dbruhn rshade98, I run my storage on my application servers as well, the rebalance operation can be a bit messy, and take forever, how often are you talking about adding nodes to the system? and how much data are you talking about storing on it?
15:31 dbruhn Here is how I like to think about your scenario, you are building an onion, it's a bunch of layers. If you ever need to work on the onion, you have to peel the layers back to get to the inside.
15:31 bugs_ joined #gluster
15:32 dbruhn This means, if you are ever working on the storage, you are probably going to have to at some point take the application down on that server too
15:32 dbruhn so if you have built a lot of redundancy into your system to deal with being able to do that, it can work well.
15:33 aixsyd jclift: heya buddy - so I think my problem is on the server hardware - not the IB cards, cable, or config. I moved hard drives around from the known good pair, but the issue seemed to stay with the physical servers
15:35 wushudoin joined #gluster
15:35 aixsyd TonySplitBrain: thats true... i mean, the problem server seems 100% okay sans this IB issue
15:36 aixsyd it could be a bad riser card
15:36 dbruhn swap your risers ;)
15:36 dbruhn finding the problem now will save you hell later
15:36 dbruhn and allow you to order the replacement part so you have a full good server
15:36 aixsyd thats true. i gotta make sure i have the right server first - im allllmost there!
15:36 dbruhn lol\
15:37 aixsyd its shocking how long it takes to move and swap and recable 5 servers
15:37 aixsyd recard, in some cases
15:37 dbruhn it's not a fun day that's for sure
15:37 aixsyd night, day and morning so far @_@
15:37 rshade98 dbruhn nice. Its good to hear someone else is doing it.
15:38 rshade98 in a worse case scenario a new node every couple of minutes.
15:38 dbruhn rshade98, one thing to keep in mind, is that if you ever want to move from gluster.org to redhat storage, redhat doesn't allow your application stack to live in the storage servers.
15:39 rshade98 a couple of gigs usually, they are small web apps.
15:39 rshade98 I think most of the time the attached ebs vol on gluster servers is around 10GB
15:39 dbruhn also, if you are adding servers, you can always mount the file system purely as a client if you don't need the space
15:39 rshade98 dbruhn, how do they stop that?
15:39 dbruhn Are you running a lot of php apps?
15:40 dbruhn redhat storage is released as an iso, and if they find  your applications on the server, they won't support it.
15:40 rshade98 depends on the client, some have 1, some have 100
15:40 rshade98 oh, got ya
15:40 dbruhn JohnMark was telling me a while back that they might be loosening up on that though
15:40 rshade98 this is all cloud
15:41 dbruhn you might want to read up on gluster as php storage
15:41 rshade98 we spent awhile working with old gluster on improving gluster on cloud
15:41 rshade98 we store the php app on the local server, the assets on gluster
15:41 dusmant joined #gluster
15:41 dbruhn ahh ok
15:41 dbruhn thats much better
15:42 rshade98 yeah, don't want to make all your page calls from remote
15:42 rshade98 so you think its feasible to test?
15:43 dbruhn I think I have almost 100TB's of data running on systems in that fashion
15:44 rshade98 oh wow nice. I think my architecture would be two standalone gluster servers, that the main guys, ebs backups,etc, then n number of app servers in the pair
15:44 dylan_ joined #gluster
15:44 dbruhn just remember those caveats and be sure you are willing to deal with them.
15:45 rshade98 yeah, I am going to test, a bunch. I know the more servers you have the faster gluster is. I think it would be super fast this way. Do you have a problem with writes taking awhile?
15:46 dbruhn Honestly my environment is 99.9% random read, I have no issues with writes at all. But I am running 15k sas, and qdr infiniband.
15:46 dbruhn so I am not the norm
15:47 aixsyd how many TB with 15K sas's?
15:47 dbruhn 30
15:47 aixsyd ill have your job plx
15:48 dbruhn you want to be a single admin in charge of a 24x7 online service provider?
15:48 aixsyd sure. im already a single admin
15:48 aixsyd not a 24x7 gig, but
15:48 bc_ joined #gluster
15:48 dbruhn lot's of sleepless nights for me, and when something goes down, I have spent days living on red bull on a data center floor
15:49 samppah dbruhn: i'm not sure if you already said it but what are you storing on glusterfs?
15:49 aixsyd yeah, but you love it
15:49 dbruhn JoeJulian has been a saving grace a couple of times for me helping me work overnight
15:49 dbruhn sampan, encrypted, compressed, and deduplicated backup and recovery data
15:50 samppah dbruhn: okay
15:50 aixsyd dbruhn: i'd take your job in a heartbeat if i didnt have one of these: https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-prn2/1504179_10153637499715713_1731949654_o.jpg
15:50 dbruhn I do love the tech, smaller than my last gig, I just would just like some backup here and there.
15:51 dbruhn Kids are awesome, I don't have a wife or kids.
15:51 aixsyd i love my lil fussy - she helps me NOT have to take 24x7 on-call jobs :P
15:51 dbruhn Ironically I used to be a touring sound engineer, and played in bands. I got off the road to try and normalize my life some....
15:51 aixsyd HA! I used to work in sound, too
15:52 aixsyd Composer/arranger for virtual orchestra
15:52 dbruhn Nice! I worked for Midas and Klark for a long time, and spent way too many hours in clubs and at my house gig.
15:53 dbruhn I do miss touring as a musician though. As much as it was a pain, some great memories
15:53 aixsyd i used to rearrange music from popular video games to make them sound like theyre being played by a real orchestra. I even got an album produced with the help and blessings of Nintendo :>
15:53 dbruhn Everyone should make sure to check out semiosis hangout this morning btw, it's the first one, and he does awesome stuff for the community.
15:54 semiosis thx!
15:54 samppah +1
15:54 dbruhn I still have my recording studio set up in the house, and I sit and compose a bunch, but nothing on that leveel.
15:55 aixsyd 3-disc, 3.5hour album. took 6 years to complete - and we recorded a live 30-piece choir via Kickstarter funding (raised like $35K)
15:55 johnmark here's the youtube link:  https://plus.google.com/events/c1e0kmili7gfqndhjdj66hnpdt4
15:55 dbruhn oh sweet!
15:55 dbruhn thanks john mark
15:55 aixsyd johnmark: thanks
15:55 zerick joined #gluster
15:56 zaitcev joined #gluster
15:57 TonySplitBrain guys, maybe you know, what throutput should i expect between two 4-port 1Gb PCI-E cards with bonding?
15:59 chirino joined #gluster
15:59 failshell joined #gluster
16:00 jclift left #gluster
16:08 daMaestro joined #gluster
16:11 jayunit100 joined #gluster
16:12 theron joined #gluster
16:17 rshade98 your bonding all 8 ports?
16:17 paulczar joined #gluster
16:17 paulczar hey fellas,  is there an official ( or preferred ) chef cookbook for glisters ?
16:17 jclift joined #gluster
16:17 paulczar glusterfs … damn autocorrect
16:18 nullck joined #gluster
16:19 * doc|holliday believes the docs on gluster.org is what you want
16:22 rshade98 paulczar, there are a couple, around the net, not sure of a preferred one: https://github.com/whitekid/chef-glusterfs, https://github.com/amccloud/glusterfs-cookbook/tree/master/recipes
16:22 glusterbot Title: glusterfs-cookbook/recipes at master · amccloud/glusterfs-cookbook · GitHub (at github.com)
16:22 rshade98 we make one, but its tightly rightscale dependent
16:22 vpshastry joined #gluster
16:22 rshade98 https://github.com/rs-services/cookbooks_internal/tree/master/cookbooks/glusterfs, but might give you some ideas on what to do
16:22 vpshastry left #gluster
16:23 TonySplitBrain rshade98: first 4-port card in one server, second one in another, 4 pachcords between ports, bonding this 4 lines to single link
16:24 vimal joined #gluster
16:24 paulczar rshade98: thanks!  I'll poke around at those
16:25 rshade98 Tony, I always think 60 to 70 percent of line speed
16:25 diegows joined #gluster
16:25 theron_ joined #gluster
16:27 rshade98 is anyone listening. do you know if they fixed the dns caching
16:27 rshade98 on the dns aliases thing.
16:29 badone joined #gluster
16:29 TonySplitBrain rshade98: sorry, no ideas
16:41 rshade98 we ran into it still cached the ip so if you updated the name you still had to restart the clients, but haven't tested in awhile
16:42 jayunit100 excellent talk by semiosis learned alot thanks !
16:48 johnbot1_ joined #gluster
16:48 johnbot1_ joined #gluster
16:58 mohankumar joined #gluster
17:01 kl4m joined #gluster
17:18 plarsen joined #gluster
17:26 Mo_ joined #gluster
17:28 mohankumar joined #gluster
17:34 _pol joined #gluster
17:37 _pol_ joined #gluster
17:38 _pol_ joined #gluster
17:38 jclift left #gluster
17:42 ProT-0-TypE joined #gluster
17:46 neofob joined #gluster
17:48 kl4m joined #gluster
17:51 mohankumar joined #gluster
18:03 LoudNoises joined #gluster
18:04 robo joined #gluster
18:06 redbeard joined #gluster
18:07 mohankumar joined #gluster
18:15 mohankumar joined #gluster
18:29 davinder joined #gluster
18:34 _pol joined #gluster
18:48 zapotah joined #gluster
18:48 zapotah joined #gluster
18:56 RedShift joined #gluster
19:12 jag3773 joined #gluster
19:27 Gugge joined #gluster
19:29 andreask joined #gluster
19:30 Technicool joined #gluster
19:39 sroy joined #gluster
19:41 dbruhn__ joined #gluster
19:52 SirHandB1nana joined #gluster
19:53 Liquid-- joined #gluster
19:58 SirHandB1nana Hey guys, anyone available to help with a problem I'm experiencing? I'm running into an issue where gluster is consistantly returing st_nlink=0 for many files within one of my gluster volumes. Running ls in the volume shows the file is there, but the number of hardlinks is 0.
19:59 SirHandB1nana This issue is persisting after remounting the volume as well.
20:02 SirHandB1nana Does gluster have a metadata cache?
20:04 RedShift joined #gluster
20:13 khushildep joined #gluster
20:15 RedShift joined #gluster
20:17 dbruhn SirHandB1nana, what happens if you stat the file?
20:18 dbruhn and this is from the client side you are seeing these issues right?
20:23 SirHandB1nana stat example.jpg
20:23 SirHandB1nana stat: cannot stat `example.jpg': No such file or directory
20:23 SirHandB1nana ls -l | grep example.jpg
20:23 SirHandB1nana -rw-rw-r--  0 apache apache   56483 Dec 20 15:25 example.jpg
20:24 dbruhn have you checked your logs for the client, and for the servers?
20:24 SirHandB1nana Yes, I wasn't able to find anything relevant
20:25 dbruhn can you find the files on the bricks?
20:27 SirHandB1nana Yes the files exist in the bricks, and have 1 or greater hardlinks
20:28 dbruhn What version are you on?
20:29 SirHandB1nana glusterfs 3.4.2 built on Jan  3 2014 12:38:06
20:29 SirHandB1nana And I actually just found something in the logs
20:29 SirHandB1nana [2014-01-10 20:26:57.197601] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: eeae472b-1d13-4590-9ead-fcfd2e7daba1: failed to resolve (Invalid argument)
20:29 SirHandB1nana [2014-01-10 20:26:57.197663] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 478541: GETATTR 140359096947008 (eeae472b-1d13-4590-9ead-fcfd2e7daba1) resolution failed
20:29 SirHandB1nana that error is logged when running a stat on the jpg
20:30 dbruhn Interesting, so what it looks like you are seeing is an error on trying to access the extended attributes
20:30 dbruhn what os?
20:30 SirHandB1nana cat /etc/redhat-release
20:30 SirHandB1nana CentOS release 6.5 (Final)
20:30 dbruhn is selinux running?
20:30 SirHandB1nana no
20:31 dbruhn gluster is running as root and has access to the fs?
20:31 SirHandB1nana Yep
20:32 dbruhn filesystem for the bricks is ifs?
20:32 dneary joined #gluster
20:32 dbruhn xfs
20:33 SirHandB1nana ext4
20:33 dbruhn shouldn't be any issues there
20:34 dbruhn can you check the extended attributes on the file in the bricks
20:34 dbruhn getfattr -m . -d -e hex /path/to/file
20:34 SirHandB1nana sec
20:36 SirHandB1nana # file: example.jpg
20:36 SirHandB1nana trusted.afr.public-client-0=0x000000000000000000000000
20:36 SirHandB1nana trusted.afr.public-client-1=0x000000000000000000000000
20:36 SirHandB1nana trusted.gfid=0xeeae472b1d1345909eadfcfd2e7daba1
20:37 SirHandB1nana Check this out http://pastebin.ca/2532952
20:37 glusterbot Title: pastebin - Untitled - post number 2532952 (at pastebin.ca)
20:38 dbruhn what was the output of that command on the second brick?
20:38 dbruhn and looking
20:39 SirHandB1nana # file: example.jpg
20:39 SirHandB1nana trusted.afr.public-client-0=0x000000000000000000000000
20:39 SirHandB1nana trusted.afr.public-client-1=0x000000000000000000000000
20:39 SirHandB1nana trusted.gfid=0xeeae472b1d1345909eadfcfd2e7daba1
20:39 SirHandB1nana exactly the same
20:40 JoeJulian All files on the bricks will have at least 1 hardlink. That 1 hardlink will not show up in the client. See ,,(.glusterfs)
20:40 glusterbot JoeJulian: Error: No factoid matches that key.
20:40 dbruhn is this only happening from your website, or if you copy files into the file system manually too?
20:41 JoeJulian @learn .glusterfs as For information about the .glusterfs directory tree, see http://joejulian.name/blog/what-is-this-new-glusterfs-directory-in-33/
20:41 glusterbot JoeJulian: The operation succeeded.
20:41 SirHandB1nana Its affecting manual files too
20:43 khushildep joined #gluster
20:43 theron_ joined #gluster
20:46 dbruhn Were you doing anything to the volume when this started happening?
20:48 JoeJulian SirHandB1nana: Can you do your test using readdirplus? I'm curious.
20:48 SirHandB1nana Nope, just noticed something was wrong due to broken images on my site.
20:48 SirHandB1nana One min @JoeJulian
20:51 SirHandB1nana $ apropos readdirplus
20:51 SirHandB1nana readdirplus: nothing appropriate
20:54 dbruhn Are you using the Fuse client, or NFS?
20:54 SirHandB1nana I'm using the Fuse client
20:55 khushildep joined #gluster
20:56 dbruhn what kind of volume btw?
20:56 dbruhn or what is the output of "gluster volume info"
20:56 SirHandB1nana Type: Replicate
20:56 SirHandB1nana Gluster replicated between 2 webheads
20:58 dbruhn this bug looks like it could be related
20:58 dbruhn https://bugzilla.redhat.com/show_bug.cgi?id=875076
20:58 glusterbot Bug 875076: unspecified, low, ---, sgowda, CLOSED NOTABUG, [RHEV-RHS] Storage domain becomes inactive after rebalance
20:58 dbruhn except it doesn't sound like you took the same actions in the bug report
20:58 SpeeR joined #gluster
20:59 _pol_ joined #gluster
21:01 SirHandB1nana And the strange thing here is that lstat(3) returns a value after readdir(3), whereas in the example case, he's getting a dirent, but no stat(2) (as evidenced by the telltale ??????????? business in his "ls" output).
21:01 dbruhn have you looked at the brick logs?
21:04 sroy_ joined #gluster
21:08 kl4m joined #gluster
21:10 SirHandB1nana [2014-01-10 09:26:42.605190] I [server-handshake.c:567:server_setvolume] 0-public-server: accepted client from somehost2-25488-2014/01/10-09:26:42:571433-public-client-1-0 (version: 3.4.2)
21:10 SirHandB1nana [2014-01-10 16:09:18.457225] I [server.c:762:server_rpc_notify] 0-public-server: disconnecting connectionfrom somehost1-10235-2014/01/10-09:24:49:332534-public-client-1-0
21:10 SirHandB1nana [2014-01-10 16:09:18.457283] I [server-helpers.c:729:server_connection_put] 0-public-server: Shutting down connection somehost1-10235-2014/01/10-09:24:49:332534-public-client-1-0
21:10 SirHandB1nana [2014-01-10 16:09:18.457332] I [server-helpers.c:617:server_connection_destroy] 0-public-server: destroyed connection of somehost1-10235-2014/01/10-09:24:49:332534-public-client-1-0
21:10 SirHandB1nana [2014-01-10 16:09:21.962245] I [server-handshake.c:567:server_setvolume] 0-public-server: accepted client from somehost1-22750-2014/01/10-16:09:21:880056-public-client-1-0 (version: 3.4.2)
21:10 SirHandB1nana Not sure if thats relevant
21:11 SirHandB1nana But that was all I was able to find
21:17 Staples84 joined #gluster
21:23 dbruhn i'm puzzled just as you are, there are several sets of logs I would start digging through them and see if you can find anything as a starting point.
21:23 dbruhn the fact that you have "trusted.afr.public-client-0" showing up to me says you had something in a split-brain state at one point, but I am not sure if that's meaningful to the conversation at this point.
21:24 dbruhn also have you run a "gluster volume status" to make sure all of the bricks are up?
21:25 SirHandB1nana All of the bricks are up according to volume status
21:26 sroy_ joined #gluster
21:33 SirHandB1nana No files are in a split brain state currently according to heal info split-brain, I dont think thats relevant.
21:39 dbruhn I didn't think so, just looked like it had been at one point in time.
21:39 dbruhn were you able to look at all the different logs?
21:39 SirHandB1nana I believe it was in a split-brain state due to an upgrade, the brick has been removed and rebuilt since.
21:40 SirHandB1nana I haven't found anything relevant yet in the logs
21:47 _pol joined #gluster
21:51 andreask joined #gluster
21:54 SirHandB1nana Now that I think about this problem more, it seems this could be a post update issue... Im going to try shutting gluster down, move the brick data to my home directory, remove the .glusterfs directories and the trusted.glusterfs.volume-id / trusted.gfid extended attributes. Then completely reinstall the current version of gluster with a clean dataset.
21:55 dbruhn It's really odd
21:55 dbruhn Sorry I haven't been of more help
21:55 SirHandB1nana Yeah lol, I dont know what else to try other than nuking from orbit
21:55 dbruhn hopefully it's not a ton of data that's a pain in the ass to move around
21:56 SirHandB1nana It's cool, thanks for helping :) Only a couple gigs so it shouldnt be too much of a pain.
21:57 dneary joined #gluster
22:13 Liquid-- joined #gluster
22:17 paulczar left #gluster
22:20 failshel_ joined #gluster
22:55 robo joined #gluster
23:25 tryggvil joined #gluster
23:29 edong23 joined #gluster
23:39 badone joined #gluster
23:52 jiphex left #gluster
23:53 MrNaviPacho joined #gluster
23:59 robo left #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary