Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-09-05

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 ttk elico - try http://www.gluster.org/community/documentation/index.php/Planning35
00:00 elico thanks!
00:01 elico but is it stable or beta? I am unsure.
00:01 elico ttk: ping *
00:03 JoeJulian elico: The devs consider it stable. My opinion is that it gets my production blessing with the release of 3.5.3 - beta of which is released next week.
00:04 elico Nice. thanks JoeJulian
00:10 ttk elico - sorry, I'm only paying intermittent attention here.  There's also http://www.gluster.org/community/documentation/index.php/Planning34 if you want the previous version's features.
00:10 ttk but it sounds like JoeJulian made you happy
00:19 gildub joined #gluster
00:19 bennyturns joined #gluster
00:23 elico indeed and thanks ttk
00:29 elitecoder joined #gluster
00:29 elitecoder What settings would you suggest to look at if you suspect your gluster setup is sending too much data?
00:32 elitecoder Bandwidth usage just seems higher than it should be and I'm wondering if I had a setting wrong or something
00:32 elitecoder Outgoing bw, from the gluster server to clients
00:35 msolo joined #gluster
00:36 msolo i'm having a problem write corruption on gluster 3.5.1
00:37 msolo if i have a process appending to a file and a second process reads that same file while writing, some writes go missing
00:37 msolo it's fairly reproducible
00:38 coredump joined #gluster
00:38 msolo i don't get the problem if the file reader is something like tail, but if it's something glibc based that calls lseek a lot, the corruption happens fairly reliably
00:39 msolo i can't tell if this is a tuning issue, where gluster is in an unsafe mode
00:39 msolo but it doesn't seem correct that having a reader should modify the behavior of a writer
00:44 ttk msolo - glibc-based file I/O buffers writes to local memory and only writes to the filesystem intermittently unless you use fflush(3) appropriately.  Are you flushing your buffers to assure that your writes are going through to the filesystem?
00:44 msolo yes, strace shows the expected coalesced writes
00:46 ttk would you be willing to share your code via http://paste.scsys.co.uk so I could take a look at what you're doing?
00:46 glusterbot Title: pasting to magnet_web (at paste.scsys.co.uk)
00:48 msolo sadly, this is part of the logging system of a much larger app that is not readily decomposed
00:49 msolo i can dump some straces
00:49 msolo when I write a trivial python program to simulate this, I don't get the failure
00:49 msolo so it is something about this application that is tickling the failure
00:53 msolo it feels like it is related to mutliple seek calls
00:53 msolo but i'm totally guessing
00:53 msolo haven't been stumped like this in a while
01:09 elitecoder What settings would you suggest to look at if you suspect your gluster setup is sending too much data?
01:12 msolo elitecoder: i would confirm that you are sending too much first. do you have some basic network interface monitoring?
01:12 msolo elitecoder: i would send a 100MB file and see how much traffic that generates in your gluster.
01:12 elitecoder well let me put it this way, gluster is sending 50 gigs a day, but our whole website only does 300 a month
01:14 elitecoder so if you're aware of a particular setting I should look at, that's what im looking for
01:15 msolo i would look at your replication factor in the volume setup, presuming it is setup as replicated
01:15 msolo that will mutliply data transfer obviously
01:16 msolo ttk: here is another interesting data point
01:16 doo joined #gluster
01:16 msolo ttk: when i read the file, i get two different errors
01:17 msolo ttk: i get an error at one offset while the files is being written, and a nearby offset once the file has been closed for writing
01:17 msolo this feels like a caching issue
01:17 Pupeno joined #gluster
01:26 Pupeno_ joined #gluster
01:27 elitecoder what's this mean?
01:27 elitecoder remote operation failed: No data available
01:30 dare333 joined #gluster
01:30 dare333 Question: Let's say I have a gluster spread across two VPC's and I want the machines in the distinct VPC's to have affinity to the more local node. How would I do that, if possible?
01:45 vimal joined #gluster
01:47 elitecoder left #gluster
01:48 toordog joined #gluster
01:59 diegows joined #gluster
02:05 RaSTar joined #gluster
02:15 JoeJulian give them chocolate and flowers.
02:16 JoeJulian dare333: in a replicated volume, a client mounted on one of the servers will try to read from the local server first since 3.4
02:25 aravindavk joined #gluster
02:30 haomaiwa_ joined #gluster
02:35 harish joined #gluster
02:35 _Bryan_ joined #gluster
02:35 side_control joined #gluster
02:37 aravindavk joined #gluster
02:41 hagarth joined #gluster
03:04 coredump joined #gluster
03:18 Pupeno joined #gluster
03:21 lalatenduM joined #gluster
03:44 nbalachandran joined #gluster
03:44 gildub joined #gluster
03:44 bharata-rao joined #gluster
03:46 shubhendu joined #gluster
03:48 spandit joined #gluster
03:50 kanagaraj joined #gluster
03:52 itisravi joined #gluster
03:52 doo joined #gluster
03:57 plarsen joined #gluster
04:12 msolo joined #gluster
04:19 ndarshan joined #gluster
04:19 dmyers joined #gluster
04:23 jiku joined #gluster
04:25 aravindavk joined #gluster
04:25 ramteid joined #gluster
04:32 kshlm joined #gluster
04:35 side_control joined #gluster
04:36 doo joined #gluster
04:39 KenShiro|MUPF joined #gluster
04:41 rafi1 joined #gluster
04:42 Rafi_kc joined #gluster
04:43 msolo joined #gluster
04:43 anoopcs joined #gluster
04:45 atinmu joined #gluster
04:50 doo joined #gluster
04:50 jiffin joined #gluster
05:04 rafi1 joined #gluster
05:16 glusterbot New news from newglusterbugs: [Bug 1125134] Not able to start glusterd <https://bugzilla.redhat.com/show_bug.cgi?id=1125134>
05:20 side_control joined #gluster
05:23 nishanth joined #gluster
05:24 ppai joined #gluster
05:28 dusmant joined #gluster
05:28 green_man joined #gluster
05:30 ws2k33 joined #gluster
05:30 side_control joined #gluster
05:30 kdhananjay joined #gluster
05:34 Paul-C joined #gluster
05:34 Paul-C left #gluster
05:41 Philambdo joined #gluster
05:41 side_control joined #gluster
05:44 kumar joined #gluster
05:55 lalatenduM joined #gluster
05:57 deepakcs joined #gluster
06:01 bala joined #gluster
06:03 saurabh joined #gluster
06:04 ndarshan joined #gluster
06:05 rjoseph joined #gluster
06:08 sputnik13 joined #gluster
06:09 rgustafs joined #gluster
06:19 jtux joined #gluster
06:19 raghu joined #gluster
06:21 gildub joined #gluster
06:24 dmyers does anyone know why it would take about 10 seconds to copy the wordpress code over to a glusterfs mount? i have 2 gluster servers running m3.large and a web server which i used which is a t2.micro
06:25 atalur joined #gluster
06:27 mariusp joined #gluster
06:30 hagarth joined #gluster
06:30 hagarth left #gluster
06:31 Alex dmyers: You might need to give a little more context than that - given no background, the wordpress code could be 2TB and that's a quick copy ;-)
06:31 hagarth joined #gluster
06:31 rjoseph left #gluster
06:31 Alex When copying, do you find that it is individual file operations that take a long time (i.e. slow throughput) or that the amount of files seems to take a long time?
06:32 ndarshan joined #gluster
06:33 rajesh joined #gluster
06:35 meghanam joined #gluster
06:35 meghanam_ joined #gluster
06:46 gildub joined #gluster
06:47 glusterbot New news from newglusterbugs: [Bug 1138541] Geo-replication configuration should have help option. <https://bugzilla.redhat.com/show_bug.cgi?id=1138541>
06:55 green_man joined #gluster
06:59 ctria joined #gluster
07:02 ctrianta joined #gluster
07:10 mhoungbo joined #gluster
07:11 nishanth joined #gluster
07:16 itisravi_ joined #gluster
07:19 getup- joined #gluster
07:21 R0ok_ joined #gluster
07:22 atinmu joined #gluster
07:23 rjoseph joined #gluster
07:24 aravindavk joined #gluster
07:30 hagarth joined #gluster
07:31 nishanth joined #gluster
07:32 rgustafs joined #gluster
07:32 siel joined #gluster
07:33 fsimonce joined #gluster
07:37 VerboEse joined #gluster
07:40 rgustafs joined #gluster
07:42 ppai joined #gluster
07:49 rjoseph joined #gluster
07:50 dmyers does anyone have some good guides on how to configure glusterfs for small files?
07:52 aravindavk joined #gluster
07:58 monotek joined #gluster
07:59 hagarth joined #gluster
08:00 atinmu joined #gluster
08:00 sputnik13 joined #gluster
08:12 getup- hi, when you're rebalancing a volume because you've expanded it's capacity, what kind of performance penalty should you expect?
08:23 karnan joined #gluster
08:25 asku joined #gluster
08:30 ricky-ti1 joined #gluster
08:33 stickyboy I have gigabytes of these messages in my volume log file:
08:33 stickyboy I [dict.c:370:dict_get] (-->/usr/lib64/glusterfs/3.5.2/xlator/performance/md-cache.so(mdc_lookup+0x318) [0x7fd3c79ec518] (-->/usr/lib64/glusterfs/3.5.2/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113) [0x7fd3c77d1c63] (-->/usr/lib64/glusterfs/3.5.2/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x233) [0x7fd3c75c33d3]))) 0-dict: !this || key=system.posix_acl_default
08:33 glusterbot stickyboy: ('s karma is now -26
08:33 glusterbot stickyboy: ('s karma is now -27
08:33 glusterbot stickyboy: ('s karma is now -28
08:33 stickyboy glusterbot++
08:33 glusterbot stickyboy: glusterbot's karma is now 6
08:34 stickyboy This is apparently an informational warning...?
08:34 gothos joined #gluster
08:35 gothos Hey! I've a question regarding performance translators. I see various configs for ie. cache-size, one for writebehind and one for io-cache
08:35 gothos I guess these should be seperate values, but I have no clue how to set them
08:36 gothos I know I can set vars via gluster volume vol_name VARIABLE VALUE
08:36 gothos but how do I specify the translator?
08:37 Slashman joined #gluster
08:40 ctria joined #gluster
08:41 cyberbootje joined #gluster
08:47 glusterbot New news from newglusterbugs: [Bug 1138567] Disabling Epoll (using poll()) - fails SSL tests on mount point <https://bugzilla.redhat.com/show_bug.cgi?id=1138567>
08:54 ctrianta joined #gluster
08:59 stickyboy gothos: We don't specify translators anymore.
09:00 stickyboy Those are probably old guides.
09:00 stickyboy Back in the pre 3.3 (?) days people used to edit their vol files manually, stacking translators to their liking.
09:05 ctria joined #gluster
09:07 ctrianta joined #gluster
09:07 shubhendu_ joined #gluster
09:10 ppai joined #gluster
09:12 gothos stickyboy: yeah, I just saw how to set cache-size differently for writebehind (window-size), but the wiki (http://gluster.org/community/documentation/index.php/Translators/performance) is not all that descriptive that way IMHO
09:17 jmarley joined #gluster
09:18 aravindavk joined #gluster
09:20 siel joined #gluster
09:21 hagarth joined #gluster
09:25 NigeyS joined #gluster
09:35 ppai joined #gluster
09:36 ndarshan joined #gluster
09:37 ctria joined #gluster
09:38 mariusp joined #gluster
09:38 kumar joined #gluster
09:41 dmyers is there a simple way to add another brick to a replica 2 gluster volume?
09:42 dmyers im not sure i understand how it works i just want every gluster server to have the same files and to add another (3rd server) to an existing 2 server setup
09:43 dmyers gothos: what is the current preferred way to set it?
09:43 dmyers gothos: do you know of the current way to configure for small files too? jw
09:45 dmyers running volume info i get this: Number of Bricks: 1 x 2 = 2 --- shouldn't it just say Number of Bricks: 2 ?
09:45 glusterbot dmyers: -'s karma is now -2
09:46 calum_ joined #gluster
09:47 glusterbot New news from newglusterbugs: [Bug 1138602] DHT: nfs.log getting filled with "I" logs <https://bugzilla.redhat.com/show_bug.cgi?id=1138602>
09:48 gothos dmyers: not sure, small files should benefit from writebehind quite a bit (setting window-size, I have that at 512MB), bit I didn't do any benchmarks here yet
09:49 gothos the problem with small files (in the write case at least) isn't the disk io, but the cpu power if you are using fuse gluster
09:49 dmyers oh i gotcha hm
09:49 gothos same goes for small read/wirtes
09:49 gothos r/w with 1MB block sizes gets me like 400MB/s here
09:50 gothos well read is at ~800MB/s
09:50 gothos write is so slow due to the replication
09:50 dmyers dang
09:50 dmyers my writes are 100MB/s on 2 nodes, how many do you run?
09:50 gothos 2 nodes
09:50 gothos each with 2x 12 drives in RAID6
09:51 dmyers im testing with SSD on 2 Digitalocean droplets
09:52 ricky-ticky1 joined #gluster
09:54 gothos I'll add another smaller node in a few months tho, I just need to migrate all the data to the new file server, which is going to be a major pain
09:55 gothos btw. do you guys actually backup your bigger gluster volumes? I've no clue how to do that efficiently so far with ~50TB
09:56 dmyers yikes not sure what i'm doing wrong now then
09:56 gothos is your networking limited to 100MB/s?
09:56 dmyers i thought i read that DO uses 1gbps between droplets
09:58 gothos hm, don't know, my DO machine at least gets 150MB/s read/write with bigger blocksizes, so that shouldn't be a problem
10:12 ricky-ticky joined #gluster
10:15 atalur joined #gluster
10:17 Norky_ joined #gluster
10:20 rjoseph joined #gluster
10:20 nbalachandran joined #gluster
10:21 atinmu joined #gluster
10:23 mariusp joined #gluster
10:28 ctria joined #gluster
10:33 mariusp joined #gluster
10:33 kanagaraj joined #gluster
10:33 spandit joined #gluster
10:37 bene2 joined #gluster
10:41 karnan joined #gluster
10:43 sputnik13 joined #gluster
10:47 glusterbot New news from newglusterbugs: [Bug 1138621] Wrong error handling for mmap() syscall in gf-changelog-process.c FILE <https://bugzilla.redhat.com/show_bug.cgi?id=1138621>
10:48 edward1 joined #gluster
10:48 sputnik13 joined #gluster
10:50 qdk joined #gluster
10:51 B21956 joined #gluster
10:58 aravindavk joined #gluster
10:58 nbalachandran joined #gluster
10:58 atinmu joined #gluster
11:00 getup- joined #gluster
11:00 elico joined #gluster
11:16 hagarth joined #gluster
11:24 liquidat joined #gluster
11:31 soumya joined #gluster
11:31 diegows joined #gluster
11:31 morse joined #gluster
11:37 chirino joined #gluster
11:38 B21956 joined #gluster
11:44 ira joined #gluster
11:48 mojibake joined #gluster
11:50 recidive joined #gluster
11:58 soumya joined #gluster
12:12 LebedevRI joined #gluster
12:26 rgustafs joined #gluster
12:26 VeggieMeat joined #gluster
12:26 elico joined #gluster
12:35 aravindavk joined #gluster
12:35 theron joined #gluster
12:35 Philambdo joined #gluster
12:36 Guest36621 joined #gluster
12:37 Guest36621 joined #gluster
12:46 soumya joined #gluster
12:51 plarsen joined #gluster
12:52 plarsen joined #gluster
13:01 eshy joined #gluster
13:02 recidive joined #gluster
13:04 mojnick joined #gluster
13:04 mojnick Hi guys i need help around gluster server problem, anyone willing to help?
13:07 mrEriksson Ask your question and someone will probably answer sooner or later
13:08 mojnick I have two servers in replication working great, i'm testing it so i can implement it on live later.
13:08 mojnick One of them is down
13:09 mojnick instance goes down to be more precise
13:09 deeville joined #gluster
13:09 mojnick then when it goes up again it obtains the new IP address
13:09 Pupeno_ joined #gluster
13:09 jmarley joined #gluster
13:09 mojnick after that i cannot start gluster server
13:09 mojnick how to fix this?
13:10 mojnick What is the solution for the gluster server when the IP is changed?
13:11 Guest36621 joined #gluster
13:11 Guest36621 joined #gluster
13:11 mojnick The other one is working, it is still on. I just need way to start the one that was down again. How, please anyone?
13:11 siel joined #gluster
13:11 mrEriksson Your other server will not recognize the rebooted server since it has a new ip
13:12 theron joined #gluster
13:12 tdasilva joined #gluster
13:12 mojnick I cannot even start the server that got the new ip address.
13:12 NigeyS mojibake can you use a static ip? if your on AWS you can assign static internal ips, and external ips via elastic ip
13:12 NigeyS mojnick*
13:12 mojnick yes i'm on aws
13:12 NigeyS dam auto complete
13:12 mojnick but it want ip to be dynamic
13:13 mojnick because my servers are in different availability zones
13:13 mojnick they cannot talk over internal ip's
13:13 NigeyS hmm, should still be able to assign a static internal ip within the VPC
13:14 mojnick probably but i want them to stay dynamic for safety purposes
13:15 NigeyS i dont see a way you can do that and have gluster know where its meant to connect to other peers if the ip keeps changing
13:19 bala joined #gluster
13:19 mojnick ok, is there a way i can manually change the ip to the new one so server can figure that out and start again
13:19 DJClean joined #gluster
13:21 dusmant joined #gluster
13:28 ndevos mojnick: instead of creating the volumes by specifying IP-addresses, you can specify the hostnames
13:28 eshy joined #gluster
13:29 ndevos mojnick: services like dyndns can provide hostnames for dynamic ip-addresses
13:29 ndevos but probably there is some AWS option for that too?
13:33 rjoseph joined #gluster
13:34 bennyturns joined #gluster
13:37 hybrid512 joined #gluster
13:40 kkeithley1 joined #gluster
13:53 johnmark_ joined #gluster
14:00 itisravi_ joined #gluster
14:02 julim joined #gluster
14:02 bene2 who maintains the gluster.org page?  I have a question about content of Downloads link not being up to date
14:02 xrandr_work joined #gluster
14:02 ndevos JustinClift: ^
14:02 mojnick ok, how to detach server from trusted pool? When i type "gluster peer detach SERVER IP" it says that bricks with peer SERVER IP exist in cluster
14:02 SpComb mojnick: you have to replace the bricks first
14:03 SpComb are you able to peer probe the old server at it's new address?
14:03 shubhendu_ joined #gluster
14:03 mojnick no, server is down, i want now to remove it from trusted pool and add new one in that place with new ip
14:03 mojnick the question is how?
14:03 elico left #gluster
14:03 elico joined #gluster
14:03 elico left #gluster
14:03 SpComb what does it error out with on startup?
14:03 elico joined #gluster
14:03 NigeyS http://pastie.org/9529515      are those erors cause for concern? everything appears to be working as it should...
14:03 glusterbot Title: #9529515 - Pastie (at pastie.org)
14:03 soumya joined #gluster
14:03 kkeithley1 joined #gluster
14:04 xrandr_work joined #gluster
14:05 xrandr_work left #gluster
14:05 xleo joined #gluster
14:05 recidive joined #gluster
14:06 gmcwhistler joined #gluster
14:08 shubhendu_ joined #gluster
14:09 skippy does mount.glusterfs support remounting?
14:09 chirino joined #gluster
14:09 skippy i'm trying to remount in order to change some mount options, but I get "GlusterFS is already mounted on ..."
14:10 hagarth joined #gluster
14:11 _Bryan_ joined #gluster
14:14 wushudoin| joined #gluster
14:16 NigeyS skippy im guessing not ... http://lists.gnu.org/archive/html/gluster-devel/2014-04/msg00075.html
14:16 glusterbot Title: [Gluster-devel] Need for -o remount support for FUSE mount (at lists.gnu.org)
14:24 skippy bummer.
14:26 harish joined #gluster
14:27 eshy joined #gluster
14:34 aronwp joined #gluster
14:37 aronwp hey guys, looking for a little advice. my current setup has 1 loadbalancer in front of 2 web servers (web1 & web2) with a DB server behind that. I have gluster running in replicate mode on both web servers and I'm running into high cpu usage on both nodes periodically
14:38 aronwp I have about 35 websites on both nodes and some get a decent amount of traffic
14:38 aronwp I'm planning to spin up some servers to run just gluster but i'm not sure what would be a good place to start
14:38 aronwp ideas
14:39 aronwp 2 2gb servers running in replicate mode or 4 1gb servers with replicate 2?
14:39 itisravi_ joined #gluster
14:40 Gugge joined #gluster
14:47 rwheeler joined #gluster
14:50 aravindavk joined #gluster
14:52 foobar aronwp: those gb's... are they ram, storage or network-link-speed ;)
14:54 aronwp lol... ram, the 1gb (ram) have 24gb ssd's, 1 vcpu & 250 Mbit out
14:55 aronwp the 2gb (ram) have 48gb sad's 2 cpu cores and 250 Mbit out
14:55 aronwp sorry the 1gb have 120 Mbit out
14:55 aronwp *125
14:56 foobar you can also replicate-3 btw...
14:56 foobar that's what I currently run with
14:56 foobar so failure of 1 will still have redundancy left over...
14:57 foobar 4 would be overkill I guess
14:58 aronwp i was thinking of replicate-3 but i heard the more replicate the worse the performance
14:58 foobar yup
14:58 aronwp are you running replicate-3 on 4 bricks?
14:58 foobar client needs to send data to every host
14:58 foobar no... just 3 servers in total
14:59 aronwp all replicated? how is the performance?
14:59 foobar with >20 bricks each... but that's due to massive amounts of data here
14:59 foobar crap
14:59 aronwp oh
14:59 foobar it's spinning metal disks too...
14:59 foobar no ssd's
14:59 foobar that might help a lot though
14:59 foobar but I need 40TB+
15:00 aronwp wow!!!
15:00 foobar so ssd's would be hugely more expensive
15:00 aronwp ya crazy expensive
15:00 foobar but gluster doesn't work very well in this use-case...
15:00 foobar for a 2 node, < 100GB setup... it will work fine though
15:01 foobar self-heal takes weeks on my setup
15:01 aronwp i bet
15:01 foobar migration of all data took 6+ months
15:02 aronwp crazy, there has to be a better way
15:02 aronwp so you think 4 1gb servers or 2 2gb (ram)
15:02 foobar I'm open to suggestions ;)
15:02 aronwp is more better?
15:03 foobar i'd go for the 2 systems then...
15:03 foobar more ram for disk-cache/inode-cache
15:03 foobar my gluster boxes have 64GB ram...
15:04 foobar Mem:  65939744k total, 19234996k used, 46704748k free,   170384k buffers
15:04 foobar Swap:        0k total,        0k used,        0k free, 11200336k cached
15:04 foobar not that they are using all of it...
15:05 nbalachandran joined #gluster
15:07 aronwp are you hosting your own servers
15:08 aronwp i was thinking of gong dedicated instead of a bunch of vps's
15:08 aronwp *going
15:08 aronwp and running XenServer
15:09 foobar aronwp: yeah... colo/physical servers
15:09 foobar db+gluster+fw+lb+appserver is physical, backend webservers are virtual boxes
15:10 foobar but this is a big _BIG_ site
15:10 foobar don't like Xen ... but all virtual stuff is running libvirt/kvm
15:11 msolo joined #gluster
15:11 foobar maybe migrate to ovirt later... but it's been around like this for years
15:12 aronwp i never played with kvm but may give it a shot
15:12 aronwp big site... massive from what I can see. 40TB
15:14 mojibake aronwp:  As for your RAM requirements.. JoeJulian stated his setup for 60volumes is 32GB ram, with the performance.cache lower from  default 32Mb to 8mb...So your CPU maybe due to disk thrashing? Maybe see about adjusting your performance.cache big time, or add much more RAM. (Don't listen to me, I am a newb).
15:14 foobar but yeah... more ram is always better ;)
15:15 foobar though with ssd's ... If you can actually get the I/O's fast.... you should need less cache
15:15 foobar but SSD's on virtual hosts... you never know how much I/O the other guests are doing
15:15 foobar and if you are getting what you can/should
15:19 aronwp would a lower cache get better performance
15:23 zerick joined #gluster
15:25 doo joined #gluster
15:29 todakure joined #gluster
15:40 Philambdo joined #gluster
15:44 jobewan joined #gluster
15:44 justyns joined #gluster
15:44 justyns joined #gluster
15:45 justyns joined #gluster
15:47 mojibake aronwp: Just a newb here.. But efficiently allocating the cache sized for the volume using will allow more active volume to cache and serve requests. You may need to do some volume profiling to see you most active volumes and see which need the cache as it is, or raised, and which volumes can be lowered.
15:51 glusterbot New news from resolvedglusterbugs: [Bug 1138382] DHT + Snapshot :- If snapshot is taken when Directory is created only on hashed sub-vol; On restoring that snapshot Directory is not listed on mount point and lookup on parent is not healing <https://bugzilla.redhat.com/show_bug.cgi?id=1138382> || [Bug 1138383] DHT + rebalance : rebalance process crashed + data loss + few Directories are present on sub-volumes but not visible on mount point + lookup
16:03 getup- joined #gluster
16:07 rotbeard joined #gluster
16:09 julim joined #gluster
16:13 aronwp joined #gluster
16:18 aronwp joined #gluster
16:26 PeterA joined #gluster
16:27 bala joined #gluster
16:27 dtrainor joined #gluster
16:31 longshot902 joined #gluster
16:32 jmarley joined #gluster
16:37 lmickh joined #gluster
16:42 recidive joined #gluster
16:43 soumya joined #gluster
16:47 dtrainor joined #gluster
16:52 zerick joined #gluster
16:54 msmith_ joined #gluster
17:12 dare333 Question: Let's say I have a gluster spread across two VPC's and I want the machines in the distinct VPC's to have affinity to the more local node. How would I do that, if possible?
17:12 dare333 Someone answered this last night for me but my ghetto IRC client didn't have enough scrollback :/
17:13 JoeJulian dare333: in a replicated volume, a client mounted on one of the servers will try to read from the local server first since 3.4
17:13 dare333 How does it know that it is the local server
17:13 JoeJulian programming
17:14 dare333 Magic!
17:14 JoeJulian In a not-smartass answer, I don't know. I haven't read that part of the code.
17:14 semiosis dare333: type /topic to see links to the channel logs
17:14 dare333 Thanks
17:14 JoeJulian Ah, right, there's that too when scrollback fails.
17:15 dare333 I'm hoping to set it manually to point to a local server then failover if needed but it definitely does that automatically?
17:15 semiosis dare333: are you trying to save costs?  why do you care which brick it reads from?
17:16 dare333 Because we have servers in two AZ's trying to hit it. Sometimes, it seems like there is cross AZ traffic
17:16 dare333 From client to server
17:16 semiosis yeah, i get that, but why is that *bad*?
17:16 dare333 Because it makes the transfer much slower
17:16 semiosis does it really?
17:17 justyns joined #gluster
17:17 semiosis you tested that?
17:17 dare333 It seems to, yeah
17:17 semiosis ok then
17:17 semiosis just checking :)
17:17 semiosis sometimes people go crazy with premature optimization
17:18 dare333 Yeah, it's quite a lot of data we're pouring through it and it's been frequent enough for others to notice.
17:21 Gues_____ joined #gluster
17:22 semiosis dare333: what instance types are you using for clients & servers?
17:22 semiosis this is unrelated to your question, but i'm curious what others are using in ec2
17:22 semiosis also, are you in us-east-1 or a different region?
17:23 dare333 I'm at c3.8xlarge for client and hi1.4xlarge for the servers
17:23 dare333 us-east-1
17:23 semiosis wow! sweet!
17:23 semiosis can I ask what you're working on with all that?
17:24 semiosis sounds like a serious gluster deployment
17:24 dare333 GIS data, Satellite imagery
17:24 semiosis cool
17:26 semiosis we had a problem a few months ago with c3/m3 instances that affected gluster.  i only tested on the cheaper end of those ranges, but the issue seems to have been quietly resolved.  AWS never responded to my inquiries
17:26 dare333 What was the issue?
17:27 semiosis glusterfs uses privileged source ports to make connections.  for example, when your client fetches the volume info, it connects to glusterd on port 24007, but the source port it uses will probably be around 990-1000
17:27 semiosis the problem was that the TCP checksums were wrong for such connections
17:27 semiosis when the source port was low
17:27 CyrilP Hi Guys, one question
17:27 dare333 That's crazy
17:27 CyrilP I have 2 gluster nodes with one replicated volume
17:27 CyrilP I need to reinstall one node
17:28 JoeJulian Oh, is that what it was? I thought it was just blocked.
17:28 CyrilP should I remove one brick from the volume or just proceed ?
17:28 CyrilP I would like to avoid any split brain situation
17:28 JoeJulian I would just proceed. Are you going to be wiping the brick?
17:29 CyrilP nope
17:30 CyrilP the brick is a raid export
17:30 JoeJulian I would make sure the "gluster volume heal $vol info" is empty and proceed. If it's not empty, it may be transient due to file activity. Judge that for yourself.
17:30 CyrilP but the last time I shutdown this machine for hardware maintenance, when I brought it up, I had a lot of heal failed
17:30 JoeJulian As long as everything's already healthy, removing the server from operation and returning it later will be fine.
17:31 JoeJulian s/will/should/
17:31 glusterbot What JoeJulian meant to say was: As long as everything's already healthy, removing the server from operation and returning it later should be fine.
17:31 * JoeJulian hates making guarantees
17:31 zerick joined #gluster
17:32 CyrilP nice bot :)
17:32 CyrilP well regarding the heal-failed I still have a bunch of them
17:32 CyrilP looks like more an history
17:33 CyrilP_ joined #gluster
17:33 CyrilP_ for example I have a bunch of gfid in heal-failed
17:34 CyrilP_ I never really understood why
17:34 CyrilP_ as the volume seems healthy
17:35 semiosis JoeJulian: tcpdump showed the packets arriving but they never made it up the stack to the listening socket
17:35 JoeJulian interesting.
17:35 CyrilP_ JoeJulian Is this normal to have like 27 entries for une brick and 55 for the other when gluster volume heal $vol info ?
17:36 JoeJulian CyrilP_: has to do with the way the heal queue works as to why you have gfid entries instead of file names.
17:37 JoeJulian Although I usually have about 300 entries in my heal queues under 3.4, they all have filenames. It's not normal to have gfid heals sitting there. Perhaps run a heal...full to see if you can make them go away.
17:38 JoeJulian btw... 3.5 makes the invalid heal queue entries go away.
17:40 CyrilP_ like a gluster vol heal vol_name full ?
17:40 JoeJulian right
17:41 JoeJulian I've seen that fix the gfids, or at least give them names.
17:41 CyrilP_ nope
17:41 CyrilP_ :p
17:41 CyrilP_ it make the filename disappear (healed)
17:41 CyrilP_ but I have like 70 gfid listed on each bricks
17:41 JoeJulian good?
17:41 zerick joined #gluster
17:41 JoeJulian Ok, do you know where to find the gfid files under .glusterfs?
17:42 JoeJulian er... .gluster
17:42 * JoeJulian needs another espresso this morning.
17:42 CyrilP_ almost, I just have to remember that
17:43 JoeJulian on the brick, .glusterfs/{first two hex digits of the gfid}/{next two digits}/{gfid}
17:44 JoeJulian stat that. If it only has a link-count of 1, you can probably delete it (or put it offline somewhere for safe keeping). If not, use the ,,(gfid resolver)
17:44 glusterbot https://gist.github.com/4392640
17:44 recidive joined #gluster
17:45 CyrilP_ for one of them
17:45 CyrilP_ only present on one brick
17:45 CyrilP_ there is only 1 link
17:47 CyrilP_ ./gfid-resolver.sh /export/raid/vol 40d0498a-073c-41d2-a3ca-cc36feee30b3
17:47 CyrilP_ 40d0498a-073c-41d2-a3ca-cc36feee30b3    ==      File:
17:47 CyrilP_ interesting...
17:48 JoeJulian If it only has a link count of one, there will not be anything for resolver to resolve.
17:48 CyrilP_ ok some sound like an orphan ?
17:49 JoeJulian probably.
17:49 CyrilP_ oh it start to put name on gfid, looks like heal process take time
17:49 JoeJulian It does
17:49 JoeJulian It has to walk the tree
17:50 CyrilP_ ok, what is the real meaning of entry under heal.. info
17:51 zerick joined #gluster
17:51 CyrilP_ because the file on the brick have the same checksum
17:52 JoeJulian It means that the gfid is marked in $brick/.glusterfs/indices/xattrop
17:52 JoeJulian The gfid files are hardlinks.
17:52 CyrilP_ which means ?
17:52 JoeJulian @lucky hardlinks
17:52 glusterbot JoeJulian: http://en.wikipedia.org/wiki/Hard_link
17:52 CyrilP_ yes I know hardlink
17:53 CyrilP_ but I wonder why they are listed with heal..info cmd
17:53 JoeJulian They probably failed to heal for some reason. Check the self-heal log for that.
17:55 CyrilP_ like that ? E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-usr_global-replicate-0: Conflicting entries for /lib/python2.7/yaml/composer.pyc
17:56 JoeJulian yeah, that looks likely.
17:56 CyrilP_ hmm I have a bunch of them
17:56 rajesh joined #gluster
17:57 JoeJulian No idea what it means. Check the ,,(extended attributes) as well as the normal attributes...
17:57 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
17:57 CyrilP_ I may have one hint... there is like 2 minutes gap between my two nodes
18:03 sputnik13 joined #gluster
18:03 CyrilP_ JoeJulian well thanks for you help, time to puppetise my entire glusterfs cluster :p
18:03 JoeJulian :)
18:09 LHinson joined #gluster
18:17 longshot902_ joined #gluster
18:21 msolo joined #gluster
18:24 recidive joined #gluster
18:27 zerick joined #gluster
18:28 JoeJulian joined #gluster
18:28 failshell joined #gluster
18:29 JustinClift bene2: That doesn't sound good (download links thing).  What needs fixing? :)
18:29 failshell with 3.5.2, when setting auth.allow to a volume, it doesn't seem to be possible to use the CIDR format. am i missing something here?
18:29 failshell like 1.2.3.4/24
18:31 bene2 JustinClift, I guess I didn't understand that glusterfs 3.6 is not officially released yet so the latest complete release is 3.5.2, hopefully that's changing soon
18:31 JustinClift failshell: Oh that's interesting.
18:31 JustinClift bene2: http://www.gluster.org/community/documentation/index.php/Planning36
18:32 failshell JustinClift: seems to only allow 1.2.3.*
18:32 JustinClift bene2: Depends on your definition of "soon"
18:32 JustinClift bene2: Don't use the 3.6.0 GA release for production data
18:32 failshell that's easy to find out for a /24, but in my case, we use /23. it requires a bit more magic on my side in Chef
18:32 JustinClift bene2: Historically, we seem to iron out imporant bugs in the .1 and .2 releases afterwards
18:33 JustinClift So, depending on how important your data is to you, is whether you should wait or not
18:34 JustinClift failshell: Would you be ok to email the gluster-users mailing list about that?  It makes sense to me that CIDR format should be supported
18:34 JustinClift Hopefully it's just a minor bug that can be fixed in the next point release
18:34 failshell JustinClift: if you could point me to it, ill gladly do it
18:34 JustinClift failshell: http://supercolony.gluster.org/mailman/listinfo/gluster-users
18:34 glusterbot Title: Gluster-users Info Page (at supercolony.gluster.org)
18:34 bene2 ok, thanks
18:35 NigeyS http://pastie.org/9529515      are those erors cause for concern? everything appears to be working as it should...
18:35 glusterbot Title: #9529515 - Pastie (at pastie.org)
18:36 zerick joined #gluster
18:36 failshell JustinClift: is there a jira/bugzilla somewhere instead? i dont really feel like subscribing to a mailing list just for a feature request
18:37 JustinClift failshell: https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
18:37 glusterbot Title: Log in to Red Hat Bugzilla (at bugzilla.redhat.com)
18:38 JustinClift ;)
18:41 dberry joined #gluster
18:41 dberry joined #gluster
18:49 glusterbot New news from newglusterbugs: [Bug 1138841] allow the use of the CIDR format with auth.allow <https://bugzilla.redhat.com/show_bug.cgi?id=1138841>
18:54 Philambdo joined #gluster
18:58 msolo joined #gluster
19:03 JoeJulian joined #gluster
19:05 ThatGraemeGuy joined #gluster
19:07 zerick joined #gluster
19:10 hagarth joined #gluster
19:11 Pupeno joined #gluster
19:11 JoeJulian joined #gluster
19:13 LHinson1 joined #gluster
19:16 _dist joined #gluster
19:23 bennyturns joined #gluster
19:28 Philambdo joined #gluster
19:30 hagarth joined #gluster
19:34 Pupeno joined #gluster
19:43 Pupeno joined #gluster
19:45 zerick joined #gluster
19:48 doo joined #gluster
19:49 elico joined #gluster
19:52 recidive joined #gluster
19:58 aronwp_ joined #gluster
20:01 rwheeler joined #gluster
20:13 hagarth joined #gluster
20:15 jobewan joined #gluster
20:34 jobewan joined #gluster
20:38 igors joined #gluster
20:41 igors hello. I have a small gluster volume I need to change the bricks because we are replacing the servers. Not sure how to proceed with the migration: http://dpaste.com/24JMDQ5
20:41 glusterbot Title: dpaste: 24JMDQ5 (at dpaste.com)
20:43 _dist igors: easiest is to make a new volume with new brick locations and migrate the data
20:44 igors _dist: and is there a way to migrate the data while the old volume is active and being used?
20:49 Spiculum rsync
20:49 semiosis what about replace-brick?
20:50 igors hehe, that's how I migrated to gluster in the first place :)
20:50 igors I'm not sure I understand the purpose of replace-brick, I tried to replace app2.rg2 with app1.web but I don't see any changes on volume info
20:53 semiosis not just replacing servers, but changing volume layout too
20:54 semiosis easiest would be to make a new volume & copy the data over.  that would require interrupting clients so they could mount the new volume
20:54 igors yes, because gluster wouldn't let me have a replica 2 with 3 bricks
20:54 semiosis otherwise it would be some combination of remove-brick & then change replica
20:54 semiosis you'd first have to remove-brick your second replica pair.  i think that can migrate data now, but i've never tried
20:55 semiosis once you migrate data onto the first replica pair from the second, then you'd do add-brick replica 3 <new-brick>
20:55 semiosis you really should try this all out on a test volume
20:56 igors hum, so if I want to remove a brick at this point I need to remove two at the same time
20:57 igors could those be brick1 and brick2? I suppose they are a replica pair
21:00 semiosis yep i mixed first/second up, you want to remove the first two, leaving only the app*.web
21:00 semiosis but again, try this out on a test volume first
21:00 semiosis before you do it in prod
21:00 semiosis i'm guessing here :)
21:01 igors sure :)
21:02 dtrainor joined #gluster
21:09 bala joined #gluster
21:12 Paul-C joined #gluster
21:16 nated here's a dumb question: ubuntu 12.04, I added _netdev to my mount options to delay mounting of my glsuter fs, sadly, that seems to be ignored and it's trying to mount at boot before anythign is up
21:19 nated (I also have the mounting-glusterfs.conf in upstart...)
21:21 semiosis try nobootwait fstab option
21:22 nated hurm, looks liek I don't have fuse in /etc/modules either
21:22 nated will try both
21:22 semiosis ubuntu kernels have fuse built in, not as a module, afaik
21:23 semiosis what glusterfs version are you using?  packages from the PPA?  either semiosis/ubuntu-glusterfs or gluster/glusterfs?
21:23 nated mine, based off yours :)
21:23 nated 3.5.2
21:23 semiosis oh hi, i knew that nick was familiar!
21:23 nated haha :)
21:26 semiosis nated: the mounting-glusterfs job isn't perfect, but it worked for most people... you should expect to see the wait-for-state process blocking your mount for ~30 seconds after boot
21:26 semiosis if you do ps ax | grep gluster immediately after boot
21:26 nated yeah, it works
21:26 semiosis so, problem solved?
21:26 nated provided I "S" the failed mount
21:26 semiosis S?
21:26 nated "Skip" at the boot prompt
21:27 semiosis oh, ok, nobootwait in fstab will fix that
21:27 nated yeah
21:27 recidive joined #gluster
21:29 nated testing now
21:38 nated yeah, nobootwait still hangs boot
21:38 JoeJulian ubuntu--
21:38 glusterbot JoeJulian: ubuntu's karma is now -1
21:38 semiosis nated: let me see your fstab
21:39 nated http://fpaste.org/131427/14099531/
21:40 glusterbot Title: #131427 Fedora Project Pastebin (at fpaste.org)
21:40 semiosis well, that seems valid.  pretty sure that defaults,_netdev is unnecessary.  try with just nobootwait in the options
21:42 nated k, trying
21:43 nated works, hurm
21:44 ninjabox1 I just like saying 'gluster cluster'
21:44 ninjabox1 that is really the only reason I use gluster.
21:46 semiosis nated: \o/
21:46 nated semiosis: looks like _netdev is the culprit
21:46 semiosis nothing but trouble, that _netdev
21:47 nated integration points are always the first to break
21:47 semiosis _netdev--
21:47 glusterbot semiosis: _netdev's karma is now -1
21:52 dtrainor joined #gluster
21:59 theron_ joined #gluster
22:12 recidive joined #gluster
22:22 JoeJulian I disagree. _netdev works perfectly in the systems that implement it. It's those systems that don't that neither continue to function correctly, nor inform the user of the unsupported nature of that options.
22:22 JoeJulian _netdev++
22:22 glusterbot JoeJulian: _netdev's karma is now 0
22:38 DV joined #gluster
22:40 semiosis @meh
22:40 glusterbot semiosis: I'm not happy about it either
22:54 dtrainor joined #gluster
23:17 xleo joined #gluster
23:18 msolo are there any known data corruption issues involving O_APPEND with Gluster 3.5.1?
23:24 plarsen joined #gluster
23:27 coredump joined #gluster
23:31 Philambdo joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary