Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-07-08

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:15 jv115 joined #gluster
00:27 gildub joined #gluster
00:27 jv115 left #gluster
00:29 Peter3 my df vs du usage on gluster NFS export is getting worse
00:29 joevartuli joined #gluster
00:29 Peter3 http://fpaste.org/116183/79394140/
00:29 glusterbot Title: #116183 Fedora Project Pastebin (at fpaste.org)
00:30 Peter3 any one knows how to find hidden files on glusterfs ??
00:31 JoeJulian What are hidden files?
00:31 Peter3 i have no idea
00:31 Peter3 seems like something consuming the space
00:31 Peter3 it's just an NFS export from gluster for backup files
00:31 Peter3 i did fuser and lsof and see nothing running on it
00:31 Peter3 unmount and remount the usage still there
00:32 Peter3 but du shows the actual usage is tiny
00:32 JoeJulian What's du -l show?
00:33 Peter3 http://fpaste.org/116184/79588140/
00:33 glusterbot Title: #116184 Fedora Project Pastebin (at fpaste.org)
00:35 Peter3 which shows the export really only using 5G
00:35 Peter3 but df shows 1.4TB usage
00:36 obelix_ joined #gluster
00:38 JoeJulian Sorry, wrong du option... -b
00:40 Peter3 http://fpaste.org/116186/79971140/
00:40 glusterbot Title: #116186 Fedora Project Pastebin (at fpaste.org)
00:41 Andreas-IPO joined #gluster
00:46 capri joined #gluster
00:49 Peter3 JoeJulian: Any clue?
00:49 Peter3 how do we list out files and inodes from gluster?
00:49 Peter3 to cross check if any files missing from du and df?
00:50 Peter3 i just umount and mount the NFS export and still using the space...
00:51 JoeJulian Odd that it's almost exactly twice as much.
00:51 Paul-C joined #gluster
00:51 Peter3 its way more then twice??
00:51 Peter3 du shows 5G and df shows 1.4TB used
00:52 JoeJulian meh, decimal places...
00:52 JoeJulian hehe
00:52 Peter3 lol
00:52 Paul-C Hi, has anyone used nfs.mem-factor to improve nfs performance, and if so what's the use case for this tweak to make the most sense?
00:53 Paul-C I have gluster nodes that have 256GB of RAM...shame to waste it :)
00:54 joevartuli joined #gluster
00:55 JoeJulian Ok, I would do a find on all the bricks (pwd inside the brick), excluding .glusterfs. Combine the lists and sort -u. Do a find on the gluster mount (pwd inside the mount) and sort -u that as well (shouldn't be anything not unique, but just to be sure). Then diff them. That'll make sure your filenames match between what's on the bricks and what's on the client.
00:55 joevartuli joined #gluster
00:56 JoeJulian Paul-C: Never heard of anyone doing that.
00:56 joevartuli joined #gluster
00:58 Peter3 i know where were some big files on that mount and it's all gone now
00:59 Peter3 i even looked up inside the bricks and no more
00:59 JoeJulian check lsof on the servers
00:59 Paul-C JoeJulian: I have a read biased workload, and have some benchmark numbers based on the default of 15. I'll run the test suite with 20 and see what magic happens...or not :)
00:59 Peter3 u mean on the gluster server?
00:59 JoeJulian Peter3: yes.
01:00 JoeJulian Paul-C: Cool. Let me know what you find out.
01:00 Paul-C JoeJulian: Will do.
01:00 Peter3 lsof all clean on gluster server too
01:01 Peter3 i also did a volume quota remove on that export and add it back and the 1.4TB usage still there
01:01 JoeJulian Oh, right... quota. If you remove quota entirely is it still wrong?
01:01 Peter3 yes
01:02 Peter3 i removed quota on that path and added back
01:02 Peter3 still wrong....
01:02 JoeJulian Without adding it back
01:02 Peter3 then how can i tell the usage?
01:02 Peter3 the du always shows the right usage
01:02 Peter3 from NFS client and gluster server
01:02 JoeJulian Now I'm confused.
01:02 Peter3 just the df is wrong
01:03 Peter3 me too
01:03 Peter3 what do u confuse at?
01:04 JoeJulian I just am wondering if df is the same without quota.
01:05 JoeJulian Maybe it's reporting the difference in total volume space and quota as being in use.
01:05 JoeJulian ?
01:06 Peter1 joined #gluster
01:06 Peter1 the usage is yes
01:07 Peter1 usage always shows the additional 1.4 TB
01:07 JoeJulian ~pasteinfo | Peter1
01:07 glusterbot Peter1: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
01:09 Peter1 http://fpaste.org/116189/78177614/
01:09 glusterbot Title: #116189 Fedora Project Pastebin (at fpaste.org)
01:11 JoeJulian Oh. You're mounting an export-dir. If you mount sas02 does du -b match df?
01:12 Peter1 http://fpaste.org/116190/40478196/
01:12 glusterbot Title: #116190 Fedora Project Pastebin (at fpaste.org)
01:12 Peter1 yes it does
01:12 JoeJulian Aha
01:14 Peter1 ya so it's not quota
01:14 Peter1 it's more like something does exist but hidden
01:14 JoeJulian Yeah, the stuff in the other directories.
01:15 Peter1 which the other directory?
01:17 JoeJulian In this example http://fpaste.org/116183/79394140/ stuff in /TrafficPrDataFc01,/TrafficDevDataFc01,/SysOpsHom​eSata03,/SdadminSata01,/KeywordMySqlBackupSata03,​/HermesSata01,/DitrackRestoreSata02,/DevMordorNet​appSata01,/DevMordorHomeSata03,/DevMordorDataSata​01,/BmcProdDBBackupSata02,/wave_roSata02,/SwDepot​Sata01,/QAMordorNetappSata01,/QAMordorHomeSata01,​/PrSEIndexSata02,/IqhqDumpSata02,/Iq06DumpSata03,​/IpcamSata02,/HyperionBackupTempSata02,/DssBa
01:17 JoeJulian ckupSata01,/DeviqIqdumpSata01,and /AseIqBackupSata02 will also be using space from the volume but not be visible when you mount NetBackupLogSata01.
01:17 glusterbot Title: #116183 Fedora Project Pastebin (at fpaste.org)
01:19 Peter1 yes but it when i turn on quota-deem-statfs
01:19 Peter1 the df should show whatever used on the path pnly
01:25 JoeJulian Dammit... This should be linked to the upstream bug tracker: https://bugzilla.redhat.co​m/show_bug.cgi?id=1020886
01:25 glusterbot Bug 1020886: high, high, ---, vshastry, CLOSED ERRATA, quota: quota-deem-statfs on  and their is mismatch in the values shown by "df -h" and quota list command
01:26 JoeJulian I know I've asked before... but what version are you using?
01:26 Peter1 3.5.1
01:29 calum_ joined #gluster
01:32 Peter1 if i created a new path on the volume and cp -pr * to the new path and share it, it shows correct usage
01:32 Peter1 if i do a move on the volume and share over the old path, i got an error while try to mount over NFS
01:35 bala joined #gluster
01:37 JoeJulian Sure sounds like a bug. I've emailed Varun for more information and I'll let you know what I hear back.
01:37 Peter1 thanks!!!
01:38 JoeJulian I think I remember something about that bug a long time ago. Where mv didn't update the quotas correctly....
01:39 Peter1 ic
01:45 haomaiwang joined #gluster
01:58 haomaiw__ joined #gluster
02:28 firemanxbr joined #gluster
02:31 joevartuli joined #gluster
02:33 bala joined #gluster
02:37 harish_ joined #gluster
02:39 RioS2 joined #gluster
02:50 Peter3 joined #gluster
02:52 Peter3 anyone experienced posix: statvfs failed on
02:52 Peter3 ?
02:52 Peter3 joeJulian u still hre?
02:52 Peter3 here?
02:53 Peter3 when i remove a directory that configured for path, i keep getting this
02:53 Peter3 [posix.c:2701:posix_statfs] 0-sas02-posix: statvfs failed on /brick02/gfs/NetBackupLogSata01old/: No such file or directory
03:00 flu_ joined #gluster
03:00 semiosis Peter3: you removed that directory through a client mount point?  not directly on the brick right?
03:00 Peter3 yup
03:01 semiosis Peter3: and no i've not experienced that (but curious)
03:01 Peter3 i rmdir on a glusterfs client
03:01 Peter3 and also removed the export with a volume set
03:01 semiosis you get that error in client log when you do the rmdir or after when you do something else?
03:01 Peter3 i got that error on the brick log
03:02 semiosis as a direct result of the rmdir?  or another command after the rmdir?
03:03 Peter3 after rmdir
03:03 semiosis what version of glusterfs?
03:04 Peter3 3.5.1
03:06 semiosis could you please pastie.org your client log file and give the link here?
03:10 semiosis what do you mean by 'removed the export with a volume set' can you show me the command?
03:10 Peter3 ls
03:12 Peter3 http://pastie.org/9366154
03:12 glusterbot Title: #9366154 - Pastie (at pastie.org)
03:12 Peter3 all i got was this
03:13 semiosis Peter3: that doesnt even include the message you mentioned!  the statvfs failed
03:13 Peter3 o that's on the gluster server
03:13 Peter3 let me paste. sorry
03:14 semiosis ah ok, thx
03:14 Peter3 http://pastie.org/9366159
03:14 glusterbot Title: #9366159 - Pastie (at pastie.org)
03:14 semiosis whats the name of the log you pasted first?
03:15 Peter3 from the gluster client's brick log
03:15 semiosis exact filename?
03:15 Peter3 gfs-sas02.log
03:16 Peter3 sas02 is the volume name
03:16 semiosis hm ok
03:16 semiosis are you getting any error/problem in the shell?
03:16 semiosis or just investigating a weird log message?
03:16 Peter3 yes the log keep poping up
03:17 Peter3 i removed the export and been getting these error msg every 10-20 mins
03:17 Peter3 and i just found out there are couple NFS client was still mounting it.
03:18 Peter3 so I just umounted and see if the error msg still pop
03:18 * semiosis confused
03:18 Peter3 so i created the NFS expot with nfs.export-dir
03:18 Peter3 i then experienced the df vs du not match and space keep being comsumed
03:19 Peter3 so i removed the export from the volume and also rm the directory via gfs client
03:19 Peter3 then start getting these error from the brick log
03:19 semiosis hmmm, sounds like new features i'm not familiar with (yet).  i probably wont be much help, sorry :(
03:20 Peter3 no prob
03:20 Peter3 thanks for looking!
03:20 Peter3 i m new to gluster too
03:20 Peter3 is nfs.export-dir new?
03:23 semiosis i assume so, because this is the first i've heard of it & i've been using gluster a while
03:24 Peter3 have u experience issue between df and du different?
03:25 Peter3 http://pastie.org/9366179
03:26 Peter3 http://pastie.org/9366181
03:26 glusterbot Title: #9366181 - Pastie (at pastie.org)
03:29 Peter3 and the quota usage keep going up without bound
03:30 semiosis no, havent seen that
03:30 Peter3 and eventually fill up the export eventho the actual du usage is very small
03:30 Peter3 gluster just keep thinking the disk has been used up
03:32 Peter3 maybe i should file a bug
03:32 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
03:33 semiosis if you want to, please provide minimal steps to reproduce the problem.  that's very helpful
03:33 Peter3 sure
03:33 semiosis good luck.  i'm off for the night.
03:33 Peter3 sure thanks!
03:33 semiosis yw
03:37 atinmu joined #gluster
03:43 ppai joined #gluster
03:47 flu__ joined #gluster
03:49 theron joined #gluster
03:49 RameshN joined #gluster
03:49 bharata-rao joined #gluster
03:51 nbalachandran joined #gluster
03:53 kanagaraj joined #gluster
03:58 MacWinner joined #gluster
03:59 davinder17 joined #gluster
03:59 kumar joined #gluster
04:01 bala joined #gluster
04:02 rejy joined #gluster
04:06 Peter3 JoeJulian u here?
04:08 ppai joined #gluster
04:10 shubhendu joined #gluster
04:11 Peter3 anyone knows how to "close" a file that no longer exist on gluster?
04:13 nbalachandran joined #gluster
04:24 RameshN joined #gluster
04:35 shubhendu joined #gluster
04:36 rjoseph joined #gluster
04:38 glusterbot New news from newglusterbugs: [Bug 1113842] Incorrect diagrams in the admin guide documentation <https://bugzilla.redhat.co​m/show_bug.cgi?id=1113842>
04:39 Peter3 q
04:39 atrius joined #gluster
04:40 Peter3 how do we find out what files are open in a brick?
04:45 saurabh joined #gluster
04:46 ndarshan joined #gluster
04:50 hagarth Peter3: ls -l /proc/<pid>/fd
04:50 Peter3 what pid is should be?
04:50 hagarth pid of the brick
04:51 Peter3 ic
04:52 ramteid joined #gluster
04:52 Peter3 i am looking at issue that df and du does not match
04:53 Peter3 http://pastie.org/9366181
04:53 glusterbot Title: #9366181 - Pastie (at pastie.org)
04:56 Peter3 hagarth: have seen these kind of issue happen?
04:57 JoeJulian hagarth: He's neglecting to mention several facts. 1. This is an nfs.export-dir. If he mounts the root share, df and du -b match. 2. He's using quota-deem-statfs. 3. If he creates a brand new export-dir the df and du match. 4. if he mv's a file from one export-dir to another export-dir through the root mount, that's what causes the mismatch.
04:57 JoeJulian He also seems to keep thinking that there's an open file issue which we've clearly identified there isn't.
04:57 hagarth JoeJulian: ok
04:58 JoeJulian and with that, I'm going to bed.
04:58 hagarth and I need to run now, tty all later
04:58 Peter3 thanks JoeJulian :)
04:58 Peter3 thanks all
05:00 kdhananjay joined #gluster
05:00 aravindavk joined #gluster
05:02 vpshastry joined #gluster
05:15 psharma joined #gluster
05:18 nishanth joined #gluster
05:21 lalatenduM joined #gluster
05:24 ekuric joined #gluster
05:28 kanagaraj joined #gluster
05:29 kshlm joined #gluster
05:29 kshlm joined #gluster
05:35 hagarth joined #gluster
05:36 aravindavk joined #gluster
05:36 raghu joined #gluster
05:41 meghanam joined #gluster
05:41 meghanam_ joined #gluster
05:43 vimal joined #gluster
05:44 prasanth joined #gluster
05:51 flu_ joined #gluster
05:51 kdhananjay joined #gluster
05:52 rjoseph joined #gluster
05:53 sahina joined #gluster
05:54 flu__ joined #gluster
06:00 ppai joined #gluster
06:07 shylesh__ joined #gluster
06:13 samsaffron joined #gluster
06:14 samsaffron hi there, sorry for the totally newbie question, but how does one ensure files all have consistent and identical times across glusterfs ?
06:14 JoeJulian ntp
06:15 samsaffron as in ensure clocks are synced? how do you rectify the issue after the fact ?
06:21 sputnik13 joined #gluster
06:23 JoeJulian I'd probably walk the tree and set the [amc]time to whatever it already is, triggering a sync of that metadata.
06:23 flu_ joined #gluster
06:23 flu__ joined #gluster
06:25 samsaffron thanks, will check that out
06:29 pdrakeweb joined #gluster
06:37 aravindavk joined #gluster
06:41 rjoseph joined #gluster
06:41 hagarth joined #gluster
06:50 fsimonce joined #gluster
06:50 bala2 joined #gluster
06:50 kanagaraj joined #gluster
07:01 eseyman joined #gluster
07:10 ctria joined #gluster
07:10 keytab joined #gluster
07:17 joevartu_ joined #gluster
07:18 haomaiwa_ joined #gluster
07:18 hybrid512 joined #gluster
07:24 Intensity joined #gluster
07:24 Intensity joined #gluster
07:32 spandit joined #gluster
07:33 haomaiw__ joined #gluster
07:35 kanagaraj joined #gluster
07:41 sputnik13 joined #gluster
07:46 Norman_M Good morning guys! Can anyone tell me where gluster stores its entries which it wants to self heal?
07:46 rgustafs joined #gluster
07:58 monotek joined #gluster
08:05 ghenry joined #gluster
08:14 liquidat joined #gluster
08:19 sputnik13 joined #gluster
08:20 Philambdo joined #gluster
08:26 bala joined #gluster
08:35 Norman_M Got some more funny errors in my nfs log
08:35 Norman_M http://pastie.org/9366991
08:35 glusterbot Title: #9366991 - Pastie (at pastie.org)
08:35 Norman_M Has anyone an idea?
08:42 lyang0 joined #gluster
08:48 Norman_M seems to be some problems with lock files for different processes. gluster cannot assert a gfid because the gfid file already exists
08:49 ndevos joined #gluster
08:50 _ndevos joined #gluster
08:50 _ndevos joined #gluster
08:59 Norman_M we already mounted the nfs with options nolock and noatime and still get there performance issues
09:04 eseyman joined #gluster
09:09 kanagaraj_ joined #gluster
09:14 haomaiwang joined #gluster
09:25 kanagaraj joined #gluster
09:30 haomai___ joined #gluster
09:32 ctria joined #gluster
09:36 karnan joined #gluster
09:36 rastar joined #gluster
09:45 sputnik13 joined #gluster
09:49 ppai joined #gluster
09:54 hagarth joined #gluster
09:55 kdhananjay joined #gluster
09:59 haomaiwa_ joined #gluster
10:02 aravindavk joined #gluster
10:10 ndevos Norman_M: what version of glusterfs do you use? if you are on a version < 3.5.1 you may want to disable nfs.drc
10:10 Norman_M we are on 3.4.4
10:10 Norman_M why do we have to disable ist and how?
10:11 ndevos oh, drc was introduced in 3.5, so you dont need to disable it :)
10:11 ndevos at least, I think so
10:12 Norman_M our colleagues are quite angry by now and our admin and me don't know what to do
10:12 Norman_M i think we have to go back on a single nfs server because this worked out fine before
10:12 ndevos well, at least its not related to nfs.drc, that really isnt in 3.4
10:12 Norman_M and it did not have any performance issues at al
10:13 Norman_M hm one possible error less ;)
10:15 haomai___ joined #gluster
10:15 sputnik13 joined #gluster
10:17 ndevos Norman_M: because it affects users running firefox, can you make sure to set the Firefox option 'storage.nfs_filesystem' in 'about:config'?
10:17 kkeithley1 joined #gluster
10:17 Norman_M hm we never thought about it. What does it do?
10:18 meghanam joined #gluster
10:18 meghanam_ joined #gluster
10:18 ndevos Norman_M: something related to locking of whatever database firefox uses... I can't remember the name atm
10:19 Norman_M Ok I'll google it up
10:21 sputnik13 joined #gluster
10:21 morse joined #gluster
10:21 Norman_M i could do it by putting it in manually for all 350 users or at least for the active 120 users...
10:22 ndevos Norman_M: https://bugzilla.mozilla.o​rg/show_bug.cgi?id=719952 and I think http://www.sqlite.org/wal.html
10:22 glusterbot Bug 719952: high, unspecified, ---, systemd-maint, ASSIGNED , reboot or shutdown commands unresponsive during systemd-fsck
10:22 ndevos uh, glusterbot, thats a bug in the mozilla bugzilla....
10:23 ndevos JoeJulian: maybe you know how to prevent such confusing answers from glusterbot?
10:30 kdhananjay joined #gluster
10:33 sputnik13 joined #gluster
10:37 kshlm joined #gluster
10:40 diegows joined #gluster
10:44 sputnik13 joined #gluster
10:47 calum_ joined #gluster
10:48 ctria joined #gluster
10:59 monotek left #gluster
11:04 RameshN_ joined #gluster
11:10 kanagaraj joined #gluster
11:21 morse joined #gluster
11:22 LebedevRI joined #gluster
11:25 julim joined #gluster
11:35 rjoseph joined #gluster
11:39 meghanam_ joined #gluster
11:40 glusterbot New news from newglusterbugs: [Bug 1117241] backport 'gluster volume status --xml' issues <https://bugzilla.redhat.co​m/show_bug.cgi?id=1117241>
11:41 meghanam joined #gluster
11:48 RameshN joined #gluster
12:04 tdasilva joined #gluster
12:07 RicardoSSP joined #gluster
12:07 RicardoSSP joined #gluster
12:10 glusterbot New news from newglusterbugs: [Bug 1117256] [3.4.4] mounting a volume over NFS (TCP) with MOUNT over UDP fails <https://bugzilla.redhat.co​m/show_bug.cgi?id=1117256>
12:24 mjsmith2 joined #gluster
12:40 aravindavk joined #gluster
12:41 japuzzo joined #gluster
12:41 kshlm joined #gluster
12:47 hagarth joined #gluster
12:52 kanagaraj joined #gluster
12:56 ppai joined #gluster
13:01 vpshastry joined #gluster
13:05 theron joined #gluster
13:06 bennyturns joined #gluster
13:13 ccha2 how can I complete upgrade client side without umount ?
13:14 aravindavk joined #gluster
13:15 sas joined #gluster
13:18 coredump joined #gluster
13:18 edward2 joined #gluster
13:21 bene2 joined #gluster
13:29 nullck_ joined #gluster
13:29 nullck__ joined #gluster
13:29 nullck_ joined #gluster
13:30 torbjorn1_ ccha2: I don't think the client will actually get upgraded then
13:30 torbjorn1_ ccha2: AFAIK you need to load the binary again from disk after updating, so you'll need to stop and start the mount
13:31 bala joined #gluster
13:55 rgustafs joined #gluster
14:01 Setsuna666_ joined #gluster
14:01 Setsuna666_ Hi
14:01 glusterbot Setsuna666_: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:02 Setsuna666_ Does anyone have experience with GlusterFS over WAN ? If it is feasable or even possible ?
14:02 Setsuna666_ Consider that both end would use 1 Gbps links
14:04 d-fence joined #gluster
14:19 samsaffron joined #gluster
14:32 davinder17 joined #gluster
14:36 theron joined #gluster
14:49 plarsen joined #gluster
14:55 chirino joined #gluster
15:02 longshot902 joined #gluster
15:13 harish_ joined #gluster
15:18 karnan joined #gluster
15:24 _dist joined #gluster
15:26 theron joined #gluster
15:32 YoungJoo_ joined #gluster
15:36 theron joined #gluster
15:47 cristov joined #gluster
15:49 kdhananjay joined #gluster
16:00 chirino joined #gluster
16:02 calum_ joined #gluster
16:04 phox left #gluster
16:09 rsavage_ joined #gluster
16:09 rsavage_ hello
16:09 glusterbot rsavage_: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
16:09 rsavage_ how is everyone?
16:10 rsavage_ I have a question.  I have recently rebuilt one of my gluster servers, and I am not seeing the expected output from 'gluster volume status' - I only see one server in the cluster and one brick.  However, when I run 'gluster volume info' I see everything... what's wrong?
16:20 pureflex joined #gluster
16:22 Mo_ joined #gluster
16:24 mortuar joined #gluster
16:25 vpshastry joined #gluster
16:30 jtdavies joined #gluster
16:32 jtdavies I am trying to set up a pair of geo-replication slaves for one of my Gluster (3.5.1, for CentOS) volumes; the first functions properly. When I attempt to start the second, Gluster claims to have successfully started the geo-replication, but querying the status gives "not started" and no data is transferred. Do you know what I am doing wrong?
16:38 vikumar joined #gluster
16:41 theron joined #gluster
16:43 longshot902_ joined #gluster
16:46 zerick joined #gluster
16:48 theron joined #gluster
16:53 fubada joined #gluster
16:53 fubada hi can someone please help figure this out? I cant "vol status" http://fpaste.org/116427/40483842/
16:53 glusterbot Title: #116427 Fedora Project Pastebin (at fpaste.org)
16:55 JoeJulian fubada: that's bizarre. I don't know what the problem is, but I suspect it could be fixed by restarting glusterd.
16:55 fubada tried :(
16:56 _dist fubada: I had that same problem recently caused by a mistake I made. I was able to resolve it without restarting glusterd, but it wasn't easy. If you can restart it, I'd do that. I believe you need to stop the volume, take all bricks down and bring them back up
16:57 JoeJulian ccha2: You cannot upgrade clients without umount. If they're VMs, upgrade one VM host, live migrate your VMs to that host, once evacuated, upgrade the client and umount/mount, rinse, repeat.
16:57 fubada so i too made a mistake and removed gluster via yum, installed newer 3.4, realized what i had done, and downgraded
16:57 _dist by you log it looks like you have a volume that is missing a peer, or at least that server thinks one is
16:58 _dist your*
16:58 fubada _dist: you siuggest restarting gluster across all 4 nodes?
16:59 JoeJulian rsavage_: It sounds like glusterd isn't started on the other server, or the two servers cannot communicate with each other.
16:59 _dist so the "missing" server (uninstalled I assume) has a lock, if taking the volume down entirely, stopping all gluster processes and bringing it all back up doesn't fix it there is another way still
16:59 _dist fubada: yes, I'd take them all down, then bring them all back up. So that there is a period where all are off at the same time
16:59 _dist (assuming you can do this in your environement)
17:00 JoeJulian fubada: what version did you downgrade to?
17:00 fubada JoeJulian:  i went from 3.3.1 to 3.4 and back to 3.3.1
17:00 _dist JoeJulian: btw my mistake was I added a replicate node, but one of my clients didn't have the dns for it, it went nuts because it couldn't find it. Would you consider that worth filing a bug over?
17:01 fubada _dist: i just did vol stop on the troubled peer, restarted glusterd and now im unable to start the volume
17:01 _dist fubada: don't just stop it, kill all the gluster processes on all bricks as well
17:01 JoeJulian _dist: If the client behaved in a way that you find illogical, yes. file a bug.
17:01 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
17:02 fubada _dist: stop vol and restart glusterd across all or just restart the daemon?
17:02 JoeJulian fubada: hang on a sec... gathering information
17:02 _dist JoeJulian: well the only thing that could have happened is the mount brick could have provided the IP, but I don't know if I'd consider it a bug rather an enhancement
17:03 _dist fubada: if JoeJulian is going to give you a hand, he'll do a better job at it than I.
17:03 jobewan joined #gluster
17:03 fubada thanks guys
17:03 JoeJulian bricks can't tell the client the ip for a number of reasons, one of which being that there are valid reasons for hostnames to resolve differently on clients vs servers.
17:04 Peter3 joined #gluster
17:05 _dist JoeJulian: that's understandable, it was an oversight as we forgot about one of the fuse mount clients. As far as I'm concerned it was an implementation mistake, not a bug.
17:06 fubada i will be back in a couple :P
17:06 fubada thanks guys
17:06 JoeJulian That's why I use dns. I'm bugging the hell out of our network people here because everythings in /etc/hosts and it requires a global chef run to make any changes, which (and this must be a major feature for chef) everyone's afraid to do outside of a change window because they're afraid chef will break something.
17:08 JoeJulian fubada: edit /var/lib/glusterd/glusterd.info. Set the operating version to 1. Run "glusterd --xlator-option *.upgrade=on -N". Start glusterd.
17:08 Peter3 is there a way to run multiple gluster commands at once?
17:09 Peter3 i keep getting management lock as monitoring tools running at the same time…
17:09 fubada JoeJulian: on all nodes?
17:09 fubada or just the broken one
17:09 JoeJulian fubada: Any that you upgraded to 3.4
17:09 fubada JoeJulian: but ive since downgraded
17:09 fubada back to 3.3.1
17:09 JoeJulian Yep
17:09 fubada okay
17:10 JoeJulian Downgrading doesn't downgrade the feature set that upgrading added.
17:10 fubada JoeJulian: all I see in glusterd.info file is the UID=
17:10 JoeJulian Ok, then just the "glusterd --xlator-option *.upgrade=on -N" should be sufficient.
17:13 fubada JoeJulian: cli is hanging on vol list
17:14 JoeJulian After running that command, you started glusterd again, right?
17:14 fubada yah
17:14 fubada Starting volume reports has been unsuccessful
17:17 fubada http://fpaste.org/116437/48398311/
17:17 glusterbot Title: #116437 Fedora Project Pastebin (at fpaste.org)
17:18 JoeJulian Unless one of your peers has a uuid of 00000000-0000-0000-0000-000000000000, I don't get that.
17:19 fubada :/
17:19 fubada they dont
17:20 _dist fubada: If I remember correctly you removed an installation by accident? JoeJulian: does each install generate a new uuid? If so you might need to put the old uuid back?
17:20 JoeJulian hagarth: How can "I [glusterd-rpc-ops.c:548:glu​sterd3_1_cluster_lock_cbk] 0-glusterd: Received RJT from uuid: 00000000-0000-0000-0000-000000000000" happen?
17:20 _dist but... I dunno that sounds dangerous, nevermind I said anything :)
17:21 hagarth JoeJulian: probably a rpc disconnect happened after a lock request was sent
17:21 JoeJulian Ah, ok.
17:21 fubada so peer status shows as good
17:22 JoeJulian fubada: Look in the glusterd log on the other server(s)
17:22 hagarth fubada: check if gluster peer status looks good on all servers too
17:23 fubada okay it does
17:26 fubada peer status looks good
17:27 MacWinne_ joined #gluster
17:28 MacWinne_ if I have 2 separate data centers, each with 4 nodes running replica2.. is it simple to setup georeplication between the 2 sites?  I currently do some stuff with lsyncd+rsync
17:28 MacWinne_ any pointers for a step-by-step would be great..
17:31 fubada JoeJulian: can i help with more detail?
17:33 fubada JoeJulian: heres the issue http://fpaste.org/116447/40484081/
17:33 glusterbot Title: #116447 Fedora Project Pastebin (at fpaste.org)
17:33 fubada the problme node has a new uuid
17:34 fubada while the cluster thinks its something else
17:34 fubada (2a93fc73-688c-414c-aae9-14738993ced0)
17:34 fubada how can I set the uuid
17:36 fubada fixed
17:36 jtdavies uuid is set in /var/lib/glusterd/glusterd.info if I recall correctly
17:36 fubada yep i put the correct uuid in there and everything is working
17:36 jtdavies just a moment late, I guess. oh well, at least yours is working
17:38 JoeJulian fubada: good job
17:38 fubada thanks man
17:38 fubada sorry to bug yall with this silly issuye
17:38 JoeJulian No, that's cool. I didn't know that's how that problem would manifest. That's good to know.
17:39 * _dist feels better that his guess wasn't crazy
17:39 fubada yah so im guessing yum install of 3.4 updated the uuid
17:39 _dist JoeJulian: I've almost got my test gluster setup in identical conditions for the vm healing issue, should have a vpn connection for you later this week.
17:39 JoeJulian No, I bet you uninstalled 3.4 before installing 3.3.
17:39 fubada i did
17:39 fubada i removed it first
17:40 JoeJulian there's probably a .rpmsave file.
17:40 fubada i removed 3.3.1, installed 3.4, removed 3.4, installed 3.3.1
17:40 fubada there isnt :(
17:41 JoeJulian Not /exactly/ a bug, but I think we should treat that file as a configuration file and rpmsave it during uninstall.
17:41 fubada http://fpaste.org/116453/40484126/
17:41 glusterbot Title: #116453 Fedora Project Pastebin (at fpaste.org)
17:41 fubada I have those .rpmsave files
17:41 fubada that is it
17:41 jag3773 joined #gluster
17:41 JoeJulian That's why you should yum upgrade or yum downgrade rather than uninstall and install.
17:41 fubada ah
17:45 cultavix joined #gluster
17:45 cultavix good evening
17:46 cultavix I've just setup 2 virtual machines with 100GB disk each and a gluster volume, replica
17:47 cultavix works great for mounting using the glusterfs.fuse
17:47 cultavix I've even got CTDB setup, seems to work
17:47 Setsuna666 joined #gluster
17:48 cultavix but I can't mount using NFS also with CIFS, it asks me for a username and password, smbpasswd fails anytime I try to create a new user (it fails after I enter the second password)
17:48 cultavix any ideas?
17:49 _dist cultavix: if your using ubuntu the CIFS thing has a specific answer, also I believe gluster has its' own NFS server which can be turned on/off via a volume option (default on) but problems can occur if you have the nfs-kernel-server installed at the same time
17:50 cultavix ah right, I'm using CentOS Minimal 6.5
17:50 cultavix so do I want to make sure that nfs is not installed?
17:50 cultavix I've got nfs-utils
17:50 cultavix I know tha tmuch
17:51 JoeJulian @nfs
17:51 glusterbot JoeJulian: To mount via nfs, most distros require the options, tcp,vers=3 -- Also an rpc port mapper (like rpcbind in EL distributions) should be running on the server, and the kernel nfs server (nfsd) should be disabled
17:51 _dist ^^ yeap, I'm not sure if new CentOS (unrelated to gluster) has the same thing where there is a separate SMB password from the user password
17:52 _dist but that was what I was thinking about with ubuntu, perhaps it's just a new smb standard?
17:52 cultavix im trying
17:52 cultavix ount -o mountproto=tcp,ver=3 -t nfs
17:53 cultavix how do I disable "kernel nfs server" on CentOS there is nothing installed at all relating to NFS
17:54 cultavix with a -v I can see the following "mount.nfs: portmap query retrying: RPC: Program not registered"
17:54 cultavix rpcbind is running on both servers and on the client
17:55 JoeJulian What version of glusterfs are you running?
17:55 cultavix are you referring to "nfsd on /proc/fs/nfsd type nfsd (rw)"
17:55 cultavix maybe this is what is causing the problem ?
17:55 JoeJulian perhaps
17:55 chirino joined #gluster
17:55 JoeJulian chkconfig nfsd off
17:55 JoeJulian service nfsd stop
17:56 cultavix 3.5.1-1.el6
17:56 cultavix gluster version
17:56 JoeJulian ok
17:56 cultavix even after stopping nfs, that "nfsd /proc/fs/nfsd" is still there
17:57 cultavix looks more like a kernel thing
17:57 cultavix I've got a debian box I can test from as well, let me check on there
17:57 JordanHackworth joined #gluster
17:59 cultavix on debian same problem "ount.nfs: portmap query failed: RPC: Program not registered
17:59 cultavix mount.nfs: requested NFS version or transport protocol is not supported"
17:59 glusterbot cultavix: make sure your volume is started. If you changed nfs.disable, restarting your volume is known to work.
17:59 cultavix glusterbot ah ok, ill try that now
18:00 JoeJulian hehe, I don't even remember programming that.
18:01 chirino_m joined #gluster
18:01 cultavix omg that worked :)
18:01 cultavix gluster:/gv0 on /mnt type nfs (rw,vers=3,addr=192.168.1.35)
18:01 cultavix w00t :)
18:02 cultavix thank you so much guys, now the only problem I have is the CIFS mount
18:02 JoeJulian you're welcome.
18:02 JoeJulian glusterbot++
18:02 glusterbot JoeJulian: glusterbot's karma is now 1
18:02 cultavix it asks for a username/password (from windows)
18:02 cultavix glusterbot++
18:02 glusterbot cultavix: glusterbot's karma is now 2
18:02 JoeJulian Yeah, that would be a samba thing.
18:02 cultavix im using ctdb
18:03 cultavix which manages samba now
18:04 jdarcy joined #gluster
18:08 pasqd joined #gluster
18:10 nishanth joined #gluster
18:13 chirino joined #gluster
18:17 stickyboy I'm looking to scale out my storage (currently replica 2 with 2 storage nodes).  Not sure if I should buy more nodes with disks, or just buy some DAS boxes + SAS connections and plug into existing boxes.
18:18 stickyboy Anyone have experience with DAS boxes?
18:19 cultavix I've done it guys.... fuck yea
18:19 cultavix glusterfs volume, mounted via glusterfs.fuse, NFS and CIFS :)
18:19 cultavix now I can actually start to play around with the more interesting stuff
18:20 purpleidea @stayinchannel
18:21 JoeJulian stickyboy: We're using Wiwynn knox2u attached to 4 port lsi sas controllers for each of our storage servers. (OCP gear)
18:22 stickyboy JoeJulian: Thanks.  That's a vote of confidence for that setup (even though "Wiwynn knox2u" has exactly *one* Google search result :P)
18:24 stickyboy JoeJulian: I'm also trying to decide if I want to stay the course with my 10GbE (Cat6a) setup or tell the boss we need something optical.  Throughput is great, but interactivity (home directories on a compute cluster, so `ls`!) is laggy.
18:25 JoeJulian To cut down on lag, decrease latency. Infiniband.
18:25 rotbeard joined #gluster
18:25 stickyboy JoeJulian: Thought so. :\
18:25 stickyboy This 10GbE Arista switch is sweet, but yeah... copper just doesn't cut it for interactivity.
18:27 _dist JoeJulian: If I used infiniteband instead of 10Gbe, where would I see the difference practically?
18:27 calum_ joined #gluster
18:28 pasqd joined #gluster
18:29 stickyboy _dist: I've got 10GbE copper, and I get 9.91 Gbit/sec... but latency is slow.  So if you have little files or do lots of "ls", don't go with 10GbE.
18:31 chirino joined #gluster
18:33 theron joined #gluster
18:34 kanagaraj joined #gluster
18:35 _dist stickyboy: we're running a 3 replicate with about 40k files and another volume (same setup) with around 32vm disk images. The only thing that is slower than normal is fsync as far as I can tell
18:36 _dist by "normal" I maen local disk speed
18:36 theron_ joined #gluster
18:39 _dist our "real" performance is around 500-900 megabytes/sec for larger writes sequential writes. If I go to smaller (1-4k writes) it does drop to around 20 Mbytes/sec
18:43 stickyboy _dist: I have 50-100 users, with some users having millions of files (grrrr, Bioinformatics cluster).  We ls and vim and untar etc allllll the time, so the latency is really annoying.
18:45 JoeJulian IB, especially w RDMA, would be much better for that workload. It eliminates a lot of context switching and TCP overhead.
18:45 theron joined #gluster
18:55 luis_silva joined #gluster
18:55 luis_silva left #gluster
19:02 stickyboy JoeJulian: How beefy are the storage nodes running those Wiwynns?  Beefy in terms of CPU / memory...
19:04 jvandewege joined #gluster
19:07 JoeJulian One of these nodes is a server http://www.penguincomputing.​com/products/relion-ocp2830 . They have Xeon E5-2650s and 128gb of ram. Two of those knox modules each populated with 30 4Tb drives.
19:07 glusterbot Title: Relion OCP 2830 (Winterfell) - Open Compute Project Solutions - Penguin Computing (at www.penguincomputing.com)
19:08 JoeJulian wait... I see 32 cores but that cpu says it only has 8...
19:10 Sunghost joined #gluster
19:10 Sunghost Hello
19:10 glusterbot Sunghost: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
19:11 osiekhan3 joined #gluster
19:11 Sunghost Short question - i copied a image on the gluster folder of my distributed gluster, direct onto the brick not the mountpoint
19:11 Sunghost than i deleted that .img file but the space is not free
19:12 Sunghost i searchd for the biggest folder and found it in .glusterfs/18
19:12 glusterbot New news from newglusterbugs: [Bug 1075417] Spelling mistakes and typos in the glusterfs source <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075417>
19:12 Sunghost the questions now is - how can i solve this and delete the unneded file?
19:15 Sunghost Any help
19:16 gehaxelt Sunghost, maybe JoeJulian can help if he's online.
19:17 Sunghost Oh JoeJulian ;) here are lot of people and JoeJulian is the only expert ??
19:18 diegows joined #gluster
19:18 JoeJulian Sunghost: In that .glusterfs/18 directory, there is one or more uuid named files (technically gfid). The one that ended up being associated with that image will also have just one link (check stat). It should also match the size of that image file.
19:18 JoeJulian ... and yes, there are other experts here. :)
19:19 Sunghost Hey JoeJulian ;) great plesure i searched the net and now on your blog with article to gfid ,)
19:19 hchiramm_ joined #gluster
19:19 JoeJulian :)
19:20 JoeJulian I do what I can to help.
19:20 Sunghost great man - ok found one file and now? how to check link stat?
19:20 JoeJulian stat
19:22 Sunghost mh stat file doesnt sho the org name
19:22 Sunghost ?!
19:22 JoeJulian Yeah, the gfid files are hardlinks to the original file.
19:23 JoeJulian So they're different directory entries pointing to the same inode chain.
19:23 JoeJulian You deleted one directory entry, so as long as that hasn't been restored you can delete this, the other directory entry. Then the inode can be released.
19:24 Sunghost no directory an image file
19:24 jag3773 joined #gluster
19:24 jag3773 left #gluster
19:25 jag3773 joined #gluster
19:25 Sunghost ok understand what you say but not how i can see it in stat
19:25 Sunghost here the stat
19:25 Sunghost stat 180884a3-eeaa-4965-8d2e-cdc96f297fc6
19:25 Sunghost Datei: 180884a3-eeaa-4965-8d2e-cdc96f297fc6
19:25 Sunghost Größe: 3000591450112Blöcke: 5860530200 EA Block: 4096   reguläre Datei
19:25 Sunghost Gerät: 902h/2306dInode: 1163        Verknüpfungen: 1
19:25 Sunghost Zugriff: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
19:25 Sunghost Zugriff    : 2014-07-05 22:31:41.374270758 +0200
19:25 glusterbot Sunghost: (0644/-rw-r--r's karma is now -1
19:25 Sunghost Modifiziert: 2014-07-08 17:03:21.125009856 +0200
19:25 Sunghost Geändert   : 2014-07-08 17:05:24.777005925 +0200
19:25 Sunghost Geburt    : -
19:26 Sunghost so which inode chain should be where?
19:26 deeville joined #gluster
19:29 Sunghost so how can i get sure that i delete the right file except from the size
19:29 deeville Is there a way to specify hosts to root-squash? Or is there no way around this global setting?
19:31 deeville The reason why I ask is I'd like to retain root access to the volume on the gluster server nodes, which are clients as well using the fuse client. I'd like to root-squash everywhere else.
19:33 Sunghost Hey JoeJulian are you testing it or what happend?
19:35 luis_silva joined #gluster
19:35 luis_silva left #gluster
19:37 stickyboy JoeJulian: Wow, impressive boxes!  Our storage nodes all have quad-core Xeon E5 26xx's with 16GB of RAM.  Our load is very sane, unless lots of people start writing to storage.
19:38 luis_silva joined #gluster
19:39 Sunghost Which boxes and where is JoeJulian ;) ?
19:41 stickyboy JoeJulian: Do you SAS daisy chain the Wywinn boxes?  I saw someone doing that at it made me nervous...
19:42 JoeJulian Sunghost: Verknüpfungen: 1 says that the inode has no other hardlinks. That's the one and it's safe to delete.
19:43 stickyboy I met a guy in Kenya last weekend who says he drank beers with the Gluster guys in Bangalore like 8 years ago. :)
19:44 JoeJulian stickyboy: No, we connect each of the 4 trays directly to our lsi sas 2308.
19:46 Sunghost Oh ok so if there is always 1 link i can delete this file? or is there a situation where i shouldnt do that?
19:47 JoeJulian For regular files, yes. Not for symlinks.
19:48 JoeJulian And I guess there's a way to do anonymous inodes through nfs. That wouldn't create a filename, just a gfid file. I know nothing about that though.
19:48 Sunghost ah ok a symlink whouldnt have 2,7tb ;) ok you helped again a lot - many thanks and a nice evening
19:52 stickyboy JoeJulian: Connect directly, gotcha.  That sounds safer than daisy chaining.
19:56 JoeJulian stickyboy: I still have one problem with the mpt2sas driver. If a drive is unresponsive during the boot process, it will hang module installation in the kernel. All modprobes from that point on will hang.
19:56 fubada anyone know how to best get xfs on rhel?
19:56 fubada nothing comes up under yum for xfs
19:57 stickyboy JoeJulian: Ouch!
19:57 JoeJulian package name is xfsprogs. It installs just fine in centos. Not sure about rhel.
19:57 fubada yah its not found in my rhel env
19:57 fubada thanks JoeJulian
19:58 JoeJulian stickyboy: yeah, I'm working with driver developers to get that fixed.
19:58 pvh_sa joined #gluster
20:01 stickyboy JoeJulian: If I recall correctly you're on Ubuntu, 12.04 perhaps?  I wonder if it works any better in CentOS 6.x.
20:01 diegows joined #gluster
20:02 JoeJulian No. Apparently this problem has existed in this driver for quite some time.
20:05 stickyboy JoeJulian: Just checked my storage servers are using mpt2sas too.  Haven't noticed those issues though.  Only with crappy Seagate drives failing. :)
20:07 _Bryan_ joined #gluster
20:08 JoeJulian stickyboy: That's another thing. Make sure you're using the bios and firmware from SAS9206-16E_Package_P19. There's a bug in prior firmware that will make drives seem to fail when they might not have really failed.
20:08 gildub joined #gluster
20:08 JoeJulian (if you're using the same lsi board)
20:09 stickyboy JoeJulian: Wow, maybe that's why Seagate sent my drives back saying they were fine.
20:09 JoeJulian likely
20:10 JoeJulian I'm expecting the same news from WD this afternoon for 15 drives.
20:10 stickyboy Although Cloudflare's data seems to backup my suspicion that Seagates are crap.
20:11 stickyboy I wanna move to Hitachis when I buy my next round of drives.
20:12 JoeJulian the best thing to do is to mix vendors.
20:12 stickyboy Good advice...
20:12 stickyboy And we're in Africa, so getting spares is a pain in the rear.
20:13 JoeJulian Yeah, I saw you mention Kenya earlier. I hope to go visit some day.
20:14 stickyboy Go to Tanzania instead. :P
20:15 stickyboy They're nicer and it's cheaper, but the culture / food / language, etc is very similar. :P
20:15 JoeJulian I just want to leave North America some time.
20:17 stickyboy JoeJulian: I was in Istanbul a few months ago.  Wow!  Also, backpacked around Thailand a bit last year... wow!
20:17 jag3773 joined #gluster
20:18 stickyboy JoeJulian: Do you have a good junior sysadmin?  I don't.  I had to go to a cyber cafe a few weeks ago when I was traveling in Tanzania.
20:18 fubada folks im new, is it possible to take a 1x4 brick and make it 2x2
20:18 fubada without removing the brick
20:19 marcoceppi joined #gluster
20:21 sjm joined #gluster
20:21 * pvh_sa knows an admin in Tanzania... "but they're rare beasts"
20:22 JoeJulian stickyboy: I'm part of the cloud team at IO (http://www.io.com) so I get to mainly focus on storage now as a Principal Cloud Engineer.
20:23 glusterbot Title: IO - THE Software-Defined Data Center (at www.io.com)
20:23 JoeJulian I'd still bring my laptop with me on a vacation though.
20:23 JoeJulian Heck, I work from home, so theoretically I can not be on vacation anywhere in the world that I can get reliable internet access.
20:24 stickyboy JoeJulian: Oh, you left the cosmetics place? :)
20:24 JoeJulian Yeah. My son now has my old job.
20:24 JoeJulian So now he's stuck. :D
20:24 stickyboy JoeJulian: Awesome. :)
20:24 fubada folks i removed a volume and now im trying to recreate it under the same path, i get /appdata/bricks/reports or a prefix of it is already part of a volume
20:24 glusterbot fubada: To clear that error, follow the instructions at http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/ or see this bug https://bugzilla.redhat.com/show_bug.cgi?id=877522
20:24 fubada is that normal>
20:24 fubada nice ;P
20:25 JoeJulian glusterbot++
20:25 glusterbot JoeJulian: glusterbot's karma is now 3
20:25 stickyboy pvh_sa: I'm involved in the Nairobi (Kenya) GNU/Linux Users Group, but we're basically the only active group in East Africa (Uganda, Tanzania, Rwanda, Ethiopia, etc).
20:26 pvh_sa so I've got a 19 node rack, 13 TB of RAIDed disk per node, was thinking GlusterFS striped replica... how do I fit that across 19 nodes though? I presume stripe 2 replica 2 would assume you need multiples of 4 bricks?
20:27 JoeJulian @stripe
20:27 glusterbot JoeJulian: Please see http://joejulian.name/blog/sho​uld-i-use-stripe-on-glusterfs/ about stripe volumes.
20:27 pvh_sa stickyboy, oh ok. yeah I'm vaguely involved there via supporting bioinformatics stuff that's connecting with nodes in the area... and I know some of the grid computing people also have links there now
20:28 pvh_sa ah thanks. ok so then... replica 2? which again... would assume an equal number of bricks ?
20:28 JoeJulian Yes, equal number of bricks.
20:28 pvh_sa hm, what to do with the 19th node then...
20:29 JoeJulian You /can/ have multiple bricks per server though, so that doesn't necessarily mean an equal number of servers.
20:29 stickyboy pvh_sa: I work at ILRI in Kenya.  We do genome sequencing and bioinformatics blah blah in Nairobi... got some links with institutes in TZ too.
20:29 stickyboy pvh_sa: (ILRI == livestock research)
20:29 pvh_sa stickyboy, you've got some of our graduates... I work at SANBI in South Africa
20:30 pvh_sa JoeJulian, ok, but multiple bricks per server means that a single node failure could lose your data (if you happen to have both replicas on the bricks on that server)
20:30 JoeJulian Yes, but you don't put both replicas on the same server. :P
20:30 JoeJulian @brick order
20:30 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
20:30 stickyboy pvh_sa: Yes! :)
20:32 pvh_sa JoeJulian, ah ok then. that's what I'll do then. this is an odd setup... astrophysics folks bought a cluster but didn't buy storage, so storage is now on DFS on the actual cluster. did cephfs on small cluster, but will try glusterfs on their big one...
20:33 fubada JoeJulian: can CIDR be used in auth.allow?
20:33 fubada or range?
20:33 fubada or just wildcards
20:33 JoeJulian just wildcards. :(
20:34 JoeJulian I think I have a really old enhancement request to use cidr.
20:34 JoeJulian Personally, I've always just managed permissions with iptables.
20:36 fubada im at a new place and they...disable iptables and selinux ;/
20:37 stickyboy <3 iptables and selinux.
20:39 * pvh_sa always disables selinux. I know, I should tune to it... but.. it always gets disabled at dev time and I never get around to re-enabling it
20:40 JoeJulian http://mhayden.spreadshirt.com/
20:40 glusterbot Title: Major's Shirt Shop (at mhayden.spreadshirt.com)
20:42 stickyboy Learn to love SELinux. :D
20:42 glusterbot New news from newglusterbugs: [Bug 1117509] Gluster peer detach does not cleanup peer records causing peer to get added back <https://bugzilla.redhat.co​m/show_bug.cgi?id=1117509>
20:44 pvh_sa i'll re-enable selinux etc when I leave the cluster to the astrophysics people. :P just a few more weeks
20:49 stickyboy pvh_sa: If you're on CentOS with recent updates, there isn't much trouble with SELinux and Gluster.
20:50 stickyboy I had to enable some bools for SMB to share FUSE volumes... but I think that's it.
20:51 pvh_sa stickyboy, yes, we're centos 6.5 here... I see glusterfs 3.4 is in repos, so I'll add the glusterfs-epel repo to get 3.5 series
20:53 stickyboy pvh_sa: Yah, I'm on CentOS 6.5 here, with GlusterFS 3.5.0.  Looking for an excuse to do a rolling update to 3.5.1. ;)
21:12 glusterbot New news from newglusterbugs: [Bug 1105277] Failure to execute gverify.sh. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1105277>
21:18 Peter4 joined #gluster
21:19 Peter3 joined #gluster
21:34 KORG joined #gluster
21:36 sjm left #gluster
21:36 KORG joined #gluster
21:50 japuzzo joined #gluster
21:59 dtrainor joined #gluster
22:03 qdk joined #gluster
22:32 Pupeno joined #gluster
22:38 gildub joined #gluster
22:49 LebedevRI joined #gluster
22:52 chirino joined #gluster
22:53 chirino_m joined #gluster
22:55 dtrainor joined #gluster
22:59 anotheral joined #gluster
22:59 anotheral Q: I lost a RAID0 brick in a distributed/replicated gluster
22:59 anotheral NBD because it's replicated
22:59 anotheral but when I restart the glusterfs service, the brick log starts piling up these errors:
23:00 anotheral http://paste.ubuntu.com/7767832/
23:00 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
23:02 anotheral i'm going by this doc, but maybe I need to remove the old brick before doing the heal command?
23:02 anotheral http://gluster.org/community/documen​tation/index.php/Gluster_3.4:_Brick_​Restoration_-_Replace_Crashed_Server
23:02 glusterbot Title: Gluster 3.4: Brick Restoration - Replace Crashed Server - GlusterDocumentation (at gluster.org)
23:19 anotheral do I run "gluster volume heal rnd0 all" from any server, or specifically the replaced one?
23:21 anotheral or should I just do a remove-brick force and add-brick?
23:39 sputnik13 joined #gluster
23:51 * anotheral tried the heal
23:52 anotheral same junk in the logs, no new files appearing on teh replaced brick
23:55 anotheral ah, looks like the folder structure is rebuilding on the replaced brick
23:55 anotheral phew
23:55 chirino joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary