Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-05-18

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:28 Pupeno joined #gluster
00:48 cyberbootje joined #gluster
00:50 julim joined #gluster
01:07 plarsen joined #gluster
01:12 halfinhalfout joined #gluster
01:25 delhage joined #gluster
01:25 necrogami joined #gluster
01:26 wkf joined #gluster
01:32 julim joined #gluster
01:38 plarsen joined #gluster
01:58 nangthang joined #gluster
02:54 gildub joined #gluster
02:55 bharata-rao joined #gluster
03:03 dusmant joined #gluster
03:18 rjoseph joined #gluster
03:22 shubhendu joined #gluster
03:36 glusterbot News from newglusterbugs: [Bug 1179179] When an unsupported AUTH_* scheme is used, the RPC-Reply should contain MSG_DENIED/AUTH_ERROR/AUTH_FAILED <https://bugzilla.redhat.com/show_bug.cgi?id=1179179>
03:36 itisravi joined #gluster
03:41 [7] joined #gluster
03:59 aaronott joined #gluster
04:06 atinmu joined #gluster
04:10 kanagaraj joined #gluster
04:11 kumar joined #gluster
04:14 suliba_ joined #gluster
04:20 RameshN joined #gluster
04:22 sripathi joined #gluster
04:26 rafi joined #gluster
04:28 kdhananjay joined #gluster
04:33 jiffin joined #gluster
04:34 prkrishn joined #gluster
04:35 nangthang joined #gluster
04:36 prabu joined #gluster
04:37 yazhini joined #gluster
04:43 sakshi joined #gluster
04:44 smohan joined #gluster
04:48 julim joined #gluster
04:50 julim joined #gluster
04:51 ramteid joined #gluster
04:55 Anjana joined #gluster
05:04 prabu joined #gluster
05:05 ndarshan joined #gluster
05:10 hagarth joined #gluster
05:10 pppp joined #gluster
05:13 deepakcs joined #gluster
05:13 prabu_ joined #gluster
05:13 yazhini joined #gluster
05:13 meghanam joined #gluster
05:14 yazhini_ joined #gluster
05:15 prabu joined #gluster
05:15 prabu_ joined #gluster
05:17 Bhaskarakiran joined #gluster
05:17 Bhaskarakiran_ joined #gluster
05:20 rjoseph joined #gluster
05:25 Apeksha joined #gluster
05:26 spandit joined #gluster
05:33 nbalacha joined #gluster
05:36 gem joined #gluster
05:36 glusterbot News from newglusterbugs: [Bug 1214822] Disperse volume: linux tarball untar fails on a fuse mounted disperse volume <https://bugzilla.redhat.com/show_bug.cgi?id=1214822>
05:42 R0ok_ joined #gluster
05:44 akay joined #gluster
05:46 kdhananjay joined #gluster
05:47 poornimag joined #gluster
05:49 ashiq joined #gluster
05:49 anrao joined #gluster
05:54 schandra joined #gluster
05:54 maveric_amitc_ joined #gluster
05:56 nsoffer joined #gluster
05:56 dusmant joined #gluster
05:57 karnan joined #gluster
06:00 Manikandan joined #gluster
06:00 Manikandan_ joined #gluster
06:03 prabu_ joined #gluster
06:04 yazhini__ joined #gluster
06:04 overclk joined #gluster
06:04 prabu joined #gluster
06:06 glusterbot News from newglusterbugs: [Bug 1221175] [geo-rep]: Session goes to faulty with "Cannot allocate memory" traceback when deletes were performed having trash translators ON <https://bugzilla.redhat.com/show_bug.cgi?id=1221175>
06:07 prabu_ joined #gluster
06:07 prabu joined #gluster
06:08 prabu left #gluster
06:10 hgowtham joined #gluster
06:11 nbalacha joined #gluster
06:13 atalur joined #gluster
06:16 hchiramm joined #gluster
06:28 jtux joined #gluster
06:34 Anjana joined #gluster
06:35 atalur joined #gluster
06:36 glusterbot News from newglusterbugs: [Bug 1221544] [Backup]: Unable to create a glusterfind session <https://bugzilla.redhat.com/show_bug.cgi?id=1221544>
06:37 yazhini joined #gluster
06:37 yazhini_ joined #gluster
06:40 raghu joined #gluster
06:40 julim joined #gluster
06:41 spalai joined #gluster
06:42 haomaiwa_ joined #gluster
06:48 glusterbot News from resolvedglusterbugs: [Bug 1218553] [Bitrot]: glusterd crashed when node was rebooted <https://bugzilla.redhat.com/show_bug.cgi?id=1218553>
06:48 glusterbot News from resolvedglusterbugs: [Bug 1214273] Attach/Detach command o/p needs refinement <https://bugzilla.redhat.com/show_bug.cgi?id=1214273>
06:49 anil_ joined #gluster
06:50 saurabh_ joined #gluster
06:51 nangthang joined #gluster
06:51 overclk joined #gluster
06:59 gem joined #gluster
07:04 al joined #gluster
07:06 glusterbot News from newglusterbugs: [Bug 1221100] Disperse volume: Directory became stale while renaming files in it. <https://bugzilla.redhat.com/show_bug.cgi?id=1221100>
07:06 glusterbot News from newglusterbugs: [Bug 1211962] Disperse volume: Input/output  errors on nfs and fuse mounts during delete operation <https://bugzilla.redhat.com/show_bug.cgi?id=1211962>
07:07 deniszh joined #gluster
07:09 spalai joined #gluster
07:11 prabu_ joined #gluster
07:11 yazhini_ joined #gluster
07:11 Slashman joined #gluster
07:13 Philambdo joined #gluster
07:13 yazhini joined #gluster
07:15 aravindavk joined #gluster
07:25 fsimonce joined #gluster
07:26 ProT-0-TypE joined #gluster
07:37 glusterbot News from newglusterbugs: [Bug 1210404] BVT; Selinux throws AVC errors while running DHT automation on Rhel6.6 <https://bugzilla.redhat.com/show_bug.cgi?id=1210404>
07:37 glusterbot News from newglusterbugs: [Bug 1222409] nfs-ganesha: HA failover happens but I/O  does not move ahead when volume has two mounts and I/O going on both mounts <https://bugzilla.redhat.com/show_bug.cgi?id=1222409>
07:53 jkroon joined #gluster
07:53 autoditac joined #gluster
07:54 jkroon hi all, is there any way to tell the gluster client that reads *from* a specific brick should not be made?  In particular, I've got a temporary situation where my two bricks are geographically separate (round-trip latency still <1ms but throughput is restricted).  so I'd like a write-only setup for the clients to each of the other sites.
07:56 gw joined #gluster
07:56 gw dear all
07:56 gw i'm newbie for glusterfs
07:57 gw i just have problem about this "https://bugzilla.redhat.com/show_bug.cgi?id=1204247"
07:57 glusterbot Bug 1204247: high, unspecified, ---, bugs, NEW , mmap failed for CEN and END part of zip file
07:57 gw anyone can help?
08:01 gw zzzzz...
08:02 overclk joined #gluster
08:02 ira_ joined #gluster
08:05 liquidat joined #gluster
08:11 ackjewt joined #gluster
08:17 ProT-0-TypE joined #gluster
08:22 Norky joined #gluster
08:35 mbukatov joined #gluster
08:37 glusterbot News from newglusterbugs: [Bug 1210205] 3.4.7 Repo not functional; repomod.xml not found!! <https://bugzilla.redhat.com/show_bug.cgi?id=1210205>
08:46 Rydekull joined #gluster
08:46 ndarshan joined #gluster
08:48 fyxim joined #gluster
08:56 kxseven joined #gluster
08:56 ctria joined #gluster
08:57 dusmant joined #gluster
08:58 gw ?
09:01 fyxim joined #gluster
09:03 johnmark joined #gluster
09:05 pppp joined #gluster
09:23 liquidat joined #gluster
09:24 ira_ joined #gluster
09:24 ira joined #gluster
09:26 prabu_ joined #gluster
09:27 julien joined #gluster
09:30 julienvey Hi, the EPEl.repo directory of the latest gluster release (3.7.0) is empty, did something change recently for this ? http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/EPEL.repo/
09:31 [Enrico] joined #gluster
09:33 hagarth julienvey: packages for 3.7.0 are being populated
09:34 overclk joined #gluster
09:35 tessier joined #gluster
09:36 julienvey hagarth: ok, great. Thanks
09:38 gw anyone see my question?
09:39 gw about "https://bugzilla.redhat.com/show_bug.cgi?id=1204247"
09:39 glusterbot Bug 1204247: high, unspecified, ---, bugs, NEW , mmap failed for CEN and END part of zip file
09:42 dusmant joined #gluster
09:43 ndarshan joined #gluster
09:47 Bhaskarakiran joined #gluster
09:48 ramteid joined #gluster
09:51 spiekey joined #gluster
09:51 spiekey Hello!
09:51 glusterbot spiekey: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:51 lalatenduM__ joined #gluster
09:52 spiekey i have a two node replica gluster setup with gluster 3.6 and i get: http://fpaste.org/222877/31942698/
09:52 lalatenduM joined #gluster
09:52 spiekey what does that error mean and how can i solve it?
09:53 pppp joined #gluster
09:55 overclk joined #gluster
09:56 gw dear all, i have the problem mentioned in this url "https://bugzilla.redhat.com/show_bug.cgi?id=1204247", can anyone get solution for this?
09:56 glusterbot Bug 1204247: high, unspecified, ---, bugs, NEW , mmap failed for CEN and END part of zip file
09:56 gw "mmap failed for CEN and END part of zip file
09:56 gw Error occurred during initialization of VM
09:56 gw java/lang/ClassNotFoundException: error in opening JAR file /glusterfs/Common/java/jre7_u51_64/lib/rt.jar"
09:57 gw i cannot run "./java -version" on gluster volume
09:57 atalur joined #gluster
10:03 spiekey anyone?
10:03 _shaps_ joined #gluster
10:06 gw i'm using centos6.6 64bit, jdk "jdk1.7.0_79" 64bit
10:07 hagarth spiekey: that seems to be coming from gluster volume status detail
10:07 glusterbot News from newglusterbugs: [Bug 1205128] Disperse volume: "df -h" on a cifs mount throws IO error and no file systems processed message <https://bugzilla.redhat.com/show_bug.cgi?id=1205128>
10:07 glusterbot News from newglusterbugs: [Bug 1221605] Scrub.log grows rapidly and the size increases upto 24GB in a span of 10 hours <https://bugzilla.redhat.com/show_bug.cgi?id=1221605>
10:07 gw gluster 3.5.3
10:08 spiekey hagarth: well, whats the problem or how can i fix it?
10:08 gw do you read my posted url?
10:09 gw bug1204247
10:09 hagarth spiekey: need to check. maybe you can determine why tune2fs is failing on your setup. The log messages are benign in nature and should not cause any disruptions in your deployment.
10:10 spiekey hagarth: hmm…looking at it they have always been there :-(
10:10 gw i dont have tune2fs failing
10:10 hagarth gw: your issue is different indeed
10:11 pkliczew_ joined #gluster
10:11 hagarth atinmu: any known problems with glusterd and volume status detail on ext*? ^^
10:11 ghenry joined #gluster
10:11 gw my issue apply to tomcat home folder on gluster volume
10:12 atinmu hagarth, I am currently looking into vol status problem which Manu reported.. he claims it ends up in a stale lock
10:13 Bhaskarakiran_ joined #gluster
10:13 pkliczew_ left #gluster
10:14 [Enrico] joined #gluster
10:15 nsoffer joined #gluster
10:15 hagarth atinmu: pinged you to check if there were any known issues with ext4/3 and volume status detail .. if you aren't then it is fine.
10:16 hagarth spiekey: what does tune2fs -l <partition> hosting the gluster bricks yield for Inode Size ?
10:16 hagarth gw: what distribution are you using?
10:16 gw centos 6.6 64bit
10:16 atinmu hagarth, if I recollect, I had seen a mail about it in gluster-users probably 6 months back
10:16 atinmu hagarth, but that problem got resolved
10:17 atinmu hagarth, need to check what was the solution
10:17 hagarth atinmu: great thanks! spiekey - maybe worth a search in the archives of gluster-users ?
10:17 DV joined #gluster
10:17 hagarth atinmu: I vaguely remember something to do with volume status detail being fixed in 3.6.x
10:19 glusterbot News from resolvedglusterbugs: [Bug 1221620] Bitd crashed on tier volume <https://bugzilla.redhat.com/show_bug.cgi?id=1221620>
10:20 autoditac joined #gluster
10:24 gw my device with xfs filesystem on lvm volume "/dev/mapper/VGGLUS-lvSYSVM01GD01 on /GLUS/SYSVM01GD01 type xfs (rw,noatime,inode64,logbufs=8,logbsize=256k,nobarrier)"
10:25 gw I then mount it by "hostname:/GLUS-SYSVM01GD on /var/lib/libvirt/images/sysvm01GD type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)"
10:29 gw so i guess "tune2fs -l" may not be helpfull
10:49 glusterbot News from resolvedglusterbugs: [Bug 1213703] geo-replication status xml output has incorrect grouping of pairs under sessions. <https://bugzilla.redhat.com/show_bug.cgi?id=1213703>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1218552] Rsync Hang and Georep fails to Sync files <https://bugzilla.redhat.com/show_bug.cgi?id=1218552>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1219937] Running status second time shows no active sessions <https://bugzilla.redhat.com/show_bug.cgi?id=1219937>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1220729] After reseting the scrubber still vol info shows scrubber throttle and frequency information and it missleads <https://bugzilla.redhat.com/show_bug.cgi?id=1220729>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1065635] dist-geo-rep: Deletes are not synced to slave <https://bugzilla.redhat.com/show_bug.cgi?id=1065635>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1091820] Changelog: Barrier in changelog for geo-replication to work with snapshot. <https://bugzilla.redhat.com/show_bug.cgi?id=1091820>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1159198] Dist-geo-rep : geo-rep doesn't log the list of skipped gfid after it failed to process the changelog. <https://bugzilla.redhat.com/show_bug.cgi?id=1159198>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1211327] Changelog: Changelog should be treated as discontinuous only on changelog enable/disable <https://bugzilla.redhat.com/show_bug.cgi?id=1211327>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1212063] [Geo-replication] cli crashed and core dump was observed while running gluster volume geo-replication vol0 status command <https://bugzilla.redhat.com/show_bug.cgi?id=1212063>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1219894] [georep]: Creating geo-rep session kills all the brick process <https://bugzilla.redhat.com/show_bug.cgi?id=1219894>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1207115] geo-rep: add debug logs to master for slave ENTRY operation failures <https://bugzilla.redhat.com/show_bug.cgi?id=1207115>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1159209] Geo-Replication Passive node is not getting promoted to active when one node of replicated slave volume goes down <https://bugzilla.redhat.com/show_bug.cgi?id=1159209>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1210965] Geo-replication very slow, not able to sync all the files to slave <https://bugzilla.redhat.com/show_bug.cgi?id=1210965>
10:49 glusterbot News from resolvedglusterbugs: [Bug 1104954] Dist-geo-rep :  geo-rep doesn't preserve the ownership while syncing entry operations to slave through mount-broker. <https://bugzilla.redhat.com/show_bug.cgi?id=1104954>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1159213] Dist-geo-rep :  geo-rep doesn't preserve the ownership while syncing entry operations to slave through mount-broker. <https://bugzilla.redhat.com/show_bug.cgi?id=1159213>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1202649] [georep]: Transition from xsync to changelog doesn't happen once the brick is brought online <https://bugzilla.redhat.com/show_bug.cgi?id=1202649>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1211037] [dist-geo-rep]:Directory not empty and Stale file handle errors in geo-rep logs during deletes from master in history/changelog crawl <https://bugzilla.redhat.com/show_bug.cgi?id=1211037>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1212410] dist-geo-rep : all the bricks of a node shows faulty in status if slave node to which atleast one of the brick connected goes down. <https://bugzilla.redhat.com/show_bug.cgi?id=1212410>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1141379] Geo-Replication - Fails to handle file renaming correctly between master and slave <https://bugzilla.redhat.com/show_bug.cgi?id=1141379>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1189363] ignore_deletes option is not something you can configure <https://bugzilla.redhat.com/show_bug.cgi?id=1189363>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1210562] Dist-geo-rep: Too many "remote operation failed: No such file or directory" warning messages in auxilary mount log on slave while executing "rm -rf" <https://bugzilla.redhat.com/show_bug.cgi?id=1210562>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1159195] dist-geo-rep: geo-rep status in one of rebooted node remains at "Stable(paused)" after session is resumed. <https://bugzilla.redhat.com/show_bug.cgi?id=1159195>
10:50 glusterbot News from resolvedglusterbugs: [Bug 1208470] [Dist-geo-rep] after snapshot in geo-rep setup, empty changelogs are  generated in the snapped brick. <https://bugzilla.redhat.com/show_bug.cgi?id=1208470>
10:51 ivan_rossi joined #gluster
10:53 pdrakeweb joined #gluster
10:53 gem joined #gluster
11:00 spalai joined #gluster
11:01 jkroon joined #gluster
11:02 DV joined #gluster
11:04 yossarianuk joined #gluster
11:05 Prilly joined #gluster
11:10 yossarianuk hi - I have a geo-rep centos 7 server setup - I cannot get local NFS to work
11:11 yossarianuk i.e - [2015-05-18 10:51:08.598144] E [rpcsvc.c:1303:rpcsvc_program_register_portmap] 0-rpc-service: Could not register with portmap 100021 4 38468
11:12 yossarianuk I have zseen this - http://www.gluster.org/pipermail/gluster-users.old/2015-January/020082.html
11:12 yossarianuk checked lock to = false
11:12 yossarianuk still occurs (after reboot)
11:12 yossarianuk and it appears the geo-repl is semi broken..
11:18 gildub joined #gluster
11:40 aaronott joined #gluster
11:58 gem joined #gluster
12:00 itisravi joined #gluster
12:01 rjoseph joined #gluster
12:02 overclk joined #gluster
12:03 jiffin joined #gluster
12:03 rafi1 joined #gluster
12:06 rafi joined #gluster
12:10 glusterbot News from newglusterbugs: [Bug 858732] glusterd does not start anymore on one node <https://bugzilla.redhat.com/show_bug.cgi?id=858732>
12:13 poornimag joined #gluster
12:14 LebedevRI joined #gluster
12:20 glusterbot News from resolvedglusterbugs: [Bug 1210205] 3.7 Repo not functional <https://bugzilla.redhat.com/show_bug.cgi?id=1210205>
12:20 glusterbot News from resolvedglusterbugs: [Bug 1210557] gluster peer probe with selinux enabled throws error <https://bugzilla.redhat.com/show_bug.cgi?id=1210557>
12:23 RameshN joined #gluster
12:29 nangthang joined #gluster
12:31 atalur joined #gluster
12:32 Anjana joined #gluster
12:44 rafi joined #gluster
12:53 dusmant joined #gluster
12:54 kanagaraj joined #gluster
12:54 necrogami joined #gluster
12:57 hagarth joined #gluster
13:01 jkroon joined #gluster
13:04 liquidat joined #gluster
13:06 Manikandan_ joined #gluster
13:11 bturner joined #gluster
13:17 wkf joined #gluster
13:17 nsoffer joined #gluster
13:19 wushudoin joined #gluster
13:20 halfinhalfout joined #gluster
13:20 Philambdo joined #gluster
13:24 dgandhi joined #gluster
13:31 firemanxbr joined #gluster
13:31 halfinhalfout left #gluster
13:34 pdrakeweb joined #gluster
13:37 Philambdo joined #gluster
13:38 hamiller joined #gluster
13:40 lexi2 joined #gluster
13:46 Philambdo1 joined #gluster
13:59 stickyboy joined #gluster
14:01 hagarth joined #gluster
14:03 chirino joined #gluster
14:15 julim joined #gluster
14:18 kdhananjay joined #gluster
14:21 spiekey_ joined #gluster
14:30 neofob joined #gluster
14:34 ira joined #gluster
14:42 Philambdo joined #gluster
14:46 rjoseph joined #gluster
14:54 nsoffer joined #gluster
15:05 aaronott joined #gluster
15:06 atinmu joined #gluster
15:07 jkroon joined #gluster
15:08 stickyboy joined #gluster
15:08 aaronott1 joined #gluster
15:09 jbrooks joined #gluster
15:12 ProT-0-TypE joined #gluster
15:18 coredump joined #gluster
15:26 nbalacha joined #gluster
15:29 ctria joined #gluster
15:35 poornimag joined #gluster
15:37 julim joined #gluster
15:39 pdrakeweb joined #gluster
15:39 JoeJulian yossarianuk: Due to bug 1181779 you need to either update rpcbind to rpcbind-0.2.0-27.el7 or start rpcbind without the -w switch.
15:39 glusterbot Bug https://bugzilla.redhat.com:443/show_bug.cgi?id=1181779 unspecified, unspecified, rc, steved, ON_QA , rpcbind prevents Gluster/NFS from registering itself after a restart/reboot
15:42 rjoseph joined #gluster
15:47 balacafalata joined #gluster
15:48 Gill joined #gluster
15:49 yossarianuk JoeJulian: thanks for the advice.
15:49 yossarianuk I have upgraded to Gluster 3.7.x - now the master is broken...
15:50 yossarianuk what ecver I do I cannot see the local brick volume status - i.e  Another transaction is in progress. Please try again after sometime.
15:50 yossarianuk cannot stop/start it
15:51 JoeJulian I don't suppose any glusterd logs show what transaction it thinks is in progress?
15:51 yossarianuk The error starts here I believ
15:51 yossarianuk E [MSGID: 106032] [glusterd-svc-mgmt.c:30:glusterd_svc_create_rundir] 0-management: Unable to create rundir /var/lib/glusterd/vols/master-vol/run [No such file or directory]
15:52 yossarianuk --> /var/lib/glusterd/vols/master-vol/run doesn't exist
15:52 yossarianuk sorry it does actually...
15:53 Twistedgrim joined #gluster
15:55 yossarianuk i.e I deleted the geo-rep volume and local brick - recreated local bricks - on the master im getting 'Another transaction is in progress. Please try again after sometime.'
15:56 yossarianuk joejulian: is this the right log file to look in
15:56 yossarianuk # /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
15:57 poornimag Is there any node in cluster executing some command and hung?
16:03 CyrilPeponnet Hey ! As we are in geo-rep topic, I have 2 questions :p
16:03 CyrilPeponnet Try to used change_detector as changelog always fallback to xsync
16:04 CyrilPeponnet And how to remove files when they are deleted from the master ? (there is the ignore_delete:true config by default, I don't know really why).
16:08 meghanam joined #gluster
16:10 yossarianuk ok removed bricks (manually/logs) seemed to be an rpcbind issue
16:11 yossarianuk JoeJulian: where can I find rpcbind-0.2.0-27.el7
16:11 gem joined #gluster
16:13 yossarianuk I have found it here -> this right ? http://buildlogs.centos.org/cah-0.0.1/rpcbind/20150422112154/0.2.0-27.atomic.0.el7.x86_64/
16:19 nbalacha joined #gluster
16:20 nbalacha joined #gluster
16:21 glusterbot News from resolvedglusterbugs: [Bug 1057292] option rpc-auth-allow-insecure should default to "on" <https://bugzilla.redhat.com/show_bug.cgi?id=1057292>
16:21 poornimag joined #gluster
16:22 mjrosenb joined #gluster
16:22 wtracz joined #gluster
16:22 wtracz Hi. Any recommendations on how to solve a directory split brain?
16:22 mjrosenb question: what perms does a file need to be in order to be considered a link-to?
16:24 mjrosenb I thought it was 000, +t
16:25 JoeJulian wtracz: If a directory shows split-brain, the only cure is to pick one and set the trusted.afr.* to 0x0
16:25 JoeJulian mjrosenb: right, mode 1000. But it also has to have the trusted.dht.linkto set.
16:25 wtracz Ok. Happy to do that, when you say set it to 0x0 do I do that on the bad node?
16:26 JoeJulian They're both bad... or both good... depends on our point of view.
16:26 JoeJulian The one with 0s will be overwritten.
16:27 JoeJulian Usually there's nothing to change so I have no idea why this gets stuck in this state.
16:27 mjrosenb JoeJulian: that's what I thought.
16:29 rwheeler joined #gluster
16:30 wtracz JoeJulian: I already see trusted.afr.vol-client-2 and -3 with 0x0 (this is a 2x2 setup)
16:30 wtracz In fact I see 0x0 on all nodes?
16:30 JoeJulian Then there's no split-brain.
16:30 CyrilPeponnet any geo-repo guru around ?
16:31 JoeJulian I don't think there is such a thing, CyrilPeponnet. ;)
16:31 CyrilPeponnet or guru like :p
16:32 mjrosenb JoeJulian: ok, better question: I have a link, but it is not being read as a link due to missing the +t
16:33 wtracz Sorry JoeJulian, if one node simply lacks trusted.afr on a child folder, then what do you suggest? Set it to 0x0 still?
16:33 mjrosenb JoeJulian: can I just copy the other file over it, and it'll be good?
16:33 JoeJulian mjrosenb: delete it.
16:34 JoeJulian ... and the gfid reference under .gluster
16:35 JoeJulian wtracz: It shouldn't matter. If that directory is showing up in "gluster volume heal $vol info split-brain" look at the timestamp. Unless new entries are showing up you don't have a problem.
16:35 poornimag joined #gluster
16:35 mjrosenb JoeJulian: just delete the link?
16:35 JoeJulian mjrosenb: yes. If it's needed it will be recreated.
16:35 mjrosenb won't it get recreated when I try to access the file again?
16:36 JoeJulian ... right, and it'll be recreated with the correct attributes.
16:37 mjrosenb JoeJulian: I'm not sure about that.
16:37 hchiramm joined #gluster
16:37 mjrosenb it looks like you can't set +t on a file on this filesystem
16:38 mjrosenb which I don't think used to be the case.
16:38 JoeJulian Ooho! Nice.
16:38 JoeJulian So it's not a posix filesystem?
16:38 JoeJulian What filesystem is it?
16:39 mjrosenb It really should be a posix filesystem
16:39 mjrosenb it is zfs
16:39 mjrosenb it was working previously,
16:39 mjrosenb but now chmod +t foo fails with
16:40 mjrosenb "Inappropriate file type or format"
16:47 wtracz JoeJulian, how would a subdirectory get different afr attributes to its parent? One has the various nodes listed (the parent), the other (the child) has none?
16:47 CyrilPeponnet any idea ? http://ur1.ca/mgw1m
16:47 mjrosenb JoeJulian: evidently, you need to be root to set +t on a file?
16:47 mjrosenb that is strange.
16:48 JoeJulian Oh, yes, you do.
16:48 mjrosenb does gluster run as root, or does it drop permissions?
16:48 JoeJulian It runs as root
16:49 mjrosenb ok, in that case, I suspect I know why it never correctly creates links, but I thought I fixed this already :-/
16:49 mjrosenb maybe I should just try upgrading to the official freebsd port.
16:50 JoeJulian wtracz: iirc, directories won't have trusted.afr attributes unless a change happens to that directory (ownership, permissions, etc) while a brick is offline.
16:51 wtracz JoeJulian, any ideas how to recover if that is the case?
16:51 JoeJulian No recovery is necessary.
16:51 JoeJulian Or at least none should be.
16:51 JoeJulian What is the problem you're trying to solve?
16:52 wtracz Ok. I have a mount point but ls fails due to I/O error on some directories.
16:52 wtracz When we run a heal, we see those folders in split brain output
16:53 JoeJulian What version?
16:53 wtracz 3.5.3 at the moment
16:53 milkyline joined #gluster
16:54 JoeJulian I'll have to take a look at that sometime.
16:54 wtracz We seem to have an awful lot of files in the split-brain entries :S
16:54 wtracz (or GFIDs)
16:55 JoeJulian I guess the solution would be to create the attributes for those directories, as you posited.
16:55 JoeJulian trusted.afr.vol-client-{0,1} for the first two bricks, trusted.afr.vol-client-{2,3} for the second two.
16:56 JoeJulian For files, use ,,(splitmount)
16:56 glusterbot https://github.com/joejulian/glusterfs-splitbrain
16:56 wtracz It seems to only be directories (not files)
16:56 wtracz Been reading your blog all day on it :)
16:57 JoeJulian :)
16:57 JoeJulian Sounds like I need to write another article on directory split-brain.
17:05 MrTink76 joined #gluster
17:05 wtracz Right, got one dir back with that
17:06 JoeJulian cool
17:06 JoeJulian wierd, but cool.
17:09 haomaiwang joined #gluster
17:11 ppai joined #gluster
17:12 kdhananjay1 joined #gluster
17:15 wtracz Hmm
17:15 wtracz Worked for one directory but not the other
17:16 mjrosenb is the upgrade from 3.3 to 3.6 painful?
17:20 JoeJulian no, and if it is, see your doctor.
17:20 mjrosenb good!
17:21 Rapture joined #gluster
17:21 mjrosenb I'll investigate the "official" port then.
17:22 mjrosenb In general, I assume that most things have gotten easier since the distributed settings mechanisim was added.
17:33 ProT-0-TypE joined #gluster
17:38 jmarley joined #gluster
17:56 spalai joined #gluster
17:59 CyrilPeponnet any idea why my geo-repo do not want to pass to rsync as engine and always falling back to xsync ? (while xsync doing the job of replication, it doesn't remove file I remove from master)
18:00 cholcombe joined #gluster
18:09 peacho joined #gluster
18:13 Gill joined #gluster
18:13 MrTink76 left #gluster
18:14 Gill_ joined #gluster
18:17 JoeJulian CyrilPeponnet: did you say which version you're running?
18:18 sage joined #gluster
18:20 Gill joined #gluster
18:21 JoeJulian That's confusing. There's never a time when the fallback_xsync function is called.
18:22 ProT-0-TypE joined #gluster
18:22 peacho_ joined #gluster
18:27 CyrilPeponnet @JoeJulian 3.5.2
18:27 CyrilPeponnet yeah I also saw in source that it's never called
18:28 rafi joined #gluster
18:29 JoeJulian If you want to figure that out, raise an exception in that function and look at the traceback.
18:29 CyrilPeponnet on the master of slave
18:29 CyrilPeponnet Master I guess
18:30 CyrilPeponnet s/of/or
18:30 JoeJulian Whichever was producing that log message.
18:30 CyrilPeponnet ok I will try that
18:32 hchiramm joined #gluster
18:34 hagarth CyrilPeponnet: if you can't figure out a reason, please drop a note on gluster-users and usually aravinda will help when he gets to read it.
18:35 ppai joined #gluster
18:44 plarsen joined #gluster
18:47 pdrakeweb joined #gluster
18:58 rafi1 joined #gluster
19:04 wtracz joined #gluster
19:04 wtracz Ok, carrying on from earlier, still have split brain issues.
19:05 wtracz One node shows afr.data-client-2=0x0....0 and afr.data-client-3=0x000...10000000, other shows 2-3 flipped.
19:09 nsoffer joined #gluster
19:10 julim joined #gluster
19:11 coredump joined #gluster
19:12 chirino joined #gluster
19:13 CyrilPeponnet @JoeJulian regarding to 3.5 src, it happens when changelog_scan trigger an exception https://github.com/gluster/glusterfs/blob/release-3.5/geo-replication/syncdaemon/master.py#L1054
19:16 CyrilPeponnet the source seems to be cls._get_api('gf_changelog_scan'), Can't find anything use full above this call
19:16 JoeJulian Ah, ok, that's why I couldn't find it. I thought you were on a different version when I went digging.
19:17 CyrilPeponnet https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L344
19:18 ppai joined #gluster
19:18 CyrilPeponnet If you have a clue on how to debug this :p maybe printing the exception will help I will try that
19:19 JoeJulian grep for "error writing changelog filename" in your logs?
19:20 CyrilPeponnet no luck
19:20 JoeJulian anything else that's wrong doesn't produce an error so I'd be stumped.
19:20 JoeJulian Talk to aravinda in gluster-users ml.
19:21 CyrilPeponnet Yeah, hope he will respond
19:22 CyrilPeponnet I had some talks previously with him but he stopped answering :p
19:22 JoeJulian Well, hopefully the summit will have re-invigorated that effort.
19:23 CyrilPeponnet :p
19:27 spalai joined #gluster
19:27 spalai left #gluster
19:35 CyrilPeponnet @JoeJulian actually https://github.com/gluster/glusterfs/blob/release-3.5/geo-replication/syncdaemon/master.py#L1076 is falling with  [Errno 2] No such file or directory as exception
19:37 JoeJulian Interesting. The answer must be in one of those arguments.
19:37 CyrilPeponnet I'm digging
19:38 JoeJulian Would be nice if that "except" gave those details.
19:38 CyrilPeponnet sure...
19:40 wtracz JoeJulian, been reading some old logs from here and seems this problem I've got happened to someone last year.
19:41 wtracz Is there any way to script up the setxattr safely?
19:41 glusterbot News from newglusterbugs: [Bug 1222678] backupvolfile-server, backup-volfile-servers options in /etc/fstab / list of volfile-server options on command line ignored when mounting <https://bugzilla.redhat.com/show_bug.cgi?id=1222678>
19:41 JoeJulian If it were me, I'd just pick one side of the replica and set the xattrs to 0 for every directory.
19:42 wtracz Ok, it is very very strange. We just took one node out to do an upgrade and upon reintroducing it hit issues everywhere
19:43 wtracz (to the OS, not the gluster)
19:43 CyrilPeponnet @JoeJulian I found in /var/run/gluster/my_vol/ssh#xxxx/uuid/changes.log: ] E [gf-changelog.c:204:gf_changelog_notification_init] 0-glusterfs: could not connect to changelog socket! bailing out...
19:43 glusterbot CyrilPeponnet: Error: Spurious "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
19:45 wtracz JoeJulian, setting it to 0x0 triggers its replacement right?
19:46 wtracz i.e. node with 0x0 is wiped in favour of non 0x0 node?
19:46 JoeJulian correct
19:47 CyrilPeponnet @JoeJulian in fact the socker for this vol seems to not exist  /var/run/gluster/changelog-ce749a38ba30d4171cd674ec00ab24f9.sock is not found
19:49 Intensity joined #gluster
19:53 wtracz Hrmmm, how to merge things? (Thanks for your help with all this too!)
19:55 JoeJulian wtracz: heal...full
19:55 CyrilPeponnet Looks this https://github.com/gluster/glusterfs/blob/release-3.5/xlators/features/changelog/lib/src/gf-changelog.c#L477 is falling
19:56 CyrilPeponnet I may be wrong my C skills are not awesome
19:58 JoeJulian CyrilPeponnet: selinux?
19:59 CyrilPeponnet disabled
19:59 JoeJulian figured
19:59 CyrilPeponnet :)
19:59 JoeJulian /var/run/gluster exists?
19:59 CyrilPeponnet yep
19:59 CyrilPeponnet but no too much things in here
20:00 CyrilPeponnet (well actually around 4000 files XSYNC-CHANGELOG
20:03 coredump joined #gluster
20:05 CyrilPeponnet looks like the socket is not created
20:05 CyrilPeponnet not sure when this should be created
20:09 JoeJulian I would think it would be part of the brick process.
20:09 JoeJulian Can you restart the brick?
20:09 CyrilPeponnet Not really...
20:10 CyrilPeponnet around 4k clients connected
20:10 CyrilPeponnet I do a restart last week btw
20:11 JoeJulian find /var/lib/glusterd/vols -name '*.vol' | xargs grep changelog
20:11 glusterbot News from newglusterbugs: [Bug 1206539] Tracker bug for GlusterFS documentation Improvement. <https://bugzilla.redhat.com/show_bug.cgi?id=1206539>
20:11 CyrilPeponnet a bunch !
20:12 JoeJulian hmm
20:12 CyrilPeponnet like changelog-dir /brickpath/.glusterds/changelogs
20:12 CyrilPeponnet changelog-on on and changelog-brick /brickpath
20:13 wtracz How much skew in stat output would make gluster think something had changed?
20:13 JoeJulian wtracz: It *should* just check the xattrs.
20:14 wtracz It is saying data changed on an empty folder.
20:14 CyrilPeponnet @JoeJulian changelog-dir contains more than 12k files CHANGELOG.timestamp
20:14 JoeJulian What about that sock?
20:15 CyrilPeponnet @wtracz could be access time
20:15 CyrilPeponnet the sock doesn't exist
20:15 JoeJulian I was hoping it was just in the wrong directory.
20:15 CyrilPeponnet mee too
20:15 wtracz stat shows no difference, just a minor difference in skew (sub ms) @CyrilPeponnet
20:16 CyrilPeponnet @wtracz make sure your nodes are using ntp sync
20:17 wtracz They should be which is what I am worried at. How tight does that sync need to be?
20:17 CyrilPeponnet @JoeJulian In fact I have several vol in replica or single, and I can't find any sock for changelog
20:18 CyrilPeponnet @JoeJulian !!! I maybe found something
20:18 CyrilPeponnet lost -> glusterfs   /var/run/ec37ce2efe4060fd89ddf5769e4922ed.socket
20:18 CyrilPeponnet lsof
20:19 CyrilPeponnet so yeah the path passed to changelog_register is wrong
20:19 JoeJulian maybe
20:19 JoeJulian there are sockets for self-heal and nfs too.
20:19 CyrilPeponnet hm
20:21 CyrilPeponnet @JoeJulian you are right and those sock must be prefixed with changelog- (ans this is not the case)
20:21 CyrilPeponnet damn
20:22 vimal joined #gluster
20:22 CyrilPeponnet I will try the ML...
20:22 wtracz @CyrilPeponnet, how close do the times need to be? I see very very minor variations in stat
20:22 CyrilPeponnet wtracz don't know
20:23 wtracz Change: 2015-05-18 13:40:58.721247917 vs Change: 2015-05-18 13:40:58.723540441
20:26 JoeJulian CyrilPeponnet: you should have had an Error, "unable to open/create changelog file %s (reason: %s). change-logging will be inactive"
20:26 JoeJulian wtracz: I've never seen a problem with that skew.
20:26 CyrilPeponnet I try to find it
20:27 wtracz @JoeJulian, seems ridiculous that it would be that but trying to identify what "data" has changed on an empty directory...
20:27 JoeJulian My guess would be just from those xattrs.
20:28 bene2 joined #gluster
20:28 CyrilPeponnet @JoeJulian No error message regargind changelog
20:30 JoeJulian That means it never got to "changelog_open"
20:30 CyrilPeponnet I am not in debug
20:30 CyrilPeponnet in which log should it be
20:30 CyrilPeponnet in volume log ?
20:31 CyrilPeponnet geo rep log ?
20:31 CyrilPeponnet brick log ?
20:31 JoeJulian It was error level
20:31 JoeJulian and I would expect it to be in brick log.
20:31 badone joined #gluster
20:35 CyrilPeponnet I found some [2015-05-14 06:16:14.059396] E [changelog-helpers.c:168:changelog_rollover_changelog] 0-usr_global-changelog: Failed to send file name to notify thread (reason: Broken pipe)
20:35 CyrilPeponnet on other nodes
20:35 CyrilPeponnet ( I have 3 nodes)
20:35 wtracz Forgive me for being stupid (again), but if I have client {0,1} do setfattr on 0 to set vol-client-1 to 0x0 does that push from 0 > 1 or 1 < 0 ?
20:37 ppai joined #gluster
20:39 JoeJulian non-zero = change pending
20:39 JoeJulian so it's marked as a change pending for whichever client is non-zero
20:42 glusterbot News from newglusterbugs: [Bug 1222614] Misleading error message during snapshot creation <https://bugzilla.redhat.com/show_bug.cgi?id=1222614>
20:44 wtracz @JoeJulian, so is non-zero mean incoming or outgoing?
20:45 wtracz i.e. push from this client out to that or a change is queued to apply to the node?
20:47 JoeJulian trusted.afr.myvol-client-0 = 0, trusted.afr.myvol-client-1 = 100. This means that for volume myvol, the second brick in a replica pair has had 100 changes since it last was able to update the first brick.
20:48 JoeJulian That's the short version.
20:48 JoeJulian @extended attributes
20:48 glusterbot JoeJulian: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://pl.atyp.us/hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
20:48 JoeJulian That last link is the long version.
20:49 wtracz My head :D So, if on myvol2 (i.e. client-1) we should be setting the attribute for client-0 to 0 if we want client-1 to be the master
20:49 JoeJulian yes
20:52 wtracz Thanks. I think with seeing 0x1 on each and having 2 nodes we're going a little bit nutty
21:06 jackdpeterson joined #gluster
21:25 Gill joined #gluster
21:28 gildub joined #gluster
21:29 wtracz Ok. Having done all the xattrs etc.
21:29 wtracz What could be the reason for still having issues (evertying looks good on glusterfsd)
21:31 JoeJulian still having read errors at the client?
21:34 wtracz Yeah
21:36 wtracz No changes, nothing in split brain, still input/output error. Incredibly weird
21:43 wtracz Aha! think we may have figured it
21:45 wtracz If LUKS was not started and we're mounting with gluster running, could that explain this
21:49 CyrilPeponnet @JoeJulian I just send an email, the funny thing is that my brick log are all empty (I guess because the log rotate rotate too much).
21:49 JoeJulian Or because they rotated the logs instead of copytruncate.
21:50 JoeJulian Which would leave the logs open and being written unless you HUP.
21:52 CyrilPeponnet could ne
21:52 CyrilPeponnet be
21:52 CyrilPeponnet is the changelog-bla.sock should exist for all volume or only for geo-replicated ones ?
21:54 jermudgeon joined #gluster
21:55 JoeJulian Should only be for geo-rep
21:56 CyrilPeponnet ok make sense, so I should be created when I start geo-rep ?
21:56 CyrilPeponnet it
21:56 JoeJulian That's what I would expect.
21:56 CyrilPeponnet :p
21:57 JoeJulian It should change the vol file and reload the brick graph.
22:05 CyrilPeponnet well /var/lib/gluster/vol/myvol/info show changeling.changelog to on
22:11 CyrilPeponnet Pfdf giving for now I will remove all file by hand
22:12 CyrilPeponnet Thanks for your time @JoeJulian, as usual :)
22:12 JoeJulian You're welcome
22:12 JoeJulian I'd help more, but I'm actually making progress with my $dayjob.
22:12 CyrilPeponnet so do I :)
22:18 wtracz joined #gluster
22:22 Gill joined #gluster
22:23 ppai_ joined #gluster
22:24 Prilly joined #gluster
22:29 ppai joined #gluster
22:33 ShaunR joined #gluster
22:57 lexi2 joined #gluster
22:59 tessier joined #gluster
23:07 Gill joined #gluster
23:17 prg3 joined #gluster
23:19 Prilly joined #gluster
23:26 coredump joined #gluster
23:27 ShaunR I have a test gluster setup on virtual servers, 3 servers being used for storage, http, and mysql.  Each server mounts the gluster volume that apache serves.  I setup the volume with a replica of 3, it seams very slow though.... a default joomla install takes 3-6 seconds to load which is horrible.
23:28 ShaunR Should i be using a replica of 3?  anything i can do to improve performance... the servers are also only linked at 100mbit
23:31 plarsen joined #gluster
23:31 aaronott joined #gluster
23:31 coredump joined #gluster
23:34 coredump joined #gluster
23:38 coredump joined #gluster
23:40 coredump joined #gluster
23:43 coredump joined #gluster
23:46 coredump joined #gluster
23:46 tessier gluster newbie here: I had to reboot one of my gluster servers for patching etc. Now it says the volume is not online:
23:46 tessier Gluster process                                         Port    Online  Pid
23:46 tessier ------------------------------------------------------------------------------
23:46 glusterbot tessier: ----------------------------------------------------------------------------'s karma is now -5
23:47 tessier Brick 10.0.2.143:/export/diskc/brick                    N/A     N       N/A
23:47 tessier How do I bring it back online? I thought it would reconnect and sync up when I restarted the machine but apparently not.
23:49 coredump joined #gluster
23:53 coredump joined #gluster
23:55 CyrilPeponnet @tessier gluster vol bla start ?
23:56 CyrilPeponnet check if daemon are running as well
23:56 coredump joined #gluster
23:57 tessier CyrilPeponnet: gluster daemons are running. gluster vol bla start results in "volume start: diskb: failed: Volume diskb already started"
23:58 CyrilPeponnet fpaste gluster vol status detail
23:59 tessier http://fpaste.org/223160/31993571/

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary