Camelia, the Perl 6 bug

IRC log for #gluster, 2013-08-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 chirino joined #gluster
00:07 verdurin_ joined #gluster
00:16 chirino joined #gluster
00:27 chirino joined #gluster
00:34 chirino joined #gluster
00:34 avati joined #gluster
00:39 yinyin joined #gluster
00:41 matthewh Ok, the problem I was having seems to be related to NAT. If the localhost tries to use the external IP (which doesn't have a hairpin) it won't work.  Changing the hosts file for 127.0.0.1 for gluster1 and so on for each host made it work
00:42 chirino joined #gluster
00:54 chirino joined #gluster
01:05 chirino joined #gluster
01:06 cyberbootje anyone here?
01:14 chirino joined #gluster
01:24 chirino joined #gluster
01:34 semiosis https://api.travis-ci.org/semios​is/glusterfs-java-filesystem.png
01:34 glusterbot <http://goo.gl/7n97Ij> (at api.travis-ci.org)
01:34 semiosis build => passing
01:34 semiosis \o/
01:35 chirino joined #gluster
01:36 semiosis chirino: https://api.travis-ci.org/semios​is/glusterfs-java-filesystem.png
01:36 glusterbot <http://goo.gl/7n97Ij> (at api.travis-ci.org)
01:36 semiosis build => passing
01:36 semiosis just said that before you got here
01:37 semiosis oh now i see you're bouncing, ok
01:43 nightwalk joined #gluster
01:46 chirino joined #gluster
01:57 kanagaraj_ joined #gluster
01:58 chirino joined #gluster
02:06 chirino joined #gluster
02:14 bharata joined #gluster
02:19 chirino joined #gluster
02:24 sprachgenerator joined #gluster
02:34 bala joined #gluster
02:38 __Bryan__ joined #gluster
02:41 chirino joined #gluster
02:43 awheele__ joined #gluster
02:46 awheele__ joined #gluster
02:50 chirino joined #gluster
02:54 shubhendu joined #gluster
02:56 jag3773 joined #gluster
02:57 chirino joined #gluster
02:58 andresmujica joined #gluster
03:09 _pol joined #gluster
03:10 chirino joined #gluster
03:18 chirino joined #gluster
03:18 CheRi joined #gluster
03:31 CheRi joined #gluster
03:31 chirino joined #gluster
03:34 kanagaraj_ joined #gluster
03:34 yinyin joined #gluster
03:36 shylesh joined #gluster
03:37 kanagaraj joined #gluster
03:40 Ramereth joined #gluster
03:42 chirino joined #gluster
03:53 chirino joined #gluster
04:01 mohankumar joined #gluster
04:03 chirino joined #gluster
04:11 RameshN joined #gluster
04:16 ppai joined #gluster
04:17 krink joined #gluster
04:18 chirino joined #gluster
04:20 _pol joined #gluster
04:22 itisravi joined #gluster
04:32 rastar joined #gluster
04:33 chirino joined #gluster
04:38 CheRi joined #gluster
04:39 yinyin joined #gluster
04:42 chirino joined #gluster
04:52 aravindavk joined #gluster
04:54 itisravi joined #gluster
05:01 chirino joined #gluster
05:13 chirino joined #gluster
05:18 nshaikh joined #gluster
05:26 satheesh1 joined #gluster
05:29 lalatenduM joined #gluster
05:30 lala_ joined #gluster
05:33 chirino joined #gluster
05:39 RameshN joined #gluster
05:45 shruti joined #gluster
05:45 hagarth joined #gluster
05:47 chirino joined #gluster
05:53 ababu joined #gluster
06:01 bulde joined #gluster
06:13 chirino joined #gluster
06:14 psharma joined #gluster
06:21 rgustafs joined #gluster
06:22 glusterbot New news from resolvedglusterbugs: [Bug 953694] Requirements of Samba VFS plugin for glusterfs <http://goo.gl/v7g29>
06:23 jtux joined #gluster
06:23 chirino joined #gluster
06:24 yinyin joined #gluster
06:25 satheesh1 joined #gluster
06:28 vshankar joined #gluster
06:31 nshaikh left #gluster
06:35 chirino joined #gluster
06:36 dusmant joined #gluster
06:38 nshaikh joined #gluster
06:40 vimal joined #gluster
06:44 chirino joined #gluster
06:50 ndarshan joined #gluster
06:52 glusterbot New news from resolvedglusterbugs: [Bug 968301] improvement in log message for self-heal failure on file/dir in fuse mount logs <http://goo.gl/GI3SX>
06:54 aravindavk joined #gluster
07:01 chirino joined #gluster
07:02 andreask joined #gluster
07:06 rjoseph joined #gluster
07:07 hybrid5121 joined #gluster
07:07 psharma joined #gluster
07:09 chirino joined #gluster
07:13 eseyman joined #gluster
07:16 chirino joined #gluster
07:24 chirino joined #gluster
07:28 foster joined #gluster
07:32 asias joined #gluster
07:36 chirino joined #gluster
07:39 ngoswami joined #gluster
07:44 chirino joined #gluster
07:50 mohankumar joined #gluster
07:51 chirino joined #gluster
07:59 chirino joined #gluster
08:05 ricky-ticky joined #gluster
08:07 chirino joined #gluster
08:10 dusmant joined #gluster
08:10 vincent_vdk joined #gluster
08:11 ntt_ joined #gluster
08:12 ababu joined #gluster
08:13 eseyman joined #gluster
08:15 ntt_ Hi. I have a glusterfs with replica = 2. If one node goes down, how can i mount the fs without change the command? eg: mount -t glusterfs server1:/vol1 /mnt/test .  If server1 goes down, i have to manually connect to server2 changing the hostname or exists an automated procedure?
08:16 samppah ntt_: use -o backupvol-fileserver=servername option to set another server
08:20 ntt_ samppah: I have: "sudo mount -t glusterfs server1:/vol1 test -o backupvol-fileserver=server2 /usr/local/bin/fusermount-glusterfs: mount failed: Invalid argument Mount failed. Please check the log file for more details."   but nothing in logs
08:21 samppah ntt_: oh, my bad.. i think the correct one is backupvolfile-server
08:22 chirino joined #gluster
08:29 ntt_ samppah: sudo mount -t glusterfs -o backupvolfile-server=server2 server1:/vol1 test ERROR: Mount point does not exist.
08:29 ntt_ but if i do sudo mount -t glusterfs server2:/vol1 test  , it works
08:29 samppah oh
08:29 samppah hmm
08:30 samppah what glusterfs version you are using?
08:30 samppah i think that option was introduced in 3.3.1
08:30 ntt_ 3.4
08:30 ujjain joined #gluster
08:32 ntt_ I'm trying also sudo mount -t glusterfs -o backupvolfile-server=server2:/vol1 server1:/vol1 test but nothing
08:32 samppah i'll test it.. just a sec
08:33 ntt_ ok. thank you
08:33 deepakcs joined #gluster
08:33 samppah ntt_: [root@boar1 mnt]# mount -t glusterfs -o backupvolfile-server=boar2 boar1:/hiomo1-ovirt1-sas1 /mnt/hiomo1-ovirt1-sas1.. this worked for me
08:35 samsamm joined #gluster
08:37 ntt_ samppah: It works! I don't known why my first command give me an error.
08:37 samppah great! :)
08:38 ntt_ thank you
08:39 jm_sa joined #gluster
08:39 jmsa joined #gluster
08:42 chirino joined #gluster
08:50 chirino joined #gluster
08:54 Hoggins joined #gluster
08:57 Hoggins Hello everyone, simple problem here, and I don't know if I need to address it on a networking point of view or on a "Glusterish" point of view : I have several servers accessing the same GlusterFS cluster (two nodes replica) through Gigabit interfaces. Problem is : when a client accesses a big file, all the other clients trying to access small or big files hang. I don't know how to address this, and it is quite annoying. Is there some
08:57 Hoggins sort of load-balancing algorithm available, allowing a client to select one of the GlusterFS nodes that is least used ?
09:02 chirino joined #gluster
09:05 samppah Hoggins: it definetly shouldn't block access to other files.. what version you are using?
09:06 Hoggins samppah: I'm using 3.4 on all nodes, clients and servers
09:07 Hoggins I mount the files with Gluster FUSE, not NFS
09:13 kanagaraj joined #gluster
09:14 kanagaraj joined #gluster
09:17 Hoggins I don't know where to look. Are there tuning options for my simple 2 replica setup ?
09:17 CheRi joined #gluster
09:17 Hoggins as you said, samppah, it should work out-of-the-box without tuning, I guess, and accesses to small files should not be impacted
09:17 Hoggins so I must be missing something
09:19 chirino joined #gluster
09:21 samppah Hoggins: i'm out of clue also.. is it blocking access until big file is released?
09:24 Hoggins yeah... actually, I guess it is not *fully* blocking... it must be able to get some data, but it looks like the big files take all the bandwidth, and accessing these files is fluid again once the big file is released, yes.
09:25 samppah is it writing or reading or does it matter at all what it is doing?
09:26 yinyin joined #gluster
09:27 mbukatov joined #gluster
09:27 Hoggins well I'm not sure about that... I guess it's only about reading in this case... would that make a difference if the heavy accesses on the big files were writing, and reading for the small files ?
09:28 samppah Hoggins: yeah i think that write barriers or reaching dirty byte limits could cause it
09:28 samppah what kind of servers you have? how many disks and are you using raid?
09:28 asias joined #gluster
09:29 chirino joined #gluster
09:30 Hoggins okay, that's interesting... my Gluster bricks are on virtual machines with plenty of RAM and CPU power, running on two separate KVM hosts using simple RAID1 arrays.
09:35 samppah Hoggins: can you use iostat or atop to see disk usage?
09:36 chirino joined #gluster
09:36 Hoggins I use nmon on both hosts, and disk usage never reaches above 30% when I encounter the problem... I don't believe the disks are the problem here.
09:46 duerF joined #gluster
09:46 Peanut Hi folks - I have a simple gluster setup with 2 servers which are bricks, and also run a bunch of KVMs. This setup is currently in testing. Later today, one of the two servers will have a terrible accident and will be re-installed from scratch. Anything I should know about rebuiling gluster after the disaster?
09:48 chirino joined #gluster
09:56 chirino joined #gluster
09:57 psharma joined #gluster
09:57 itisravi joined #gluster
09:58 xymox joined #gluster
09:59 ndarshan joined #gluster
10:06 mooperd joined #gluster
10:08 chirino joined #gluster
10:23 rjoseph joined #gluster
10:35 chirino joined #gluster
10:38 satheesh joined #gluster
10:45 chirino joined #gluster
10:56 spider_fingers joined #gluster
10:57 spider_fingers left #gluster
10:57 kanagaraj joined #gluster
11:03 chirino joined #gluster
11:08 RicardoSSP joined #gluster
11:08 DV joined #gluster
11:11 ntt_ Hi. I'm trying to compile a c program with #include api/glfs.h  and I have a linker error. What is the -l argument for glusterfs with gcc?
11:14 zetheroo joined #gluster
11:15 chirino joined #gluster
11:16 zetheroo so I moved one of the VM's onto the testbed and one of the gluster servers - sure enough the filesystem on the VM went into read-only
11:16 zetheroo unfortunately this makes gluster utterly useless for us :(
11:19 Peanut zetheroo: what exactly happened?
11:20 zetheroo when one brick goes down (like when the host stops/is shutdown ...) etc), the VM's running on the host which is still up go into read-only mode
11:20 hagarth zetheroo: can you please log a bug for this with the gluster logs attached? This does not happen in my setup.
11:21 ngoswami joined #gluster
11:21 zetheroo to get the VM back to read/write mode we have to get the brick that went down back up and restart the VM's
11:22 zetheroo hagarth: thing is that this is pretty urgent ... we have production VM's running on this setup and did not anticipate this issue
11:22 zetheroo also, where does one log a bug?
11:23 hagarth zetheroo: I understand but unless more details are provided, it would be difficult to analyse your issue.
11:23 zetheroo ok
11:23 hagarth @bugs
11:23 zetheroo what kind of details are needed?
11:23 hagarth zetheroo: http://bugzilla.redhat.com
11:23 glusterbot Title: Red Hat Bugzilla Main Page (at bugzilla.redhat.com)
11:24 hagarth zetheroo: glusterfs logs from all nodes would be useful
11:26 zetheroo in the bug report what do I select as the Component?
11:28 hagarth replicate/afr
11:28 zetheroo ok
11:31 zetheroo there are a fair few logs in /var/log/glusterfs/  .... any ones in particular needed?
11:32 zetheroo looks like this http://paste.ubuntu.com/5992376/
11:32 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
11:32 chirino joined #gluster
11:33 satheesh1 joined #gluster
11:33 hagarth zetheroo: it would be great if you can tgz the directory and upload it.
11:34 zetheroo ok
11:35 Peanut In general, this is not what should happen, I hope? If I lose one brick in a replicated setup, I should still have R/W access to it? I'm about to perform that test myself today actually.
11:36 hagarth Peanut: yes, as long as one brick of a replica set is available, R/W access should continue.
11:36 zetheroo Peanut: it seems like the brick itself remains r/w ... but the filesystems in the VM's go into read-only
11:36 Peanut Oh.. how would the VMs even notice?
11:38 hagarth zetheroo: the filesystems in VMs are going read only because of an I/O failure which should not normally happen if a single brick in a replicated set goes offline
11:38 samppah zetheroo: have you reduced network.ping-timeout value?
11:38 zetheroo I think the brick goes into a temporary read/only mode and then comes out of it after a certain amount of time ... but the VM's get stuck in read-only during that time
11:38 zetheroo samppah: yes, I have tried that
11:40 psharma joined #gluster
11:43 chirino joined #gluster
11:44 RameshN_ joined #gluster
11:44 manik joined #gluster
11:48 Peanut "ipmitool -H cluster0 -I lanplus -U ADMIN chassis power off" - here we go *grin*
11:50 zetheroo ok it's there ... https://bugzilla.redhat.com/show_bug.cgi?id=997889
11:50 glusterbot <http://goo.gl/dhuUGy> (at bugzilla.redhat.com)
11:50 glusterbot Bug 997889: urgent, unspecified, ---, pkarampu, NEW , VM filesystem read-only
11:50 zetheroo hope I did it correctly ... :)
11:53 Peanut zetheroo: I just pulled the plug on one of my two cluster nodes - the VMs on the other server see no problems as far as I can tell, everything is still nicely mounted rw, and I can create new files.
11:54 zetheroo Peanut: here is an interesting twist .... on a test I did today I had 4 Linux VM's running on host2 and I shutdown host1.
11:54 zetheroo 3 out of 4 of the VM's were still r/w
11:54 chirino joined #gluster
11:56 zetheroo the 4th one which dropped to read-only was one of our production VM's which serves up our internal wiki, intranet etc ...
11:56 zetheroo the other 3 VM's do nothing of the sort
11:57 zetheroo also, the 3 VM's which were ok all have the Gnome Desktop installed ... whereas the 4th that failed does not
11:57 zetheroo not sure if any of that is useful ... but those where the only differences that I could come up with
12:01 satheesh joined #gluster
12:02 aliguori joined #gluster
12:04 Peanut What is the glusterfs command that would show me the status of my volumes, in the sense that I can see I'm short one brick?
12:06 kanagaraj Peanut, gluster volume status [all|VOLNAME [BRICKNAME]]
12:06 zetheroo gluster volume status
12:06 zetheroo ?
12:07 Peanut I tried those commands, thanks - doesn't actually show any warning (that I can recognise) that the replica is degraded in any way?
12:08 B21956 joined #gluster
12:08 psharma joined #gluster
12:09 chirino joined #gluster
12:11 zetheroo I am seeing this :
12:11 zetheroo Self-heal Daemon on localhostN/ANN/A
12:11 zetheroo shouldn't this be online?
12:12 sprachgenerator joined #gluster
12:13 zetheroo also seeing this:
12:13 zetheroo Brick saturn:/mnt/brick24009NN/A
12:13 zetheroo how do you get a brick back online?
12:16 robo joined #gluster
12:18 kanagaraj gluster volume start <volname> force
12:18 hagarth joined #gluster
12:23 zetheroo is a brick and a volume the same thing?
12:24 chirino joined #gluster
12:24 kanagaraj no, start force will try to bring up the bricks which are down
12:25 zetheroo ok thanks - that worked
12:29 nshaikh joined #gluster
12:33 tjikkun_work joined #gluster
12:34 satheesh1 joined #gluster
12:39 sjoeboo joined #gluster
12:40 captainvmos joined #gluster
12:42 captainvmos hello, I require some assistance. I've got a new build of gluster3.3 on xfs/ubuntu. Everything went well with the build and I can mount it, but when I write files into the mount, they don't appear, although they do appear in the brick, albiet empty
12:42 captainvmos any pointers?
12:43 chirino joined #gluster
12:44 bala1 joined #gluster
12:49 plarsen joined #gluster
12:52 andreask joined #gluster
12:53 msvbhat captainvmos: Anything in gluster mount logs?
12:53 msvbhat captainvmos: Is it a fuse mount or nfs mount?
12:53 awheeler joined #gluster
12:54 captainvmos msvbhat: it's glusterfs, hang on I can pastebin logs
12:54 awheeler joined #gluster
12:55 social hmm I'm having issue on gluster 3.4.0 where I create file, do something with it, delete it, create same file, try to open it and i get permission denied
12:55 social if I add sleep between second create and open it's OK
12:55 captainvmos how about that?
12:55 captainvmos http://pastebin.com/qFzFZXnN
12:55 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
12:56 eseyman joined #gluster
12:57 kkeithley @paste
12:57 glusterbot kkeithley: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
12:57 captainvmos alrighty, dpaste it is https://dpaste.de/GW82w/
12:57 glusterbot Title: dpaste.de: Snippet #237197 (at dpaste.de)
12:58 kkeithley @later tell JoeJulian that glusterbot is still telling people to use dpaste; it should be telling them to use pastebinit
12:58 glusterbot kkeithley: The operation succeeded.
12:58 * kkeithley wasn't trying to be a nag
13:00 captainvmos this is what i get when using this in fstab to mount
13:00 captainvmos store-01-mail:/data     /data   glusterfs       0       0
13:00 captainvmos but I also get the same problem when following the mounting intructions here
13:00 captainvmos http://www.gluster.org/category/ubuntu/
13:00 glusterbot Title: Glorious Gluster – How to setup GlusterFS on Rackspace Cloud and Ubuntu 12.10 | Gluster Community Website (at www.gluster.org)
13:01 kkeithley @ppa
13:01 glusterbot kkeithley: The official glusterfs packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.4 stable: http://goo.gl/u33hy
13:01 captainvmos but when I do that I also get stuff like this in the logs
13:02 captainvmos [2013-08-16 11:08:45.324840] E [stripe-helpers.c:268:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
13:02 captainvmos [2013-08-16 11:08:45.333404] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 3314: FLUSH() ERR => -1 (Invalid argument)
13:02 msvbhat captainvmos: Okay. So it's a striped volume?
13:02 captainvmos striped and replicated
13:02 satheesh joined #gluster
13:03 captainvmos 4 bricks
13:03 kkeithley striped because you expect to write files bigger than any single brick?
13:03 duerF joined #gluster
13:03 msvbhat captainvmos: As I understand, the issue is, when you write a file on the mount point, ls doesn't show the file. But one of the backend bricks have that file?
13:03 captainvmos not really, but we expect the space requirements to increase so I want to build in the striping at this oint
13:04 captainvmos msvbhat: that's pretty much it
13:04 kkeithley okay, as long as you're not expecting some magic RAID-like performance boost.
13:04 captainvmos i can touch or mkdir, if I cp or mv then the file will appear in the brick, but be empty
13:06 bennyturns joined #gluster
13:06 chirino joined #gluster
13:06 captainvmos yesterday I built on 3.2 and had similar issues, then I totally ballsed it up so I started again with fresh servers
13:07 msvbhat captainvmos: I don't see anything bad/unexpected in the glusterfs mount log.
13:07 tjikkun_work joined #gluster
13:07 kkeithley and if you turn on the stripe coalesce option?
13:08 captainvmos ?how do I do that?
13:11 kkeithley gluster volume set $vol cluster.strip-coalesce on
13:11 kkeithley s/strip/stripe/
13:11 glusterbot What kkeithley meant to say was: and if you turn on the stripee coalesce option?
13:11 social kkeithley: have you heared about some xattr cache issues in 3.4.0? I'm getting W [fuse-bridge.c:875:fuse_fd_cbk] 0-glusterfs-fuse: 2927: OPEN() /a6/kachna/tab_one.csv => -1 (Permission denied) on open from postgres on file which apache wrote; if I add sleep 1 into code before opening from postgres it works fine :/
13:11 kkeithley social: doesn't ring a bell
13:12 social anything particular where I should look now?
13:12 social like finding out what happends during that one second :/
13:13 captainvmos stripe coalesce doesn't seem to make any difference, hang on let me just reboot the client and see
13:14 kkeithley stripe-coalesce affects how the server writes the files on the bricks. You'd have to write new files and I wouldn't think it would affect the clients.
13:14 kkeithley social: not sure. Maybe turn off any write caching.
13:15 kkeithley i.e. disable performance.io-cache, or shrink the size of the cache?
13:16 captainvmos nope, still the same
13:17 kkeithley I can't imagine it would be any different.  Are you able to try using the .debs from the @ppa?
13:17 captainvmos I used this
13:17 captainvmos add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
13:18 kkeithley oh, thought you'd said you built gluster from source
13:19 kkeithley social: or try changing any of the other cache settings, e.g. performance.cache-refresh-timeout.
13:19 captainvmos no, that's how I ballsed it up yesterday, so I decided to go by the book today
13:20 B21956 joined #gluster
13:21 msvbhat captainvmos: You don't see the file in the mountpoint, and there are no errors seen in log file?
13:22 msvbhat captainvmos: Not sure what's going on.
13:22 chirino joined #gluster
13:22 Hoggins hey... I have the feeling that my Gluster problems appeared upgrading from 3.3 to 3.4. I'm sure this is stupid, but I would not want to leave this behind without making sure it *is* stupid. So... has anyone ever tried to downgrade from 3.4 to 3.3 ? How does it go ?
13:23 captainvmos since the reboot (of the client) I'm getting this
13:23 captainvmos add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
13:23 captainvmos I mean
13:23 captainvmos 6 13:15:04.289879] E [stripe-helpers.c:268:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
13:23 captainvmos and this
13:23 captainvmos [2013-08-16 13:13:59.199760] I [client-handshake.c:1423:client_setvolume_cbk] 0-data-client-3: Server and Client lk-version numbers are not same, reopening the fds
13:23 glusterbot captainvmos: This is normal behavior and can safely be ignored.
13:23 social kkeithley: http://paste.fedoraproject.org/32557/37665940 and still nothing
13:23 glusterbot Title: #32557 Fedora Project Pastebin (at paste.fedoraproject.org)
13:23 msvbhat captainvmos: Can you try setting stripe size?
13:24 social Hoggins: fd leak or xattr issues?
13:24 captainvmos I'll try anything once
13:24 msvbhat captainvmos: gluster volume set $vol cluster.stripe-size <value>
13:24 social Hoggins: I know about two things we had to patch 3.4.0 (backports from master)
13:24 captainvmos what size? the filesystem size or number of bricks or what?
13:25 msvbhat captainvmos: s/stripe-size/stripe-block-size
13:25 kkeithley social: file a bug report!
13:25 glusterbot http://goo.gl/UUuCq
13:25 msvbhat captainvmos: stripe size is the size of each chuck of the striped file.
13:26 social kkeithley: sure I will but I want to have some usable data for the dev to fix it
13:26 captainvmos so the size of the original brick mount point?
13:26 kkeithley social: yup
13:27 hagarth joined #gluster
13:27 semiosis captainvmos: are you sure you need ,,(stripe)?
13:27 glusterbot captainvmos: Please see http://goo.gl/5ohqd about stripe volumes.
13:27 social Hoggins: and we tested as an DR downgrade. I suggest stoping everithing and unmounting everything. Don't try to make it without outage :/
13:27 semiosis maybe distributed-replicated would do
13:28 semiosis sorry if i missed that part of the conversation
13:28 Hoggins social: not really sure what my problems are... but I have this *strange* feeling my cluster was way smoother "before".
13:28 Hoggins okay, I will not try anything stupid
13:28 captainvmos I'm not 100% that I do need striped, I'm makign this up as i go along
13:28 captainvmos also,
13:28 captainvmos volume set data cluster.stripe-size 391g
13:28 captainvmos option : cluster.stripe-size does not exist
13:28 captainvmos Did you mean cluster.stripe-block-size?
13:28 msvbhat captainvmos: No. The size of each chunk of the file. If the stripe-block-size is 4k then each chunk of the file would be 4k.
13:29 msvbhat captainvmos: Yeah It's stripe-block-size
13:29 social Hoggins: lsof on bricks does what? do you use acls?
13:29 semiosis captainvmos: i'd suggest trying a distributed-replicated volume and seeing if that would do before using stripe.
13:29 semiosis read that article glusterbot linked
13:30 semiosis if you don't know you need stripe, you probably dont need it
13:30 captainvmos ?
13:30 captainvmos gluster> volume set data cluster.stripe-block-size 4
13:30 captainvmos option block-size 4: '4' is not a valid size-list
13:30 captainvmos Set volume unsuccessful
13:30 captainvmos gluster> volume set data cluster.stripe-block-size 4098
13:30 captainvmos option block-size 4098: '4098' is not a valid size-list
13:31 Excolo joined #gluster
13:31 Excolo anyone know anything about this? /usr/local/lib/glusterfs/3.​3.2/rpc-transport/rdma.so: cannot open shared object
13:31 msvbhat captainvmos: Try 4k.
13:31 Excolo cant find much on google
13:31 msvbhat captainvmos: But I second semiosis here.
13:32 captainvmos nope, it's not having it
13:32 Excolo the volume is tcp, not rdma
13:32 captainvmos sod it, I'll try ditching the striping
13:32 captainvmos thanks for your time, hopefully I won't be back
13:32 kkeithley Excolo: /usr/_local_/lib? Did you build from source?
13:32 rcheleguini joined #gluster
13:33 Excolo yes
13:33 bennyturns Excolo, could be a logging error, is that in a log?
13:33 Excolo yes
13:33 Excolo i built from source, and thats from the logfile
13:35 Hoggins social: do you suggest I should post the whole output of "lsof" on one of my bricks ? I did not set up any ACL.
13:35 social Hoggins: no, I suggest that if you had fdleak you'd notice that output of lsof is strange, AKA holding tons of deleted fd
13:35 Hoggins aw ok
13:35 Hoggins I'll check that
13:39 msvbhat captainvmos: Try 16KB or something?
13:39 Excolo thats wierd, didnt try actually going into the source/rpc-transport/rdma and running make
13:39 Excolo when that compiles it gives an error (but when i compiled source it didnt)
13:43 kkeithley Excolo: did you run autogen.sh and configure? configure would have told you it was or wasn't going to build rdma (Infiniband verbs). You'll need some Infiniband -dev or -devel prerequisites
13:44 Excolo didnt run autogen.sh
13:44 kkeithley yeah, the src tarball probably has configure already
13:45 Excolo this is the summary after the configure
13:45 Excolo GlusterFS configure summary
13:45 Excolo ===========================
13:45 Excolo FUSE client        : yes
13:45 Excolo Infiniband verbs   : no
13:45 Excolo epoll IO multiplex : yes
13:45 Excolo argp-standalone    : no
13:45 Excolo fusermount         : no
13:45 Excolo readline           : yes
13:45 Excolo georeplication     : yes
13:45 Excolo Linux-AIO          : no
13:46 Excolo i should also note that 3.2.5 worked on this server fine
13:46 semiosis please use pastie.org or ,,(paste)
13:46 glusterbot For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
13:46 Excolo sry
13:47 kkeithley And 3.3.2 will probably work fine too, once you actually build the rdma support
13:47 kkeithley Or you could use pre-built rpms or .debs from ,,(repos)
13:47 glusterbot See @yum, @ppa or @git repo
13:49 captainvmos right, that worked, except I've missed a symlink or made a typo somewhere and my entire volume is 7gb, arse. Having another go
13:58 jmalm joined #gluster
13:59 shylesh joined #gluster
14:03 chirino joined #gluster
14:06 Excolo ok, now i have /usr/local/lib/glusterfs/3​.3.2/rpc-transport/rdma.so
14:06 Excolo however, it is still saying it cant connect
14:07 Excolo [glusterfsd-mgmt.c:1778:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Transport endpoint is not connected
14:08 semiosis Excolo: pastie some context lines
14:09 Excolo http://pastie.org/8242271
14:09 glusterbot Title: #8242271 - Pastie (at pastie.org)
14:10 Excolo I have cleared all iptables rules on the server im on, and both bricks
14:10 Excolo and i can ping the server
14:10 semiosis Excolo: can you telnet to port 24007?
14:10 semiosis ,,(ports)
14:10 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
14:11 Excolo the volume is tcp
14:14 Excolo holy fuck i got it mounted
14:14 Excolo (sorry for the language)
14:14 bugs_ joined #gluster
14:14 semiosis rock & roll
14:14 Excolo ive litterally been having a nightmare at work (did 80 hours OVERTIME last week alone) and these issues have just been the cherry on top to a crappy few weeks
14:17 manik joined #gluster
14:17 lpabon joined #gluster
14:25 Excolo thank god, ok now i can mount it again with all firewalls re-enabled
14:26 Excolo (and yes i dropped the connection and re-mounted, i know it would have probably hung on to an established connection)
14:27 Peanut semiosis: did something change recently in the glustefs-package from the PPA? I'm trying to install it, after adding your repository, and it says that the packe is 'not available'.
14:28 semiosis Peanut: not that i know of.  which package exactly?
14:29 deepakcs joined #gluster
14:29 Peanut I'm rebuilding a machine. So it's a fresh install, I just did 'add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.4', apt-get update, then apt-get install glusterfs.
14:29 semiosis Peanut: did you apt-get update after adding the ppa?
14:30 semiosis the package names are glusterfs-server, glusterfs-client, and glusterfs-common (depended on by the others)
14:30 Peanut "Package glusterfs is not available, but is referred to by another package. However, the following packages replace it: glusterfs-common, glustefs-server, glusterfs-client" (and a :i386 version of each is listed, too)
14:30 chirino joined #gluster
14:30 semiosis those have always been the names in that ppa
14:30 Peanut Ah ok, I'll just install glusterfs-server then, see what it pulls in. Perhaps I did run into this the last time, but forgot to document it.
14:32 ababu joined #gluster
14:33 captainvmos Hurray! distributed/replicated works like a charm, thanks very much
14:34 Peanut So I have 1 empty, rebuilt brick now - how do I get the healthy brick to rebuild the replica onto that?
14:35 __Bryan__ joined #gluster
14:36 Peanut (sorry, I'll Google first)
14:37 jfield joined #gluster
14:37 JoeJulian gluster volume heal $vol full
14:38 Peanut Do I need to get the old UUID in glusterd.info first?
14:39 JoeJulian Ah, I didn't scroll back to see what you were doing. When you said empty brick, I thought you were referring to the brick, not the whole server.
14:40 Peanut Yeah, I completely re-installed the server, for science - well, and because /etc/group between the two machines different.
14:40 JoeJulian http://gluster.org/community/documen​tation/index.php/Gluster_3.2:_Brick_​Restoration_-_Replace_Crashed_Server
14:40 glusterbot <http://goo.gl/60uJV> (at gluster.org)
14:41 Peanut Yeah, I'm on that page now, not too much different for 3.4 I hope?
14:41 zerick joined #gluster
14:41 JoeJulian ... hmm... may need a slight update for 3.4...
14:46 neofob joined #gluster
14:47 samsamm joined #gluster
14:48 Peanut I've done everything on that page, and a heal full, but the brick remains empty.
14:48 JoeJulian Probably so...
14:48 JoeJulian "gluster volume status" will probably show that the brick isn't started.
14:49 Peanut No, it shows both bricks, and "Status: Started"
14:49 JoeJulian fpaste it
14:51 Peanut http://loopy.jive.nl/~boven​/gluster_volume_status.txt
14:51 glusterbot <http://goo.gl/Kai8Qb> (at loopy.jive.nl)
14:52 Peanut Hmm.. there's a N for 'Online' at the rebuilt brick.
14:53 JoeJulian Yeah, that's what I was looking for. http://www.joejulian.name/blog/re​placing-a-brick-on-glusterfs-340/
14:53 glusterbot <http://goo.gl/VoSJ6x> (at www.joejulian.name)
14:56 chirino joined #gluster
14:57 B21956 joined #gluster
14:57 B21956 left #gluster
14:58 mbukatov joined #gluster
14:58 Peanut Thanks, JoeJulian. Doesn't seem to be quite the case here (yet): looks like I missed a subdir. The brick mountpoint is /export/brick0, but it complains about '/export/brick0/sdb1' missing.
14:59 Peanut Ah, and _now_ I get the 'extended attribute is absent' error.
15:04 Peanut Should I be able to see that fattr on the other brick? "getfattr /export/brick0{,sdb1}" doesn't show it.
15:05 Peanut Neither does getfattr -d
15:06 JoeJulian @extended attributes
15:06 glusterbot JoeJulian: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
15:09 Peanut Ah, thank you
15:11 Peanut Replication is working again, yay!
15:13 JoeJulian awesome
15:20 zetheroo seems like gluster's self heal makes it impossible to run a VM  ...
15:20 zetheroo :P
15:20 JoeJulian ok
15:20 Peanut zetheroo: why that?
15:20 JoeJulian That would seem incorrect from my perspective. What's yours?
15:21 Peanut Unless you run the same VM twice, on each node?
15:22 zetheroo just powered up both gluster servers and can see glusterfsd using a lot of CPU ... if I try to stay a VM with virsh or virt-manager it hangs
15:22 zetheroo either the command hangs or virt-manager
15:22 whodidthis joined #gluster
15:22 JoeJulian What version?
15:22 zetheroo gluster 3.3.2
15:22 zetheroo there is a lot of activity on the glustered bricks ...
15:23 zetheroo and I have 2 glusterfs processes and one glusterfsd ...
15:24 JoeJulian Hmm, I haven't experienced that problem in 3.3. Any clues in your client log?
15:24 zetheroo I cannot even shutdown the hosts
15:24 LoudNoises joined #gluster
15:25 zetheroo is that the cli.log ?
15:25 chirino joined #gluster
15:26 zetheroo and then libvirt completely crashes ... :P
15:26 zetheroo total mess
15:27 kaptk2 joined #gluster
15:28 zetheroo have to pull the plug on the hosts ... damn
15:31 zetheroo so while self-heal is going on the performance is drastically compromised it would seem
15:33 zetheroo is there any way to know how long the self-heal will take?
15:33 manik joined #gluster
15:37 wirewater joined #gluster
15:40 zetheroo should/can I disable the self-heal process?
15:45 zetheroo shouldn't both bricks have the same amount of used space? http://paste.ubuntu.com/5993192/
15:45 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
15:56 chirino joined #gluster
15:59 krink joined #gluster
15:59 Peanut Ladies and gentlemen - I've just killed a brick, re-built it from scratch, resynched it (with your help), and just live-migrated a guest back to that brick. Quite nice, I can go into the weekend feeling good about this project.
16:02 semiosis awesome
16:02 zetheroo if a brick is rebuilding and during this time the brick goes offline, when it comes back online does it have to start rebuilding from scratch again?
16:03 zetheroo or does it continue where it left off from?
16:03 semiosis zetheroo: it's a per-file thing
16:03 zetheroo I see
16:03 semiosis should pick up where it left off
16:04 jclift_ joined #gluster
16:04 zetheroo I guess for some reason my volume had self-healing offline for a long time ... so there was a huge mismatch between the bricks ...
16:04 zetheroo it's now trying to rebuild about 1.1TB of data
16:04 zetheroo going to be a looooong night for me ...
16:04 jbrooks joined #gluster
16:05 zetheroo is it normal that during the rebuild process the gluster volume is pretty much useless?
16:06 semiosis you can reduce the number of parallel heals.  see background-self-heal-count (or something like that) in ,,(undocumented options)
16:06 glusterbot Undocumented options for 3.4: http://goo.gl/Lkekw
16:07 zetheroo this? key = "cluster.background-self-heal-count"
16:09 zetheroo is the default Max?
16:11 zetheroo if I set this on one of two servers do I need to set it on the second? or is it applied from server1 to server2 automatically?
16:12 semiosis i think default is 16
16:12 zetheroo ok
16:12 semiosis volume options are cluster-wide
16:13 zetheroo I set it to 2 ... I can now access the gluster volume without issue
16:13 semiosis thats what i do
16:13 semiosis i also use the full heal alg, works better for me
16:13 zetheroo does this mean that it will only copy 2 files at time?
16:13 semiosis yes
16:14 zetheroo ok
16:14 Hoggins left #gluster
16:15 zetheroo just tried to start a VM and virt-manager greys out still ... :P
16:16 Peanut zetheroo: I would use the virsh commandline, virt-manager gets confused a bit too easily.
16:17 zetheroo ok will try
16:17 duerF joined #gluster
16:18 zetheroo hmm ... even trying to start the vm with virsh ... the command just sits there ...
16:20 zetheroo I keep getting this issue with libvirtd creating a duplicate process ... which leads to virsh no longer working ... then the process becomes defunct and the host has to be rebooted ...
16:23 zetheroo interestingly enough - if I stop the gluster volume and start it again the duplicate libvirtd process disappears ...
16:23 chirino joined #gluster
16:23 zetheroo and virsh works again
16:34 stevenlokie_ joined #gluster
16:35 zetheroo even with "cluster.background-self-heal-count" set to 1, I cannot start any VM's
16:36 zetheroo if I set it to 0 does this stop the self-heal process altogether?
16:39 Mo___ joined #gluster
16:41 bdeb4 joined #gluster
16:47 chirino joined #gluster
16:55 ababu joined #gluster
16:55 Technicool joined #gluster
16:57 _pol joined #gluster
16:58 _pol joined #gluster
16:59 mbukatov joined #gluster
17:04 bdeb4 @ports
17:04 glusterbot bdeb4: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
17:13 mooperd joined #gluster
17:13 hagarth joined #gluster
17:16 chirino joined #gluster
17:23 chirino joined #gluster
17:28 NuxRo guys, what bonding options do you use between peers? (not clients)
17:32 chirino joined #gluster
17:40 kkeithley semiosis: in the set of Quantal, Raring, and Saucy, which is the most interesting, Raring?
17:40 semiosis sounds like an SAT question
17:40 chirino joined #gluster
17:40 semiosis as in... need more context
17:40 semiosis brb
17:41 bennyturns joined #gluster
17:41 kkeithley I'm fighting a losing battle getting f19 onto my Beagleboard xM. As an experiment I just put Raring on it. As long as I'm here maybe I should build a set of glusterfs .debs for it.
17:42 harish joined #gluster
17:43 JoeJulian zetheroo: I believe that if you attempt to access more than cluster.background-self-heal-count images simultaneously, you'll block on cluster.background-self-heal-count + 1 until it's healed.
17:44 zetheroo ok
17:45 zetheroo I think I am going to be giving Proxmox a try as soon as this fiasco is over ... I can't have something like this happening again ...
17:47 kkeithley JoeJulian: did glusterbot tell you about the discrepancy between dpaste/pastebinit?
17:49 JoeJulian no
17:50 JoeJulian Ah, I see
17:50 kkeithley when someone uses pastebin, glusterbot still tells them to use dpaste
17:50 JoeJulian No, I think @later only works if I ever leave the channel.
17:50 kkeithley yeah
17:50 JoeJulian I should see if I can edit it to do that the next time that nick speaks...
17:51 kkeithley ???
17:52 JoeJulian ie an @later either waits for a join or waits until that nick posts a message into the channel. It's a supybot thing.
17:53 JoeJulian Oh! nevermind. It does to that I just didn't notice the PM. :D
17:55 chirino joined #gluster
17:55 JoeJulian @paste
17:55 glusterbot JoeJulian: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
17:56 kkeithley not that one
17:56 JoeJulian yeah, I was looking up the debian tool...
17:56 kkeithley http://pastebin.com/qFzFZXnN
17:56 glusterbot Title: [2013-08-16 11:57:46.235739] I [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-data-client - Pastebin.com (at pastebin.com)
17:57 kkeithley oh, sorry
17:57 JoeJulian Ugh... pastebinit uses pastebin.com. :(
18:02 semiosis JoeJulian: on what distro?  on my ubuntu it uses the ubuntu paste site
18:08 kkeithley semiosis: I'm fighting a losing battle getting f19 onto my Beagleboard xM. As an experiment I just put Raring on it. As long as I'm here maybe I should build a set of glusterfs .debs for it.
18:09 semiosis kkeithley: probablyr aring
18:09 semiosis s/yr /y r/
18:09 glusterbot What semiosis meant to say was: kkeithley: probably raring
18:09 semiosis saucy will be GA in late october
18:10 chirino joined #gluster
18:10 kkeithley please point me to your recipe for building .debs for ubuntu (again)? Thanks
18:10 JoeJulian semiosis: Oh, cool. I was just doing some cursory searches and saw that referenced.
18:11 semiosis JoeJulian: from the man page: -b [pastebin url] (default is distro-specific with a fallback to pastebin.com)
18:12 semiosis kkeithley: extract the glusterfs source tree; then extract the debian folder into it from https://launchpad.net/~semiosis/+archive​/ubuntu-glusterfs-3.4/+files/glusterfs_3​.4.0final-ubuntu1~raring1.debian.tar.gz
18:12 glusterbot <http://goo.gl/f2U1eq> (at launchpad.net)
18:14 * JoeJulian thinks that should be merged upstream...
18:14 semiosis kkeithley: you'll need to rename the glusterfs tarball glusterfs_3.4.0final.orig.tar.gz or make a symlink with that name pointing at it, this must be in the parent folder to the source tree
18:15 semiosis then from in the root of the source tree run 'debuild -S -sa -uc'; go up a folder, and run 'pbuilder --build glusterfs_3.4.0final.....dsc'
18:15 semiosis roughly speaking
18:15 semiosis JoeJulian: what upstream?
18:15 semiosis JoeJulian: you mean put a debian folder in the glusterfs source tree?
18:15 JoeJulian yeah
18:16 kkeithley agreed. next to extras/LinuxRPM we could have extras/UbuntuDeb(s)
18:19 chirino joined #gluster
18:19 semiosis oh right
18:21 semiosis kkeithley: before running pbuilder --build you'll need to run pbuilder --create with the appropriate --distribution & --arch arguments
18:25 _pol joined #gluster
18:35 chirino joined #gluster
18:40 applio joined #gluster
18:43 applio can anyone point me at docs/other regarding data locality support in the HDFS compatibility layer?
18:45 applio (my google-foo is proving inadequate)
18:46 JoeJulian You've found a gap in my knowledge. I haven't done anything at all with HDFS.
18:47 applio thanks -- know anyone who's more into that sort of thing?
18:48 JoeJulian jclift_: Any ideas there? ^
18:48 chirino joined #gluster
18:48 jclift_ Hmmmm, good question.
18:49 jclift_ Give me a sec, going to scan email archives. :D
18:49 applio http://gluster.org/pipermail/glust​er-users/2012-January/009524.html
18:49 glusterbot <http://goo.gl/VGXrfD> (at gluster.org)
18:49 applio ^^ as close as I've come
18:49 PatNarciso Happy Friday everyone.
18:50 PatNarciso The examples I've seen for a replicated gluster - all the brick mount points include a fresh, formatted ext4 (or xfs) file system.  Is this required?  As long as all the bricks are the same partition type, is there any issue with the brick being mounted within an existing mount?
18:51 JoeJulian They don't even have to be the same filesystem type (I presume that's what you meant by partition type).
18:51 PatNarciso I did.
18:51 JoeJulian You just can't have a brick within a brick.
18:52 PatNarciso Ok -- slick!
18:53 jclift_ applio: I seems like the Gluster + Hadoop guru person best showing up in my email archives is Jay Vyas
18:53 jclift_ s/I/It/
18:53 glusterbot What jclift_ meant to say was: applio: It seems like the Gluster + Hadoop guru person best showing up in my email archives is Jay Vyas
18:54 ntt_ joined #gluster
18:54 JoeJulian yeah, that's the name I couldn't come up with...
18:54 JoeJulian @seen jvyas
18:54 glusterbot JoeJulian: jvyas was last seen in #gluster 30 weeks, 4 days, 2 hours, 30 minutes, and 32 seconds ago: <jvyas> woohoo bigdata hartord gluster RHS meetup this jan/feb :) hope to see some of you guys there.
18:54 jclift_ I'm not sure of his IRC nick though.
18:54 jclift_ Hmmm.
18:54 Technicool jayunit100
18:55 JoeJulian @seen jayunit100
18:55 glusterbot JoeJulian: jayunit100 was last seen in #gluster 27 weeks, 1 day, 15 hours, 50 minutes, and 40 seconds ago: <jayunit100> JoeJulian, nope i dont work in tech support.  :) more r&d type stuff i guess.
18:55 jclift_ Heh
18:55 jclift_ I'll see if I can rouse him and get him onto IRC :D
18:55 applio jclift_: very cool -- thanks!
18:55 JoeJulian I could have sword I'd seen him more recently than that.
18:55 JoeJulian Since Summit even...
18:56 JoeJulian Nope, I'm wrong. :D
18:56 sprachgenerator joined #gluster
18:57 jclift_ JoeJulian: Same here.  I _know_ I've chatted with him about Gluster stuff not long ago on IRC
18:58 jclift_ I've just emailed him, asking him to join if he's got the time.  I have no idea what timezone he's in though, so he may not even get it in a reasonable time period
18:58 jclift_ Technicool: Thx btw. :)
18:58 JoeJulian I'm sure he's on one of the US coasts...
18:58 applio jclift_: thanks again
18:59 JoeJulian Mostly sure it's the left one...
18:59 ntt_ Hi. I'm trying to use glusterfs api example (gfapi.py from git repository). I have a "Request received from non-privileged port. Failing request" error. Why? It uses port 24007
18:59 zetheroo left #gluster
18:59 JoeJulian "received from" ... you're not running gfapi.py as root
19:01 jayunit100 joined #gluster
19:01 jayunit100 hi gluster
19:01 Technicool jclift_, afaik he is on west coast time, and there he is
19:02 jayunit100 someone asking about hadoop on gluster?
19:02 jclift_ applio: ^^^ :D
19:02 applio jayunit100: yes!  thanks for joining!
19:02 jayunit100 applio: :)
19:02 applio I was looking for a pointer to any docs/other regarding data locality in the HDFS compatibility layer
19:03 applio I saw it mentioned as a future need in stuff older than the 3.3-beta
19:03 PatNarciso In a distributed gluster, where it's ideal for a particular brick (small-brick) to have as much free space as possible -- is there a setting/preference that could be set so that files are relocated from small-brick to larger-brick?
19:03 applio but wasn't sure of its current status
19:03 jayunit100 regarding locality, we are actually testing and improving locality
19:04 applio ok
19:04 jayunit100 one of the important parts of it is configuring cluster to be greedy about where it places files when you open a file stream, etc.
19:05 applio that sounds like an on-demand action?  or can this be communicated before a MR job to the scheduler?
19:05 jayunit100 applio: also, lets call it "HCFS" - have you read http://wiki.apache.org/hadoop/HCFS :)
19:05 glusterbot Title: HCFS - Hadoop Wiki (at wiki.apache.org)
19:05 applio no -- looking
19:07 applio actually, I think I have read that before in an earlier edit
19:10 applio jayunit100: I've previously gotten QFS and CassandraFS plugged into hadoop and used them in jobs
19:11 applio but I am not familiar with the specifics of what must be exposed via the HDFS API, if anything, to aid the scheduler in sending MR tasks to appropriate nodes
19:12 applio (for optimal data locality)
19:14 applio when you say that you're testing and improving locality, can you describe a bit more how you communicate locality upstream (towards the scheduler for MR jobs)?
19:14 applio or perhaps that should be phrased in the passive: "how locality info gets pulled in by the scheduler"
19:16 bdeb4 any ideas why i am receiving this error message when trying to mount a volume? mount.nfs: Connection timed out
19:16 bdeb4 i can't find the log file to see which port it's trying to connect on
19:20 applio really, I'd been under the (perhaps mistaken) impression that by merely implementing HDFS compatibility that information about what nodes contain what files/blocks is queried and maintained by the NameNode
19:21 chirino joined #gluster
19:21 applio and that a NameNode-like service would be one of the things you might supply as part of the HDFS compatibility layer for gluster
19:21 jayunit100 applio: hi sorry was getting coffee
19:21 * jayunit100 reads back
19:21 ntt_ There is a web application for glusterfs monitoring and management?
19:21 jayunit100 applio: okay, here is how it works
19:22 mattf applio, there's no namenode involved in the hcfs feature of gluster. each node in the volume knows about location information. so the HDFS API calls (getblocklocations etc) for accessing it simply can ask the underlying gluster volume.
19:22 jayunit100 the Hadoop API defines a FileSystem abstract class.
19:23 jayunit100 This abstract class has no notion of HDFS, and so there is a getBlockLocations, which is part of the FileSystem abstract class' contract.  Gluster stores this meta data in extended attributes, and so - the Gluster Hadoop Plugin can use these xattrs to "guess" where a file block is.  Then the InputSplits which the JobTracker assigns to tasks are assigned such that InputSplit.blockLocation = Task location.  Makes sense?
19:23 jayunit100 applio: ^^
19:23 applio yep -- with you so far
19:24 mattf there's no guessing involved. the xattr points directly to which node has the file.
19:25 jayunit100 okay i thought there were cases where xattr could be stale
19:26 jayunit100 like a pointer to another location or something.
19:26 rcheleguini joined #gluster
19:26 applio the JobTracker talks to what service to get the InputSplits?
19:26 jayunit100 great question
19:27 applio your description makes sense -- though it also suggests that the necessary information is already being made available to the JobTracker
19:27 jayunit100 JobTracker directly can getBlockLocation from an InputSplit
19:28 applio (necessary for ensuring data-local execution)
19:28 jayunit100 InputSplits take Mappers for processing.  InputFormat is the handler which manages the InputSplits.   and in hadoop, each InputSplit has a location.  In a FileInputSplit, the location is implemented by querying the FileSystem API -
19:29 jayunit100 So, here is the path:
19:29 jayunit100 JobTracker gets a "job.xml" file created by a hadoop client.
19:29 applio ok
19:30 jayunit100 JobTracker then loads up the FileInputFormat  - which yields the InputSplits[]
19:30 jayunit100 then , for each InputSplit, the JobTRacker asks "WHERE ARE YOU???"
19:32 jayunit100 For a FileInputFormat, which provides a list of FileSplit instances, each FileSplit has a block location.  That BlockLocation is provided by the FileSystem  (abstract class)
19:32 jayunit100 that is generic to any FileSystem implementation … HDFS , Kopsmos, S3, or GlusterFS
19:32 applio gotchya
19:32 * semiosis waves to jayunit100
19:33 jayunit100 NOW the GlusterFS magic happens here:
19:33 jayunit100 hi semiosis
19:33 semiosis jayunit100: when you have time ping me re: https://github.com/semiosis/libgfapi-jni/issues/1
19:33 glusterbot <http://goo.gl/VLpzjq> (at github.com)
19:33 jayunit100 semiosis: okay will do :)
19:33 semiosis thanks
19:34 jayunit100 When the InputSplits are created from the FileInputFormat  , they will return ass mattf said, the result of getFileBlockLocations, which calls the underlying xattrs
19:35 jayunit100 So , in conclusion, the GlusterFileSYstem class makes system calls to get meta data, and fulfills its FileSystem contract that way.  the Hadoop Framework takes care of the rest :)
19:35 applio cool
19:35 avati semiosis: looks like there is some overlap of effort..
19:35 applio that helps -- thanks
19:36 semiosis avati: there may be.  i heard you've been busy with swig?
19:36 jayunit100 semiosis: okay i responded.  I'll run that build again in fedora.
19:36 avati semiosis: semiosis yes.. let me share my work
19:36 applio I think my view of how HDFS chose to chop of responsibilities was clouding my view a bit
19:37 applio but I think that helps quite a lot
19:37 jayunit100 applio: cool.
19:37 applio but that does come back to my earlier question
19:37 avati semiosis: https://github.com/avati/glus​terfs/tree/java/api/src/java
19:37 glusterbot <http://goo.gl/dxZtQr> (at github.com)
19:37 avati semiosis: i'm doing this w/ a deadline for a customer requirement
19:37 jayunit100 avati: hi :) this looks cool
19:37 applio about what remains to be done / optimized regarding data locality in planning jobs?
19:38 avati semiosis: trying to model the classes as similar to java.io package, so that a user has to make minimal changes to the app
19:38 avati jayunit100: hey!
19:38 jayunit100 i had no idea you had this working !
19:39 RicardoSSP joined #gluster
19:39 RicardoSSP joined #gluster
19:39 jayunit100 avati: we could start testing this :)
19:39 semiosis avati: that makes sense for supporting someone with existing (legacy) codebase.  java.io is ooold though.  java7 has new IO (NIO.2) which supports most of the common filesystem semantics, incl symlinks, perms, owners, etc
19:39 semiosis even renaming!
19:39 jayunit100 I mean hack testing it, nothing formal
19:39 semiosis none of that is is java.io
19:39 semiosis so my focus is on implementing an nio.2 filesystemprovider
19:40 avati semiosis: yes.. i know.. the code too is just casting void * to byte[] now. modelling to nio will be cool, by using directbytebuffer
19:40 semiosis https://github.com/semiosis/glusterfs-​java-filesystem/blob/master/readme.md
19:40 glusterbot <http://goo.gl/lWdA6h> (at github.com)
19:40 semiosis i recently added a roadmap to the readme
19:40 avati semiosis: we should merge the effort
19:41 semiosis never mind the build status... travis-ci is a work in progress.  the tests pass on my machine of course
19:41 semiosis avati: ok
19:41 jayunit100 avati: semiosis we discussed this on our end to
19:42 jayunit100 after all its very relevant to us (the hadoop people).  :)  any thing we can do to lend a hand let us know
19:42 avati semiosis: a logical way to structure our work is, the stuff you've done can focus on java.nio[.2] similar-ness, the stuff in my github will be focusing on java.io similar-ness?
19:42 avati semiosis: what are you using for jni? not swig?
19:43 semiosis avati: hawtjni
19:43 semiosis avati: which was written by chirino
19:43 avati how does it compare to swig?
19:43 jayunit100 seems like stream objects in swig would be very hard to implement
19:43 avati jayunit100: why?
19:43 semiosis it's a modern java framework... annotation driven & totally maven-ized
19:44 applio jayunit100: thanks again -- really appreciate your taking time to explain so much
19:44 avati semiosis: oh, ok.. i loved swig for the customization hooks, and for the fact that it is multi-language
19:44 jayunit100 i thought SWIG was more for data
19:44 jayunit100 and JNI has more flexibility for actual crosstalk between languages
19:45 avati jayunit100: the glusterinputstream and glusteroutputstream classes are using swig generated jni internally
19:45 jayunit100 i have to look to see
19:45 jayunit100 applio: no problem !
19:46 avati jayunit100: see the typemap customization in https://github.com/avati/glusterfs/bl​ob/java/api/src/generate-interface.sh
19:46 glusterbot <http://goo.gl/DRosmm> (at github.com)
19:46 avati jayunit100: swig is very powerful in the way it lets you customize input / output marshalling and type mapping between two languages
19:46 jayunit100 okay cool
19:46 avati of course, i have never used hawtjni. it may be far better
19:47 semiosis avati: see for example https://github.com/semiosis/libgfapi-jni/bl​ob/master/libgfapi-jni/src/main/java/com/pe​ircean/libgfapi_jni/internal/GLFS.java#L152
19:47 glusterbot <http://goo.gl/YGJ4Up> (at github.com)
19:47 semiosis that is the main jni binding class, it resembls glfs.h
19:47 semiosis there's a couple other classes for structs & some helper/utilities
19:48 jayunit100 i was assuming that the hard stuff was the JNI part, which would require java intensive stuff anyways.  So the other interfaces (again just speculating) wouldn't really have much value. avati
19:48 avati semiosis: where is stat and statvfs defined?
19:48 semiosis https://github.com/semiosis/libgfapi-jni/​tree/master/libgfapi-jni/src/main/java/co​m/peircean/libgfapi_jni/internal/structs
19:48 glusterbot <http://goo.gl/ICTLNX> (at github.com)
19:49 semiosis i am using a library called Lombok (http://projectlombok.org/) which generates all the getters/setters/tostring/equals/hashcode methods in the structs
19:49 glusterbot Title: Project Lombok (at projectlombok.org)
19:49 semiosis so the structs really look like structs, no boilerplate java code :)
19:50 avati semiosis: where is the marshalling happening?
19:50 semiosis somewhere behind the scenes, thanks to hawtjni
19:50 avati i'm wondering if the ordering of members must match the ordering in the system headers
19:50 semiosis dont think so
19:50 chirino avati: it does not
19:50 semiosis hi chirino
19:50 chirino hey!
19:51 chirino GLFS.java is looking good!
19:51 semiosis thx :)
19:51 avati how do the lifecycles of the java object and the C structure map?
19:52 avati for e.g, glfs_lstat() expects a structure whose lifecycle is outside its scope, and just fills members
19:52 chirino They get copied in and out
19:52 avati so as a user of glfs.java, how do I use lstat
19:53 avati or, glfs_lstat rather
19:53 chirino we copy the java object to a local struct in the JNI method. call the actual C method, then copy any updates to the local struct back to the original java object
19:53 semiosis avati: https://github.com/semiosis/glusterfs-jav​a-filesystem/blob/master/glusterfs-java-f​ilesystem/src/main/java/com/peircean/glus​terfs/GlusterFileSystemProvider.java#L191
19:53 glusterbot <http://goo.gl/hFw5Y2> (at github.com)
19:53 chirino semiosis: fyi if the stat  arg in glfs_lstat is a pure 'out' variable, you should annotate it as such.
19:54 semiosis chirino: thanks i'll do that
19:54 chirino that flags = NO_IN thingy.
19:54 semiosis i get it :)
19:55 semiosis avati: actually this is the right example... https://github.com/semiosis/glusterfs-jav​a-filesystem/blob/master/glusterfs-java-f​ilesystem/src/main/java/com/peircean/glus​terfs/GlusterFileSystemProvider.java#L250
19:55 glusterbot <http://goo.gl/gyugRY> (at github.com)
19:55 semiosis that's the implementation of the NIO.2 api which relies on calling lstat via jni
19:57 avati semiosis, chirino: ok.. looks like that should "technically work"
19:57 avati but as a java programmer, does such an interface look natural?
19:58 avati i'm no java programmer :-) jayunit100?
19:58 jayunit100 haha
19:58 chirino it's more natural than writing JNI code in c/c== :)
19:58 semiosis i've seen worse!
19:58 chirino ++ even
19:59 avati :-)
19:59 jayunit100 what is the classyour referring to ? this one ? https://github.com/semiosis/glusterfs-jav​a-filesystem/blob/master/glusterfs-java-f​ilesystem/src/main/java/com/peircean/glus​terfs/GlusterFileSystemProvider.java#L250
19:59 glusterbot <http://goo.gl/gyugRY> (at github.com)
19:59 semiosis jayunit100: ?
20:00 jayunit100 "but as a java programmer, does such an interface look natural?"
20:01 avati i see what's happening
20:01 jayunit100 semiosis: ah ok so lombok is a maven task that precedes the build for th ejni bindings, huh?
20:01 semiosis lombok is magic :)
20:01 avati GLFS.java is raelly an internally class used within GlusterFileSystemProvider (and others)?
20:01 avati not the one you are expecting the library user to use?
20:01 semiosis GLFS.java is a straight mapping of the libgfapi c api
20:02 semiosis anyone java program/lib should be able to use it to interact with the libgfapi c library
20:02 chirino avati: correct.  It's acutally quite dangerous to use if you don't use it carefully.
20:03 jayunit100 WOW lombok is awesome !
20:03 semiosis +1
20:03 chirino Note that it's passing around real memory pointers around and stuff like that.
20:03 jayunit100 I'm looking at the Cleanuup Annotation
20:03 jayunit100 @Cleanup
20:03 jayunit100 for streams
20:03 semiosis jayunit100: java7 has try-with-resources to do something similar, i think
20:03 semiosis seen that?
20:04 jayunit100 http://projectlombok.org/features/index.html
20:04 glusterbot Title: Lombok feature overview (at projectlombok.org)
20:04 jayunit100 i haven't used TryWith .  i just switched to clojure which gives you all that declarative goodness for free :)
20:04 semiosis nice
20:05 jayunit100 lombok is essentially a like a macro layer on top of java with all the main benefits that you get from declarative languages.  i really like it.  thanks for sharing LoudNoises
20:05 jayunit100 oops sorry
20:05 jayunit100 didn't meant to ping loud noises
20:05 semiosis yw
20:05 avati chirino: got it!
20:06 avati it's analogous to the swig approach's glfsJNI.java and GlusterFileSystem.java
20:07 avati semiosis: you have made good progress here already. at some point we will need to merge our works.. my proposal is this:
20:07 avati (let me know what you think of the proposal)
20:08 avati I will continue my current work on the swig based java.io-ish bindings for now [there is a customer waiting on it with a fast approaching deadline]
20:09 avati once the classes are finalized in this approach, we can look at rebasing those classes to use your JNI
20:09 avati this way the apps which are written based on the swig based bindings continue to work
20:10 avati the good thing is you seem to be working on getting java.nio-ish interfaces (no?)
20:10 avati and the swig based stuff is java.io-ish.. so there shouldn't be API clashes (or very minimal)
20:11 avati does that make sense?
20:12 semiosis yes
20:12 semiosis sounds good to me
20:13 avati semiosis: what are your plans of release and packaging of your work?
20:13 semiosis as far as api clashes go, my goal is simply to implement the java api.  i'm not interested in making any new classes/interfaces except for internal stuff necessary to facilitate the spec implementation
20:14 semiosis as far as release & packaging... maven central
20:14 avati which spec are you referring to?
20:14 bdeb4 can anyone tell me why i would be getting this error after starting the server:  0-: Unknown key: brick-0
20:14 semiosis the java spec, this stuff: http://docs.oracle.com/javase/tu​torial/essential/io/fileio.html
20:14 glusterbot <http://goo.gl/PdEiX> (at docs.oracle.com)
20:15 avati semiosis: you mean your class names will be exactly the same as that in the spec?
20:16 semiosis avati: most of the time i prefix the class name with Gluster, but classes are in packages, and all of my packages are prefixed with com.peircean so there shouldn't literally be any clashes
20:17 semiosis users should never see my Gluster* classes though
20:17 semiosis once they load my jar into their app they interact with my classes through the java nio.2 api
20:17 avati hmm, how will a user who wants to access some files in gluster and some files in local filesystem write their function?
20:18 avati my question is probably too dumb :-) please bear with me :-) i don't understand how the same nio.2 api will work in thise case?
20:18 semiosis Path glusterFile = Paths.get(new URI("gluster://server:volume/some/file"));
20:19 semiosis that's one way to do it
20:19 semiosis see my example: https://github.com/semiosis/glusterfs-jav​a-filesystem/blob/master/glusterfs-java-f​ilesystem-example/src/main/java/com/peirc​ean/glusterfs/example/Example.java#L52
20:19 glusterbot <http://goo.gl/uPHrg3> (at github.com)
20:19 semiosis that class is a demonstration of a high level api consumer
20:20 avati i see.. so you just implement a provider so that the standard java uri parser can understand the path belongs to you
20:20 semiosis right, then implement all the methods java will call to do work
20:20 semiosis (there's a bunch!)
20:20 avati i get the general idea.. this sounds great
20:21 semiosis indeed
20:21 avati is that "provider" concept available in java.io as well?
20:21 semiosis no
20:21 avati or just an nio feature?
20:21 semiosis strictly speaking, NIO was in java5 iirc, this stuff is NIO.2 which was introduced with java7
20:21 semiosis it is brand new
20:22 avati ah
20:22 avati how widely is java7 adopted? will serious users resist this just because they want to continue using jav5 or java6? or is java7 already widely used?
20:23 semiosis the java7 jre is common enough, but programs using the new java7 features are still pretty rare afaict
20:24 chirino joined #gluster
20:24 semiosis in my research i only found two real world implementations of an nio.2 filesystem provider... the zip fs provider by sun/oracle as an example, and an in-memory filesystem https://github.com/marschall/memoryfilesystem
20:24 glusterbot Title: marschall/memoryfilesystem · GitHub (at github.com)
20:24 semiosis we're on the cutting edge :)
20:24 avati i see.. do you plan to provide new classes for the old jdk users?
20:24 avati i suspect the majority are still on older jdk..
20:24 semiosis haven't looked into it at all
20:25 avati it can be incremental work on top of what's already built
20:25 semiosis there are parts of the nio.2 api which provide compatibility with java.io, it may be possible to use those to connect to legacy code, though the code would probably need some amount of changes
20:26 semiosis Path.toFile() for example
20:26 avati first we can provide the java.io'ish by rebasing my classes to your jni and package them together.. then we can create new classes for java.nio for those whoare still on java6 etc.
20:26 avati ok
20:27 avati but just calling .toFile() will internally rewire everything to call your read / write methods?
20:27 semiosis actually, Path.toFile is only for the default provider... scratch that idea
20:27 avati yeah.. that's what I thought
20:27 semiosis hehe
20:28 avati the nio.2's filesystemprovider certainly sounds like the best technical approach
20:28 avati but we will need to cater to the "older" group too somehow
20:31 avati semiosis: are you working off github on this sstuff?
20:32 semiosis yes, and i mirror to the gluster forge
20:33 avati cool, ok
20:33 semiosis planning to put up a web site in preparation for a wider announcement soon
20:34 avati cool.. do think about how you plan to package the "new classes" which will be java.io-ish and java.nio-ish for the older users.. I think that will be important for serious adoption
20:35 avati I will try to rebase my classes on top of your JNI soon (once I'm done with the deadline)
20:35 semiosis good plan :)
20:36 avati let me know what's the right location for such "new classes" in the repo
20:36 semiosis tbh i am an "older" user too... been running java apps written around java.io File's on glusterfs for 2+ years now
20:37 semiosis so i will draw on my experience with that :)
20:38 semiosis avati: perhaps a new project would be the best place for your classes.  are you familiar with maven?
20:38 avati i have only heard its name :-)
20:38 semiosis ok, let me try to give you the main points as i see them...
20:39 avati semiosis: i was wondering if a single package for all glusterfs access would be better?
20:39 avati it may be a totally busted idea.. but thought it would reduce confusion for a user
20:39 semiosis maven manages dependencies.  if you want to provide a capability to other programs, you package that with maven, then others (also using maven) can import your library.
20:40 semiosis there is a distribution network for these maven packages... the biggest node is maven central (http://search.maven.org/)
20:40 glusterbot Title: The Central Repository Search Engine (at search.maven.org)
20:40 avati is your stuff already on search.maven.org? (if not, do you intend to publish it there?)
20:41 semiosis i will publish it there when it is ready.  anyone can host their own maven repository, it's just a web site with a particular structure.  i have maven configured to publish my packages to an s3 bucket, which anyone can access over http
20:42 semiosis maven projects are controlled by a Project Object Model, pom.xml, like this: https://github.com/semiosis/glusterfs​-java-filesystem/blob/master/pom.xml
20:42 glusterbot <http://goo.gl/rLo8M7> (at github.com)
20:42 avati i see
20:42 semiosis it declares how to build, test, execute, and deploy the code
20:42 semiosis and some other stuff too
20:43 semiosis maven takes a lot of flak, mostly because the pom.xml of a mature project is very big.  but maven does amazing things... turns intractable problems into managable ones
20:44 avati ok
20:44 semiosis the trend is to break projects up into modules which provide specific feature sets... in our case, a jni binding, an nio.2 provider, a java.io-like api
20:45 avati so your jni is really a separate package.. isn't it "dangerous" to make it an independently installable package?
20:46 avati just because it is too low level and someone might be tempted to use it directly?
20:46 semiosis it is what it is :)
20:46 avati or is it a common convention to name something libXYZ-jni as a warning to say "it is too low level"?
20:46 semiosis for example, you're considering using it to build a java.io-like interface
20:47 semiosis yes -jni is a convention
20:47 avati right.. but i'm using it only to make it more sanely usable (and not using it like an end app)
20:47 avati btw, what about the shared library (libgfapi.so)
20:47 avati how is that dependency expressed?
20:47 semiosis things fail
20:47 avati ok
20:48 avati and does it do a loadLibrary() and expect it in a standard path?
20:48 semiosis i set up CI for the glusterfs-java-filesystem project using travis-ci and it blew up.  i had to install glusterfs (coincidentally from a PPA that I happen to manage) in the travis pre-test phase
20:49 avati where is your CI running?
20:49 semiosis yes there is a standard path, and a way to change that path, but i haven't explored that
20:49 semiosis https://travis-ci.org/semiosis/g​lusterfs-java-filesystem/builds
20:49 glusterbot <http://goo.gl/54mr8c> (at travis-ci.org)
20:49 semiosis they provide free CI for open source projects
20:50 semiosis not as nice as jenkins though
20:50 avati oh cool..
20:50 avati ok
20:51 avati so I create a new project / repo with the java.io-ish classes, and declare in the pom.xml that it depends on libgfapi-jni package, that's it?
20:52 avati and when everything's in maven, it should "just work"
20:52 semiosis in theory
20:52 semiosis ping me if you have any maven questions.  it can be kind of confusing at first
20:53 avati glusterfs-java-filesystem <-- is that name following some convention?
20:53 semiosis no
20:53 avati and com.peircean.glusterfs .. is that class hierarchy implying somehow that it is a filesystem provider for nio.2?
20:53 social avati: Hi, I'm looking into anothe strange one with acls. in short I have acl set on path, I do rm -rf path/* , create file; delete ; create new file and try to read it from process which should have right to read by acls but I fail. If I add sleep 1s between creation and read it works fine
20:53 semiosis not at all
20:54 semiosis avati: this file declares that the library is a filesystem provider: https://github.com/semiosis/glusterfs-java​-filesystem/blob/master/glusterfs-java-fil​esystem/src/main/resources/META-INF/servic​es/java.nio.file.spi.FileSystemProvider
20:54 glusterbot <http://goo.gl/DZonft> (at github.com)
20:54 avati i see
20:55 semiosis as for the name... maven projects are identified by "GAV" - group-artifact-version.
20:55 social avati: and the error I get is from fuse:  W [fuse-bridge.c:875:fuse_fd_cbk] 0-glusterfs-fuse: 3230: OPEN() /a6/kachna/tab_one.csv => -1 (Permission denied)
20:55 avati social: have you moutned with -o acl?
20:55 semiosis group is usually an internet domain name, reversed.  i'm using my peircean.com name, so i make the group com.peircean, then artifact is the project, and version is version
20:56 social avati: sure, it wouldn't work without -o acl even with added sleep 1 :)
20:56 bennyturns joined #gluster
20:56 avati social: what version are you on? try disabling stat-prefetch (there is a fix in master to be backported to 3.4 w.r.t ACLs)
20:57 social avati: I have disabled stat-prefetch and it had no effect also we have backported 0001-md-cache-fix-xattr-cac​hing-code-in-getxattr.patch
20:57 avati semiosis: I see.. I'm guessing my classes would be in org.gluster.glusterfs (for the java.io'ish stuff)
20:58 avati semiosis: does that sound OK? there would be a com.peircean.glusterfs and an org.gluster.glusterfs.. is that too confusing?
20:58 social avati: I tried to make sure there are no cache http://paste.fedoraproject.org/32557/37665940/
20:58 glusterbot Title: #32557 Fedora Project Pastebin (at paste.fedoraproject.org)
20:59 avati social: have you tried mounting with -o entry-timeout=0 and attribute-timeout=0?
21:00 social avati: I'll try
21:02 avati social: are you trying create from one client and access it from another?
21:02 social at this test everything is on one node
21:04 avati hmm
21:04 avati you can show the sequnce of commands executed, highighting the one which failed?
21:05 avati verbatim shell commands (with just paths mangled)
21:05 semiosis avati: seems fine to me.  maybe call the artifact java-io-glusterfs?
21:05 semiosis just an idea
21:05 avati semiosis: that means org.gluster.java-io-glusterfs ?
21:06 semiosis pretty much.  also to be clear the maven GAV does not have to match the java class packages
21:06 semiosis so your maven project could be org.gluster.java-io-glusterfs while the java classes in that project are org.gluster.glusterfs.*
21:06 semiosis or whatever
21:06 avati hmm
21:06 social avati: can't as it's complex test for whole subsystem, I didn't get to minimal reproducer but I got quite close with pure while true; do echo 'lala' > omg ; done on one user and while true; do cat omg ; done on another user where directory was owned by root but it had acls for apache write and postgres read
21:07 social avati: some of the cats hit Permission denied
21:07 semiosis hyphen isn't allowed in java package names, but it is allowed -- and very common -- in maven GAVs
21:07 avati semiosis: ok
21:08 avati social: so omg is a file which has acls set on it before the loops started, and the acl is just remaining unmodified through the test?
21:09 avati what's the output of 'ls -l omg' 'getfacl omg' ?
21:09 social avati: cat omg ; unlink omg
21:09 social it should inderit acls of parrent dir
21:09 avati social: ok..
21:11 avati and you say the problem happens even without stat-prefetch loaded eh
21:11 social avati: yes
21:12 avati is the open() failure sporadic?
21:13 avati how frequent isit? once in how many iterations does open() fail?
21:14 social avati: with test it always fails otherwise it's quite sporadic. acls http://paste.fedoraproject.org/32692/13766876
21:14 glusterbot Title: #32692 Fedora Project Pastebin (at paste.fedoraproject.org)
21:16 avati oh, it fails always?
21:16 avati every single iteration?
21:17 social at the test yes. and it does not if I add sleep 1 between creation and read
21:17 avati the creater and reader are separate threads right?
21:17 avati separate users too
21:17 social creator is post to apache webdav which writes to gluster and read is read from postgres
21:18 social so definetly
21:19 avati did setting entry-timeout=0 on the mount change the behavior?
21:20 social I set this into fstab entry-timeout=0,attribute-timeout=0,noauto,acl​,context=system_u:object_r:gdc_staging_rw_t:s0
21:20 avati y3qh
21:20 chirino joined #gluster
21:20 avati yeah
21:22 social hmm now the test passed O.o
21:26 social ok it's something about entry-timeout=0,attribute-timeout=0
21:26 social I'll try it with only attribute-timeout=0 now
21:28 avati social: checking what could be happening
21:28 social attribute-timeout=0 didn't help :/
21:29 social so entry-timeout=0 does the trick
21:30 avati do you have replication enabled ?
21:31 chirino joined #gluster
21:31 social at this case Type: Distribute
21:32 avati oh
21:32 avati social: what happens with this:
21:33 avati sh -c "echo test > newfile; getfacl newfile"
21:33 avati or even "touch newfile; getface newfile"
21:33 avati *getfacl
21:34 social with entry-timeout=0 or without
21:34 avati without
21:37 avati
21:38 social avati: from one user it seems fine http://paste.fedoraproject.org/32706/66890601
21:38 glusterbot Title: #32706 Fedora Project Pastebin (at paste.fedoraproject.org)
21:40 avati hmm
21:40 avati can you try this:
21:40 avati sudo user1 touch file; sudoe user2 getfacl file
21:44 chirino joined #gluster
21:53 whodidthis left #gluster
21:56 chirino joined #gluster
21:56 social avati: I don't get it, it seems to work fine even if I loop it
21:59 piotrektt joined #gluster
21:59 piotrektt joined #gluster
22:06 GypsyMage joined #gluster
22:06 GypsyMage 'lo. Anyone here?
22:07 GypsyMage Quick Q & A... How can I add an existing disk to a gluster dist?
22:09 GypsyMage Uhg. All these idle clients and no Gluster help
22:11 JoeJulian GypsyMage: Patience, grasshopper. We have $dayjobs... To add a new brick (or set of bricks) to a gluster volume, use "gluster volume $vol add-brick"
22:12 GypsyMage I want to know what happens to the data on the brick I'm adding?
22:12 GypsyMage When I add it now, it doesn't show on my gluster mount point. I see the root-level files only.
22:13 JoeJulian That's technically "undefined" behavior. If, however, you create a new volume using a populated disk in the left-hand brick of a replica set, that data will be included in the volume and replicated.
22:13 GypsyMage What about a distributed volume?
22:14 GypsyMage Ie, trying to move from a single softRaid to a unify/dist gluster
22:14 JoeJulian Roughly the same thing. To take advantage of distribution, you'd need to rebalance.
22:14 JoeJulian But you'd need to create your volume with that populated disk as the first disk.
22:14 GypsyMage So instead of creating my new dist volume with the empty softraids and adding my old softraid... create the volume on the old first
22:14 JoeJulian yes
22:14 GypsyMage excellent. I'll try that right now.
22:15 chirino joined #gluster
22:16 social avati: from printfs inserted into acl_permits when I hit fuse-bridge.c:875:fuse_fd_cbk  OPEN()  path  => -1 (Permission denied) error it :
22:16 social posix_acl_get (inode, frame->this, &acl, NULL); goes into if (!acl)
22:16 social so calls acl = posix_acl_ref (frame->this, conf->minimal_acl);
22:17 social at that particular case
22:17 avati hmm
22:17 avati that's what i suspected
22:18 social so I should go down the posix_acl_get road now I guess
22:18 GypsyMage JoeJulian, excellent advice. Worked like a charm. How come that isn't in any documentation anywhere?
22:18 avati posix acl should get created in the file in posix_acl_create() with the helper posix_acl_inherit_file()
22:19 avati so the acl should be set in the inode even before the inode is linked in the table
22:19 JoeJulian GypsyMage: Because it's unsupported behavior that may go away some day.
22:22 jebba joined #gluster
22:22 GypsyMage Thank you sir. Have a good night.
22:22 JoeJulian You're welcome. You have a good night too.  :)
22:25 social avati: oh noes. wrong
22:25 social avati: I can't count in printfs :) it hits the if (acl->count > 3) and says acl_present = 1;
22:27 social avati: after that it goes 2 times into for loop and on second time jumps on POSIX_ACL_USER:,  goes into mask_check and ends up in red :/
22:42 awheele__ joined #gluster
23:03 social avati: if ((ace->perm & perm & want) == want) { in mask_check gets ace->perm:0 perm:5 want:4
23:06 social avati: and with entry-timeout=0 I get 6 5 4
23:24 awheeler joined #gluster
23:43 avati social: that's interesting
23:44 avati i'm a bit confused though.. are you seeing the 0/5/4 vs 6/5/4 problem, or the conf->minimal_acl problem?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary