Camelia, the Perl 6 bug

IRC log for #gluster-dev, 2013-05-03

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 yinyin joined #gluster-dev
02:15 portante|afk joined #gluster-dev
02:15 kkeithley joined #gluster-dev
03:17 badone joined #gluster-dev
03:45 itisravi joined #gluster-dev
03:47 bulde joined #gluster-dev
04:26 mohankumar joined #gluster-dev
04:32 bala1 joined #gluster-dev
04:40 bulde joined #gluster-dev
04:56 aravindavk joined #gluster-dev
04:59 sgowda joined #gluster-dev
05:07 bala1 joined #gluster-dev
05:10 hagarth joined #gluster-dev
05:15 bala1 joined #gluster-dev
05:30 raghu joined #gluster-dev
05:32 bulde joined #gluster-dev
05:39 lalatenduM joined #gluster-dev
05:49 vshankar joined #gluster-dev
06:23 rgustafs joined #gluster-dev
06:33 deepakcs joined #gluster-dev
06:36 mohankumar hagarth: bala1: is there any way to get the transport type of a gluster volume?
06:36 mohankumar for example vdsm needs to know the transport type of a volume
06:38 deepakcs hagarth, bala1 in the past i was told to use gluster volume info to deduce the transport type, but for that to work, VDSM host has to be a peer of glsuter volume
06:38 deepakcs hagarth, bala1 that may not be the case always. In fact VDSM gluster storage domain doesn't need vdsm host to be part of volume peer, so using gluster cli is adding constraint
07:22 hagarth deepakcs: you can probably use the powerCLI/python bindings that oVirt exposes for deriving volume info
08:02 deepakcs hagarth, sorry was away.. powerCLI is a diff set of CLIs ?
08:02 deepakcs hagarth, how does it get the info if the vdsm host is not a peer in gluster volume ?
08:07 bala1 deepakcs: to get info thru gluster cli or vdsm-gluster (which uses gluster cli inside), the host should be part of gluster cluster
08:09 bala1 deepakcs: when I saw gluster cli code, I thought it would be cool to have this output of gluster cluster, where an user can fire up gluster cli command anywhere
08:09 bala1 deepakcs: however this work involves lot of security enhancement :)
08:24 mohankumar why gluster volume set <volname> debug.trace on is not supported?
08:25 mohankumar vshankar: sgowda: hagarth: bulde: ^^
08:25 mohankumar i need to enable tracing for debugging my bd xlator patches, but above command does not add debug/trace xlator to the volume file
08:25 ndevos mohankumar: can you not set the <something>.log-level to TRACE with 'gluster volume set VOL ...'?
08:27 ndevos well, that is something different than the trace xlator, I think...
08:27 rastar joined #gluster-dev
08:28 bulde mohankumar: where do you want to load it? debug.trace takes a translator name (on brick side) or 'client' (not 'on/off')
08:28 bulde for example, supported values are 'posix, locks, acl, marker, io-threads, client'
08:30 gbrand_ joined #gluster-dev
08:30 mohankumar ndevos: thanks, it gives lots of traces
08:30 ndevos mohankumar: yeah, I know, thats what I tend to use while debugging...
08:31 ndevos never used the trace xlator though, sounds interesting
08:31 mohankumar ndevos: but it lists traces related to rpc, but i am looking xlator specific traces
08:32 ndevos mohankumar: it just changes the log level, any "gf_log (..  GF_TRACE ..)" and more important messages will be captured
08:32 mohankumar ndevos: thanks!
08:33 mohankumar # gluster volume set bd diagnostics.brick-log-level trace
08:33 mohankumar Segmentation fault (core dumped)
08:33 mohankumar ndevos: ^^ when i pass 'trace' as input, it segfaults
08:33 mohankumar bug
08:34 ndevos ouch!
08:35 ndevos mohankumar: I currently dont have time to look at that, sorry!
08:37 mohankumar ndevos: np
08:37 rgustafs joined #gluster-dev
08:38 deepakcs joined #gluster-dev
08:40 deepakcs bala1, sorry.. my comp shut off.. just saw your msgs
08:41 deepakcs bala1, so powercli is a new set of cli tools  ? is it available or being planned ?
08:55 bala1 deepakcs: i am not aware of powercli
09:01 deepakcs bala1, this is what hagart said.. which i was referring... "hagarth> deepakcs: you can probably use the powerCLI/python bindings that oVirt exposes for deriving volume info"
09:03 deepakcs bala1, Does the ability to run gluster cli cmd from anywhere exists today, or is in the plan ? Its not clear to me
09:05 hagarth deepakcs: you can use the ovirt CLI right?
09:05 deepakcs hagarth, i am in vdsm context
09:05 hagarth mohankumar: which version are you using?
09:05 mohankumar hagarth: is it regarding gluster cli seg fault?
09:06 deepakcs hagarth, oVirt CLI will go via vdsm-gluster, which again has the constraint that vdsm host shud be part of volume peer
09:06 hagarth mohankumar: yeah
09:07 mohankumar hagarth: my recent git commit is fc8aa43d4689b3945681a2ab27427daebac297c7
09:07 hagarth deepakcs: I seem to not understand your requirement. You are on a host outside the gluster cluster and would want to figure out the transport type of a volume?
09:07 deepakcs hagarth, yes
09:09 hagarth deepakcs: so from the host, can't you not use the oVirt CLI? then the request will eventually reach a vdsm host that is part of the gluster cluster.
09:09 deepakcs hagarth, in ovirt when user create gluster storgae domain, user provides volfileserver:volname as the input..
09:10 deepakcs hagarth, nope.. its possible that gluster cluster and vdsm / ovirt cluster are totally separate.. with no common host between them
09:11 ilbot_bck joined #gluster-dev
09:11 deepakcs deepakcs> hagarth, nope.. its possible that gluster cluster and vdsm / ovirt cluster are totally separate.. with no common host between them
09:14 deepakcs hagarth, u r hinting that the glsuter cluster when being managed by ovirt will have vdsm-gluster running, so ovirt cli should help
09:14 deepakcs hagarth, but not sure if i can call ovirt CLI from vdsm host.. i don't think its installed there
09:15 deepakcs bala1, Does the ability to run gluster cli cmd from anywhere exists today, or is in the plan ?
09:15 hagarth deepakcs: yeah. Is there a good reason to not have ovirt CLI installed on a vdsm host?
09:16 deepakcs hagarth, IIUC ovirt CLI goes thru REST API and vdsm install on host won't have it.. since its needde on the engine , not on vdsm
09:16 deepakcs hagarth, are u suggesting we install it on vdsm or put it as a dep ?
09:18 hagarth deepakcs: it would be a very thin install. You would not need the same installables as needed on the engine.
09:19 deepakcs hagarth, what if the glsuter volume being supplied isn't managed by ovirt.. its was manually created and doesn't have vdsm-gluster running on gluster host... ovirt cli won't help in that case
09:22 hagarth deepakcs: that is correct. you can make use of gluster --remote-host=<HOSTNAME> <cmd>
09:23 hagarth for talking to remote glusterd but I am not sure if we will retain this interface for a while.
09:23 hagarth mohankumar: what was the exact command that induced the crash?
09:25 mohankumar [root@host ~]# gluster volume set bd diagnostics.brick-log-level trace
09:25 mohankumar Segmentation fault (core dumped
09:26 mohankumar [root@host ~]# gluster volume set bd diagnostics.brick-log-level off
09:26 mohankumar volume set: failed: option log-level off: 'off' is not valid (possible options are DEBUG, WARNING, ERROR, INFO, CRITICAL, NONE, TRACE.)
09:26 mohankumar hagarth: ^^
09:30 deepakcs hagarth, ok, let me try that... so using --remotehost means glusterd doesn't have to be running on the vdsm host, rite ?
09:30 deepakcs hagarth, if this works.. its best suited for vdsm gluster domain usecase. and given the usecase.. i think you should continue to support this interface !
09:30 hagarth deepakcs: yes, glusterd would not be necessary on the vdsm host.
09:31 hagarth deepakcs: let us see.. I would like a better authentication model between gluster CLI and remote glusterd to continue supporting this interface.
09:31 deepakcs hagarth, ok, i will try that in vdsm context and get back..
09:31 hagarth mohankumar: you can use INFO to set it back to default
09:35 deepakcs hagarth, thats something to work on i guess.. but the usecase for the remote gluster cli exists today :)
09:36 mohankumar hagarth: INFO? i didnt get that
09:36 hagarth deepakcs: i agree, more help on retaining that interface would be useful :)
09:36 hagarth mohankumar: diagnostics.brick-log-level INFO
09:36 hagarth instead of diagnostics.brick-log-level off
09:37 mohankumar hagarth: ah got that, i was trying that to check if gluster segfaults for every bad input
09:37 mohankumar but doesnot look like
09:37 deepakcs hagarth, :)
09:37 hagarth mohankumar: trace and error-gen are critical keywords that trigger this crash :)
09:38 mohankumar :)
09:46 rastar joined #gluster-dev
09:51 deepakcs hagarth, related Q.. during glsuter mount.. (on vdsm host) ... the volfile would be fetched from the remote gluster host, isn't it possible to expose the transport type from the volfile fetched ?
09:51 deepakcs hagarth, via some method ?
09:52 deepakcs hagarth, and how does fetching volfile in this case doens't pose any auth issue ... like you mentioned for gluster cli talkign with remote glusterd ?
10:04 hagarth deepakcs: you can lookup the volfile to parse the transport type.
10:05 deepakcs hagarth, a gluster cli/cmd to do so would have been ideal.. otherwise vdsm will need to know too much details abt the volfile path etc
10:05 hagarth deepakcs: normally, volume set auth-allow/reject take care of auth issue. However for cli-glusterd, all commands can be issued and that is dangerous.
10:06 hagarth deepakcs: you can use gluster ::system <volname> to fetch a volume file.
10:07 deepakcs hagarth, ok, and thsi gives the volfile contents.. which i will have to parse then
10:13 sgowda joined #gluster-dev
10:13 hagarth deepakcs: yeah
10:43 edward joined #gluster-dev
11:05 mohankumar bulde: thanks for pointing about debug.trace posix
11:05 mohankumar looks like i should add support to bd also, but it should be xlator specific
11:53 kkeithley johnmark: you can't set it up in your cube, there's too much and it's too heavy for the floor.
11:53 kkeithley and I was in training yesterday and will be all day today too.
12:43 yinyin joined #gluster-dev
12:52 nickw joined #gluster-dev
12:54 awheeler_ portante|afk: Ping
13:14 bala joined #gluster-dev
13:30 hagarth joined #gluster-dev
13:33 sghosh joined #gluster-dev
13:57 itisravi joined #gluster-dev
13:58 wushudoin joined #gluster-dev
14:07 Supermathie I want to help profile GlusterFS for hotspots... what's the best way? I did a callgrind run and got some results: http://pastie.org/private/5mx681oi3imclsmownsemg but don't know how useful that is.
14:08 portante|ltp joined #gluster-dev
14:25 yinyin joined #gluster-dev
14:51 johnmark kkeithley: wow, seriously?
14:51 * johnmark didn't realize it was that huge
14:54 * johnmark reads scrollback... wow
14:59 Supermathie Is there already a known issue wrt the gluster nfs server GETTING 'stale file handle' when a directory that *used* to be mounted gets recreated?
15:07 yinyin_ joined #gluster-dev
15:16 wushudoin joined #gluster-dev
15:38 portante awheeler_: pong
15:39 awheeler_ portante: Any chance the 3.4 multivolume patch will make it into the beta release?
15:39 portante I hope
15:39 portante what do you think the hold up is?
15:40 awheeler_ you posted: Found why the unit tests are failing for the UFO directory tree in release-3.4. Please don't submit...
15:40 portante yes, so somebody needs to back port the change I posted upstream to 3.4
15:40 awheeler_ So waiting on you for that before moving forward.
15:41 portante after we get the change upstream accepted
15:41 portante I have it out for review, but the team here seems to be a bit distracted ... not sure what is happening
15:41 awheeler_ Right, which is part of moving gluster-swift to it's on repo, yes?
15:41 portante you might want to ping kaleb and luis and found out is the hold up
15:41 portante yes
15:42 portante we really want to march gluster-swift to its own rhythm
15:42 awheeler_ The link in the comment includes . as part of the link, so it's a broken link.
15:42 portante ugh, sorry
15:42 portante it is a simple fix
15:42 awheeler_ no worries, but that could be part of the holdup.  Not sure though -- I figured it out.  :)
15:43 portante so I would be okay if you took that, back ported it to 3.4 and we just get that in
15:43 awheeler_ w/o 4909?
15:44 awheeler_ The only build failure I've seen was unrelated to ufo, on this one: http://review.gluster.org/#/c/4857/
15:45 awheeler_ And that was a failure of this test: /opt/qa/tools/posix-compliance/tests
15:46 awheeler_ So, I haven't seen the UFO issues you mention in 4919
15:46 awheeler_ Or is that the one where UFO doesn't build at all?
15:47 portante take a look at the UFO test runs in Jenkins, they appear to be all noops
15:48 awheeler_ Right, that must be what you are talking about.  I can backport 4919, no problem
15:48 awheeler_ I misread what you meant by 'failing', you meant 'failing to run' not resulting in a failed test.  lol
15:48 portante both, actually, because when they do run, one test fails, which is what that fix is for
15:49 awheeler_ Oh, I haven't seen that bit.
15:49 johnmark avati: ^^^ can you shed some light on the backport?
15:49 johnmark or maybe that's hagarth
15:50 johnmark portante: you can't backport it?
15:51 portante johnmark: I am out the rest of today, and all day Monday, and would rather have the other team members work on getting unit tests working again and fixing things that are broken, so that we have more knowledge transfer here
15:51 portante I am concerned that things are all piling up on one person to do this sort of thing
15:53 johnmark portante: +1
15:53 johnmark understood
15:54 johnmark portante: I'll lpabon
15:55 Supermathie Any post-3.4.0a3 patch suggestions to fix crashes? http://pastie.org/private/uamfqlqwe1lgjvszw5cw
16:02 johnmark portante|afk: bleh... I meant 'I'll ping lpabon'
16:08 yinyin_ joined #gluster-dev
16:20 nickw joined #gluster-dev
16:25 awheeler_ portante|afk: The referenced file doesn't exist in 3.4: http://review.gluster.org/#/c/4919​/1/test/unit/common/test_Glusterfs.py
16:29 awheeler_ or do you mean the backport should include that as well?
16:44 awheeler_ portante|afk: http://review.gluster.org/#/c/4945/ and http://review.gluster.org/#/c/4946/
16:44 awheeler_ portante|afk: Let me know if that's what you wanted.
16:57 awheeler_ portante|afk: That change didn't result in the ufotests being run.
16:57 awheeler_ portante|afk: Do we need to backport the tox tests?
17:10 awheeler_ portante|afk: It appears that tox depends on the multivolume patch -- not mine, but Junaid's: https://github.com/gluster/glusterfs/commit​s/master/ufo/bin/gluster-swift-gen-builders
17:23 _ilbot joined #gluster-dev
17:38 awheeler_ portante|afk: Still not running the UFO tests
17:39 hagarth Supermathie: why is io-cache part of the NFS server?
17:40 hagarth apart from write-behind, no performance xlators are normally recommended for the NFS server.
17:40 Supermathie hagarth: Can't answer that, all I can say is I created it on 3.4.0a3+ and opened the floodgates
17:42 hagarth Supermathie: this is a preventive fix - http://review.gluster.org/#/c/4916/
17:45 Supermathie Oh, wait, my bad, the core I was looking at was from the fuse mount
17:46 hagarth Supermathie: that explains why ioc was part of the xlator stack.
17:46 Supermathie I also have logs from glusterfs/nfs running under valgrind when it crashed
17:48 hagarth Supermathie: does valgrind refer to invalid reads/writes?
17:49 Supermathie oh yeah tons
17:49 Supermathie I also have a core for the nfs crash
17:49 hagarth Supermathie: have you logged a bug for this?
17:50 Supermathie http://pastie.org/private/68se6djlr7silniun7f4kq
17:50 Supermathie can't remember if I did that one...
17:50 Supermathie no apparently not
17:51 Supermathie Look like anything for which there's a fix?
17:51 hagarth I don't seem to remember anything right away. Would be good to track this.
17:51 Supermathie I can replicate it easily by throwing a load at it.
17:52 hagarth what kind of a load is that?
17:56 Supermathie Oracle
17:56 Supermathie OH HAI! https://github.com/Supermathie/glusterfs/blob​/master/xlators/nfs/server/src/mount3.c#L529
17:57 Supermathie mres->resolveloc.inode->gfid is used when 6 lines up, nfs_loc_wipe (&mres->resolveloc) was called
17:58 Supermathie that gf_log call should probably use the local copy of gfid, yes?
17:59 hagarth checking nfs_entry_loc_fill
18:02 hagarth yeah, it does seem logical to use the copied gfid in gf_log() as we are interested in dumping the parent gfid.
18:03 Supermathie yeah, mres->resolveloc.inode is zeroed or invalid at that point - that gf_log is what crashed
18:03 Supermathie I'll test it and submit a patch
18:04 hagarth Supermathie: when you refer to Oracle as the workload, what is being stored in the gluster volume?
18:05 Supermathie everything; database, control files, redo logs
18:06 hagarth interesting, how has the performance been?
18:07 Supermathie hagarth: Pretty good actually. Lots of bugs tickled out of gluster.
18:07 Supermathie But perf has been decent. About to start doing some real testing.
18:08 hagarth Supermathie: cool. Let us know how it goes.
18:08 Supermathie sooo many "0-gv0-replicate-3: no active sinks for performing self-heal on file"
18:09 hagarth is this message being seen when some of the bricks of a replica set are down?
18:11 Supermathie No, not at all. Something odd is happening where the afr xattrs are getting set on the files underneath. Looks the same as https://bugzilla.redhat.com​/show_bug.cgi?id=955753#c1 but it's a different cause.
18:11 glusterbot Bug 955753: high, unspecified, ---, vraman, NEW , NFS SETATTR call with a truncate and chmod 440 fails
18:13 hagarth need to check that out.
18:18 Supermathie GAH now nfs server seems deadlocked
18:20 Supermathie oh
18:21 Supermathie deadlocked while dumping backtrace :)
18:22 Supermathie http://pastie.org/private/j2sc5ahioekt0xyje46gza
18:23 lalatenduM joined #gluster-dev
18:24 Supermathie also glusterfs/nfs is 13GB of RSS
18:24 * Supermathie smells a leak
18:24 hagarth yeah and the backtrace points to a corruption somewhere
18:24 hagarth valgrind should help here
18:27 Supermathie killed and restarted under valgrind
18:28 hagarth ok
18:44 Supermathie heh gluster/nfs got oom-killed
18:46 kkeithley johnmark: The Sunnyvale lab stuff is 20+ 2U machines...
18:58 johnmark kkeithley: oh! holy crap. ok
18:58 * johnmark didn't know that
19:25 Supermathie hagarth_afk: Yeah there is a bad bad leak somewhere...
19:28 Supermathie oh god, this is bad.... same file on two different bricks... (no, not a replica)
19:28 Supermathie wtf
19:30 johnmark Supermathie: are you using the latest backported 3.4? or is this alpha3?
19:30 Supermathie johnmark: a3
19:31 Supermathie of course all those crashes could have really messed up the disk
19:31 Supermathie might blow it all away, restore data and try again
19:34 johnmark Supermathie: if you do that, I would incorporate the latest backports and build
19:34 Supermathie backports of?
19:34 Supermathie rebase against latest release-3.4 tag?
19:35 johnmark Supermathie: yes
19:44 Supermathie heh only one change to pickup (bae32a5affd514e5a78ba3af6cc644cd5cd6814a)
19:45 Supermathie ==31395== LEAK SUMMARY:
19:45 Supermathie ==31395==    definitely lost: 22,068,769,377 bytes in 8,187,288 blocks
19:45 Supermathie ==31395==    indirectly lost: 13,662,771 bytes in 1,085,883 blocks
19:45 Supermathie ==31395==      possibly lost: 27,949,328 bytes in 10,696 blocks
19:46 Supermathie ==31395==    still reachable: 117,519,648 bytes in 18,496 blocks
19:47 Supermathie Maybe I'll go back to 3.3.1...
19:47 Supermathie Way way way too leaky
19:47 johnmark Supermathie: hrm.
19:47 johnmark that doesn't sound right - there should be more than that added to 3.4
19:47 johnmark Supermathie: can you please file a bug report?
19:47 Supermathie I was already based off the release-3.4 tag
19:48 johnmark oh oh ok
19:48 johnmark I thought you were just using the alpha3 build from a couple of weeks ago
19:48 johnmark Supermathie: still, I think this merits a bug report and a post to the gluster-devel list
19:48 johnmark pretty please :)
19:48 Supermathie will do :)
19:48 johnmark thank you!
19:50 Supermathie johnmark: The actual code I'm running is https://github.com/Supermathie/g​lusterfs/tree/release-3.4-oracle
19:57 Supermathie johnmark: It's a little terse - I'm kind of pressed for time ;/
19:57 Supermathie How's release-3.3 latest?
20:01 gbrand_ joined #gluster-dev
20:40 Supermathie answer: pretty stanky
20:40 Supermathie trying my changes rebased onto 3.3.1
20:58 Supermathie One of these patches (possibly the stable write ones) is making the gluster/nfs effectively single-threaded I think... I'm untarring a whole bunch of stuff and while it's processing a big file, the nfs server is unresponsive
21:13 Supermathie Naw, something is just going horribly wrong
21:13 Supermathie 75.352204 192.168.11.1 -> 192.168.11.1 NFS V3 WRITE Call, FH:0x88cc3b7d Offset:1563058176 Len:8192 UNSTABLE
21:13 Supermathie 75.387227 192.168.11.1 -> 192.168.11.1 NFS V3 WRITE Call, FH:0x88cc3b7d Offset:1563074560 Len:8192 UNSTABLE
21:13 Supermathie 75.436722 192.168.11.1 -> 192.168.11.1 NFS V3 WRITE Call, FH:0x88cc3b7d Offset:1563090944 Len:8192 UNSTABLE
21:14 Supermathie 75.454644 192.168.11.1 -> 192.168.11.1 NFS [RPC retransmission of #12145]V3 WRITE Call, FH:0x88cc3b7d Offset:1529634816 Len:8192 UNSTABLE
21:14 Supermathie 75.462015 192.168.11.1 -> 192.168.11.1 NFS [RPC retransmission of #12139]V3 WRITE Call, FH:0x88cc3b7d Offset:1529618432 Len:8192 UNSTABLE
21:14 Supermathie 75.469156 192.168.11.1 -> 192.168.11.1 NFS [RPC retransmission of #12133]V3 WRITE Call, FH:0x88cc3b7d Offset:1529602048 Len:8192 UNSTABLE
21:14 Supermathie 75.472866 192.168.11.1 -> 192.168.11.1 NFS [RPC retransmission of #12128]V3 WRITE Call, FH:0x88cc3b7d Offset:1529585664 Len:8192 UNSTABLE
21:21 Supermathie that's a local nfs mount
21:21 Supermathie going to ensure v3.3.1 is good with what I'm trying to do and bisect
21:59 badone joined #gluster-dev
22:04 portante|ltp joined #gluster-dev
23:22 badone joined #gluster-dev
23:56 badone joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary