Camelia, the Perl 6 bug

IRC log for #gluster-dev, 2013-06-02

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 badone_ joined #gluster-dev
02:01 mjrosenb joined #gluster-dev
02:01 mjrosenb morning all.
02:02 mjrosenb in the dht callback code, is there a way to determine what the file being operated on is
02:02 mjrosenb and by file, I mean filename
04:56 hagarth mjrosenb: which operation are you referring to?
05:01 mjrosenb hagarth: let me start from the beginning
05:01 mjrosenb hagarth: linkto appears to not be working.
05:01 mjrosenb i'm trying to figure out where something goes wrong.
05:02 mjrosenb I added some patches that enable me to run the server on a freebsd box
05:02 mjrosenb and it mostly works, but renaming files just breaks everything.
05:02 mjrosenb thusfar i've verified that the brick is correctly reading the linkto xattr.
05:07 hagarth mjrosenb: ok and after the linkto reaches dht, does it proceed?
05:09 mjrosenb hagarth: I've been trying to puzzle where that data flows.
05:10 hagarth mjrosenb: what does the cached_subvolume for this inode indicate?
05:10 hagarth mjrosenb: you can probably load the trace xlator below and above dht to understand the flow..
05:13 mjrosenb hagarth: ack doesn't seem to recognize cached_subvolume
05:13 mjrosenb hagarth: that second one sounds useful.  guess I'll need to edit the config file for it?
05:13 mjrosenb also, this is with 3.3.0
05:14 hagarth mjrosenb: yes, you will need to edit the client volfile.
05:14 hagarth mjrosenb: any plans of moving on to 3.4.0?
05:15 hagarth 3.4.0 contains improvements for trace xlator and can be enabled from the CLI (though the location of the xlator in the graph is static)..
05:16 hagarth bbiab
05:17 mjrosenb hagarth: I should get this fixed, then try to get the patches into the actual tree.
05:21 mjrosenb https://gist.github.com/5692698 -- I copied the config file that was printed in the logs, and added this to it.
05:26 mjrosenb ahh. debug/trace
05:26 mjrosenb and subvolumes
05:28 mjrosenb https://gist.github.com/5692709 -- this is the logging output.  The name of the file that should have a linkto is 'bad.txt'
05:31 hagarth mjrosenb: yes, getting patches in would help with a moving codebase.
05:33 mjrosenb I meant to do it several months ago
05:33 mjrosenb but I noticed this bug
05:33 mjrosenb and I just haven't taken the time to actually fix it.
05:34 hagarth was bad.txt renamed before loading trace?
05:34 mjrosenb yes.
05:34 hagarth you might also find some collaborators after the code is in the repo - more motivation to get it in :).
05:35 hagarth is this the gfid of bad.txt --> 43623f40-c78d-461e-910f-1bd8dee8986c ?
05:36 mjrosenb how do I determine that?
05:36 mjrosenb oh, extattr, I bet?
05:37 hagarth yeah! trusted.gfid on bad.txt in the bricks would yield you that.
05:38 mjrosenb is that supposed to be stored as raw binary, or text
05:38 mjrosenb oh
05:38 mjrosenb this is awkward
05:39 mjrosenb memoryalpha# getextattr -x user trusted.gfid /local/tmp/bad.txt
05:39 mjrosenb /local/tmp/bad.txt43 62 3f 40 ffffffc7 ffffff8d 46 1e ffffff91 0f 1b ffffffd8 ffffffde ffffffe8 ffffff98 6c
05:39 mjrosenb all negative bytes have been sign extended to words?
05:39 mjrosenb how does that even happen?
05:40 mjrosenb oh, maybe tha is just with the display
05:40 hagarth you can use -e hex to format the o/p of getfattr
05:40 mjrosenb https://gist.github.com/5692734 -- much better.
05:41 mjrosenb -x should have caused it to display in hex
05:41 hagarth mode=100000, nlink=1, uid=0, gid=0, size=0 does seem to indicate to dht that a link file is present on client-0 which corresponds to memoryalpha
05:42 hagarth the actual file should be present on memorybeta. does this reflect the placement on disk?
05:42 mjrosenb yes.
05:45 mjrosenb on memorybeta, we have
05:45 mjrosenb Attribute "gfid" has a 16 byte value for /local/tmp/bad.txt
05:45 hagarth anything in the client log files which indicate this gfid or  bad.txt?
05:45 * mjrosenb wonders why it isn't trusted.gfid
05:46 hagarth was that from attr?
05:46 mjrosenb yes.
05:47 hagarth attr doesn't list namespace usually.
05:47 hagarth so that should be fine.
05:48 hagarth mjrosenb: anything in the client log file when you perform a stat on bad.txt?
05:50 mjrosenb hagarth: https://gist.github.com/5692752
05:51 hagarth no logs at all from dht? Does stat return on the mount point with an error?
05:51 mjrosenb I may not have logging turned on for dht?
05:52 hagarth all logs should be enabled by default.
05:52 mjrosenb both of the shell commands stat /data/tmp/bad.txt and stat /data/ did not return error codes
05:53 mjrosenb unless there is something else that you meant?
05:53 hagarth rather, what is the o/p seen for stat?
05:55 mjrosenb I showed you the log for stat /data/tmp/bad.txt.  you want the same for stat /data?
05:56 hagarth mjrosenb: let us get back a step, when you indicate that rename affects operations - what is the behavior? the file does not get listed in ls or is it something else?
05:57 mjrosenb it gets listed twice
05:57 mjrosenb presumably, once from each node.
05:58 mjrosenb err, once from each brick
05:58 hagarth ah i see .. there is a check in dht to filter duplicates, that is probably not being effective.
05:59 hagarth mjrosenb: check_is_linkfile is the macro which does that. might be worthwhile to see if all conditions there are being met in your case.
06:01 mjrosenb should I just replace that macro with a function?
06:01 mjrosenb it seems like it would be the easiest way to debug this.
06:01 hagarth for gdb'ing in?
06:02 hagarth yes, macros are evil (for debugging at least) !
06:02 mjrosenb the arguments are i, s, x
06:02 mjrosenb I'd assume that i is inode* and x is xlator*
06:03 hagarth s is struct iatt *
06:06 mjrosenb can I just treat a mode_t as an integer for printing purposes?
06:10 hagarth mjrosenb: yeah you can treat it as unsigned int.
06:10 hagarth formatting it in octal would make it more readable.
06:11 mjrosenb can printf do octal?
06:11 mjrosenb %o
06:11 mjrosenb so it can
06:12 hagarth yeah
06:12 mjrosenb mode doesn't match: 755
06:12 mjrosenb mode doesn't match: 0
06:13 mjrosenb now, if only I knew the name of the file it was checking for link-ness
06:14 hagarth hmm in the trace dump, mode seems to be 100
06:14 hagarth which is correct for a linkfile..
06:14 mjrosenb this was called 3 times
06:15 mjrosenb we got 755 twice, and 0 once.
06:16 hagarth for single execution of a ls command - do we see 3 logs from check_is_linkfile?
06:17 mjrosenb this is for stating the offending file.
06:18 Supermathie joined #gluster-dev
06:18 ndevos joined #gluster-dev
06:19 mjrosenb so with i, a, and x, is there a way for me to print out the filename that we're operating on?
06:19 mjrosenb or is that long gone at this point?
06:20 hagarth so we don't observe a sticky bit in the mode at all (in all 3 cases)?
06:21 semiosis joined #gluster-dev
06:21 hagarth__ joined #gluster-dev
06:21 hagarth you can perform an inode_path on the inode.
06:22 mjrosenb ok, I get the inode and buf_t arguents
06:23 mjrosenb what is the const char *name argument?
06:23 mjrosenb oh, looks like it can be null.
06:24 hagarth you can pass NULL to that.
06:26 mjrosenb I can only assume that it allocates a chunk of memory, and writes it into che char ** I provided
06:26 mjrosenb so not freeing it is a memory leak
06:26 mjrosenb which I don't care about right now.
06:26 mjrosenb ... or not
06:26 mjrosenb Segmentation fault (core dumped)
06:27 hagarth oops
06:27 hagarth can you paste the diff somewhere?
06:28 mjrosenb https://gist.github.com/5692818 is the logging function I added in to replace the macro.
06:28 mjrosenb (gdb) p buffer
06:28 mjrosenb $1 = 0x1000000000 <Address 0x1000000000 out of bounds>
06:29 mjrosenb I guess it returned an error that I ignored?
06:29 mjrosenb yeah, it returned -1.
06:29 hagarth yeah
06:30 mjrosenb guess that means there isn't a filename present?
06:32 mjrosenb ok, so with that in place, we see if / is a link twice
06:32 mjrosenb then we check something that we inode_name can't turn into a name, and get 0
06:32 hagarth it would be better to use st_mode_from_ia in your patch
06:33 hagarth instead of doing a direct comparison with DHT_LINKFILE_MODE
06:33 hagarth oops
06:33 hagarth i ignored line 3, so the diff is fine.
06:33 mjrosenb memoryalpha# ls -l /local/tmp/bad.txt
06:33 mjrosenb ----------  2 root  wheel  0 Sep  8  2012 /local/tmp/bad.txt
06:33 mjrosenb that is a problem
06:33 mjrosenb the perms *are* 0.
06:33 hagarth sticky bit is missing
06:34 mjrosenb I seem to remember something about this.
06:34 mjrosenb ok, this makes sense
06:35 mjrosenb the issue is on the freebsd brick, rather than the linux client.
06:37 hagarth you can probably workaround that for now by not checking mode ( just size 0 and the linkto xattr) to filter duplicates.
06:38 hagarth however, the larger issue of why sticky bit was not set on the brick is something that needs to be sorted out.
06:38 mjrosenb where should we be setting that
06:38 mjrosenb iirc, if you try to set it at creation time, it doesn't work, but if you set it later, it is a-ok
06:39 mjrosenb and in a now lost version of the patchset, I just unconditionally set the permissions after any file is created.
06:41 hagarth maybe worth tracing dht_rename_create_links to see where it fails
06:45 mjrosenb does dht_rename_create_links make a single rpc to make the link, or does it do many steps?
06:47 hagarth it might be simpler to trace from dht_linkfile_create
06:48 hagarth it does a mknod and the retval from mknod should indicate what has happened on the brick.
06:49 semiosis joined #gluster-dev
06:49 hagarth__ joined #gluster-dev
06:49 mjrosenb wait, linkfiles are made with mknod, not creat/open?
06:51 hagarth yes, check dht_linkfile_create and the WIND macro there.
06:56 mjrosenb ../../../../../xlators/stora​ge/posix/src/posix.c:808:2: error: attempt to use poisoned "system"
06:56 mjrosenb :(
06:58 hagarth darn
07:00 hagarth so does mknod fail on freebsd?
07:00 mjrosenb no, the node is created, just I think you can't set the sticky bit at creation time.
07:02 hagarth could it be a problem with extattr_set_link or the arguments we are passing to it?
07:03 mjrosenb the major/minor seem to be 0,1 on the node for links?
07:04 mjrosenb #define S_ISVTX  0001000  /* save swapped text even after use */
07:04 mjrosenb that's the sticky bet?
07:04 hagarth yeah .. that's the sticky bit
07:06 hagarth this is how the sticky bit seen on linux (from ls -l): ---------T. 2 root root    0 May 16 22:26 a
07:11 mjrosenb man mknod on linux says
07:11 mjrosenb EINVAL mode requested creation of something other than a regular file, device special file, FIFO or socket.
07:11 mjrosenb man mknod on freebsd says
07:11 mjrosenb [EINVAL]           Creating anything else than a block or character spe‐
07:11 mjrosenb cial file (or a whiteout) is not supported.
07:13 mjrosenb oh wait
07:13 mjrosenb this is already handled
07:13 mjrosenb I should like read more than 3 lines of code at a time.
07:14 hagarth how is it handled? by the #idef __NetBSD__?
07:15 mjrosenb no, by the check, and fallback to creat afterwards.
07:15 hagarth the fallback to creat is only for OSX/Darwin?
07:15 hagarth oops no
07:16 mjrosenb yeah, creat just ignores that bit.
07:16 mjrosenb just tested it with: https://gist.github.com/5692894
07:18 mjrosenb evidently, creat just ignores all permission bits.
07:19 mjrosenb no.
07:19 mjrosenb i forgot to remove the temp file.
07:19 mjrosenb it sets premission bits, just not the sticky bit.
07:21 mjrosenb just augmented my test case tothis: https://gist.github.com/5692908
07:21 mjrosenb the output was: -rwx------  1 root  wheel  0 Jun  2 03:19 /root/tmp.tmp
07:21 mjrosenb -rwx-----T  1 root  wheel  0 Jun  2 03:19 /root/tmp.tmp
07:25 hagarth what happens if you do umask (0) before creating in your test program?
07:26 mjrosenb oh, that works.
07:27 mjrosenb i'm guessing gluster has umask(0)'ed before it starts doing stuff?
07:27 mjrosenb blast.  still creating files without T
07:30 mjrosenb oh wait,
07:30 mjrosenb I haven't been actually updating the binary
07:30 mjrosenb i've been installing to a different directory
07:30 mjrosenb where are the config files stored, so I can point this build at them?
07:33 hagarth config files are in /var/lib/glusterd
07:33 mjrosenb even with --prefix=foobar?
07:34 hagarth yeah .. unless you have altered glusterd.vol, /var/lib/glusterd would be the default.
07:36 mjrosenb ok, I'll look at this more tomorrow
07:36 mjrosenb sleep for now.
07:37 hagarth I will also head out now. keep me posted on how this goes.
07:38 hagarth mjrosenb: good night.
07:38 mjrosenb thanks for all your help.
12:59 badone_ joined #gluster-dev
13:15 edward1 joined #gluster-dev
17:50 hagarth joined #gluster-dev
18:37 mjrosenb hah-hah!
23:35 badone joined #gluster-dev
23:51 lpabon joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary