Camelia, the Perl 6 bug

IRC log for #gluster-dev, 2013-05-14

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:29 yinyin joined #gluster-dev
01:08 jules_ joined #gluster-dev
01:19 bala joined #gluster-dev
01:56 nickw joined #gluster-dev
02:04 nickw joined #gluster-dev
02:42 lalatenduM joined #gluster-dev
03:14 bharata joined #gluster-dev
03:17 aravindavk joined #gluster-dev
03:20 shubhendu joined #gluster-dev
04:23 yinyin joined #gluster-dev
04:35 lalatenduM joined #gluster-dev
04:44 hagarth joined #gluster-dev
04:52 mohankumar__ joined #gluster-dev
05:00 bulde joined #gluster-dev
05:20 raghu joined #gluster-dev
05:22 bharata joined #gluster-dev
05:25 johnmark joined #gluster-dev
05:50 bala joined #gluster-dev
05:55 rastar joined #gluster-dev
06:12 aravindavk joined #gluster-dev
06:25 mohankumar__ joined #gluster-dev
06:32 kshlm joined #gluster-dev
06:32 mohankumar joined #gluster-dev
06:34 yinyin_ joined #gluster-dev
06:45 bharata joined #gluster-dev
06:50 shubhendu joined #gluster-dev
07:12 aravindavk joined #gluster-dev
07:18 nickw joined #gluster-dev
07:47 mohankumar joined #gluster-dev
07:48 shubhendu joined #gluster-dev
08:31 shubhendu joined #gluster-dev
08:35 rastar joined #gluster-dev
08:40 mohankumar__ joined #gluster-dev
09:28 aravindavk joined #gluster-dev
09:31 rgustafs joined #gluster-dev
09:36 kshlm joined #gluster-dev
09:39 deepakcs joined #gluster-dev
09:40 nickw joined #gluster-dev
09:41 rastar joined #gluster-dev
10:01 shubhendu joined #gluster-dev
10:28 hagarth1 joined #gluster-dev
10:31 yinyin_ joined #gluster-dev
10:42 aravindavk joined #gluster-dev
10:44 shubhendu joined #gluster-dev
11:02 kshlm joined #gluster-dev
11:04 kkeithley1 joined #gluster-dev
11:12 vshankar joined #gluster-dev
11:32 yinyin_ joined #gluster-dev
11:34 yinyin- joined #gluster-dev
11:35 nicolasw joined #gluster-dev
11:42 lkoranda_ joined #gluster-dev
12:11 avati joined #gluster-dev
12:29 edward1 joined #gluster-dev
12:47 lpabon joined #gluster-dev
12:48 nickw joined #gluster-dev
12:59 mohankumar__ joined #gluster-dev
13:01 lalatenduM joined #gluster-dev
13:07 bala joined #gluster-dev
13:39 kkeithley| ahem. so...  trying to undo collateral damage from BZ 851092.  On release-3.4 bug-884455.t passes on my dev machine, on master it passes on build.gluster.org.
13:40 kkeithley| but on release-3.4 it's failing on build.gluster.org. Is it failing because test 11 took longer than 15 seconds? Or because it didn't get '"0" rebalance_completed'?
13:41 kkeithley| I've tried extending it to EXPECT_WITHIN 20 and now running EXPECT_WITHIN 30.
13:41 kkeithley| Is there some way to see what it's really failing?
13:47 jbrooks joined #gluster-dev
13:59 wushudoin joined #gluster-dev
14:07 foster avati: just got an interesting result w/ a 30s delayed/batched fsync... got about 50% of the perf. drop back in the smallfile benchmark
14:07 foster trying again with 60s
14:08 sghosh joined #gluster-dev
14:18 shubhendu joined #gluster-dev
14:26 yinyin_ joined #gluster-dev
14:30 bala1 joined #gluster-dev
14:30 sghosh joined #gluster-dev
14:31 kkeithley1 joined #gluster-dev
14:36 badone joined #gluster-dev
14:37 foster_ joined #gluster-dev
14:37 nixpanic_ joined #gluster-dev
14:37 nixpanic_ joined #gluster-dev
14:38 bfoster_ joined #gluster-dev
14:40 rastar joined #gluster-dev
14:57 awheeler_ joined #gluster-dev
15:14 kshlm joined #gluster-dev
15:29 avati joined #gluster-dev
15:34 hagarth joined #gluster-dev
15:37 hagarth joined #gluster-dev
16:01 portante` joined #gluster-dev
16:19 bala joined #gluster-dev
17:22 _ilbot joined #gluster-dev
17:55 aravindavk joined #gluster-dev
17:56 lalatenduM joined #gluster-dev
18:21 aravindavk joined #gluster-dev
18:39 lbalbalba joined #gluster-dev
18:39 lbalbalba ty1
18:40 lbalbalba left #gluster-dev
19:41 jbrooks joined #gluster-dev
19:44 _ilbot joined #gluster-dev
20:42 avati foster, is that with syncfs?
20:46 foster avati: no, just running fsync()s, but my 50% metric is off, it's only slightly better
20:47 foster my afr-fsync baseline tests appear to jump around, making it difficult to measure the delta consistently
20:48 foster I should also mention I'm running on a fairly new upstream kernel, where the performance seems to better overall as well
20:57 lbalbalba joined #gluster-dev
20:58 lbalbalba left #gluster-dev
21:06 avati oh ok
21:07 avati foster, what's the expense of doing fsync() on a file which has no outstanding changes in it (either data or metadata)?
21:07 avati is it a (close to) true NOP?
21:07 foster this is also based on the broken fxattrop behind thing, fwiw. I'm just using it as a research vehicle at this point
21:08 foster I think it should be an empty syscall, I'm actually using some hacked up tracepoints in xfs to measure two things...
21:08 foster 1.) whether writeback occurs due to the fsync
21:08 foster 2.) whether we push on the log
21:09 avati foster, yeah.. good things to measure
21:09 foster but I haven't gone and explicitly measured a request in that state
21:09 avati btw
21:09 foster oh and there is a flush request if barriers are enabled
21:10 avati in the latest patch i submitted (#4746), the default mode is to syncfs() on behalf of a bunch of fsync()s
21:10 avati what i noticed was, even doing syncfs() had a "side-effect" on unrelated mkdir() and create() calls
21:10 foster ok, this went in only recently, right?
21:10 avati i.e, doing one syncfs() on behalf of (hundreds of) fsync()s
21:10 avati no, still in reivew
21:10 avati not merged
21:11 foster oh, ok. good enough, I asked just to make sure I wasn't accidently testing it :)
21:11 avati so it appears, even doing a single syncfs() has spillover effects.. is that expected?
21:11 avati (effects on unrelated mkdir/create i.e)
21:12 foster we're talking about a sync() here, right?
21:12 avati syncfs() [which is sync() on a targetted single fs]
21:13 foster ah, so wouldn't you expect that to be global wrt to the fs?
21:13 foster IOW, what do you consider "unrelated" ?
21:13 avati what i meant was
21:14 avati an ongoing syncfs() should not affect a mkdir() [at least that was my expectation]
21:14 avati mkdir() should take the same amount of time whether or not a syncfs() is happening in parallel
21:15 avati no?
21:15 foster hmm
21:16 foster so syncfs() appears to do an inode writeback followed by an super->sync_fs()
21:17 foster in xfs, sync_fs() looks like it does a log flush
21:17 avati i thought sync() was just an iteration on every mounted fs by calling ->syncfs(), i maybe wrong
21:18 foster that's what it looks like
21:18 avati so when a log flush is underway, new transactions are blocked?
21:18 avati [when a log flush is underway because syncfs() was called, not because log space got filled]
21:18 foster I don't think so, I believe there are multiple log buffers for this reason
21:19 foster one gets flushed, the other is opened for business
21:19 foster so there might be something interesting going on there
21:20 foster do you have a test that exhibits that behavior?
21:21 avati hangon
21:22 avati http://review.gluster.org/#/c/4746/6/x​lators/storage/posix/src/posix-helpers.c
21:22 foster also note that it looks like syncfs() makes a couple passes on inode writeback
21:22 foster nonblocking followed by blocking
21:22 avati yes.. i think that is expected
21:23 avati initiate non-blocking writeback on dirty inodes.. and then wait for each one to finish
21:24 avati [followed by flush if barriers is enabled?]
21:24 foster well, the second iteration can still flush something not caught by the first, no?
21:25 avati it could.. but from gluster's p.o.v it doesn't care
21:25 avati syncfs() could take longer to return, but that's ok
21:26 avati the issue here seems to be that even with the algo shown in that patch, where one syncfs() is performed on behalf of a bunch of fsync() requests [and fsync() never performed], there is spillover effect of mkdir() becoming slow
21:26 avati the test i did was -
21:27 avati time (for j in {1..3}; do for i in {1..10000}; do mkdir -p subdir.$j$i; dd if=/dev/zero of=subdir.$j$i/file.$i bs=4k count=1 >/dev/null 2>&1; done & done ; wait)
21:27 avati create subdir, and file within it
21:27 avati and i was seeing 'interval stats' in gluster volume profile info
21:28 avati you will see that some mkdirs take a _really long_ time
21:28 avati and this latency is measured at io-stats xlator (so rpc congestion is discounted)
21:30 foster and how often are syncfs()s occurring here?
21:31 avati it is issued in the fsyncer thread serially.. there is an artificial enforced usleep(batch_fsync_delay_usecs) before picking up the current outstanding fsync requests
21:31 avati and after the current syncfs() completes, those requests are unwound, then another usleep() to pick/stage the next batch
21:32 avati so, it is batch_fsync_delay_usecs + time_to_perform_current_syncfs + stack_unwind_overhead
21:35 foster ok, any idea if that is xfs specific?
21:35 avati i dont know :(
21:35 avati the two drives had other data and i couldn't format them with other fs
21:36 avati you might be able to experiment this outside gluster too
21:36 foster I guess I'm wondering if it's an fs/vfs locking thing, or a real I/O thing
21:36 foster i.e., a buffer read getting starved
21:37 avati if you wrote a program to create dirs and create files in a loop, printin progress reate every second.. and while it is running run sync command from a separate shell.. we should be able to see the effect on progress rate
21:37 avati or would such a drop be expected??
21:39 foster we could also put your looping syncfs() call into a program and run the command above, no?
21:41 avati yes.. correct [though we would have to make some change to code to make the fsync() requests NO-OPs]
21:41 foster then you might be able to perf record the mkdir
21:43 avati the natural background synchronization (of create, write, mkdir) does not seem to have such severe impact on ongoing mkdir() and create() as much as the effect by syncfs()
21:44 avati some mkdir() would take longer than few seconds
21:50 foster hmm, i wonder if the constant syncing could lead to effectively thrashing the log buffers
21:50 foster e.g., having two (or a few?) just isn't enough
21:50 avati foster, trying to understand terminology.. is a "log flush" the writing of in-memory log to on-disk journal? or performing the already committed log enttries to actual changes to inodes in the disk
21:51 foster by log flush, I mean committing the in memory log to the on disk log
21:51 avati when is the change to actual inodes performed, and does that action have a well-known name?
21:52 foster writeback?
21:52 avati foster, i tried to set the usleep() delay higher (up to 10 secs) and there was still mkdirs() which took long
21:52 avati 10s sleep after previous syncfs() completed
21:53 foster ok
22:22 foster on a quick test, I'm not reproducing a delta on that command above with a 'while [ true ]; do sync; done' in the bg
22:23 foster (directly on an fs, running 3.10.0-rc1)
22:23 foster and I was only creating 5k files, I'll run it longer...
22:29 avati hmm
22:30 avati maybe mkdirs and xattrs need to be involved to make a difference?
22:31 foster mkdirs beyond what that command does? I'm assuming that mkdir -p subdir.* is the command that was delayed..
22:32 avati right
22:33 avati it's the same mkdir
22:33 avati maybe xattrs make a difference?
22:34 foster could be
22:38 avati bbl
22:49 portante|ltp joined #gluster-dev
23:08 kkeithley1 OMG, it worked
23:08 kkeithley1 both worked even, woohoo
23:35 jclift_ joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary