Perl 6 - the future is here, just unevenly distributed

IRC log for #pdl, 2012-12-12

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:15 Molaf_ joined #pdl
01:14 jberger joined #pdl
09:55 sivoais joined #pdl
11:19 sivoais joined #pdl
12:36 Molaf__ joined #pdl
14:04 Mithaldu run4flat: small bug
14:04 Mithaldu \%header should be [\%header]
14:04 Mithaldu that said
14:04 Mithaldu 0.0220520496368408 seconds # perl
14:04 Mithaldu vs
14:04 Mithaldu 0.00248408317565918 seconds # FlexRaw
14:04 Mithaldu that rocks, thank you! :D
14:22 Mithaldu my @sample = $values->slice( "$start:$end" )->list;
14:22 Mithaldu how do i do this better?
14:50 run4flat Mithaldu, that depends on what you mean by better
14:50 run4flat If you just want to unpack those values into Perl scalars, that's about the best way
14:51 Mithaldu well for the immediate task: how do i get a slice of an array piddle as a perl list?
14:51 Mithaldu the docs say list is deprecated
14:51 Mithaldu but are unclear as to alternatives
14:51 Mithaldu for the non-immediate task: how do i dump a slice of an array piddle into a file as raw data?
14:51 run4flat alas, the docs could be reworded
14:52 run4flat list isn't deprecated, it's just "dumb"
14:52 Mithaldu you mean naive
14:52 run4flat For example, one could easily imagine writing a tied array interface
14:52 run4flat which would be cheaper in memory, in the general case
14:52 Mithaldu i only tie things when i intend to be cruel :)
14:52 run4flat right, naive
14:52 run4flat haha
14:53 run4flat but, "list" isn't deprecated, so feel safe to use it
14:53 Mithaldu sweet
14:53 run4flat also consider "unroll"
14:53 run4flat which I believe is in 2.4.11_1
14:53 run4flat (i.e. not yet on CPAN)
14:54 run4flat but that's for creating multidim arrays from multidim piddles
14:54 run4flat which is not your goal here
14:54 run4flat sorry, babbling
14:54 * Mithaldu nods
14:54 Mithaldu it's ok
14:54 run4flat Dumping a slice of an array piddle into a file as raw data: look at PDL::IO::FastRaw's writefraw or PDL::IO::FlexRaw's writeflex
14:54 Mithaldu i also saw get_dataref
14:55 Mithaldu that sounds like could help with dumping
14:55 Mithaldu ha, or that :)
14:55 Mithaldu btw, just so you get an idea what i'm doing, take a look at this: https://dl.dropbox.com/u/10190786​/NB_S1L03_010807_jpod101.mp3.png
14:55 Mithaldu guess what it is :)
14:55 run4flat yeah, actually dumping the contents of get_dataref is pretty much what both of those do
14:56 Mithaldu ah
14:56 run4flat and they create the header file for you
14:56 run4flat are you trying to pull out the "interesting" parts of this time series?
14:56 run4flat :-)
14:56 Mithaldu depends on how you define interesting
14:56 run4flat touche
14:56 Mithaldu as the title says, it's an mp3
14:57 Mithaldu i'm searching for the location of a certain sample within the mp3
14:57 run4flat ah, ok
14:57 Mithaldu and i'm doing that with cross correlation
14:57 run4flat oh, cool
14:57 run4flat cross correlation with another sample?
14:57 Mithaldu yeah
14:57 Mithaldu i have the exact location of the sample in a certain file
14:58 run4flat I see, and you need to get just that part out
14:58 Mithaldu i convert that into raw data with a sample rate of 1200 Hz
14:58 Mithaldu then i load that into a piddle
14:58 Mithaldu and convert search windows of other files into piddles as well and cross-correlate
14:59 Mithaldu the end goal here is to cut off the intro of each mp3
14:59 run4flat you mean, you aim to remove the leading silence of the mp3s?
15:00 Mithaldu not silence
15:00 Mithaldu each of those mp3s has a voice + music intro that's somewhere between 16 and 25 seconds long
15:00 Mithaldu since each mp3 is only 5-10 minutes that gets old fast
15:00 run4flat ... and which you don't care for
15:00 run4flat :-)
15:00 Mithaldu luckily the intro ends with a distinct sound that i can find
15:01 Mithaldu and that graph shows the exact location :D
15:01 run4flat ah, perfect
15:01 run4flat btw, this reminds me about a conversation about signal processing that happened not long ago on Stack Overflow
15:01 run4flat and somebody suggested John Lapeyre's cpan contributions: http://search.cpan.org/~jlapeyre/
15:02 run4flat might be worth looking through to see if there's anything useful
15:02 run4flat though it sounds like you have most everything worked out already
15:03 Mithaldu i kinda don't want to have to study advanced maths courses to help my japanese learning a bit :)
15:04 run4flat Sure, just figured I'd point it out, in case it was helpful
15:05 run4flat and in case anybody else was paying attention
15:05 Mithaldu right
15:05 Mithaldu the issue for me is that i don't even know what any of those modules do
15:06 run4flat yeah, it's pretty close to the edge of my understanding of time series, too
15:07 run4flat you just sounded like you knew all this stuff
15:07 run4flat what with all your talk about cross-correlation and such
15:07 run4flat :-)
15:07 Mithaldu hahaha
15:07 Mithaldu welcome to google
15:07 * run4flat shakes hands with "google"
15:07 Mithaldu i actually don't even understand how the cross correlation algorithm works
15:08 Mithaldu i found a mailing list entry with an example function
15:08 Mithaldu and reduced it to what i need
15:08 Mithaldu all i know is
15:08 Mithaldu i give it an array
15:08 Mithaldu then an offset and a second array
15:08 Mithaldu it aligns those arrays along the offset and calculates a correlation value
15:09 Mithaldu and by marching the offset across the first array i get all the correlation values
15:09 run4flat yep
15:09 run4flat and the highest is what you want
15:10 run4flat Is that cross-correlation code written in pure Perl, or does it use PDL?
15:10 Mithaldu that uses pdl
15:10 Mithaldu perl would be way slow :)
15:11 run4flat ah good. :-)
15:11 run4flat just had to check
15:11 Mithaldu https://gist.github.com/15c7caabb09e3f7a3f78
15:11 * run4flat checks
15:14 * run4flat likes Smart::Comments
15:16 Mithaldu run4flat: i have to disable it at times though, because it messes up the debugger
15:16 run4flat There's probably a way to replace the loop at line 118 with a call to fftconvolve
15:17 run4flat but that's ok
15:17 Mithaldu well if that makes things faster i am highly interested :D
15:17 run4flat it's something to consider if/when you want to learn more about convolutions, cross-correlation, and time series stuff
15:18 run4flat Is this a bottleneck in your process?
15:18 Mithaldu absolutely
15:18 run4flat or do you just enjoy making things faster?
15:18 run4flat ah, ok
15:18 run4flat let's see here...
15:18 run4flat help fftconvolve
15:18 liddle_piddle_bot run4flat: http://pdl.perl.org/?docs=fftcon​volve&title=PDL::fftconvolve
15:18 Mithaldu right now it takes 10 seconds per file or something like that
15:18 run4flat I don't think that's right...
15:20 run4flat here's the page I was looking for: http://pdl.perl.org/?docs=​FFT&title=fftconvolve
15:20 run4flat brb
15:20 * Mithaldu looks
15:22 Mithaldu yeah, way above my level for now
15:32 run4flat ok, I *think* you can achieve the same thing using conv1d: http://pdl.perl.org/?docs=Primitiv​e&title=PDL::Primitive#conv1d
15:32 run4flat replace lines 117-125
15:33 run4flat with something like...
15:34 run4flat $search_space->conv1d($sample)
15:34 run4flat er, sorry...
15:34 run4flat $cross_correlations = $search_space->conv1d($sample)
15:34 run4flat and then save $cross_correlations to a file, or just return the results as a piddle
15:35 run4flat I'm not sure if that'll do all the proper normalization for you, though, so that may give trouble
15:36 run4flat but it's surmountable trouble
15:36 run4flat and it should be faster than the manual loop + calculations that you're doing now
15:37 run4flat anyway, try both the original way and the way using conv1d
15:37 run4flat and see if they match
15:37 run4flat :-)
15:38 * run4flat is gonna close this window and try to work, feel free to ping to get his attention
15:41 Mithaldu err
15:41 Mithaldu i don't understand how i get conv1d to do the same thing sub cross_cor currently does
15:41 Mithaldu the algorithm is kind of involved
15:42 Mithaldu or to be more specific:
15:42 Mithaldu the kernel to be applied depends on both the slice of the search space we're looking at, and the sample itself
15:43 Mithaldu so i cannot just generate one data set once and apply it multiple times
15:47 Mithaldu http://pdl.perl.org/?docs=Primitiv​e&amp;title=PDL::Primitive#conv1d <- the linked convolve is a dead link
16:08 run4flat conv1d does 90% of the stuff that cross_corr does; the only thing it doesn't do is the normalization, but there are ways to apply normalization after the fact
16:08 run4flat and, I'd bet money that you don't need the normalization
16:12 Mithaldu huh
16:12 Mithaldu i guess i can play around and see what happens
16:16 run4flat If you could post your data somewhere, I could try playing with it and see what I get. :-)
16:20 Mithaldu sure
16:22 Mithaldu (generating the data i'm currently getting)
16:33 Mithaldu run4flat: have a zip of a .git dir: https://dl.dropbox.com/u/1​0190786/audiofind.zip.rar
16:33 Mithaldu the git history should explain what's what
16:33 run4flat OK, thanks
16:34 run4flat I may not be able to look at this for a little while
16:34 run4flat but this'll be fun. :-)
16:34 Mithaldu no hurry :)

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary