Perl 6 - the future is here, just unevenly distributed

IRC log for #pdl, 2014-01-04

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:20 sivoais vicash: <https://gist.github.com/zmughal/8249456> You can put the function in the the PDL namespace and export it from your module to support both ways of calling it
00:22 sivoais also, why isn't Test::PDL part of the PDL dist itself? :-) It would be very useful. <http://p3rl.org/Test::PDL>
00:22 vicash sivoais: Can the dims[0] of the pdl be different from the nelem of the PDL ?
00:24 sivoais vicash: yes, when you have multiple dimensions
00:24 sivoais [   (sequence(2,3)->dims())[0], sequence(2,3)->nelem ]
00:25 sivoais # gives [ 2 , 6 ]
00:25 vicash but i have a PDL with 1 dimension and am getting the dims[0] to be 46 but the actual nelem = 50
00:26 vicash actually my PDLPP code might have a problem. maybe i shud use slice
00:34 jberger_ Test::PDL was being considered for the PDL core but it took too long so the author released it separately
00:34 jberger_ iirc
00:49 jberger_ joined #pdl
00:50 jberger__ joined #pdl
02:16 webart joined #pdl
03:13 vicash what is the best way to take some data shaped like a CSV file where there are say 4-6 columns of maybe 100_000 rows of data points into a single PDL without having to store those rows/columns as arrays
03:18 * vicash finds the PDL::IO module
03:50 jberger_ actually, isn't rcols enough?
03:50 jberger_ with appropriate delimiter?
04:07 vicash jberger_: yes you're right. i had actually solved this problem about 9 months ago and forgot about it
04:11 jberger_ joined #pdl
04:11 jberger joined #pdl
04:11 Bender1 joined #pdl
04:11 osfameron joined #pdl
04:11 sivoais joined #pdl
04:11 ribasushi joined #pdl
04:11 lungching joined #pdl
04:11 Mithaldu joined #pdl
05:28 vicash Has anyone had success installing PGPLOT or getting PDL::Graphics::PGPLOT to work on Windows using Strawberry Perl ?
05:56 vicash http://blogs.perl.org/users/_vikas_ku​mar/2014/01/technical-analysis-in-fin​ance-using-pdl---an-introduction.html
05:57 drrho joined #pdl
12:21 drrho joined #pdl
12:46 drrho joined #pdl
13:26 sivoais vicash++ nice post!
13:26 sivoais I need to blog about PDL. But first I need to finish my wrappers.
13:27 jberger_ vicash++
13:31 sivoais you can also put images on MetaCPAN <http://questhub.io/realm/perl/s​tencil/5216f85b7e4319690e00000a> :-D
13:34 jberger joined #pdl
13:48 sivoais cool, I got the example working on my Linux box. I didn't have PGPLOT nor Finance::QuoteHist, so I installed those.
14:11 drrho joined #pdl
14:28 pdurbin nice post
14:30 chm joined #pdl
14:32 chm vicash : There was some discussion about updating the PGPLOT build process to work  more portably but I don't know where it stands.
14:32 chm I stopped working with PGPLOT once it stopped building out of the box on the latest cygwin.
14:33 chm One challenge for PDL is the lack of a single standard GUI but there are a number of alternatives
14:33 chm for current PDL versions: PDL::Graphics::Gnuplot, PDL::Graphics::Prima.
14:34 chm The latest release of the plplot library now builds on cygwin so I plan to update the existing PDL::Graphics::PLplot
14:35 chm to work again (I hope).  My goal for PGPLOT and PLplot was to implement an OpenGL based output
14:35 chm driver as a way to provide an available-anywhere interface for graphics where you could mix
14:35 chm plotting/charts with real time images and graphics as well.
14:36 chm vicash : I took a look at TA_lib and it appears that there is a lot of glue code in the library.
14:36 chm Much is to implement stuff that PDL already supports more flexibly.  It should be possible to make a PDL::PP
14:37 chm bindings to the library itself but considering that the computational parts of the routines seems to be
14:38 chm relatively small---it might be simpler to implement them directly in PDL::PP and to provide a TA_lib like
14:38 chm interface for non-PDL users to understand.
14:38 chm For example: the standard math routines are already in PDL.  We also have support for many more data types than
14:39 chm only float/double.  PDL also can thread (vectorize) over more than one dimension so you can apply
14:39 chm methods to more than one data set at a time with the same syntax as applying to a 1D dataset.
14:40 chm For more complete and useful discussion and help on things PDL, please join the perldl mailing list.
14:41 chm It is read by all the PDL developers and will usually get a much more correct response than
14:41 chm you might get from a quick chat discussion.  It also has the benefit of being archived so that there
14:41 chm is an institutional memory of sorts that can be used to set PDL development plans.
14:43 chm sivoais : You can install Test::PDL via CPAN or your favorite perl module installation method.
14:43 chm It provides some nice functionality and PDL authors and developers are encouraged to use it.
14:44 chm At the moment for the PDL core distribution, as tests are updated to use Test::PDL, we could easily
14:46 sivoais Yes, that's what I did. imo, having it in the core encourages using it across documentation / tutorials
14:46 chm add the needed support in the build process.
14:47 sivoais But the default Perl way is *use the CPAN* :-)
14:47 chm There does appear to be a chicken-and-egg problem with having T
14:47 chm Test::PDL used in the PDL core since it has PDL itself as a dependency.
14:48 chm The on-going plan for PDL is to slim down the core distribution to just what is needed for the basics,
14:49 chm and to provide a way for non-core distributions to add-in completely.
14:50 chm I would like to see the many, bulky and balky external dependency modules spun off to CPAN stand-alone modules.
14:50 sivoais I believe BioPerl did/is doing the same thing, last I heard
14:50 chm A
14:50 chm At that point, the existing PDL distribution would become a bundle instead with the side benefit
14:51 chm that the fixes and updates for the component modules would no longer require an entire PDL distribution
14:52 chm re-release with all the unrelated components.  I would like to see folks stop re-inventing block array objects
14:52 chm and be able to use PDL3::Core::Kernel or some such and build on that instead.  That could provide
14:53 chm transparent upgrade to full-PDL capabilities without a lot of duplicate efforts....
14:55 sivoais that would be very powerful. I've used PDL for programs that weren't really mathematical because the vectorised operations made the most sense.
14:56 chm The final major effort for the PDL-2.x distribution sequence is to complete the 64bit index support
14:56 chm so that it correctly handles the long long data type.  At that point, PDL-2.x will be more of a stable, maintenance
14:56 vicash chm,sivoais: thanks
14:56 chm branch and PDL3 work will begin in earnest starting with some major clean up and refactoring.
14:57 chm The goal is to be backwards compatable as possible so that disruption and duplicate effort is minimal.
14:57 vicash chm: I actually have a PDL::PP version of movavg that uses TALib as well. it is part of PDL::Finance::TA and is available as PDL::Finance::TA::TALib. however, the moving average part in TA-Lib is definitely much faster as per the Benchmark I wrote in examples/benchmark.pl in the repo.
14:57 chm We don't want to experience the NumPy->Numeric->NumArray->NumPy challenge (not sure I got it right)
14:58 vicash chm: my plan is to write pure PDL/Perl versions of everything in the TA-lib and also provide PDL::PP wrappers for as many as i can
14:59 chm vicash : What I mean by PDL::PP is the native bindings that generate the code that is used by all the PDL computational routines.
15:00 vicash chm: ah, ok. i will look into that. i assume you mean i can natively call the conv1d() function for example
15:00 chm I never did see some direct timings for TA_lib but the conv1d was 25X faster than the original code.
15:00 chm If you look at PDL::PP (the PDL Book as a good intro tutorial by David Mertens) you'll see
15:01 chm pretty much the same type of code generation capability as what is hand-rolled in the TA_lib source.
15:01 vicash chm: yes, i have not provided timings in the post. but i will look at PDL::PP . I use PDL::PP to call TA_Lib functions right now in the TALib version of the module
15:02 chm The conv1d function was writted in PDL::PP, see primitive.pd in PDL/Basic/Primitive in the PDL sources.
15:02 vicash ok
15:02 chm ...but you might want to start with the tutorial since conv1d does a lot of guru tricks---sometimes developers can be almost too clever...
15:03 chm I would expect using just the high level bindings would still be faster but if you would like a more consistent interface
15:03 chm with the native TAlib, using PDL::PP would be simpler since you could probably start from the existing C code for the CODE blocks in the PDL::PP implementation
15:04 chm Oops, talking too long, got to run some errands.  Please continue and I'll touch back later.  Ciao.  o>
15:04 chm o/
15:04 vicash ok
15:19 vicash sivoais: Finance::QuoteHist and PDL::Graphics::PGPLOT are recommended packages to be installed in Build.PL
15:19 vicash i can add PGPLOT to that mix too
15:21 sivoais ah, I didn't install the recommended packages
15:21 vicash yea i figured since it is required only in examples/ i put them in recommended.
16:07 drrho joined #pdl
17:19 vicash suppose I am retrieving  10_000+ rows from a database like Postgresql and I have to use DBIx where I am using fetch_row or similar function to fetch a row at a time. What is the best way to append values to a PDL in a for loop without having to dump to CSV and use rcols() ?
17:20 vicash the scenario in my case is that I do not know how many rows are there and need PDL to take care of expansion of memory without having to use arrays and then create a PDL from them
17:30 rindolf joined #pdl
17:45 sivoais vicash: is it not feasible to call ->count on the resultset?
17:49 sivoais I haven't played with the underlying C API for any DB driver. Do they let you access the results as a contiguous block of memory?
17:51 sivoais Perhaps a better question to ask on the mailing list
18:16 chm vicash : Here are two examples of PDL::PP implementations for TA_ACOS and TA_AD routines: http://paste.scsys.co.uk/289475 and http://paste.scsys.co.uk/289476
18:17 chm I would expect that you could implement almost all of the TAlib routines in the same fashion.
18:25 vicash chm: thanks. i have implemented TA_MA in PDL:PP already but i willl look at your style of implementation and see how it is done
18:25 vicash i actually plan to write both Pure Perl/PDL and TALib implementations so that on Windows where TALib doesn't compile at least the Pure PDL version can be used
18:28 jberger vicash, installing PDL requires a C compiler too though
18:29 jberger unless you install from rob's repo
18:40 chm vicash : I used Inline::Pdlpp to simplify the perl demo part.  If you were to implement all of TAlib you could make a full module and Makefile.PL to go with it
18:59 vicash jberger,chm: Actually I have created the Alien::TALib module to load TALib dependencies on a system but TALib uses ./configure and that doesnt work on Strawberry Perl
19:00 vicash i have not tried cygwin as i dont like using it
19:02 jberger vicash, you can bundle a pre-built binary version with your alien module for use on windows
19:13 chm vicash : TA_lib builds just fine on cygwin.  I'm not sure what the best alternative is for SPP installs.
19:35 jberger_ joined #pdl
19:38 chm joined #pdl
19:48 vicash sivoais: even if i call ->count on the resultset i will still need to populate a PDL one element at a time which can be very slow if i want each column of the resultset to be part of a PDL as a separate dimension
19:49 vicash that's my real question. how do i populate a PDL one element at a time ?
19:49 jberger__ joined #pdl
19:50 jberger___ joined #pdl
19:57 jberger__ joined #pdl
19:57 jberger___ joined #pdl
20:02 sivoais vicash: you could use append? But that would be inefficient. It'd be better to preallocate a piddle and then replace each row
20:02 jberger_ joined #pdl
20:02 sivoais DBI also has fetchall_arrayref()
20:03 vicash yes.. i was just looking to see if something like rcols() existed for data in memory rather than having to use a file .. rcols i have to say is a great function and allows for easy loading
20:04 sivoais you can actually use the $max_row parameter on that and maybe tweak the size of that to get better performance
20:05 vicash i will populate the pdl directly one item at a time for now..
20:08 jberger__ joined #pdl
20:08 jberger___ joined #pdl
20:08 jberger____ joined #pdl
20:08 jberger_____ joined #pdl
20:09 jberger______ joined #pdl
20:09 jberger_ joined #pdl
20:10 jberger sorry for the noise :-/
20:12 jberger_ joined #pdl
20:30 jberger sorry, I had an IRC client go nuts on me

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary