Perl 6 - the future is here, just unevenly distributed

IRC log for #pdl, 2016-03-31

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
15:51 gtodd when PDL is reading *massive* data sets from files does it more tools to moderate impact on memory resources than say plain perl does?
15:59 gtodd e.g. c.f. discussion of perl read/sysread chunking and the like http://stackoverflow.com/questions/36201884/count-subsequences-in-hundreds-of-gb-of-data focussing on reading a really really long string of nucleotide data.  It is safe to say this would *not* be a job for PDL ?
16:22 vicash read it in C
16:24 vicash *using C and optimizing for file system issues. maybe use a parallel read method such as with MPI over multiple disks mirroring the same data. using a single disk for multiple GBs of data, your bottleneck is mainly disk not just RAM/CPU
16:24 vicash run iotop to see what is causing I/O delays
16:33 gtodd so piddles aren't for "strings" perse
16:34 gtodd since a billion character string as a PDL::Char will be ... in memory
16:36 gtodd I guess a string that big is not *massive*  but a trillion characters or a 500gb file is not something you roll around in RAM without thinking about it first
16:36 gtodd :)
16:38 sivoais gtodd: you can using mmap to keep things to disk. See PDL::IO::FastRaw
16:38 sivoais only works on Unices
16:38 * sivoais wonders what the equivalent is on Windows
16:40 sivoais Ah, it's CreateFileMapping
16:40 gtodd cOOL
16:40 gtodd oops capslock
16:40 gtodd but worth it
16:40 gtodd thanks
16:40 sivoais hAHA :-P
16:40 sivoais no problem
16:42 sivoais nice SO link. I'm looking over it now
16:45 sivoais using mmap was one of the tricks that made Perl the fastest and among the shortest entries to the Widefinder project
16:45 sivoais <https://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder>
16:47 vicash sivoais: are you done with your PhD now ?
16:49 sivoais Master's. I passed my defense in January. Just doing some finishing up to make things release-worthy. You know how bad things get when the deadline is approaching. :-P
16:49 vicash cool!
16:50 vicash congrats on passing the defense
16:50 sivoais Thank you!
16:56 gtodd sivoais: excellent reference ... thanks
17:09 gtodd markov
17:09 gtodd argh
18:42 jberger sivoais__
18:42 jberger err
18:42 jberger sivoais++
18:42 jberger sivoais: are you going to go for the Ph.D.?
18:54 sivoais no, I'm trying to go for an M.D. Want to move closer to working with patients (and if I get an opportunity, translational research)
19:02 jberger what is translational research?
19:04 sivoais It's turning research into health outcomes. Like the development side of R&D, but for medicine
19:05 jberger ah cool
19:05 jberger with a data focus, I assume
19:24 sivoais yeah, that would be very awesome
21:42 Mithaldu did you guys change anything with the github repos?
21:43 Mithaldu i'm getting tons of error mails from the synch scripts
21:45 Mithaldu oh wow, all of sourceforge is 500ing
21:53 sivoais oh, that's why I wasn't able to access a page just a bit ago. I thought that was just one bad link. I didn't check if it was the rest of the site.

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary