Perl 6 - the future is here, just unevenly distributed

IRC log for #pdl, 2013-05-30

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:57 Meiermann joined #pdl
05:10 Meiermann joined #pdl
05:57 LimitSupremum joined #pdl
13:59 MeierM joined #pdl
16:19 mgrimes joined #pdl
16:21 mgrimes Anyone have an experience with a "(in cleanup) INVALID TRANS MAGIC NO 19098640 0" error message?
16:21 run4flat yeah, a bit
16:21 run4flat though I can't remember what caused it
16:21 run4flat gimme a sec, lemme take a look at the code
16:22 mgrimes Thanks run4flat. I think I'm getting the right output, but that intermittent error worries me.
16:26 run4flat mgrimes, like I said, I've seen that error before, but I've never traced its cause.
16:27 run4flat I'm looking at pdlapi.c at the moment, and I can't see what would cause it to occur.
16:28 run4flat ah, ok, this might be it
16:28 run4flat Have you recently upgraded PDL?
16:29 mgrimes I have a couple of different version in different local::libs.
16:29 run4flat there are two situations where I could occur
16:30 run4flat First, if you upgraded PDL and we recently changed the magic number, you'd have problems invoking versions compiled against older PDLs
16:30 run4flat I know that Chris recently changed some of the internal data layouts to accommodate the 64 bit support
16:30 run4flat That could be a cause
16:30 run4flat another possibility is that the magic number is not being set or changed at the right time
16:31 run4flat which would almost certainly be an internal error in PDL
16:31 run4flat since nobody is ever supposed to set the magic number by hand
16:32 run4flat Or, it could be that your have some faulty RAM. :-)
16:32 mgrimes Ahh. I think I've seen different results invoking with differnt includes (prove -b) vs -I...
16:32 run4flat oh, but that's (in cleanup)
16:32 run4flat which probably means it's happening after your script is done
16:33 * run4flat digs some more
16:33 mgrimes Could it be picking up a the wrong version of a module? I'll try rm -rf my local::lib and reinstalling.
16:34 mgrimes I'm hoping it isn't some pointer error resulting in a fortran library overwritting the magic number of a pdl.
16:34 run4flat I don't think it's the fortran
16:35 run4flat usually, when you invoke a fortran method, you pass it the data pointer
16:35 run4flat and that is almost always nowhere near the PDL's thread magic number slot
16:35 mgrimes oh good.
16:35 run4flat But, for the life of me, I cannot remember what leads to this problem in my cod
16:35 run4flat code
16:36 run4flat I think this could be triggered by an unexpected exit from the threadloop
16:36 run4flat which can happen when I write a PDL::PP method and I'm not careful about exit conditions.
16:37 run4flat ... maybe...
16:39 mgrimes I think we are exiting cleanly, but I honestly don't know if I need to be doing some cleanup.
16:42 run4flat Do you know which of your code is causing the problem?
16:42 run4flat I'd be happy to take a look if you can share it
16:44 mgrimes That would be great. It in PDL::Opt::QP, which is up on cpan (http://bit.ly/1151kyG) and github (http://bit.ly/16sQEmq).
16:44 * run4flat looks
16:45 mgrimes The t/solve-qp.t test exercises the qp() routine which calls qpgen2 (Fortran).
16:47 run4flat which fortran file contains qpgen2?
16:47 mgrimes src/solve.QP.f
16:47 run4flat cool
16:48 run4flat wow, this is a monster of a function
16:48 mgrimes I think the issue is in my qp sub, b/c qp_orig calls qpgen2 as well (and I don't think I've seen the error there). They just expose a slightly different api.
16:50 run4flat Hmm
16:50 mgrimes I wouldn't want to support the Fortran. It is a library used by a both R and Python tools.
16:50 run4flat I've just looked over the .pd file and the fortran code
16:50 run4flat I'll take a look at the test suite
16:50 run4flat yeah, but it's good to know if it's causing problems that PDL isn't handling cleanly for some reason
16:51 run4flat but if those tools handle it, then it's probably just fine.
16:52 run4flat hmm, consider PDL::Core's approx instead of this: https://github.com/mvgrimes/pdl-opt-qp/blob/master/t/solve-qp.t#L12
16:52 LimitSupremum joined #pdl
16:52 run4flat though, I think that came from one of the PDL test suite files
16:52 run4flat if I had to guess.
16:53 mgrimes It did. Didn't know about approx. I'll check it out.
16:54 mgrimes I just nuked my local::lib and reinstalled. Still getting the error with "perl -Iblib/arch -Iblib/lib t/solve-qp.t", but not with "prove -bv t/solve-qp.t".
16:55 run4flat what about "perl -Mblib t/solve-qp.t" ?
16:55 run4flat that's how I would normally include the blib directory
16:56 run4flat Oh, hey, you're the one that's been hitting the mailing list recently with new work, right?
16:56 run4flat :-)
16:57 run4flat I just made that connection
16:57 mgrimes Yeah. Just started using PDL in the last couple of months. I has been excellent!
16:57 mgrimes s/I/It/
16:58 run4flat Well, welcome!
16:59 run4flat anyway, let me see if I can reproduce the problem on my system
16:59 mgrimes Thanks for the welcome.
16:59 mgrimes Same error with -Mblib (I knew there was an easier way to do the includes)
17:00 * run4flat clones
17:02 * run4flat is installing dependencies
17:03 run4flat wow, M::B 0.4005 is taking forever to install
17:04 mgrimes lots of tests and dependencies
17:05 run4flat Just finished
17:05 run4flat does this really require perl 5.14?
17:06 mgrimes I don't think so....
17:07 run4flat the test files say "use 5.014"
17:07 run4flat commenting that out...
17:07 run4flat haha, memory problems...
17:08 run4flat yeah, I'm getting troubles. :-)
17:09 mgrimes Do you get error with both solve-orig.t and -qp.t?
17:09 run4flat I get troubles
17:09 run4flat t/solve-qp.t segfaults after the first test output
17:10 run4flat t/solve-orig.t complains with INVALID MAGIC NO... after printing "ok 1 - Got expeced solution"
17:10 mgrimes Wow. Not good. I've never gotten the segfault. What os?
17:10 run4flat Ubuntu 10.10
17:10 run4flat 32 bit
17:10 run4flat fwiw
17:11 run4flat Hmm
17:12 run4flat interesting that you supply nulls to qp
17:12 run4flat I never do that sorta thing myself, though I know there's a good use for it
17:13 run4flat wait wait wait
17:13 * run4flat is looking at the signature
17:14 run4flat I don't think you can call qpgen2 that way given your signature
17:14 run4flat oh wait
17:16 run4flat Hmm, what does this do? https://github.com/mvgrimes/pdl-opt-qp/blob/master/lib/PDL/Opt/QP.pd#L196
17:16 run4flat I've never seen pdl()->reshape before
17:16 run4flat :)
17:17 mgrimes Creates a [ 1 x $q ] (2-dim) matrix (err pdl). There is probably a better way.
17:17 run4flat I had no idea that would work
17:18 run4flat I thought you could only reshape something that already existed
17:18 run4flat I mean, something that already had the memory allocated
17:18 run4flat but, reshape probably allocates new memory as necessary
17:18 mgrimes I think pdl() creates it, then ->reshape changes it.
17:18 run4flat yeah
17:18 run4flat I would just use zeroes
17:19 run4flat zeroes(1$q)
17:19 run4flat oop
17:19 run4flat zeroes(1, $q)
17:19 run4flat there's also PDL::new_from_specification
17:19 run4flat if you don't want to zero-out the data
17:19 run4flat but, I feel like this is working too hard
17:19 run4flat all of this could be handled for you by PDL::PP
17:20 mgrimes I think the reshape(1,$m) was just to be consistent with reshape(0,$n).
17:20 run4flat reshape(0, $n) ???
17:20 run4flat where is that?
17:21 run4flat agh, why would you do that?
17:21 mgrimes https://github.com/mvgrimes/pdl-opt-qp/blob/master/lib/PDL/Opt/QP.pd#L281
17:21 mgrimes inexperience? :-)
17:21 run4flat hahaha
17:22 mgrimes Honestly not sure. Seems like they should all use zeros()
17:23 run4flat the only reason to use reshape is if it handles the zero-sized dimension better than zeroes
17:23 run4flat no, zeroes handles that fine
17:23 run4flat that's what I'd use
17:23 run4flat but it shouldn't be necessary
17:23 run4flat PDL::PP should be able to take care of all of this
17:24 run4flat BTW, what's the difference between qp_orig and qp?
17:25 mgrimes qpgen2 wants two kinds of constraints (== and >) combined into a matrix and a vector. qp_orig mirrors that. qp() combined the matrix and vectors for you.
17:26 run4flat ah, ok
17:26 mgrimes its just a slightly simpler api.
17:27 run4flat got it
17:27 run4flat I think that your code can be simplified a bit, using the built-in capabilities of PDL::PP
17:28 run4flat and, I think that could solve the thread magic number issues
17:29 mgrimes I've read PDL::PPs pod a coupld of times, guess I need to read it again.
17:29 run4flat ha, I know that feeling
17:29 run4flat For example, you shouldn't need to allocate the return piddle memory
17:30 run4flat PDL::PP will take care of that for you
17:30 run4flat *if* you say ($ret_pdl_1, $ret_pdl_2, ...) = qpgen2($input_pdl_1, $input_pdl_2, ...)
17:31 run4flat Have you read the PDL::PP chapter out of the book?
17:32 mgrimes A while ago. Lately I've been looking at the pod.
17:33 run4flat alright, time for lunch;
17:33 mgrimes Thanks for your help!
17:33 run4flat If we can get rid of the problem by changing some code...
17:33 run4flat that'd be what I'd do
17:33 run4flat I'll see what I can do
17:33 run4flat :-)
17:33 run4flat o/
18:24 mgrimes run4flat: I've rewritten QP to let PDL::PP take care of allocating more of the storage.
18:25 mgrimes Solves the error message on my dev box (Mac), but on Linux it still fails hard. I pushed the changes to github.
18:33 * run4flat pulls
18:36 mgrimes Actually, stepping through with the debugger it fails at: https://github.com/mvgrimes/pdl-opt-qp/blob/master/t/solve-qp.t#L17
18:36 mgrimes That just setup for the test.
18:37 run4flat hmm
18:38 run4flat so you want $mu = pdl q[0.0427; 0.0015; 0.0285],
18:38 run4flat and $amat to be...
18:39 run4flat pdl q[ 0.0427, 1; 0.0015, 1; 0.0285, 0]
18:39 run4flat right?
18:39 mgrimes yes
18:39 run4flat hmm, the pdl shell seems to indicate that everything is ok here for me
18:39 run4flat line 17 shouldn't give trouble
18:40 run4flat btw, the copy shouldn't be necessary there
18:40 run4flat glue doesn't create a dataflow'ing piddle
18:41 mgrimes I got overzealous on the copys.
18:41 run4flat That test doesn't pass on my machine because "qp: constraints are inconsistent, no solution! at t/solve-qp.t line 51."
18:43 run4flat but I don't get memory troubles anymore
18:45 mgrimes crazy... passes w/o warnings on my mac, but this segfaults on linux
18:45 shadowpaste "mgrimes" at 217.168.150.38 pasted "segfaults on linux" (3 lines) at http://paste.scsys.co.uk/251741
18:45 * run4flat tries
18:46 run4flat that works fine on my machine
18:46 run4flat that's very odd
18:47 mgrimes Must be an issue with module versions or something.
18:47 run4flat perhaps
18:47 run4flat what version of PDL do you have?
18:47 mgrimes 2.006
18:47 * run4flat scratches head
18:48 mgrimes I'll try reinstalling PDL, etc.
18:48 mgrimes Thanks again for your help!
18:50 run4flat :-)
18:50 run4flat Hmm, looks like I didn't cover RedoDims in the book
18:50 run4flat It also looks like there is some cleanup to do in the text
18:50 run4flat like, you can now comment out sections of code using /* */ blocks
18:50 run4flat that used to break
19:29 jberger goodness guys, keep it down! I'm trying to sleep!
19:29 jberger :-)
19:50 run4flat mgrimes, I'm looking at qp at the moment
19:50 run4flat what do you think about changing the interface so that the sub takes key/value pairs instead of accepting null piddles?
19:50 mgrimes I just started looking at your commits. Very nice!
19:50 run4flat thanks! :-)
19:51 mgrimes I generally prefer k/v pairs.
19:51 run4flat mind if I take a crack at it?
19:51 run4flat :-)
19:52 mgrimes Please do!
19:52 run4flat k
19:54 mgrimes I'm not sure I understand the change to threadloop. Would this then handle a 3 dim Dmatix, 2d dvec, etc, returning a 2d solution vec?
19:54 run4flat yes
19:54 run4flat well
19:54 run4flat it would handle a "stack" of 2d Dmatrix inputs
19:54 mgrimes Right. Sweet!
19:54 run4flat and a similarly stacked 1d dvec, etc
19:55 run4flat trying to figure out all of that stuff always makes my head spin
19:55 run4flat so I try to make PDL::PP do as much of it for me as possible
19:55 run4flat :-)
19:57 mgrimes How about ierr? The fortran code checks uses a <> 0 value to indicate that the Dmatrix has been decomposed already. I haven't made use of that, but I think it needs to remain io.
19:57 run4flat oh, ok
19:58 run4flat The question then becomes: how do you supply meaningful input so the user doesn't have to specify everything
19:58 mgrimes I don't like the double duty, but I really don't want to change the Fortran code.
19:59 run4flat which could be handled using k/v pairs in qp
19:59 run4flat right, no need to change the fortran code
19:59 run4flat I would use different piddles for it, I think
19:59 run4flat I would assign the input piddle to a temporary integer
20:00 run4flat which you pass by address to the fortran code
20:00 run4flat and then assign the modified integer to the output ierr
20:00 run4flat that way, the output $ierr has the full output shape
20:00 mgrimes Agreed. Instead of passing Dmat. One would pass R_transpose
20:00 run4flat even if the input piddle is small
20:00 * run4flat isn't sure if he's making sense
20:05 run4flat mgrimes, can the fortran code really handle a matrix with zero memory allocated?
20:05 run4flat i.e. zeroes(0, $n) will have no memory
20:06 mgrimes I would expect it wouldn't handle that.
20:07 run4flat but, in that case, $meq would be zero
20:07 run4flat so maybe it would handle it right?
20:08 mgrimes That sounds right.
20:09 mgrimes The only issue would come if both Aeq and Aneq where 0x$n, but I don't think that would make much sense to try to optimize.
20:12 run4flat I'm not sure what's going on with the fortran code to know
20:12 run4flat Do you know if $A or $a is modified by the fortran code?
20:13 mgrimes I am not certain, but I believe they so. I know D is.
20:14 run4flat oh, hey, the Fortran is nicely documented
20:14 * run4flat reads
20:14 mgrimes Yes, both can change.
20:16 run4flat yeah, but if the fotran is used in other libraries, we probably should leave it untouched
20:16 run4flat so that any upstream changes can be easily incorporated
20:17 mgrimes I meant both A and a may be changed by the fortran code.
20:17 run4flat oh oh oh
20:17 run4flat ok
20:17 run4flat :-)
20:55 run4flat mgrimes, I think I figured out the INVALID MAGIC NO problem
20:56 run4flat or at least, I could trip it
20:56 run4flat it happens for me with this error: Error in qpgen2:Wrong dims
20:56 run4flat which, I am pretty sure, causes the PDL method call to terminate without cleaning up
20:57 mgrimes Ah. Sounds reasonable. I tried to add code to ensure the correct dims before passing off to qpgen2
20:58 run4flat yeah
20:58 run4flat when I disabled that code, I got the errors
20:58 run4flat :)
20:58 run4flat that may be a red hearing
20:58 run4flat but still, I guess it's worth knowing
20:59 mgrimes I was actually getting good ouput from qp(). Passing tests, etc.
20:59 run4flat I have gotten the first test to pass
20:59 run4flat but not the second
21:11 mgrimes Nice progress. I've been toying with some tests of the "stacked" problems.
21:11 mgrimes The dim checking needs to be fixed, but the threading seems to work nicely.
21:11 run4flat cool
21:13 * pdurbin hears a herring
21:13 run4flat cool
21:13 run4flat mgrimes, I have had a lot of issues getting the dims to line up
21:14 run4flat so, I haven't pushed my args processing
21:14 mgrimes I'm happy to look at it working or not.
21:14 run4flat but, I gotta switch to other things, so I'll push it, knowing that it's not working quite right
21:14 run4flat cool
21:15 mgrimes Really appreciate all your help. Happy to return the favor it I can.
21:15 run4flat I'm sure we'll find something to work on
21:16 run4flat I am also interested getting Module::Build to play nicely with PDL
21:17 run4flat so I may take a closer look at your M::B::Pluggable::PDL
21:17 run4flat :-)
21:17 mgrimes It isn't perfect, but its been working for me, and I think the handling of pod is better then MakeMaker dists.
22:16 LimitSupremum joined #pdl

| Channels | #pdl index | Today | | Search | Google Search | Plain-Text | summary