Perl 6 - the future is here, just unevenly distributed

IRC log for #webwork, 2012-05-30

| Channels | #webwork index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
13:31 aubreyja joined #webwork
13:53 aubreyja Hi -
13:58 aubreyja mgage, mgage_: around?
16:46 theseb joined #webwork
16:48 theseb where/how find docs on how to access student data in webwork database outside of webwork?
16:57 aubreyja hi theseb
16:58 aubreyja What do you mean 'outside of webwork'?  Would you like to export the data for analysis, or just be able to manipulate the database outside of the web interface (e.g. from the command line)
16:59 theseb aubreyja: i have my own web app in which i want to display what homeworks students have completed
17:00 theseb aubreyja: so in other words....in a user interface that is different than webwork's
17:00 theseb aubreyja: command line would be great
17:00 theseb aubreyja: if i could export data in a script to analyze it ..that is what i need i guess
17:01 aubreyja ok, well, if you're using something like php, you can just use php to query the database directly
17:01 aubreyja in your webapp
17:01 aubreyja or you can export the data using mysqldump to, say, csv, and then manipulate it however you would like
17:02 theseb aubreyja: i'm using a python web app (django)
17:02 theseb aubreyja: i am aware of python libraries to access MySQL databases directly.....i just need to know details of webwork student tables
17:03 aubreyja ah, ok, I think there is a schema posted on the wiki - I'll have a look - just a sec
17:05 theseb aubreyja: yay! thanks!
17:15 theseb left #webwork
17:15 theseb joined #webwork
17:15 theseb oops
17:15 theseb aubreyja: still there?
17:16 aubreyja sorry - visitor in my office - back
17:16 theseb :)
17:16 aubreyja just a sec
17:21 _ilbot joined #webwork
17:21 Topic for #webwork is now WeBWorK (http://webwork.maa.org) is an open-source online homework system for math and sciences courses. WeBWorK is supported by the MAA and the NSF. | Release notes: http://goo.gl/Ry5HN | Channel logged at http://goo.gl/jELTn
17:23 theseb aubreyja: must switch computers...back in 3min
17:27 theseb_ joined #webwork
17:29 aubreyja ok, phone call - sorry - all of the sudden a lot of interruptions
17:30 aubreyja So, here are the tables: http://webwork.maa.org/wiki/Database_Layouts
17:30 aubreyja I actually put up a complete database schema some time ago but I'm having a hard time finding it….
17:31 theseb_ aubreyja: thanks..i actually stumbled on that page before but didn't understand why you "select" a database layout...i was more looking for the current layout
17:32 aubreyja yeah, that's old info - currently there is only the standard sql_single layout. The idea of selecting the db layout was more of a goal than a reality although for a time you could select between the sql_single and the sql_moodle layouts
17:33 aubreyja I'm going to look for the images I uploaded for the db schema, but I created them with this:
17:33 aubreyja http://www.mysql.com/products/workbench/
17:33 aubreyja let me see if I can find the images...
17:34 dbwalton joined #webwork
17:34 aubreyja Hi Brian
17:34 dbwalton Hi Jason
17:35 aubreyja ok, theseb_ - I found the images - I'll upload them to the wiki - just a sec
17:36 theseb_ k
17:36 aubreyja http://webwork.maa.org/wiki/File:Course_db_diagram.png#.T8ZalJlYslM
17:37 dbwalton New Question:  So I'm looking at spending some time trying to track down a server slowdown issue here at our campus.  I'm hoping for some tips on what I can use to accomplish this, such as artificially stressing the server, or tools to track where the server gets stalled.
17:37 aubreyja http://webwork.maa.org/wiki/File:NPL_db_diagram.png#.T8ZawplYslM
17:38 aubreyja well, if you can test it offline, NYT::DevelProf is really nice and gives nice charts, images, bar graphs, etc of function calls and how long they take, as well as other things like regex's, etc. that might be taking time
17:38 aubreyja You can use ab to send zillions of fake requests
17:39 aubreyja sorry Devel::NYTProf
17:40 theseb_ aubreyja: very nice.....thanks....Math01_set holds an assignment and Math01_user holds a user but where is user *score* kept?
17:40 theseb_ aubreyja: wait...perhaps Math01_set_user?
17:40 dbwalton I'll look into those tools.  Thanks.
17:40 aubreyja http://webwork.maa.org/wiki/SqlSingle#.T8ZbmplYslM
17:41 aubreyja So in Math_01_problem_user is the "status"
17:41 aubreyja that's the percentage correct the user has
17:41 aubreyja status times value  = score
17:41 aubreyja for an individual problem
17:45 aubreyja also, dbwalton: check out the timing.log in webwork2/logs and the check_timing_log.pl script in webwork2/bin.  You could probably modify it to scrape out requests that take a long time and then reconstruct what was going on from that information
17:47 dbwalton So here is a funny thing about the timing.log -- I've noticed that the time reported back in the log file does not necessarily match the real time taken for serving the request.  I was looking at this file during a slow period and actually timed a request physically as taking over 1 minute, but the timing.log entry was only a fraction of a second.
17:47 dbwalton or maybe it was only a few seconds -- but far from the actual time.
17:48 dbwalton I haven't looked at "check_timing_log.pl" -- I'll go find that.
17:48 aubreyja ah, I haven't looked carefully at the script, but I'll bet it is just timing the response part of the request
17:49 aubreyja oops - it's timing_log_check.pl
17:49 dbwalton thx
17:50 aubreyja actually, it's not that helpful - it just reads the log - the real question is how is that log being generated...
17:51 aubreyja …webwork2/lib/WeBWorK/PG/Local.pm...
17:52 aubreyja actually it's all commented out there…:)
17:53 dbwalton One of the things I've been trying to figure out is to identify what questions/requests have been made at the time the slowdown occurs.  I found the apache2 log files, but haven't spent a lot of time trying to interpret them.
17:54 aubreyja is the slow down recent? was it after an upgrade (i.e. did anything change?)
17:56 dbwalton We had an issue about a year ago when the server updated some PERL modules and a bunch of processing stuff broke.  But the slowdowns continued occasionally , without any clear updates or changes.  Sometimes, instructors have complained because they were in the Library Browser and it slowed down; they have wondered if the Library Browser itself or the homework sets editing process might contribute.
17:56 aubreyja ah, WeBWorK.pm - it looks like the timing log just times the time from figuring out what display module to use until it returns from rendering the display module
17:57 aubreyja we had an issue last year with the library browser too
17:58 aubreyja it only seemed to happen to someone teaching college algebra, but he would go to display a list of algebra problems and it would hang for a while
17:58 aubreyja sometimes it would clear up, sometimes it would remain hung, and sometimes he would get an error message
17:59 aubreyja That was on a 32 bit freebsd server with 4GB of ram. We upgraded to a 64 bit freebsd server with 16GB of ram and we haven't had any problems even after doubling the number of students on the server
17:59 dbwalton How many students does that serve?
18:00 aubreyja so, the easy solution might be to get more ram if you can.
18:00 aubreyja We went from about 1200 to about 2300
18:00 dbwalton I need to do a survey to figure out how many students we are working with, but I really don't think we're even close to 400 yet.
18:01 dbwalton We're using a virtual server, and it looks like it has about 7.5 GB ram assigned.
18:01 dbwalton I'm not sure how to find out if the virtual server system itself has anything to do with our problems.
18:02 aubreyja my guess is that 7.5 GB of ram is plenty for you then
18:02 aubreyja I'm on a virtual server too, but I don't know enough about that to even speculate if it could be causing your problem
18:03 aubreyja I imagine that if there are a lot of other vms on the host they could be using up resources when you need them, but again I don't know much about virtual hosting
18:04 dbwalton I've been told that the server we are using shouldn't be seeing a lot of stress.
18:04 aubreyja if you know that your campus uses vmware for example, then you could probably look into their documentation about this or see if they have tools for measuring it
18:05 dbwalton One strategy I've thought about pursuing is to set up a clone of the server and try to send it a lot of artificial requests and see if I can find any patterns.
18:05 dbwalton Yes, we are using vmware.  I'll have to look into that possibiility.
18:06 aubreyja that could work - maybe the would even set you up with a mirror dev site on the same server
18:06 aubreyja I tried out Devel::NYTProf and ti was really neat to see the info it sent back - I'll see if I can dig it up
18:07 dbwalton That sounds like a promising route.  I was wondering if there was a profile tool.  I'm definitely going to look into that.
18:08 aubreyja I'll send you what I have if I can find it again
18:09 dbwalton New question:  Has there been any development about changing the display format depending on device?  (i.e., creating a more suitable interface for mobile devices)
18:11 aubreyja Actually, there has been a bit of progress on that - there is some use of jquery and jquery mobile in new versions of some of the display modules - and there has been a lot of talk about wanting to go that route
18:12 aubreyja I actually had a grad student here that we hired to write an iOS app and he made a bit of progress but quit working on it when job season started and never picked it up again
18:17 dbwalton Sounds good.  I've seen a number of students doing their work on phones or kindles or ipads.
18:17 dbwalton Thanks for your comments
18:19 aubreyja np - actually now when a student asks about a problem in class he's just as likely to have a hard copy as to pull it up on his/her phone (actually, the phone is somewhat more likely…)
19:04 aubreyja mgage, mgage_: hi Mike -around?
19:16 alex_basyrov joined #webwork
19:17 alex_basyrov Brian (dbwalton), I just saw your post about webwork server slowdowns... Did you get anything new about possible cause?
19:18 dbwalton No, I'm really just starting to figure out HOW to go looking for answers.
19:20 alex_basyrov this could be a long shot, but mod_perl by default uses only 5 perl interpreters
19:20 alex_basyrov and the max number of perl interpreters is also low
19:20 alex_basyrov if all of the perl interpreters are taken, then the request to webwork could hang for a while
19:20 alex_basyrov until perl interpreter is available
19:20 dbwalton Which config file sets these settings?
19:21 alex_basyrov good question. I'll look it up
19:22 aubreyja that's an interesting point alex -the setting is PerlInterpMax
19:22 alex_basyrov I guess it could be set in the webwork.apache2-config file
19:22 alex_basyrov or anywhere in the apache config files
19:23 aubreyja so, perhaps setting PerlIterpMax 10 (or something) - yes in webwork.apache2-conf I guess
19:24 alex_basyrov this page talks about the number of perl interpreters
19:24 alex_basyrov http://perl.apache.org/docs/2.0/user/config/config.html#Threads_Mode_Specific_Directives
19:24 alex_basyrov I'm not sure if the parameters apply to webwork set up though
19:25 aubreyja right, as I look at this it seems to be only for threaded MPM...?
19:27 aubreyja I don't really understand the relationship between this setting, threaded vs pre fork MPM, and perl compiled with -Duseithreads or not...
19:27 alex_basyrov anyways, it might make sense to play with parameters PerlInterpStart, PerlInterpMax, PerlInterpMaxSpare, PerlInterpMinSpare  and see if they make any difference
19:28 dbwalton In looking at the webwork.apach2-conf file, I don't see these set anywhere.  Does this mean that new lines need to b added?
19:28 dbwalton Is there a status command that will report how many interpreters are open at any given moment?
19:29 alex_basyrov yes, these parameters are not set anywhere at all
19:29 alex_basyrov I don't know about the status command
19:33 alex_basyrov what might help in finding the bottlenecks, is some sort of real-time count of apache processes, and real-time status of RAM use.
19:34 aubreyja sorry - phone call - yes, add a line..let's see
19:34 alex_basyrov and maybe even the status of use of swap
19:35 alex_basyrov we are using 'monit' that collects data every minute, and sends and email when RAM use is above xx% and the count of apache children is above yy, and etc., etc.
19:36 alex_basyrov i can share the config file we use if that could help
19:36 aubreyja ok, I think it would go exactly after PerlModule mod_perl2 in webwork-apache2.config
19:36 dbwalton Yes, that type of information would be helpful.
19:37 alex_basyrov it looks like monit could check resources even every second
19:38 alex_basyrov the default installation has a really good sample config file
19:41 aubreyja got to run guys -ttyl
19:41 alex_basyrov to check system load, one would have something like this:
19:41 alex_basyrov check system webwork.host.edu if memory usage > 75% then alert if cpu usage (user) > 70% then alert if cpu usage (system) > 30% then alert if cpu usage (wait) > 20% then alert check process apache with pidfile /var/run/apache2.pid if totalcpu > 50% then alert    if totalmem > 10 GB then alert if children > 10 then alert
19:41 alex_basyrov oops
19:42 alex_basyrov bad formatting
19:42 dbwalton I was finding some of this at http://mmonit.com/wiki/Monit/MonitorApacheStatus
19:42 alex_basyrov that would work as well
19:43 alex_basyrov if you set pretty low limits for alerts, you will get tons of emails, but you will also have a lot of information
19:43 alex_basyrov something to experiment with
19:44 dbwalton That sounds like an interesting approach.  I'm still trying to design a strategy for exploration, and this sounds like it might be a good place to start.  Jason also suggested using a perl profiler that might give some indication about where time is being spent.
19:45 alex_basyrov But if the timing log does not have large witing times, then perl profiler might not help that much
19:45 alex_basyrov Also, using system profiler on a production server is not a good idea
19:46 dbwalton I was planning on creating a clone of the server and send artificial requests.
19:46 dbwalton But there might be some process that the profiler might catch in spite of the timing log showing reasonable times for most requests.
19:47 alex_basyrov another minor thing: if you have 'ExtendedStatus On' on the apache configs, that also could slow server down
19:48 alex_basyrov yes, using a perl profiler is better than not having any profiler
19:59 alex_basyrov got to go. bye
19:59 alex_basyrov left #webwork
20:00 dbwalton left #webwork
21:20 aubreyja joined #webwork
21:27 Guest96368 joined #webwork
22:28 mgage_ hi -- been at the FITL conference at RIT all day.
22:28 mgage_ aubreyja: just catching up
22:28 aubreyja Hi - fitl?
22:29 mgage_ Faculty Institute on Teaching and Learning  http://www2.rit.edu/fitl/
22:29 mgage_ only moderately interesting I'm afraid
22:30 aubreyja ah, fair amount of activity on irc as it turns out :)
22:30 mgage_ yep -- still reading
22:32 aubreyja I ran into a couple of minor issues in your github code today as I was working with my instructors and our college algebra coordinator.
22:34 mgage_ ok -- you can practice posting them to issues.  I'm not sure whether it should be done on your github site or mine.  David Gage has been posting to issues and he has a new library browser 3 out.  what is the gist of the issues?
22:34 aubreyja the first was that in the score report summary, the statement "You have … attempts remaining" was coming out as "You have -x remaining" where x = (I think) number of attempts, not number of attempts remaining
22:34 mgage_ ok -- that's pretty easy to check -- and could easily have been introduced from Grant's rewrites
22:35 aubreyja The second was that the very nice date picker in in the...
22:35 aubreyja just a sec phone call
22:36 aubreyja ok…when editing the dates for homework sets, the date picker was behaving a little strangely, but I can just show you next week
22:36 aubreyja it mostly worked, but some dates didn't come out right
22:37 mgage_ the date picker needs work -- then it also needs to be added to other places using dates in the homework editor and perhaps elsewhere
22:37 aubreyja oh, I have periodicReranomization.pl working very nicely
22:37 mgage_ good project for next week
22:37 mgage_ that sounds good. -- a bunch of people like to use that.
22:37 aubreyja I set it up so that, for example,  PeriodicRerandomization(3) actually works on a mod 4 cycle,
22:38 aubreyja but on 0 mod 4 the regenerate button is displayed and the text entry box for the answer is made non-editiable (with a little javascript insertion).
22:39 aubreyja I could do the same to make the submit button disappear if I could get something like HEADER_TEXT, but which is inserted at the bottom of the page, or at least below the buttons - is there anything like that? I didn't see anything
22:39 mgage_ my memory is that you always have some slight ambiguity as to how to start the next cycle.  pressing "submit answer" to get a new problem doesn't seem quite right.  -- maybe the text of that button could be modified with javaScript?
22:40 aubreyja yes, I can do that, but only if the javascript is inserted after the button is already on the page I think.
22:41 aubreyja for example, adding the code in HEADER_TEXT didn't work, but adding it to the output of the PeriodicStatus function did since that occurs below the problem text and so below the answer blanks
22:41 mgage_ no -- but there is a precedent.  we have the post_header_text item that is after the header but before the form creating the problem, we might as well add a post_form_text item for javascript code.
22:41 mgage_ I'd like better names for both of those
22:41 aubreyja that sounds like what I need
22:42 aubreyja another task for next week then
22:42 mgage_ yes -- or that one could fit just as well with the winona code camp.
22:43 aubreyja I went back to using up submissions with the "get a new version" button because of this, but maybe we can not do that if we are disabling the submit button too
22:43 aubreyja thanks - I've got to head home, but I'll be around later and tomorrow, etc.
22:44 aubreyja oh, and btw, Chris Sangwin called me on the phone - interesting guy - author of dragmath as it turns out
22:46 mgage_ yes -- he sent me an email and asked some questions (I think you saw my answer)
22:46 aubreyja yep, ttyl
22:46 mgage_ ttyl -- I'll be working tonight
22:51 theseb joined #webwork
22:51 mgage_ hi
22:51 theseb why is the webwork question suffix *.pg instead of *.pl like PERL files?
22:53 mgage_ long story -- but it is not a pure perl file -- the file is written in the .pg language which contains a lot of perl subroutines/macros and also processes TeX commands.  The questions for webwork are in .pg files   the macros for processing those files are in .pl files
22:54 theseb mgage: wait...what is the "pg language"?
22:54 theseb postgres?
22:54 theseb but it uses mysql?!
22:54 theseb mgage: is the pg language something webwork made up ?
22:55 mgage_ yes.  https://math.webwork.rochester.edu/docs/docs/pglanguage/
22:56 mgage_ (what just sent is a historical document -- no longer being updated)
23:01 theseb wow...no kidding...thanks!
23:02 mgage_ the main source of information is webwork.maa.org/wiki   -- there is a lot of information there but organizing is an ongoing community effort.
23:02 mgage_ where are you working from?  (I'm Mike Gage at University of Rochester)
23:02 theseb I in houston...i'm the one that started the python thread
23:03 theseb thanks for the Inline::Python idea...(I think that was you)
23:03 theseb I'm*
23:04 mgage_ oh right --- sorry,  I was up north in Canton, NY helping out my parents so I was only half paying attention.  I'm back in Rochester now and trying to catch up
23:04 theseb I replied to your post...I got Inline::Python to work but there is a incompatibility w/ PG language...basically webworks wants ENDDOCUMENT to be last thing....inline python wants to be last...see the problem?
23:05 mgage_ several people are interested in using python.  I'll be interested to see what can be done.
23:06 theseb mgage: ideally webwork would be an "open tent" app that let everyone submit stuff in many languages yet everyone could share in the growing NPL
23:06 mgage_ ENDDOCUMENT() has to be the last call because the entire file is evaluated as one long subroutine that returns  some data.  ENDDOCUMENT() is the function that gathers all the data produced while executing the .pg file and returns it from the PG question renderer to the webwork2 LMS which formats the data for the student.
23:07 theseb another idea i had to avoid fighting webwork's perlisms is to just convert a problem spec in python to pg syntax....i guess i'm basically proposing writing a python2pg "compiler" of sorts
23:08 theseb it would be more secure/safe/..../ to write these "Compilers" for ruby, python, etc.
23:08 mgage_ that's ok with me.  -- we even considered python briefly when we started webwork2 in 2002 but python wasn't as robust as perl then. (When we started in Webwork 1996 python may not even have existed).
23:09 theseb mgage: do you think more people ask for python nowadays than perl?  i'd guess there are more python fans now
23:10 mgage_ I think that is a pretty good idea.  The very first implementation of webwork was actually a precompiler for CAPA ,  but eventually turned out that using webbrowsers, CGI and perl was  better than trying to get CAPA to work for large numbers of students. (CAPA has since scaled up to lon-CAPA in much the same way)
23:12 mgage_ I've had suggestions to rewrite webwork in nearly every language (java, scheme, some CAS language, ..., php, ruby, python) (I was kidding about Forth although I rather like Mops -- the object oriented forth language).  they are all pretty similar (well forth and scheme are different)  -- there is a webwork clone in php
23:13 mgage_ the main problem is the amount of rewriting you have to do, and you need to balance this agains the gains.  if you want a simpler language to write questions in did you look at PGML?  It's not been widely used yet but it shows promise.
23:14 theseb mgage: i haven't heard of pgml ...should i look into it? yes i do want simplest language
23:14 theseb and i'm only learning enough perl to write pg files
23:14 theseb never used it much
23:14 mgage_ yes -- stands for pgml   (written by davide cervone)
23:15 mgage_ http://webwork.maa.org/wiki/PGLabs#PGML_markup_lab  sorry   stands for PG markup language
23:15 mgage_ :-) typing too fast
23:16 theseb thanks..i'll look into it
23:16 theseb i really like webwork
23:16 theseb i run a small teaching business and hope to use it for a long time on lots of classes
23:17 mgage_ why does INLINE::Python have to be last?    thanks.    I have a former graduate student working at U. of H Clear Lake
23:17 theseb mgage: i have no idea :).....i only ever used inline::python yesterday per your suggestion....i never got it to work otherwise
23:18 mgage_ sounds good -- I hope it will work well for you.  Feel free to jump in and help when you feel comfortable.  as you can see from the map we have lots and lots of users but not that many developers for an open source project.
23:18 theseb i hope i can help
23:18 mgage_ ok -- I wonder if it really needs to be last.
23:19 theseb i'll check a little more ...maybe i'll get lucky
23:19 mgage_ great.  hang out on the irc and sign up on the wiki/forum at    webwork.maa.org/wiki if you haven't already
23:19 theseb mgage: is that the place i signed up to post on forums? i've had that for a while
23:20 mgage_ yes -- you are all set then
23:20 theseb k great...well thanks again for all your help
23:21 mgage_ the forums is where most general communication goes on.  then there is the IRC and  the ww-devel mail lists (see development section of wiki for address) which is mostly just developers
23:21 mgage_ np -- ttyl

| Channels | #webwork index | Today | | Search | Google Search | Plain-Text | summary