Camelia, the Perl 6 bug

IRC log for #parrot, 2012-08-31

Parrot | source cross referenced

| Channels | #parrot index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:16 whiteknight joined #parrot
00:20 kid51 joined #parrot
00:30 whiteknight good evening, #parrot
00:32 benabik o/ whiteknight
00:33 whiteknight hello benabik
01:02 dalek Rosella/commandline_2: 2476c29 | Whiteknight++ | src/commandline/ (2 files):
01:02 dalek Rosella/commandline_2: [CommandLine] Break ArgumentDef out into two subclasses, so we don't need to do an if/else dispatch on every single method call
01:02 dalek Rosella/commandline_2: review: https://github.com/Whiteknig​ht/Rosella/commit/2476c29003
04:14 wagle joined #parrot
04:53 kurahaupo joined #parrot
05:23 benabik joined #parrot
05:55 wagle joined #parrot
07:37 kurahaupo joined #parrot
07:37 kurahaupo left #parrot
08:11 lucian joined #parrot
08:19 Psyche^ joined #parrot
09:49 schmooster joined #parrot
09:54 schmooster joined #parrot
10:26 woosley left #parrot
10:55 ligne joined #parrot
11:25 schm00ster joined #parrot
11:34 schmoo joined #parrot
11:52 JimmyZ joined #parrot
13:03 nemesys anyone else working on threads currently?
13:03 nemesys except nine
13:03 dngor Are you polling for a lock on that feature? :)
13:04 nemesys not really, no - rather trying to understand more of it
13:04 nemesys read nine's thesis about it, now trying to get the current picture
13:06 nemesys if I understood it correctly, parrot itself uses 2 threads anyways, the io_thread for dispatching signals and a "main" thread
13:07 nemesys the main thread than allows running more "tasks"
13:08 benabik joined #parrot
13:10 nemesys i'd rather say I do lack architecture understand of parrot
13:10 nemesys s/understand/understanding/
13:18 PacoAir joined #parrot
14:17 JimmyZ joined #parrot
14:18 JimmyZ nemesys:  here are some posts about threads on parrot, FYI: http://whiteknight.github.com/allposts.html
14:40 bluescreen joined #parrot
14:42 nnunley joined #parrot
15:13 dmalcolm joined #parrot
15:33 JimmyZ joined #parrot
16:12 contingencyplan joined #parrot
16:55 whiteknight joined #parrot
17:08 dalek nqp/toqast: a94c294 | jnthn++ | src/NQP/Actions.pm:
17:08 dalek nqp/toqast: Fix an $/ handling issue, which fixes two more of the Rakudo spectest file regressions.
17:08 dalek nqp/toqast: review: https://github.com/perl6/nqp/commit/a94c2944cb
17:09 dalek rakudo/nqpqast: 44f06cf | jnthn++ | src/Perl6/Grammar.pm:
17:09 dalek rakudo/nqpqast: Update a workaround for a bug (that needs a proper fix).
17:09 dalek rakudo/nqpqast: review: https://github.com/rakudo/rakudo/commit/44f06cf0cf
17:09 dalek rakudo/nqpqast: 6c8ef80 | jnthn++ | src/Perl6/Metamodel/AttributeContainer.pm:
17:09 dalek rakudo/nqpqast: Fix subtly buggy code that used to work for entirely the wrong reason. Deals with another regression.
17:09 dalek rakudo/nqpqast: review: https://github.com/rakudo/rakudo/commit/6c8ef80eb3
17:09 jlaire joined #parrot
17:29 whiteknight good afternoon, #parrot
17:45 japhb (Carried over from #perl6): What's the current state of the thread project in Parrot?  Is that now done, or "coming soon", or still a ways out?
17:46 whiteknight soming soon. I plan to merge it on tuesday
17:46 tadzik ooh yes
17:46 japhb Ah, has a date even, excellent.
17:46 whiteknight why, is rakudo itching to start parallelizing?
17:47 japhb We're itching for 6mojo, which means either good non-blocking IO or good threads.
17:47 whiteknight non-blocking IO will come after threads
17:48 japhb whiteknight, implemented in terms of threads, or using async primitives from the OS?
17:48 sri how expensive will threads be?
17:49 whiteknight japhb: I don't know. Threads gives us a toolset to handle asynchronous callbacks, so threads will be involved somehow
17:49 sri along the lines of jvm threads (1kb) or perl5 (1mb+)?
17:49 whiteknight doing asynch IO in a platform-neutral way without using threads too much may be a hassle
17:49 whiteknight sri: for a common, single-threaded application there's negligible cost for adding it
17:50 whiteknight sri: nine has measured some nice speedups for parallelizable tasks with threads
17:50 japhb whiteknight, I think sri was asking about per-thread overhead ...?
17:50 sri NIO in the jvm is not a bad platform independent way to abstract non-blocking I/O
17:51 whiteknight I don't know what per-thread overhead will be. Probably closer to the jvm version than the p5 version
17:51 japhb good to hear!  :-)
17:51 sri right, i meant per-thread overhead
17:51 whiteknight Parrot uses a hybrid approach of tasks+threads, so you can schedule multiple tasks on a single thread for virtually zero overhead
17:51 whiteknight it's like stackless threads in python
17:52 whiteknight keep in mind that the initial version found in the merge may need to be tuned, optimized, etc. So things will get better if they aren't great initially
17:52 japhb nodnod
17:54 sri interesting
17:55 whiteknight sri: if threading is used, you'll get a pool of threads. You create tasks and the tasks are dispatched onto threads. In a 1:1 system, it could be relatively heavyweight, but in an n:1 system you get much lower overhead
17:56 whiteknight especially since n tasks running on a single thread are able to share data without locks
17:56 whiteknight so there's no risk of memory corruption
17:59 sri ah, tasks are pretty much coroutines?
18:00 sri or green threads
18:02 * sri was hoping for real threads like in jruby and rubinius
18:06 lizmat sri: fwiw, the weight of an Perl 5 ithread depends on the amount of code compiler / number of modules loaded
18:06 lizmat as *everytihing* (apart from optrees) is copied to a new thread when a thread is started
18:10 sri which makes them mostly useless
18:10 sri the ruby folks have been doing some pretty exciting stuff with their cheap real threads http://celluloid.io
18:15 whiteknight sri: yes, tasks are green threads
18:15 whiteknight sri: but they get dispatched to real threads
18:15 whiteknight But the system is much more flexible than saying every task must be a thread
18:15 sri and each real thread contains a whole interpreter?
18:15 whiteknight yes, believe so
18:15 sri :(
18:16 whiteknight although it doesn't get a copy of all the interp data
18:16 sri tbh. that kills concurrency in perl6 for me
18:16 lizmat sri: what does?
18:17 whiteknight and what would you prefer it do?
18:17 whiteknight keep in mind, this is only the first go at it
18:17 whiteknight well, that's not quite right. It's a flexible system that we can definitely build on and modify
18:18 whiteknight What we need most are real use-cases from real users
18:18 dalek rakudo/nqpqast: f962cf8 | jnthn++ | src/Perl6/Grammar.pm:
18:18 dalek rakudo/nqpqast: Fix $< parsing (just replace what we had before with what STD has). Fixes the S05 failures.
18:18 dalek rakudo/nqpqast: review: https://github.com/rakudo/rakudo/commit/f962cf8fd9
18:18 sri jruby threads for example are not isolated, if you modify a data structure from multiple threads you're responsible for locking
18:19 whiteknight so, what does that buy you? That pushes consistency requirements onto the programmer, but doesn't make them go away
18:20 sri real concurrent threads that are cheap
18:20 sri yes, you can use actors for that
18:20 sri see Akka for example
18:20 whiteknight sure the threads are cheap, and all the locks and deadlock potential and spinwaiting are expensive
18:20 whiteknight it just pushes the costs onto the programmer
18:21 sri right, but it makes you much more flexible
18:21 whiteknight What do you mean?
18:21 sri http://akka.io
18:22 whiteknight The hybrid task/thread model we're preparing to merge was very much inspired by actors and concurrency models like python stackless and erlang
18:23 whiteknight I'll keep saying that it's not perfect in the first version, but much of the potential is there to do more, more efficiently
18:23 sri perhaps i'm not understanding how task scheduling works then
18:23 whiteknight A task is basically just a continuation: small and cheap to invoke. Tasks are queued to threads, where they are run
18:24 whiteknight create tasks, schedule them to threads.
18:24 whiteknight if you have multiple tasks on a single thread, they preempt each other
18:24 whiteknight or, they can
18:25 whiteknight if you're on a single-threaded platform, this system degrades very gracefully with no changes needed to most programs
18:25 whiteknight It also really doesn't have locks, because data shared across threads is basically read-only
18:26 sri you're basically back to non-blocking I/O if you want to scale a server to 10k active sockets
18:26 whiteknight what do you mean?
18:27 sri i just don't see scheduling of 10k green threads working where each is watching a socket
18:27 whiteknight having 10k active sockets in almost any configuration is going to be a problem
18:28 sri perl5 does it without problems on my old macbook :)
18:28 sri (non-blocking I/O of course)
18:28 whiteknight how do you have perl5 doing it, with 10k ithreads, each with a socket to watch?
18:29 sri that would be impossible i imagine :)
18:29 whiteknight right, you wouldn't do it that way in parrot either
18:29 sri but you do it that way in jruby
18:29 sri or rather... can do it that way
18:30 sri or Java/Scala
18:30 whiteknight in parrot you could easily attach all your sockets to a Select, then use that to dispatch short-lived tasks to handle incoming requests
18:30 whiteknight you wouldn't want to create 10k tasks up front and have them all busy-wait
18:32 whiteknight since tasks are just continuations and they have a very small memory footprint, you can create them quick and maintain a large number of them in a queue somewhere
18:32 whiteknight I don't remember the exact numbers, but nine did benchmarks with 10k tasks or more running in a single process
18:33 sri i guess you can already have that in perl5 with ithreads and Coro
18:35 whiteknight sri: If you have some specific ideas you want us to consider, I'd definitely like to hear all about them
18:35 whiteknight our system will get better with feedback
18:39 sri guess i was hoping for something that doesn't make huge sacrifices to prevent data corruption
18:40 whiteknight I wouldn't call too much of what we've done "huge sacrifices"
18:40 sri yielding between coroutines can be a huge pita
18:40 whiteknight it's just a continuation invoke, no more expensive (theoretically) than a method call/return
18:40 whiteknight of course, method call/return in our system is a little more expensive than it needs to be
18:41 sri real threads could just use blocking syscalls and not worry about yielding
18:41 whiteknight Yeah, and you can use real threads and blocking syscalls
18:41 whiteknight or, you can use a blocking syscall and preempt a new green thread to run during it
18:41 sri but that would be expensive again :)
18:42 whiteknight I think you and I have very different definitions of "expensive"
18:42 sri cheap = in the 1-2kb overhead range
18:42 sri Coro in perl5 is 4kb i believe per coroutine
18:43 whiteknight I don't know how big a parrot task is, but I doubt it's 4k
18:44 whiteknight I would estimate it's 1k or less per task
18:44 whiteknight per thread interp memory usage is higher because it allocates thread-local pools
18:44 whiteknight so that's hard to measure what is actually being used
18:45 sri i get where you're coming from, but making sure that green threads don't block each other might be just as annoying as corrupted data structures (if you forget locking)
18:47 sri you basically make scheduling green threads your users problem ;p
18:48 sri it's either... schedule your tasks... or protect your data
18:48 whiteknight Green threads don't block each other, they're preemptive. Users don't schedule them, they run automatically when the preemption signal is received or when the current task is blocking
18:48 whiteknight users don't have to do any scheduling themselves
18:49 sri well, a blocking syscall would block all tasks in the same thread
18:50 whiteknight yes, that's true. But then you wouldn't make blocking syscalls
18:50 sri right, that's what i mean, you have to be aware of it and use non-blocking syscalls instead
18:50 whiteknight and internally, things like the IO system will automatically be able to convert blocking operations into nonblocking+context switch
18:51 sri aaah
18:51 sri that's very important information :)
18:51 whiteknight Like if you do a socket.recv, it wouldn't block if you had data in the buffer, but would block if you had to recv. So it would be smart enough to do a context switch while the syscall was waiting
18:51 whiteknight theoretically, in the future, hand-waving
18:51 sri well... that changes everything of course
18:52 whiteknight see? We're not completely incompetent
18:52 whiteknight we just appear that way on the internet
18:52 * sri pats whiteknight on the back
18:54 whiteknight so that's how you to 10k sockets with parrot: Create tasks, use non-blocking IO requests, and let the scheduler keep things moving while they wait
18:54 whiteknight async io requests create callbacks, which become new tasks to pick up where the request left off
18:55 sri still leaves open the question of syscalls without non-blocking analog... fs I/O
18:55 whiteknight most of our platforms have non-blocking filesystem io, just not in a standard way with a standard api
18:55 whiteknight so putting that in place is going to be harder
18:55 sri (libuv has a separate thread pool for that)
18:55 whiteknight that's an idea we've kicked around as well
18:56 whiteknight of course if you serialize multiple blocking requests to a single requests thread, you still end up with a bottleneck
18:56 sri right, it seems to work for node.js though
18:57 * sri wonders how erlang handles the problem
18:58 whiteknight sure, there are ways we can make it work if that's the direction we choose
18:59 whiteknight erlang probably does something close to what I said: use non-blocking requests as much as possible
19:05 sri quite ambitious plans, but i'm intrigued
19:14 autark joined #parrot
19:25 wagle joined #parrot
20:03 wagle joined #parrot
20:05 dalek nqp/toqast: c07af7a | jnthn++ | src/ops/nqp.ops:
20:05 dalek nqp/toqast: RSA should also report true in nqp::islist, otherwise the result of split does not count as a list, which caused some breakage.
20:05 dalek nqp/toqast: review: https://github.com/perl6/nqp/commit/c07af7a4a5
20:10 dalek rakudo/nqpqast: 9c1534d | jnthn++ | tools/build/NQP_REVISION:
20:10 dalek rakudo/nqpqast: Update NQP_REVISION to ease testing.
20:10 dalek rakudo/nqpqast: review: https://github.com/rakudo/rakudo/commit/9c1534d3ac
20:27 cotto joined #parrot
20:34 lucian joined #parrot
20:39 dalek nqp/toqast: c0fbe4d | jnthn++ | src/6model/reprs/KnowHOWREPR.c:
20:39 dalek nqp/toqast: Add missing mark in KnowHOWREPR that could lead to segfaults and other corruption.
20:39 dalek nqp/toqast: review: https://github.com/perl6/nqp/commit/c0fbe4d73f
21:07 dalek rakudo/nqpqast: b79e158 | jnthn++ | tools/build/NQP_REVISION:
21:07 dalek rakudo/nqpqast: Another revision bump.
21:07 dalek rakudo/nqpqast: review: https://github.com/rakudo/rakudo/commit/b79e1588b3
21:12 dalek rakudo/nqpqast: 119716f | jnthn++ | src/core/Exception.pm:
21:12 dalek rakudo/nqpqast: Fix is_runtime to unbust backtrace stuff in some cases.
21:12 dalek rakudo/nqpqast: review: https://github.com/rakudo/rakudo/commit/119716f349
23:04 lucian joined #parrot
23:29 benabik joined #parrot

| Channels | #parrot index | Today | | Search | Google Search | Plain-Text | summary

Parrot | source cross referenced