The web in a box - a next generation web framework for the Perl programming language

IRC log for #mojo, 2017-02-22

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 PopeFelix And there are two "finish" events happening for the initial POST
00:10 lluad joined #mojo
00:12 lluad joined #mojo
00:30 lluad joined #mojo
00:46 Janos_ joined #mojo
00:49 sugar joined #mojo
00:49 PopeFelix ok, i have a fix, but I'd sure like some eyes on it
00:50 PopeFelix https://github.com/PopeFelix/Mojo-UserAg​ent-Mockable/tree/issue-9-proxy-support
00:50 PopeFelix Um.. just post comments here i guess and I'll look for them in the scroll tomorrow
01:40 itaipu joined #mojo
02:39 asarch joined #mojo
03:28 noganex joined #mojo
04:11 sh14 joined #mojo
04:42 inokenty-w joined #mojo
05:04 dboehmer joined #mojo
05:18 Janos_ joined #mojo
06:32 kaare_ joined #mojo
06:41 eseyman joined #mojo
07:12 polettix joined #mojo
07:15 Vandal joined #mojo
08:06 AndrewIsh joined #mojo
08:17 trone joined #mojo
08:44 rshadow joined #mojo
08:51 rshadow joined #mojo
09:11 sri tinita: there's already tests
09:11 sri they are just wrong :)
09:12 sri now of course i'm wondering if we actually need the blib code
09:24 sri the fact that tinita noticed makes me believe we might need it
09:43 coolo sri: postgresql question - can I have an increment call to a counter that defaults to 0 without having to write select+insert|update with locking?
09:43 coolo sounds like generic enough, but I don't even know what to google
09:43 good_news_everyon joined #mojo
09:43 good_news_everyon [mojo] kraih pushed 1 new commit to master: https://git.io/vDpYu
09:43 good_news_everyon mojo/master 95b3769 Sebastian Riedel: fix blib handling in Mojo::Home
09:43 good_news_everyon left #mojo
09:45 sugar joined #mojo
09:45 sri coolo: update set foo = foo +1 where ....;
09:48 coolo will this return 0 if it failed and I can then insert the initial foo?
09:50 CHYC coolo: Sounds like you want something like a CTE
09:50 Jonis coolo: you're looking for https://wiki.postgresql.org/wiki/UPSERT
09:51 sri Jonis: his postgres is too old
09:51 Jonis ah :/
09:51 sri coolo: you can check for the number of rows affected, yes
09:52 sri real upsert was quite complicated in old versions of postgres, CHYC is right with his CTE suggestion
09:52 coolo UPSERT sounds good - and postgresql95 reasonable :)
09:53 sri well, if you can switch to postgres 9.5, then hell yes, do it! :)
09:53 coolo (94 is the only one supported right now, but who cares for supported :)
09:53 coolo sri: I'm in the process of slimming down the data and will then re-import
09:53 sri 9.6 id kinda meh, but 9.5 was huge
09:53 sri *is
09:54 CHYC ON CONFLICT is amazing, but you need to have unique constraints in the database for it to work.
09:55 sri example for a 9.5 upsert being used https://github.com/kraih/minion/blob/​master/lib/Minion/Backend/Pg.pm#L110
09:56 sri although, the next_val thing is a bit "creative" ;p
09:56 sri s/_//
10:02 rshadow joined #mojo
10:37 itaipu joined #mojo
10:52 sugar joined #mojo
11:00 tchaves joined #mojo
11:19 marty joined #mojo
11:34 go|dfish joined #mojo
11:37 sugar joined #mojo
12:03 kes joined #mojo
12:20 sri looking into salvas pull request, the server code is totally borked anyway
12:35 tinita sri: moin. just saw you fixed the blib handling, thanks
12:35 tinita i just wanted to have a look into t/moho/home.t and was wondering why it worked
12:36 tinita PopeFelix: so, as you can see, just mentioning it and going to bed *can* be successful ;-)
12:40 tinita PopeFelix: and apart from that, i have been contributing to open source projects long enough that i cannot just excpect people to fix something if i don't provide a detailed test case
12:42 itaipu joined #mojo
12:42 sri can confirm, know tinita like forever
12:43 tinita =)
12:44 coolo sri: one more PG question - how to mass insert with mojo::pg ?
12:45 * sri remembers announcing the first catalyst releases on her german perl forum back in the days
12:45 tinita haha
12:45 sri coolo: not actually done that yet :p
12:46 tinita sri: and i think #perlde was started around 2003 or so?
12:46 stephan48 yea there no really efficient way
12:46 stephan48 i recently stumbled over the same problem and really need to make a feature request about that
12:47 sri tinita: yea
12:47 stephan48 basically our temporary solution(cause time) was to skip the nonblocking insertion and just use the dbh to manually create a prepared statement
12:47 sri coolo: maybe the COPY stuff works best, but support for that will depend on DBD::Pg support for it
12:48 sri stephan48: non-blocking inserts are pointless
12:48 coolo sri: not *that* many rows. multirow inserts would do
12:48 stephan48 either prepared statements or COPY where the solutions
12:49 tchaves joined #mojo
12:49 sri coolo: unless SQL::Abstract can somehow generate those, i don't think there's a comfortable solution
12:49 stephan48 we first tried a query with 80k placeholders :D but the driver died somewhere around 30k.
12:49 coolo sri: damn
12:50 stephan48 i looked into it, it does not really support multiple inserts in one.
12:50 tchaves joined #mojo
12:50 coolo sri: wait, dbix::class's populate does something about bulk
12:50 coolo I need to look this up
12:51 stephan48 if you come up with a nice way - please shout
12:51 coolo http://search.cpan.org/~ribasushi/DBIx-Class-​0.082840/lib/DBIx/Class/ResultSet.pm#populate
12:51 * coolo is using this in openqa
12:52 coolo (and forgot about it for a moment :)
12:53 nic When I do big inserts (in MySQL) I bind local variables in the prepared statement and zip through the list
12:53 sri coolo: looking at the code, it does not appear efficient
12:53 nic column inserts can be optimised and fast, but in the wild I never find the source data is in column form, and doing a matrix flip defeats the aim of being fast
12:54 nic so the traditional DBD::?? method is usually the fastest
12:54 sri coolo: populate does not appear to actually batch inserts
12:54 nic (I don't think you mentioned performance, so I'm off on a tangent there)
12:54 coolo sri: according to the log it does
12:56 sri coolo: like this? INSERT INTO films (code, title, did, date_prod, kind) VALUES ('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'), ('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy');
12:57 sri i mean, if you just want to batch it into one transaction you can do my $tx = $db->begin; $db->insert(...) for ...; $tx->commit;
12:57 sri i specifically meant effiencient as in one statement :)
12:58 coolo sri: this is what I get from DBIC_TRACE:
12:58 coolo INSERT INTO screenshot_links ( job_id, screenshot_id) VALUES ( ?, ? ): '__BULK_INSERT__'
12:58 sri btw. this stuff might be the most effiencient way https://metacpan.org/pod/DBD::Pg#COPY-support
12:59 sri coolo: yea, that's just one prepared insert statement and multiple executes in the same transaction, you can do that with Mojo::Pg easily
12:59 stephan48 sri: it looks like it does delegate to http://cpansearch.perl.org/src/RIBASUSHI/DBIx​-Class-0.082840/lib/DBIx/Class/Storage/DBI.pm -  _insert_bulk which then delegates to _dbh_execute_for_fetch. this function will then run a prepared statement and call execute_for_fetch
12:59 stephan48 oh?
13:00 sri coolo: Mojo::Pg will automatically use a prepared statement if you run the same sql twice or more times
13:00 sri like i said before
13:01 sri my $tx = $db->begin; $db->insert({foo => 'bar'}); $db->insert({foo => 'baz'}); $tx->commit;
13:01 sri will do the same
13:02 stephan48 ah, via prepare_cached
13:02 sugar joined #mojo
13:02 coolo sri: I will try the next time.
13:02 coolo I hope my current job will finish somewhen :)
13:03 stephan48 sri++ # i read the code like twice last week but it did not stick in my head that it will do exactly what i want. i even read the docu for prepare_chaced to find out what the third param was. /o\
13:03 stephan48 thank you :)
13:03 stephan48 will make the code much more readable
13:04 sri always assume Mojo::Pg does the most effiecient thing possible :)
13:04 sri it's heavily optimized
13:08 nic In the MySQL world, a single-statement insert (VALUES (..), (..), (..)) is considerably faster than multi-insert, which is why mysqldump uses it
13:10 coolo in mysql you also have insert delayed
13:11 sri btw. re unix domain sockets, there's more uses of http+unix https://github.com/msabramo/requests-unixsocket
13:15 sri and it's actually in docker-py https://github.com/docker/docker-py/blob​/master/docker/transport/unixconn.py#L55
13:18 salv0 sri: I have documented the client side of the http+unix patch: https://github.com/salva/p5-Mojo/commit/​0a91f3bf27798b725dd2870e7160700989dfc5f8
13:18 nic I've never encountered a scenario where 'delayed' was appropriate.  It's a highly niche flavour of insert
13:19 sri salv0: i've already started on a new patch, so don't invest too much in it
13:19 sri your server code was very borked anyway, and there were security issues
13:24 salv0 ok
13:26 sri salv0: the whole thing with "'http+unix://'.url_escape($socket_path).'/test'" is a terrible idea
13:27 sri "http+unix://%2Ftmp%2Fmy.socket/test" is much easier on the eyes
13:28 Janos_ joined #mojo
13:41 marty joined #mojo
13:44 Janos_ joined #mojo
13:49 Janos_ joined #mojo
13:53 sugar joined #mojo
13:56 gryphon joined #mojo
14:14 gizmomathboy joined #mojo
14:22 marty joined #mojo
14:24 sri can someone run this on windows please "perl -MIO::Socket::UNIX=AF_UNIX -E 'say AF_UNIX'"
14:25 sri active/strawberry and possibly cygwin
14:27 sh14 joined #mojo
14:32 stephan48 https://paste.stejau.de/7ad2280d3338
14:32 stephan48 windows 10 x64
14:32 sugar joined #mojo
14:32 stephan48 the '' need to be swapped to " otherwhise windows complains about the cmdline
14:32 sri thanks
14:33 sri bummer
14:33 sri i need to test somehow if unix domain sockets are available
14:33 stephan48 ah
14:33 sri was hoping i could do it without trying to open one
14:33 stephan48 you hoped that it would be undef?
14:33 sri ye
14:33 pink_mist $ perl -MIO::Socket::UNIX=AF_UNIX -E 'say AF_UNIX'
14:33 pink_mist 1
14:33 pink_mist on cygwin
14:33 marty joined #mojo
14:35 sri perl -MIO::Socket::UNIX -E 'say IO::Socket::UNIX->new'
14:35 sri maybe that? :)
14:35 sri die!
14:36 pink_mist $ perl -MIO::Socket::UNIX -E 'say IO::Socket::UNIX->new'
14:36 pink_mist IO::Socket::UNIX=GLOB(0x600078690)
14:36 sri or in the words of reaper... die, die, die!
14:36 pink_mist (cygwin still)
14:36 sri ok, on cygwin it works anyway, no?
14:36 pink_mist it should, yes
14:37 stephan48 same result as for pink_mist. same perl version as before
14:37 sri argh
14:38 sri perl -MIO::Socket::UNIX -E 'say IO::Socket::UNIX->new(Listen => 1, Local => "foo.sock")'
14:38 sri now it has to die though
14:38 pink_mist still same result on cygwin
14:38 sri as expected
14:39 stephan48 C:\Users\stephanj>perl -MData::Dumper -MIO::Socket::UNIX -E "say Dumper(IO::Socket::UNIX->new(Listen => 1, Local => "foo.sock"))"
14:39 stephan48 $VAR1 = undef;
14:39 stephan48 no dead but atleast an undef
14:39 sri interesting
14:39 sri thanks both of you
14:40 stephan48 you are welcome :)
14:40 pink_mist now fire up that WSL? :P
14:41 stephan48 mh. i haven't actually checked that out
14:43 Janos_ joined #mojo
14:45 irqq joined #mojo
14:48 vicash joined #mojo
15:04 mcsnolte joined #mojo
15:20 stryx` joined #mojo
15:31 PopeFelix tinita, I stand corrected. :)
15:37 sri PopeFelix: it doesn't always work though
15:37 PopeFelix As amply demonstrated by the lack of comments on the thing I posted. ;)
15:38 * sri quickly vanishes in a cloud of smoke
15:41 sri hmm, stumbled over an interesting problem with Mojo::URL
15:41 sri or rather multiple problems
15:41 PopeFelix oh?
15:42 sri a) it generates invalid URLs for http://foo%2Fbar.baz/yada
15:42 sri becomes http://foo/bar.baz/yada
15:43 jabberwok A cloud of smoke?  I got this!   > RING BELL. GET CANDLES. LIGHT MATCH. LIGHT CANDLES WITH MATCH. READ BOOK.
15:43 sri b) it lowercases the host after applying punycode encoding
15:44 sri so http://%2FUsers%2Fsri/foo becomes  http:///Users/sri/foo
15:44 sri oops
15:44 sri so http://%2FUsers%2Fsri/foo becomes  http:///users/sri/foo
15:44 sri that way
15:44 purl rumour has it that way is too black magic and even more exposure of internals of a subsystem to completely unrelated parts
15:45 sri so it's really bad
15:47 marty_ joined #mojo
15:52 sri where the hell did percent encoding of the host get lost?
15:52 sri this is a pretty big bug
16:26 sri think i have a reasonable patch that takes care of everything
16:30 good_news_everyon joined #mojo
16:30 good_news_everyon [mojo] kraih pushed 1 new commit to master: https://git.io/vDhms
16:30 good_news_everyon mojo/master 8f6e214 Sebastian Riedel: add support for UNIX domain sockets and fix a few bugs
16:30 good_news_everyon left #mojo
16:32 pink_mist sri++
16:35 jabberwok with »$host =~ /[^\x00-\x7f]/« … are control characters \x00-\x1f (NUL, BEL, VT) legal in hostnames?
16:35 jabberwok (( somewhere in the background an ASR-33 clacks away ))
16:35 lluad joined #mojo
16:37 nic alphanumeric plus hyphen
16:37 nic although today I encountered a customer with surname unicode non-breaking-space, so...
16:39 jberger sri++
16:40 nic "I think Sebastian may be on fire.  Can you check?"
16:40 nic [lights match.  sets Sebastian's coat ablaze]
16:40 nic "Yes, I can confirm he's on fire."
16:41 sri also suse++ for letting me work on fun stuff for a week
16:43 sri btw. there is one thing i'm not entirely sure about https://github.com/kraih/mojo/commit/8f6​e21410f240b3a055d3c42a270382a7fc3a676#di​ff-a9009b150b211dbbfc0a7453ca27e660R60
16:43 sri how to clean up unix sockets
16:44 sri i can't actually unlink them on Mojo::IOLoop::Server destruction, because that breaks hypnotoad
16:44 sugar joined #mojo
16:44 sri since you can delete unix sockets while they are in use
16:46 good_news_everyon joined #mojo
16:46 good_news_everyon [mojo] kraih pushed 1 new commit to master: https://git.io/vDhGO
16:46 good_news_everyon mojo/master d49db3c Sebastian Riedel: test UNIX domain sockets on Travis too
16:46 good_news_everyon left #mojo
16:47 sri looks good on appveyor too https://ci.appveyor.com/project/kraih/mojo/history
16:48 gryphon joined #mojo
16:48 sri (not that the tests are activated there)
16:48 sri one of these days i should maybe try to find out how to set env vars there :)
17:02 dboehmer sri: you removed support for X-Forwarded-Host in https://github.com/kraih/mojo/commit/f2​dec76039b783e9c49c9682c05c4c2c242b85d7 . I am interested in how a well configured reverse proxy should transport the original host name. can you explain or link to discussion fom back then?
17:05 Grinnz dboehmer: https://metacpan.org/pod/Mojol​icious::Guides::Cookbook#Nginx
17:08 sugar joined #mojo
17:22 dboehmer Grinnz: I am not that familiar with nginx.conf but I read it like "Host:" will be the original Host:, right? I think that might be an issue with backend servers serving multiple domains (don't know if anyone does this) and would recommend setting X-Forwarded-Host. got it working for my setup now by setting ProxyPreserveHost in Apache
17:22 Grinnz no, i was referring to X-Forwarded-For and X-Forwarded-Proto, which are the headers mojo uses
17:22 Grinnz the apache section is right below that
17:23 dboehmer but For is client IP and Proto is protocol. I am interested in the Host
17:23 Grinnz but yes, Host will also be maintained as the original host correctly
17:24 Grinnz ProxyPreserveHost is set in the example for Apache there
17:24 dboehmer i saw it
17:24 dboehmer my personal problem is solved.
17:25 dboehmer Grinnz: thanks for linking to this example of how you define "well configured"
17:36 jnbek joined #mojo
17:42 sri http+unix proxy support is not officially supported yet, we'll see if i get around to making a test
17:50 sugar joined #mojo
18:01 lluad joined #mojo
18:11 dod joined #mojo
18:13 asarch joined #mojo
18:19 rshadow joined #mojo
18:23 Grinnz unfortunate that SQL::Abstract doesn't have a way to do multiple-insert by default, doesn't seem like it would be as complicated as LIMIT at least
18:23 Grinnz but i don't know if oracle/mssql/etc do it differently
18:40 jberger seems like there could be some way to insert from postgres arrays as rows or else json as rows
18:41 Grinnz that doesn't really help me for sqlite or mysql :P
18:44 jberger json_to_recordset looks handy
18:44 jberger no, of course that's only going to work in pg
19:30 blonewolfs joined #mojo
19:36 disputin joined #mojo
19:42 tyldis I wish SQL::Abstract would recognize an AOH for efficient insert
19:42 tyldis Unless I missed something this morning when I gave it my first spin
19:45 tyldis I often import from CSV to SQL, and Text::CSV_XS + SQL::Abstract (through Mojo, of course) makes it far too easy ;)
19:50 lluad joined #mojo

| Channels | #mojo index | Today | | Search | Google Search | Plain-Text | summary