Perl 6 - the future is here, just unevenly distributed

IRC log for #crimsonfu, 2015-12-02

crimsonfu - sysadmins who code

| Channels | #crimsonfu index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
08:04 Azgarech joined #crimsonfu
15:00 pdurbin dotplus bene larsks et al.: any experience with https://www.globus.org ?
15:00 larsks pdurbin: not me!
15:01 pdurbin larsks: have you heard of it?
15:01 bene yeah man
15:01 bene we installed globus at hms
15:01 bene wrote a paper on it somewhere
15:01 larsks pdurbin: I think we looked at it many years ago, but I wasn't directly involved.
15:01 pdurbin hmm, so you can install it. ok
15:01 larsks And we didn't ultimately use it...
15:02 bene http://journals.iucr.org/s/issues/2012/03/00/ie5073/index.html
15:03 bene my info is at least 2 years out of date
15:03 bene but it was under heavy development at the time
15:03 bene it was all x509 based for user auth
15:03 bene and i think ian foster (lead PI) and the team were trying to add other user auth schemes
15:03 bene it's really slick when you get it working
15:04 * pdurbin prints the paper
15:04 bene log in to the web interface, choose your ~ at one site, browse and select the files, then select the destination and click Transfer
15:05 bene and it uses gridftp on the backend to make best use of your long fat networks
15:05 bene i'm not sure the paper is worth reading
15:05 pdurbin oh
15:05 bene in the end everyone decided that it was too much freaking work doing the x509 thing or the alternative posix user -> x509 mapping thing
15:06 bene i would just check out the software today
15:06 bene i bet they have RPMs
15:06 bene install a couple nodes in some vagrant boxes and see what it looks like
15:06 pdurbin bene: in your architecture drawing, globus is on the side by itself: http://journals.iucr.org/s/issues/2012/03/00/ie5073/ie5073fig2.html
15:06 bene yeah
15:07 bene it's actually the thing that is behind those arrows showing the data flows i think
15:07 bene "we wrote a paper" was the royal we
15:07 pdurbin "Scott can use Globus Online to access his data at NE-CAT and transfer it to his personal space on the laboratory file server scott@sliz.harvard.edu:/home/scott or to his own laptop."
15:07 * bene casts Summon ijstokes
15:08 bene just saw your ABCD post
15:08 pdurbin bene: so you can run Globus entirely on your own hardware? And it's free as in beer? And free as in freedom?
15:10 bene no idea about the license
15:10 bene we ran it on some shitty hardware at hms, so yeah
15:12 bene interesting
15:12 bene it seems like they have evolved somewhat
15:12 pdurbin yeah?
15:12 bene globus connect looks like it does dynamic endpoint mapping?
15:12 bene like bittorrent sync or something
15:12 bene https://docs.globus.org/how-to/globus-connect-personal-linux/
15:13 bene and you only need an SSH key and not the x509 insanity
15:13 pdurbin heh
15:13 bene yeah
15:13 bene it looks kinda slick
15:13 bene they were headed in this direction when we did that prototype thing
15:14 pdurbin part of the abcd post was to get a sanity check on https://github.com/IQSS/dataverse/issues/2728
15:14 bene ian foster got some $$$ from NSF to keep building this thing
15:14 bene and it looks like it may have paid off
15:15 bene get back to us in 2 days with a report
15:15 * bene delegates
15:15 pdurbin :)
15:15 pdurbin bene: thanks for your thoughts on this. and the link to the paper
15:16 pdurbin larsks: why didn't you use it?
15:16 bene git lfs/git annex for large file DVN
15:16 bene is there a CLI interface to DVN?
15:16 bene totally should be able to 'dvn status' 'dvn add' 'dvn commit' for file management
15:17 pdurbin bene: we have APIs. and Python and R client libraries: http://guides.dataverse.org/en/4.2.2/api/client-libraries.html
15:17 bene the CLI _is_ an API
15:17 pdurbin bene: and we don't call it "DVN" anymore. Now we call it "Dataverse".
15:17 bene E_TOOMANYLETTERS
15:18 bene DVS then?
15:18 pdurbin sure
15:18 pdurbin DvObject in the code :)
15:19 pdurbin this paper should say "its" instead of "it's" on page 466
15:26 pdurbin "The development of the Globus Online service... has now also made the previously complicated task of using high-performance parallel file transfer tools such as GridFTP... accessible through simple web interfaces."
15:26 dotplus I was part of the team that globus @ornl.gov, although it was setup before my time and mostly just worked so I didn't get much exposure. IIRC, we had a reasonably automated [set of] processes to deal with auth mgmt
15:26 dotplus that *supported* globus @ornl.gov
15:28 pdurbin just worked is good
15:29 dotplus short version of my experience is that the scientists wanted it so we provided and, once setup well, it seemed to work well for them.
15:30 pdurbin if you run Globus yourself, where is the data stored? on any file system?
15:31 dotplus I think so, I don't see any reason why it can't go anywhere. We used Lustre.
15:32 dotplus But there's not much point pouring data through a massive parallel pipes if you can only write through a straw
16:12 Whoop joined #crimsonfu
16:12 larsks joined #crimsonfu
16:12 hydrajump joined #crimsonfu
16:12 bear joined #crimsonfu
16:12 semiosis joined #crimsonfu
16:12 chasmo77 joined #crimsonfu
16:12 dotplus joined #crimsonfu
16:12 mhayden joined #crimsonfu
16:12 arcanine joined #crimsonfu
16:12 pdurbin joined #crimsonfu
16:12 searchbot joined #crimsonfu
16:28 prologic joined #crimsonfu
16:28 westmaas joined #crimsonfu
16:28 JoeJulian joined #crimsonfu
16:28 bene joined #crimsonfu
16:47 Whoop joined #crimsonfu
16:47 larsks joined #crimsonfu
16:47 searchbot joined #crimsonfu
16:47 pdurbin joined #crimsonfu
16:47 arcanine joined #crimsonfu
16:47 mhayden joined #crimsonfu
16:47 dotplus joined #crimsonfu
16:47 chasmo77 joined #crimsonfu
16:47 semiosis joined #crimsonfu
16:47 bear joined #crimsonfu
16:47 hydrajump joined #crimsonfu
16:51 ilbot3 joined #crimsonfu
16:51 Topic for #crimsonfu is now http://crimsonfu.github.com - ConfiguRatIon Management of Systems Or Network kung FU | logs at http://irclog.perlgeek.de/crimsonfu/today
16:56 sivoais joined #crimsonfu
16:57 pdurbin dotplus: right. makes sense. sorry for the delayed response. netsplits
16:59 dotplus meh. irc is often async for me anyway:)
16:59 bene i think disk bandwidth is both cheaper and more reliable than network bandwidth in most cases
17:01 bene and when you add 50ms of latency, BDP for traditional file transfer methods becomes even more of a limiting factor
17:20 pdurbin bene: I assume you mean https://en.wikipedia.org/wiki/Bandwidth-delay_product
17:27 bene yes
17:28 bene not boogie down productions
17:28 bene although now i want to listen to some KRS One
17:28 * pdurbin boogies down
17:49 bene yeah, i switched over from the rolling stones to return of the boom bap
18:43 dotplus bene: definitely. but you still gotta have it
18:44 sivoais joined #crimsonfu
19:00 bene joined #crimsonfu
19:19 sivoais joined #crimsonfu
19:28 JoeJulian joined #crimsonfu
20:04 arcanine joined #crimsonfu
22:35 Azgarech1 joined #crimsonfu

| Channels | #crimsonfu index | Today | | Search | Google Search | Plain-Text | summary

crimsonfu - sysadmins who code