Perl 6 - the future is here, just unevenly distributed

IRC log for #opentreeoflife, 2015-06-09

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:58 jar286 joined #opentreeoflife
10:52 jar286 joined #opentreeoflife
12:06 kcranstn joined #opentreeoflife
12:45 kcranstn jar286?
12:45 jar286 hi
12:45 kcranstn just realized that I need to activate the SSL
12:45 kcranstn cert
12:46 jar286 oh.
12:46 jar286 and i had remembered, then forgot
12:46 jar286 I didn’t do it before, jimallman did…
12:47 jar286 a matter of downloading the right files and putting them in the right place on the 4 servers
12:47 jar286 I have about 10 minutes now
12:47 jar286 the site seems to be still working
12:48 kcranstn expires tomorrow
12:48 jar286 not sure but I think you will need to download the files (I can check though)
12:48 kcranstn I need a CSR code
12:49 jar286 “Please visit our Knowledgebase to find instructions on how to create a CSR and install the certificate on your server.”
12:49 jar286 http://namecheap.simplekb.com/kb.show?show=category&categoryid=14
12:51 jar286 vague memories
12:51 jar286 “a CSR generated on your web server for your domain name”
12:53 jar286 fortunately i’m in the habit of taking notes, to compensate for the sieve-like nature of my mental memory
12:53 kcranstn yay
12:53 kcranstn so CSR generation happens on the AWS server?
12:54 jar286 yes, using the openssl command
12:56 jar286 top google hit: http://info.ssl.com/article.aspx?id=10082    (going to google only to confirm the notes that I already have)
12:56 kcranstn on ot14?
12:56 jar286 yes, ot14
12:56 kcranstn also https://www.namecheap.com/support/knowledgebase/article.aspx/9446/0/apache-opensslmodsslnginx
12:56 jar286 (although could be done on ot10, ot18, or ot20)
12:57 kcranstn user is opentree?
12:57 jar286 no, user is admin (same ssh key though)
12:57 jar286 are you going to attempt it?  I’ll paste in my notes
12:57 kcranstn I don’t have the public key
12:58 jar286 I guess we have a choice, we can use the old key, or make a new one
12:58 jar286 I don’t see any reason not to make a new one, other than cache delays
12:59 jar286 the public key is somewhere under /etc/ssl/
13:00 kcranstn which I can’t access
13:00 jar286 if you’re logged in as admin, do ‘sudo bash’
13:00 kcranstn I can’t log in as admin
13:00 kcranstn Permission denied (publickey).
13:01 jar286 oh, *that* key
13:01 jar286 right, jim and i have it , I don’t know if I ever sent it to you
13:02 jar286 hmm… is gmail.com to gmail.com a secure channel?
13:03 kcranstn I don’t know
13:03 jar286 do you have a PGP key?
13:03 kcranstn yes
13:04 jar286 is it on the MIT keyserver?
13:04 kcranstn I doubt it
13:04 jar286 or on the web?
13:04 jar286 the public key of course
13:04 jar286 I can send you PGP email
13:05 jar286 or, you can email it to me in gmail, that’s good enough for me.
13:05 jar286 easiest of course is to upload it to pgp.mit.edu
13:05 jar286 back in 2 mins
13:14 kcranstn on MIT key server
13:15 jar286 here’s a much lower tech approach: I just did scp -p production.pem opentree@ot16:for-kcranstn
13:16 jar286 delete it when you’ve copied it locally
13:16 jar286 i’m off, talk to you around 11
13:16 kcranstn ok
13:33 jimallman kcranstn: i’m late to the party. anything i can do to help?
13:33 kcranstn I think I have an old opentree.pem file
13:35 jimallman let me see if i can find it on ot16… or i can upload my latest using scp (as jar286 described above)
13:36 kcranstn can’t log into servers with old pem file
13:36 * jimallman nods
13:39 jimallman i don’t recall this key changing… i’m trying some of the older servers...
13:44 kcranstn joined #opentreeoflife
13:46 kcranstn working for you on old servers?
13:48 jimallman yes, with the same opentree.pem
13:49 jimallman i’m looking for a secure way to send mine… i haven’t used gmail for this in the past.
13:50 jimallman how do you feel about dropbox with an obscure URL (sent via email)? then we can trash it as soon as you have it
13:51 jimallman for that matter, i suppose i can make a copy in “public space” on an ot# server, with an obscure filename, and do the same
13:52 kcranstn joined #opentreeoflife
13:53 jimallman re-sending last two messages, in case you didn’t see these:
13:53 jimallman how do you feel about dropbox with an obscure URL (sent via email)? then we can trash it as soon as you have it
13:53 kcranstn got them
13:53 jimallman dropbox?
13:53 kcranstn what are the first 4 letters of the opentree.pem that you have?
13:53 jimallman MIIE
13:54 kcranstn ok, its the same file. not sure why I am getting permission errors
13:54 kcranstn ssh opentree@ot16.opentreeoflife.org -i ~/.ssh/opentree/opentree.pem
13:54 jimallman ah, ok. are you using .ssh/config to define access? or cmd-line options?
13:54 jimallman got it
13:54 kcranstn I’ve tried both
13:54 kcranstn Permission denied (publickey).
13:55 jimallman let me try the same…
13:55 kcranstn host ot14
13:55 kcranstn hostname ot14.opentreeoflife.org
13:55 kcranstn user opentree
13:55 kcranstn IdentityFile ~/.ssh/opentree/opentree.pem
13:55 kcranstn on ot14 and ot 16
13:55 kcranstn gotta transport to nescent. be back in ~20 minutes
13:56 jimallman hm, your one-liner above worked for me
14:13 kcranstn joined #opentreeoflife
14:17 kcranstn back
14:17 kcranstn something strange going on
14:42 jimallman hi, back now
14:43 jimallman your one-liner (above) works for me: ssh opentree@ot16.opentreeoflife.org -i ~/.ssh/opentree/opentree.pem
14:43 jimallman kcranstn: ^
14:43 kcranstn hmmm
14:44 jimallman any chance you have a modified HOSTS file? ‘ping ot16.opentreeoflife.org’ should respond with this IP address: 54.190.10.24
14:49 jimallman kcranstn: if you’re seeing the expected IP address, we might try the -v option for ssh: ssh -v …
14:49 kcranstn PING ot16.opentreeoflife.org (54.190.10.24): 56 data bytes
14:50 jimallman ok, so it’s the right machine…
14:51 jimallman and this path is correct to your local key? ~/.ssh/opentree/opentree.pem
14:51 kcranstn I can log into ot16 but not ot14
14:51 jimallman same here~
14:52 jimallman ah, ot14 expects the production.pem
14:52 kcranstn I get it
14:52 jimallman so we’re actually in good shape, if what you need is on ot16
14:52 kcranstn *goes back to refresh memory about what she needs*
14:53 jar286 joined #opentreeoflife
14:53 jimallman according to my ssh config, these servers expect production.pem: ot14, ot18, ot20
14:53 jimallman (all others expect opentree.pem)
14:56 jar286 I think ot18 is kaput
14:56 jar286 if kcranstn can log in to ot16, she can pick up production.pem from there
14:56 jar286 and then use it with ot14
14:58 * jimallman nods
15:00 kcranstn I am on 14
15:00 kcranstn generating CSR
15:04 kcranstn done
15:09 kcranstn ok, awaiting confirmation email
15:17 kcranstn now awaiting certificate
15:38 kcranstn have zip file with many *.crt files
15:41 kcranstn suggestion for instructions?
15:41 kcranstn (finding things that don’t seem to match the current setup)
16:00 jimallman hi kcranstn, catching up now...
16:01 jimallman you probably got the entire chain of certs, down to a trusted root cert. hopefully this is largely the same as what we already use, since deployment will copy them all (iirc) and point to them in apache config.
16:01 kcranstn AddTrustExternalCARoot.crt
16:01 kcranstn COMODORSAAddTrustCA.crt
16:01 kcranstn COMODORSADomainValidationSecureServerCA.crt
16:01 kcranstn STAR_opentreeoflife_org.crt
16:02 jimallman see related notes here:  https://github.com/OpenTreeOfLife/opentree/blob/b8fbcf7dd03c54f07ee1f6e06fd965b07c23be7a/deploy/setup/ssl-certs/README.md
16:04 jimallman note that we sort-of ignore their chain (i forgot about this) and build our own all-inclusive STAR_opentreeoflife_org.pem instead.
16:04 kcranstn ok
16:08 jimallman assuming your new STAR_opentreeoflife_org.pem is different from the old one, we should store it in the repo, replacing https://github.com/OpenTreeOfLife/opentree/blob/b8fbcf7dd03c54f07ee1f6e06fd965b07c23be7a/deploy/setup/ssl-certs/STAR_opentreeoflife_org.pem
17:14 kcranstn joined #opentreeoflife
18:10 kcranstn so I can overwrite the STAR_opentreeoflife_org.crt file in setup/ssl-certs/
19:10 mtholder joined #opentreeoflife
19:11 mtholder kcranstn, do you know how much memory our "big" instances on amazon have?
19:11 kcranstn just a sec
19:12 kcranstn https://aws.amazon.com/ec2/instance-types/
19:12 kcranstn m3.large
19:12 mtholder thanks
19:13 kcranstn ot19 and ot20
19:13 kcranstn 14, 16, 17 are m3.medium
19:13 mtholder I was just playing around with http://chokkan.org/software/simstring/ for fuzzy matching of strings.
19:14 mtholder that + in memory OTT may be very fast.
19:15 kcranstn nice
19:25 kcranstn are you thinking of a replacement for taxomachine?
19:25 kcranstn or about the make-based pipeline?
19:28 jar286 joined #opentreeoflife
19:32 mtholder sorry, didn't see your query....
19:32 mtholder more a replacement for oti.
19:33 mtholder we have some long standing bugs there. (and I think that it does fuzzy matching).
19:33 mtholder if the performance is good, then we might be able to replace taxomachine for the sake of a smaller code base.
19:33 mtholder kcranstn ^
19:34 kcranstn ah, I didn’t realize the fuzzy matching was in oti, not taxomachine
19:34 kcranstn makes more sense
19:34 kcranstn agree about oti
19:34 mtholder I think that it is in both.
19:39 jimallman joined #opentreeoflife
19:40 kcranstn jimallman and jar286 - I’ve run that resolve.sh script with the new crt file
19:40 kcranstn not sure what else needs to be done
19:41 jimallman hi kcranstn.. assuming that the files have the expected names, i believe you’re ready to deploy.
19:42 kcranstn scary
19:45 jar286 deploy ot10 and ot16 (dev) first.
19:46 kcranstn you say that in a casual way that seems to assume I have done this before
19:48 jar286 you probably have
19:48 jar286 it’s designed to be pretty easy
19:49 jar286 update your deployed-systems and opentree repos
19:49 jar286 cd opentree/deploy
19:49 kcranstn yup
19:49 jar286 ./push.sh -c ../../deployed-systems/development/devtree.config
19:50 jar286 that should be all
19:50 jar286 I’m investigating how you can check the SSL handshake to see whether the new stuff is seen
19:51 jar286 (looking at http://blog.taddong.com/2010/04/manual-verification-of-ssltls.html )
19:51 jimallman jar286: i’m reviewing the deployment tools, and i can’t see when/how the new .crt and .pem files will be pushed to the server. is this a preliminary (manual) step when setting up a new machine?
19:52 kcranstn that’s what the docs say
19:52 jar286 I would expect it to be manual...
19:52 kcranstn crap, I only copied to production
19:53 kcranstn so my redeployment on dev did nothing
19:55 kcranstn I should also put those in the opentree repo
19:55 jar286 that’s fine, just need to try it on dev first to make sure it all works & steps are understood, then try production
19:55 jar286 clobbering files on production has no effect until apache restarts there
19:58 kcranstn https://github.com/OpenTreeOfLife/opentree/pull/654
19:59 kcranstn do I need to copy the pem file to ot10 and ot16?
19:59 jar286 dev server config gets development branch of opentree repo.  should push that commit to development branch on github.
20:00 jar286 yes
20:00 jar286 whatever the instructions say…
20:02 kcranstn https://github.com/OpenTreeOfLife/opentree/pull/655
20:02 kcranstn the instructions say “put the required files on the server"
20:03 kcranstn not clear which servers need the SSL certs
20:03 jimallman we (I, anyway) usually skip the PR and just merge the new branch to development.
20:04 jimallman for testing, i mean.
20:04 kcranstn ok
20:05 jimallman regarding which servers, i’m pretty sure that *all* would need these certs, yes?
20:05 jimallman since all are responding to secure traffic on *.opentreeoflife.org
20:05 kcranstn ot10 and ot16 should use opentree.pem to connect, right?
20:06 kcranstn getting a permission denied error trying to scp to opentree@ot16.opentreeoflife.org:/etc/ssl/certs/opentree/
20:07 jimallman try as admin?
20:07 jimallman worst case, we’ll need to scp to ~, then log in as admin and use sudo mv
20:08 jar286 what he says.
20:08 jar286 only root can write to certs/ , and only admin can sudo to root
20:09 kcranstn I don’t know the sudo pwd
20:09 kcranstn the file is in /home/opentree on ot16
20:09 jimallman hm, user admin can sudo w/o password i think
20:09 jimallman ssh admin@otBLAH….
20:10 * jimallman just confirmed on ot10
20:11 kcranstn done
20:13 kcranstn on ot16 and ot10
20:14 kcranstn can someone else merge that PR?
20:14 jar286 ok
20:15 * jimallman is wondering: 654 or 655?
20:15 jar286 done
20:16 jar286 I ignored the one that was closed, and did the merge to development
20:16 jar286 kcranst, how did you tell github that the merge was supposed to be to development instead of master?
20:16 jar286 kcranstn that is
20:17 kcranstn picked development from the pulldown menu
20:17 jar286 ohh… I never use the GUI
20:17 jar286 oh wait, you mean on github
20:17 kcranstn yes
20:18 jimallman yes, you can choose ‘base’ and ‘compare’ branches in the GitHub UI
20:18 jar286 what meant was, I never noticed that there was a pulldown, will look for it
20:19 jimallman in the compare view, for example https://github.com/OpenTreeOfLife/opentree/compare/master...profiles-and-collections
20:19 jimallman (prelim to creating a PR)
20:20 kcranstn redeployed development
20:21 jar286 not broken, at least
20:22 kcranstn yay
20:22 kcranstn how do we judge whether it is ok to push to production?
20:22 kcranstn the SSL certs expire tomorrow
20:23 kcranstn (my bad. I renewed them but didn’t realize that I also had to do all of this activation and installation)
20:24 kcranstn commuting to home. will check back in 20 min
20:24 jar286 I sort of wanted to do something to look at the TLS handshake; that’s why I was looking at that blog post
20:24 jimallman right, reading this now...
20:25 jimallman can we simply use a (sensitive) web browser to do this?
20:25 jimallman fwiw, the resolve.sh script seems to do a pretty thorough job of checking cert-chain integrity
20:26 jar286 checking ff to see if it gives handshake details
20:28 jar286 hmm, Firefox can't establish a connection to the server at devtree.opentreeoflife.org
20:29 jimallman indeed, it’s not running? i’ll check for obvious errors (apache, etc)
20:29 jar286 apache not running on ot16
20:30 jar286 RSA server certificate wildcard CommonName (CN) `*.opentreeoflife.org' does NOT match server name!?
20:32 jar286 I see the .pem file but not the .crt file
20:32 jar286 the .crt file is the old one, from last June
20:33 jimallman hm, maybe that’s the reason for the mismatch
20:33 jar286 yes I would expect so.
20:33 jar286 maybe it’s on ot14, looking
20:34 jar286 nope
20:35 jimallman should be on the latest development branch: https://github.com/OpenTreeOfLife/opentree/tree/development/deploy/setup/ssl-certs
20:35 jar286 Not sure what the .pem and .crt are.  Is the .pem the public key, and the .crt the signed cert (that vouches for it)?
20:35 jimallman see https://raw.githubusercontent.com/OpenTreeOfLife/opentree/development/deploy/setup/ssl-certs/STAR_opentreeoflife_org.crt
20:36 jar286 ah, I bet there’s a test -e
20:36 jimallman STAR_opentreeoflife_org.crt is the public key, and STAR_opentreeoflife_org.pem is the (combined) chain of certs to a trusted root
20:37 jar286 oh, I got it backwards
20:37 jar286 as-admin.sh copies the .pem but not the .crt
20:37 jar286 and it only does it if the .pem doesn’t already exist
20:37 jar286 so this is effectively manual setup
20:38 jar286 if we copy the .crt from the repo to ot16, and then restart apache, we should be good
20:38 jimallman hm, i’m checking to see if the initial cert is rolled into the .pem (truly all-in-one)…
20:38 jimallman we might just use the .crt as an input to .pem creation
20:39 jimallman confirmed! STAR_opentreeoflife_org.crt is incorporated as the first cert in STAR_opentreeoflife_org.pem
20:39 jimallman it’s probably not refered to in our apache config
20:39 jar286 hmm.
20:40 jar286 hmm.
20:41 jimallman confirmed: https://github.com/OpenTreeOfLife/opentree/blob/master/deploy/setup/apache-config-vhost-ssl#L22-L24
20:41 jimallman sorry for adding to the confusion
20:42 jar286 well I don’t see why apache is unhappy.  unless KC didn’t get a star cert, which seems really unlikely (it was a renewal, and you’d expect a renewal to continue what you already have)
20:44 jimallman we can convert these certs to human-readable with “openssl x509 -noout -text”
20:44 jar286 you mean the .pem file?…
20:44 jimallman i’m rusty with these commands, but following the steps in resolve.sh  https://github.com/OpenTreeOfLife/opentree/blob/master/deploy/setup/ssl-certs/resolve.sh
20:44 jimallman the main .crt file should tell its domain name and wildcard status (i guess)..?
20:47 jar286 oh I see, as-admin.sh doesn’t install the .pem, it just checks for its presence
20:47 jar286 I would think so
20:49 jar286 Subject: OU=Domain Control Validated, OU=PositiveSSL Wildcard, CN=*.opentreeoflife.org
20:49 jar286 looks fine
20:50 jar286 Validity
20:50 jar286 Not Before: Jun 10 00:00:00 2014 GMT
20:50 jar286 Not After : Jun 10 23:59:59 2015 GMT
20:50 jar286 ???
20:50 jar286 that’s from the git repo… development branch… just updated…
20:50 jimallman ruh-roh
20:51 jar286 I don’t get it… maybe my local development branch is wrong...
20:51 jimallman first few letters of the cert?
20:51 jar286 my repo was wrong.  trying again.
20:52 jar286 Validity
20:52 jar286 Not Before: Jun  9 00:00:00 2015 GMT
20:52 jar286 Not After : Jun 10 23:59:59 2017 GMT
20:52 jar286 Subject: OU=Domain Control Validated, OU=PositiveSSL Wildcard, CN=*.opentreeoflife.org
20:52 jar286 that’s all ok, whew.
20:52 jimallman diffed certs, just in case:  https://github.com/OpenTreeOfLife/opentree/commit/958286c542ea16006b6ba7ccb4d28aeae85e7ff0
20:53 jar286 might want to diff the text versions… but I really don’t think there’s a cert problem
20:53 jimallman but yes, this all looks ok. so what’s the “mismatch”. perhaps the server reckons its own domain name wrong?
20:54 jar286 the server doesn’t know its own domain name
20:55 jimallman or (based on some quick googling) this is reporting a mismatch between public and the private .key file
20:57 jar286 ahh… karen created the CSR… had to fill in a bunch of info… maybe there was a problem with that
20:57 jimallman OR it’s trying to match ServerName specified in apache config
21:00 jar286 just compared metadata in old & new certs, looks same
21:00 jar286 the question is what changed between old and new
21:00 jimallman ok. i don’t see any instance of ServerName in our apache config files on ot16, so that’s apparently a red herring
21:00 jimallman what about the private .key file?
21:00 jimallman is this new or changed?
21:01 jimallman from apache config:    SSLCertificateKeyFile /etc/ssl/private/opentreeoflife.org.key
21:01 jar286 the private key should be unchanged (that’s what I advised kc and that’s what the file dates say)
21:01 jimallman ok
21:04 jar286 there is no ServerName in ot16 apache ssl config
21:04 jar286 but there is on ot20…
21:05 jar286 hmm, something about having two versions of the apache configs, wonder if that could be it… but no, that is not new
21:06 jar286 I didn’t update all servers to debian 8.  but again I can’t see how that could be related
21:06 jar286 since we’re comparing ot16 before the recent PR to ot16 after the recent PR
21:07 jar286 I guess if I were trying to be thorough I would reinstate the old .pem file on ot16 and see if that fixes the problem, to make absolutely sure it has something to do with the .pem file
21:08 jimallman i like it!
21:08 jimallman meanwhile, i’m trying to check for consistency between the .key and .crt, as described here: http://stackoverflow.com/questions/17990537/apache2-ssl-certificate-key-mismatch
21:08 jar286 ok
21:14 jar286 not the pem file.
21:15 jar286 looks like we haven’t been running the curator tool on devtree very much!
21:15 jimallman i was going to protest, but of course most of my testing is on my local “devtree”.
21:16 jimallman but i do periodically check there when building a pull request, to make sure a feature is visible/working
21:16 jar286 yes, this is puzzling
21:17 jimallman i just tried ‘apache2ctl configtest’ and it comes back clean. where did you see the “mismatch” error above?
21:17 jar286 this is on ot16, when I do apache2ctl start
21:18 jar286 and then I look in the log file
21:18 jimallman mind if i try, just to see the fireworks?
21:18 jar286 but the same error is happening all the way back to July 2014
21:18 jar286 sure, go ahead, can’t hurt
21:18 jimallman this might be a red herring.
21:18 jimallman the warning, i mean. i saw some threads where this was discussed, apache being paranoid when a setup is actually fine.
21:19 jar286 right
21:20 jar286 I see.  the historical error is different from what we see now.  old ssl_error.logs say “RSA server certificate wildcard CommonName (CN) `*.opentreeoflife.org' does NOT match server name!?”
21:21 jar286 now we have “SSL Library Error: 185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch”
21:21 jimallman yep, i tried some tests as described in the stackoverflow link above...
21:22 jar286 but remember we’re using the old .pem file, and it’s still failing…
21:22 jimallman it looks like a failing result, but i didn’t want to mention it until i had a better understanding…
21:22 jimallman hm, do we think we have the same .key file as before?
21:23 jimallman looks like it, the file ot16:/etc/ssl/private/opentreeoflife.org.key is dated July 17, 2014
21:24 jar286 on ot16? look at /etc/ssl/certs/private
21:24 jar286 I mean /etc/ssl/private
21:24 jar286 written July 2014
21:25 jimallman agreed, see above ^
21:25 jimallman so is it possible this key wasn’t used to make the new certs?
21:25 jar286 it’s possible, but how would that explain the server failing now?
21:25 jimallman unlikely, i realize. and again, i need to do some more reading to understand this test. here’s the gist:  https://gist.github.com/jimallman/60c7836b3b802867c809
21:26 jar286 it’s got the old .pem file, which worked earlier today as far as we know
21:26 jimallman ah right, so you’re saying the old .pem file also fails?
21:27 jimallman let’s check file permissions, i guess…
21:27 jar286 oh, you’re saying the *old* .pem doesn’t match the private key?
21:27 jimallman no, that was a check against the new .pem on ot16 (sorry for confusion)
21:28 jar286 you mean, the new .pem when it was ot16. it’s not there now
21:28 jimallman i guess so, yeah. i ran this test before you had reverted to the old .pem
21:29 jar286 hmm… file is owned by opentree instead of root, wonder if that makes a difference
21:29 jar286 (but the apache error suggests it actually looked at the field values…)
21:29 jar286 oh foo, I think I screwed up
21:29 jar286 going senile
21:30 jar286 looks like we still have the new .pem on ot16
21:30 jar286 let me try the experiment once again
21:30 jimallman cool, thanks
21:39 jar286 ok.  old .pem is installed on ot16, and ot16 apache starts up just fine.
21:40 jar286 so the problem *is* with the .pem.
21:40 jar286 and it is possible that kc created the CSR with the wrong private key.
21:41 jar286 I don’t have authority to create new csrs… I don’t think… but I can check
21:41 jimallman ok, grab me if i can help
21:42 jar286 gotta run, forgot about something. will check in later tonight
21:42 jar286 jimalman ^
22:42 jar286 joined #opentreeoflife
22:43 jar286 what I meant was, I can generate csrs, but I don’t think i can get namecheap to do the signing.
23:15 kcranstn joined #opentreeoflife
23:20 kcranstn joined #opentreeoflife
23:28 kcranstn joined #opentreeoflife

| Channels | #opentreeoflife index | Today | | Search | Google Search | Plain-Text | summary