Google's New Cloud Technologies

• google storage for developers
  • api compatible with amazon s3
• prediction api (machine learning)
• bigquery

• store your data in google's cloud
  • any format, any amount, any time
• you control access to your data
  • private, shared, public
• access via google apis or third party tools/libraries
• sample use cases
  • static content hosting, e.g. static html, images, music, video
  • backup and recovery
  • sharing
  • data storage for applications
    • e.g. used as storage backend for android, appengine, cloud based apps
  • storage for computation
    • bigquery, prediction api

• high performance and scalability
  • backed by google infrastructure
• strong security and privacy
  • control access to your data
• easy to use
  • get started fast with google and third party tools

• restful api
  • get, put, post, head, delete
  • resources identified by uri
  • compatible with s3
• buckets -- flat containers
• objects
  • any type
  • size: 100 gb / object
• access control for google accounts
  • for individuals and groups
• two ways to authenticate requests
  • sign request using access keys
  • ???

• objects of any type and 100GB/object
• unlimited numbers of objects, 1000s of buckets
• all data replicated to multiple US data centers
• leveraging google's worldwide network for data delivery
• only you can use bucket names with your domain names
• read-your-writes data consistency
• range get

• key-based authentication
• authenticated downloads from a browser

• go to http://code.google.com for basic info
• http://code.google.com/apis/storage (currently in preview mode)
  • getting started guide, docs, etc.
  • can sign up for an account
• command line tool available -- gsutil -- low-level access from the command line, scripting
• google storage manager -- web-based tool for managing google storage

• google bigquery
• google prediction api
• google.org -- imagery
• google patents
• panoramio
• picnik
• vmware
• US Navy
• theguardian
• socialwok
• xylabs
• etc.

• storage: 0.17/gb/month
• also costs for up/downloads
• similar pricing to amazon s3
• preview in US
  • 100GB free storage and network from google per account
  • sign up for waitlist at http://code.google.com/apis/storage
• non-US preview available on case-by-case basis

• google's sophisticated machine learning technology
• available as an on-demand restful http web service
• provide a bit of text and "train" the algorithm in the service to predict outcomes based on patterns
• simple example: language detection
  • provide series of examples of english, spanish, french, etc. and train the prediction api to recognize the language
• endless number of applications
  • customer sentiment
  • transaction risk
  • etc

• predict and respond to emails in an automated way

• three step process
  • upload training data to google storage
  • build a model from your data
  • make new predictions

• POST prediciton/v1.1/training?data=mybucket...
• can respond when the prediction engine is ready and gives an estimate of accuracy

• apply the trained model to make predictions on new data
• returns json data
• includes scores indicating confidence of prediction

• data
  • input features: numeric or unstructured text
  • output: up to hundreds of discrete categories
• Training
  • many machine learning techniques

• cuisine predictor
• spreadsheet of type of food (e.g. mexican, italian, french) and food description as training data
• upload spreadsheet to google data storage
• kick off training process, then can check to see if it's done
• pretty accurate predictions even on a limited training dataset

• also resides on top of google storage
• can have large amounts of data that you can quickly analyze using sql-like language
• fast, simple to use

• interative tools
• spam
• trends detection
• web dashboards
• network optimization

• scalable to billions of rows
• fast--response in seconds
• simple--queries in sql
• webservice based--rest, json

• upload to google storage
• call bigquery service to import raw data into bigquery table
• perform sql queries on table

• google accounts
• oauth
• https

• bigquery shell utility available -- just type sql commands and get responses back
• can tie in a google spreadsheet and point it to a bigquery table

• not really an advocacy position
• still in engineering, but work a lot more with users directly
• go out to companies to help them be successful

• lots of different definitions
• pyramid of (bottom up):
  • infrastructure as a service
    • joyent, rackspace, vmware, amazon web services
    • provides cooling, power, networking
  • application platform as a service
    • GAE falls in this category
    • tools to build apps
  • software as a service
    • google docs, etc.

• easy to build
• easy to maintain
• easy to scale
  • appengine resides in google's overall infrastructure so will scale up as needed
• started with only python
• with java support, opened the doors for java enterprise developers

• launched in 2008
• 250,000 developers
• 100,000+ apps
• 500M+ daily pageviews
  • 19,000 queries per second -- has almost doubled since January

• best buy
• socialwok
• xylabs
• ebay
• android developer challenge
• forbes
• buddypoke
  • 62 million users
• gigya
  • do social integration for large media events (movie launches, sports events) -- huge spikes in traffic so GAE just handles it
• ubisoft
• google lab
• ilike
• walk score
• gigapan
• others
• point here is it's very easy to drop specific apps on GAE without running litearlly everything on GAE
• very popular among social networking apps because of easy scalability

• managing everything is hard
• diy hosting means hidden costs
  • idle capacity
  • software patches & upgrades
  • license fees
• "cloud development in a box"

• collection of services
  • memcache, datastore, url fetch, mail, xmpp, task queue, images, blobstore, user service
• ensuring portability -- follows java standards
  • servlets -> webapp container
  • jdo/jpa -> datasource api
  • java.net.URL -> URL fetch
  • javax.mail -> Mail API
  • javax.cache -> memcache
• extended language support through jvm
  • java, scala, jruby, groovy, quercus (php), javascript (rhino)
• always free to get started
• liberal quotas for free applications
  • 5M pageviews/month
  • 6.5 CPU hours/day

• download and install SDK
  • Eclipse plugin also available
• build app and then deploy to the public GAE servers
• app engine dashboard
• app engine health history
  • shows status of each service individually across GAE as a whole

• google app engine launcher for python
• sdk console
  • local version of the app engine dashboard
• google plugin for eclipse
  • wizard for building new app engine apps
  • can run the entire gae environment locally within eclipse
  • easy deployment to app engine servers
  • in process of building a new version of this with more features

• aggressive schedule for providing new features
• may 2010 -- app engine for business announced

• multi-tenant apps with namespace API
• high performance image serving
• openid/oauth integration
• custom error pages
• increased quotas
• app.yaml now usable in java apps
• can pause task queues
• dashboard graphs now show 30 days
• more -- see http://googleappengine.blogpost.com

• go to http://code.google.com/appengine for basic information
• can link gmail account to appengine
• download sdk and optionally the eclipse plugin
• http://appengine.google.com is the url for the actual service
  • log in here with google account and create new applications

• demoing eclipse plugin
• can create a new Google Web Application, optionally with GWT
• projects follow the typical java webapp structure
• before deployment, can test/debug locally just like any Java project in eclipse
• even the datastore is available locally for development/testing
• new features tend to be introduced in python first, then java gets them later
• to deploy, right click the project, choose "google," then deploy
  • this brings up a window where you put in your application ID and version, then uploads to the GAE servers
• can log into GAE dashboard and configure billing with maximum charges if your app will exceed the free quotas
• can use your own custom domains, this ties into google apps
• can assign additional developers to GAE applications by email address
• can deploy new versions of applications and keep the old ones as well, can toggle between versions and choose one as default

• GAE for Business
• same scalable cloud hosting platform, but designed for the enterprise
• not production quite yet
• enterprise application management
  • centralized domain console (preview available today)
• enterprise reliability and support
  • 99.9% SLA
  • direct support
    • tickets tracked, phone support, etc.
• hosted SQL (preview available today)
  • managed relational sql database in the cloud
  • doesn't replace the datastore--available in addition to the datastore
• ssl on your domain
  • current core product doesn't offer this
• secure by default
  • integrated single signon
• pricing that makes sense
  • apps cost $8/user, up to a max of $1000 per month

• GAE for Business
• Google Apps for Business
• Google Apps Marketplace
• Firewall tunneling technology available (Secure Data Connector)

• enterprise admin console (preview)
• direct support (preview)
• hosted sql (limited release q4 2010)
• sla (q4 2010)
• enterprise billing (q4 2010)
• custom domain ssl (2010 - 2011)

• can run this all locally in eclipse
• demo of spring mvc travel app running on GAE with the SQL database
  • have to explicitly enable sessions
  • had to disable flow-managed persistence

• http://bit.ly/gae4btt

If you currently contact me on IM at mpwoodward at gmail dot com, please replace that with matt at mattwoodward dot com. I'll run both for a while in case anyone has any issues.

Thanks!

After the initial setup was complete and I switched my @mattwoodward.com account to Google Apps for Domains, I added that account to the Droid, but in the sync settings screen only Contacts and Mail were showing up as options. There was no checkbox to have it sync the calendar. I deleted and re-added the account a couple of times to no avail.

I started thinking maybe the problem was that I used my @gmail.com account as the "primary" account, though I'm not sure why it wouldn't support multiple calendars. Maybe it does and this was just a glitch. At any rate, I reinitialized the phone to factory defaults since that's the only way you can delete whatever account you chose as your primary account when you first set up the phone. Then when I booted the phone and it went back into the initial setup mode, I used my @mattwoodward.com account and now the calendar sync shows up.

I wasn't using my @gmail.com account for anything anyway and since everything was already in my Google account as far as contacts, etc. are concerned, this wasn't a big deal. I do need to reinstall the applications I downloaded but there were only about three of those to worry about.

More on the Droid later--loving it so far even with this little glitch!

I've been rather vocal about "getting off Google" in the past, and it's a battle I've been winning with my email for a few years now. But today I got my Droid (more on that later), and I thought long and hard before even ordering one because I knew what this would entail.

Obviously the Droid is using the Andoird OS, which means (surprise!) it's Google-centric. Google offers a huge amount of convenience, and the Android OS itself is absolutely spectacular. Temptation rears its head.

I've been researching mobile phones like crazy over the last few months, and when it comes to "freedom" there really are no great choices. The Neo FreeRunner keeps showing promise, but ultimately you're still going to be at the mercy of a cellular carrier, and picking the least amongst evils there isn't easy.

The iPhone is a COMPLETE non-starter for me. AT&T coverage is horrendously bad, particularly where I live, and there is no way I'm diving into Apple's cesspool of control. No offense to iPhone owners; if you're happy with it, great! But it's absolutely not for me.

I've had Verizon for years and been nothing but happy for years, so I've been suffering the crappy phones until the Droid came along. But the Droid has the Google problem I've been avoiding for so long, so what to do?

Well, on the Google fight I give up. I'm keeping my mattwoodward.com email address, but I changed that domain over to Google Apps for Domains today. That way I can get this all working seamlessly with my phone, and I guess I just have to trust that Google isn't all THAT evil. I'll keep using Scroogle for my searches when I'm at my desk though.

I guess the bottom line of this post is if you want the benefit of living in the cloud, you gotta trust somebody. So I suggest doing your research and choosing an option that works for you on all levels. Just make sure to have an out if the solution you align yourself with turns up the evil dial at a later date.