Tuesday, June 29, 2010

Tomcat 7 Released



Apache Tomcat 7.0, the latest version of the popular open source Java Web server, is available Tuesday from the Apache Software Foundation.


The upgrade constitutes the first major release of the project since 2006. To ease Web framework integration, version 7.0 implements the Java Servlet 3.0, JavaServer Pages 2.2 and Expression Language 2.2 specifications.




Big changes are the support of Servlet 3.0, JSP 2.2, and EL 2.2, but from what I've read there are lots of nice fixes and performance improvements as well. I'm about to set up two new servers so this is great timing!


Full details at https://blogs.apache.org/tomcat/entry/apache_tomcat_7_released


Monday, June 28, 2010

Has Oracle been a disaster for Sun's open source?



As others have noted, this is a good demonstration of the fact that open source projects are effectively "immortal": provided there is sufficient interest among users, they can always be forked. It should also serve as a reminder to Oracle that they are the guardians of the open source projects formerly managed by Sun, not the owners (well, they own the copyright, but that's not quite the same.) If it fails to move the projects forward in the way that many users would like, it may well be faced with more forks.


The problem is that Oracle is naturally trying to optimise its acquisition of Sun for its own shareholders, but seems to have forgotten that there are other stakeholders too: the larger open source communities that have formed around the code. That may make sense in the short term, but is undoubtedly fatal in the long term: free software cannot continue to grow and thrive without an engaged community.




This is a really insightful analysis by Glyn Moody of what the Oracle acquisition of Sun may mean for important open source technologies big and small, from things like OpenSSO and OpenSolaris to OpenOffice.org, MySQL, and even Java.


The mention of the "immortality" of free software and open source projects is really key in my mind. A lot of people wondered about the future of MySQL when Sun bought it, and many are outright panicked now that MySQL is in the hands of Oracle.


But that's what's so great about free software: none of these things are truly in Oracle's control. They belong to the communities around them. So ultimately it wouldn't be wise for Oracle to piss off the communities that made these projects what they are.


That doesn't mean Oracle won't do something terribly unwise but if they do, the code's out there and the free software community will ultimately do what's right and best for the project in the long run, even if that means forking.


musicmumblr.com


With his vocal cords soaked in whiskey and cheap cigarettes, he moves through stories of small towns and blue collar dreams of getting out one day. Oh and the girl that busted up his heart… she is in there too.



My good friend and former co-worker Brandon Culpepper just started this music blog and it's off to a great start! He has an amazingly deep and broad knowledge of music, not to mention impeccable taste, so this is definitely a blog to follow.

Make sure and follow on Twitter as well @musicmumblr

Cutting Waste by Reforming IT | The White House


While a productivity boom has transformed private sector performance over the past two decades, the federal government has almost entirely missed this transformation and now lags far behind on efficiency and service quality.  We are wasting billions of dollars a year, and more importantly are missing out on the huge productively improvements other sectors have benefited from.


Quite simply, we can’t significantly improve the efficiency and effectiveness of the federal government without fixing IT.




Some rather astounding (albeit not surprising) numbers on some of the horrendously wasteful projects in government IT. I hope this starts a similar movement in the other branches of government, because all told this would make a massive difference in government spending, and that's rather critical right now.

Using free and open source software for all projects where it's feasible would certainly help as well.

'Java 4-Ever' Expertly Skewers Movie Trailers and Microsoft





A co-worker shared this with me--absolutely brilliant. (Note that there's one brief potentially NSFW moment towards the end.)

Monday, June 14, 2010

The Latest Reason I Hate SQL Server

When is a copy of a database not a copy of a database? When it was made using SQL Server's "copy database" feature, that's when.

Having run into this before I should have known better, but I needed to make a copy of a production database for testing purposes. Normally I'd take a backup and restore it to a different database, but I noticed SQL 2005 has a "copy database" function I hadn't seen before. So I figured I'd give it a try.

Totally and utterly pointless. It looks like it copies the database until you start looking at all the fields using auto-increment IDs in your old database that magically aren't still auto-increment IDs in the "copy" of the database.

Thanks for another worthless feature Microsoft. I only use SQL Server under duress anyway, but why can't they get even such basic things like "copy database" right? Inexcusable.

Friday, June 11, 2010

Slashdot Linux Story | Adobe (Temporarily?) Kills 64-Bit Flash For Linux


"It seems that with the release of the 10.1 security patches, Adobe has, at least temporarily, killed 64-bit Flash for Linux. The statement says: 'The Flash Player 10.1 64-bit Linux beta is closed. We remain committed to delivering 64-bit support in a future release of Flash Player. No further information is available at this time. Please feel free to continue your discussions on the Flash Player 10.1 desktop forums.' The 64-bit forum has been set to read-only."



Very forward-thinking Adobe. Just awesome! You did hear that Google recently announced that their employees can no longer use the one operating system Flash actually works well on, right? You think they'll be the only ones who do that?

Doesn't really matter since Flash is on a death march at this point anyway, but just highlights what nonsense Adobe's latest "one app, any device" marketing crap is.

So long, Flash. Can't say I'll miss you.

Saturday, June 5, 2010

Open BlueDragon on Google App Engine - Sample App and Presentation PDF

This is long-delayed after cf.Objective() 2010, but here is a zip of the sample app and the presentation PDF from the presentation Peter and I gave on running CFML applications on Google App Engine.

We also gave this presentation at the CFMeetup a couple of weeks ago (you can watch the recording), and are doing it again for the Mid-Michigan CFUG on Tuesday, June 8, at 7 pm Eastern US time.

To use the sample app create a new Google App Engine project in Eclipse, then copy the unzipped files into that project.

If you need help getting up and running with GAE on Eclipse, check this blog post on the Detroit Area Adobe UG's blog, or check the GAE for Java page. The MMCFUG may also be streaming this meeting live so I'll post again with a URL if that happens.


I didn't add this to the PDF or code before I uploaded them, but both are released under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. Creative Commons License





openbd_gae_sample_app.zip
Download this file






OpenBD_on_GAE.pdf
Download this file


Thursday, June 3, 2010

Open Source Bridge - The Story of Spaz: How to Give Away Everything, Make No Money, and Still Win

Edward Finkler http://getspaz.com @funkatron http://funkatron.com

  • non-technical talk--"why I do what I do"

    • learned how I define what success and happiness are



  • Spaz is a personal project -- about 3 years old

    • twitter client -- "your special twitter friend"

      • based on original logo people thought he was making fun of mentally retarded people

      • actually is a picture of clay aiken from early american idol





  • Early 2007

    • work started on Spaz

    • had done a couple of open source PHP libraries in the past

    • had only done open source for other developers, not for end users

    • started work in RealBasic on Mac

      • also the author of LameBrain, wrapper for LAME



    • started getting interested in twitter API

    • not many other Mac clients for twitter at the time

    • article written up in ars technica about it

    • pownce came along about the same time

      • came out with desktop client, written in apollo (now AIR)



    • RealBasic wasn't set up to make it easy to make it look non-standard or skin things

    • silverlight was also being released about this time, also Java FX

    • particularly interesting aspect of AIR was that you could make apps in HTML/CSS/JS in addition to Flash

      • looked at using Flex, but not a fan of monotheistic technology

      • Flex community is a lot like the MS development community--one official source of information, not a lot of community sharing going on

        • quite a contrast with something like PHP where everyone shares in a bunch of different ways



      • didn't like having to go through the official channels to try and figure things out

        • really hard to simply search for things on the internet and find examples with Flex





    • twitter was really flaky at this point--had to write the code very defensively

    • learned a ton by digging into this--crash course in javascript

    • driven because he was interested in it and enjoyed working on it

      • "I liked what it did, and I used it"





  • November 2007

    • Adobe had AIR Developer Derby in mid-2007

      • submitted Spaz on a whim

      • got a call in October 2007 from Adobe saying he won

        • got to go to Adobe MAX conference, lots of PR around it





    • Adobe didn't give a rat's ass that it was open source

      • they liked it because it was pretty

      • only got about 5 submissions on the HTML side of things



    • was at the AIR booth--people asking tons of questions that frankly Adobe should have been answering

      • turned into evangelist of sorts



    • cool that Spaz got known in the wider world



  • 2008-2009

    • started to get emails criticizing the name--"why do you hate the disabled?"

      • happened quite a bit



    • tons of twitter clients released in 2008

      • lots of them written in AIR

      • a few clients were a lot better with marketing themselves -- biggest example is TweetDeck

        • people spending full-time on it, went after venture capital, etc.

        • cool if you want a twitter client that fills your whole screen, but wasn't what he wanted to do





    • seeing success of tweetdeck was hard


    • the thing I like about open source is creating something and getting ideas from people, hearing about how they're using the product, etc.

    • lots of end users were saying stupid things like "Spaz sucks"

      • big disconnect--creating products for end users is pretty different from creating developer libraries

      • people were dismissive and mean -- made it not fun

      • eventually learned to respond with humor to baseless criticisms

        • helped to not get as upset about things





    • started seeing companies that were hanging off twitter and providing services that twitter doesn't provide

      • twitter has started to stomp out all the ancillary business around twitter



    • started getting offers from companies around spaz

      • people offering money to add specific services to spaz, e.g. twitpic



    • wanted to continue to provide an open source, transparent twitter client

      • in one case took money as a donation to offset overall development time into spaz

      • wasn't interesting in doing this in general



    • this underlined a specific way that he wasn't willing or able to compete with other clients that were out there

      • would sacrifice transparency around the project

      • realized how screwed up and evil tech journalism is--fundamentally wrong

        • actively manipulate how people perceive things, don't give coverage to things that don't play their game





    • started to realize he couldn't compete with other clients because of all the extraneous junk that goes on



  • palm webos -- jan 2009 - june 2009

    • got call to see if he was interested in working on apps for the new webos

    • palm wasn't really known as an open source company

    • said up front he wanted to keep the app open source--palm didn't object

    • web development as a rule is pretty open, but palm didn't really have this understanding in their DNA

    • had to sign really strict NDA

    • palm was rushing things out, documentation was terrible

    • went to a developer event at palm--could talk about things, but couldn't share the code he had written--against the NDA

    • this six months sucked because couldn't make something and share it with other people

    • palm didn't tell him another company was also working on a twitter app for webos

      • found out from someone else--no information sharing going on

      • bunch of secret stuff--wasn't fun



    • didn't get a development device--emulator ran 3X faster than the actual phone

      • didn't write specifically to their proprietary platform so it ran slow

      • another twitter app that did write to the proprietary bits ran faster



    • got email from palm asking not to release source code

      • had told them first thing he was going to open source it

      • lot of wtf moments

      • decided he would never work for another company like this under an NDA

      • palm didn't get the sharing culture--they get it much better now



    • two guys from ajaxian got hired to better interface with the development community

      • immediate changes for the better--hope HP doesn't screw it up



    • reminded me of lessons i should have learned the first time around

    • friend recommended book -- "ignore everybody and 39 other keys to creativity"

      • the thing that you really like doing, if you rely on it for income, it'll become less enjoyable and it will change how you interact

      • when spaz became something that was being defined by standard definitions of success, the process became less enjoyable

      • had to redefine what success was





  • july 2009

    • spaz statement of purpose

      • can keep going back to this and referencing it to remind himself of why he's doing what he's doing



    • spaz was built for the sake of building it. it is not a means to an end. however, creating it has had several good consequences.

      • point of it was to build something and build something good



    • spaz demonstrates that making things is good, and sharing how you make them is better.

      • creative endeavor in and of itself is good

      • sharing is what makes me happy



    • spaz is a necessary counter to closed, hidden technologies. spaz must always be open.

      • source code must always be open and have to be transparent about the intentions of the project

      • important that we have options, even if they aren't the most popular or defeat other projects



    • the value of spaz does not lie in the judgments of others, but in the process of building it, and the enjoyment derived by those who use it.

    • we welcome anyone who wishes to participate in the spaz project with open arms, as long as they understand and respect the purposes of the project.

    • the spaz project values clear and open communication between participants.

    • having the statement of purpose helped keep things going when it was frustrating



  • 2nd half of 2009

    • not a lot happened

    • worked on getting webos version better

    • started on spazcore to share common things between all the different platforms

    • discovered didn't have time to try to keep multiple versions going

    • started working on community building



  • diversity statement

    • women are under-represented -- 10-15% in IT as a whole, open source it's more like 2-3%

    • worked with a group called phpwomen

    • largely lifted from python's diversity statement



  • rest of 2010 -- cultivating a community

    • initially was using google code for everything

    • have found that github works better

      • a lot of people use git (particularly js people), more social aspects oriented around git (easier to fork, etc.)

      • potentially attracts more people




    • tenderapp.com and lighthouseapp.com

      • tender oriented towards end-user support -- offer free accounts for open source projects

      • lighthouse more oriented towards developer issue tracking, setting milestones, etc. -- also have free accounts for open source projects



    • if you only have roadmap, etc. in your head it's very hard to get people to help you

      • need to break things down into issues of various size

      • schedule people's time for them in the sense that you give them bite-sized tasks



    • hackathon

      • great way to get more people involved

      • helps to have a schedule

      • done online with irc



    • have someone who's more or less dedicated to doing community related tasks



iizip - Hacking Together Your Own Dropbox

Ben Dechran, Sputnik Agency (Australia)
  • works for a marketing agency, lots of photoshop files thrown around
  • file server which works great most of the time unless ...
    • people forget to put the files on the file server
    • people forget to rename a file before putting it on the file server and delete the old file
    • people don't put a file on the file server and go on vacation
  • dropbox nice solution, but has issues
    • works well for small companies, small groups, unconferences, etc.
  • problem #1 -- storage space
    • in our case, needed 3TB of space
  • problem #2 -- need internet connection
    • if connection goes down, you have the files but you lose the sync
  • problem #3 -- dependent upon third party
    • what if they go down?
  • problem #4 -- all or nothing
    • every person replicates everything
    • dropbox supposedly doing selective sync
  • problem #5 -- user management
    • users can't be managed by your company's infrastructure
  • problem #6 - big bad world
    • security and privacy
    • it's encrypted, but you're sending data out to someone else's servers
  • other options
    • ifolder - open source project by novell, written in .net
      • tricky to set up
      • slow
      • packaged for windows, mac, opensuse
  • decided to roll my own--needed ...
    • local files
    • monitor for changes
    • versioning
    • conflict resolution
    • transparent -- install and forget
  • attempted solutions
    • tortoisesvn -- people forgot to check in, etc.
    • cron job to push changes to svn
      • some of the changes were hard to detect
      • if someone deletes a directory on their hard drive, deletes the metadata
    • tried git -- since it's distributed it solved some of the issues with svn
  • for synchronization
    • inotify, inotifywait -- in the linux kernel
    • notify-send -- could use this for bubble type notifications in dropbox
  • for conflict resolution
    • grep the git status
  • iizip is the combination of all these various tools
    • currently the project is a bunch of scripts; eventual goal is to have a package that would be deployed/installed on local machines
    • iizip-init -- creates git repository and iizip directories
    • the scripts mostly map to the git commands
      • e.g. send checks in locally and also pushes to remote repository
  • someone in the audience suggested looking into using couchdb in similar fashion to how ubuntu one works
  • what's in the pipeline?
    • partial checkout so not everyone has to have every file locally
      • idea of subscribing to specific directories--IMAP already does this
    • push changes from other machines
      • currently there's a cron job that runs on the local machine
      • dropbox immediately pushes changes to all machines
      • considered using xmpp to handle this, but IMAP has persistent connections and also supports directories
    • multi-user
      • currently can't run more than once on the same machine since the directories are hard-coded
  • unix tools philosophy -- many small tools linked together to achieve a larger goal
    • also makes it easier to port to other operating systems
  • unison -- would in theory do all of this so worth checking out
  • potential issues -- git doesn't track empty directories
    • if it's empty you may not care

Open Source Bridge - Relational vs. Non-Relational

Josh Berkus, PostgreSQL Experts Inc.
  • overview focused on choosing what type of database you need vs. investigation of any specific database
  • up until a few years ago there were only a handful of options for open source databases
    • most were sql/relational
    • a few written in java
    • only really exciting thing going on in the relational world is postgres vs mysql
  • today there are many more open source databases
    • as many as 5 dozen now?
  • databases for lots of different purposes, but lots of people want to lump a lot of the new ones under the "nosql movement" label
    • not so fond of this term
    • has implication that every database that doesn't have a sql interface is more or less identical
    • all non-relational databases aren't the same
    • have graph, document, key-value, distributed, hierarchical ... quite different from one another
    • some of the non-relational databases have sql interfaces
  • all relational databases aren't the same either
    • embedded, oltp, mpp, streaming, c-store ...
  • mythbusting
    • "revolutionary" is bandied about a lot but database technology goes back a long way
    • not really any new database designs in terms of fundamental architecture
    • are new implementations and combinations of design
    • last "new" thing was map-reduce in 2002
    • even couchdb is largely similar to Pick, which was created in 1965
    • when looking at new databases, don't look for revolutionary concepts, look for good implementations
    • what's going on right now is actually a renaissance of non-relational databases
  • myth: "non-relational databases are toys"
    • google - bigtable
    • amazon - dynamo
    • facebook - memcached
    • us veterans administration - pick, cache
  • myth: "relational databases will become obsolete"
    • xml databases were supposed to replace rdbms ca. 2001 -- didn't happen
    • rdbmses evolved to include xml functionality
    • one of the things we'll see out of the current non-relational innovation is that some of the implementations will hybridize with one another
  • myth: "relational databases are for when you need ACID transactions"
    • transactions != relational -- orthagonal features
    • robust transactions without relationality: berkelydb, amazon dynamo
    • sql without transactions: mysql isam, ms access
  • myth: "users are adopting nosql for web-scale performance"
    • sometimes it is, sometimes it isn't
    • performance test done by myyearbook.com
      • benchmark of key/value storage and retrieval
      • only real difference in performance is between databases that guarantee durability and those that don't
    • horizontal scalability
      • some non-relational databases are built for horizontal scalability and some aren't
      • complexity of implementation rises with the ability to scale out to a massive number of nodes
  • myth: "one ring theory of database selection"
    • "what's the best database to use?" - wrong question
    • don't need to use only one database
    • choose the db that meets your applicaton's goals, or use more than one together
    • use a hybrid
      • mysql ndb
      • postgresql hstore
      • hadoopdb
  • but what about choosing between relational and non-relational?
  • relational oltp databases
    • transactions: more mature support
    • constraints: enforce data rules absolutely
    • consistency: enforce structure 100%
    • complex reporting: keep management happy!
    • vertical scaling (but not horizontal)
  • sql vs. no sql--sql promotes ...
    • portability
    • managed changes over time (ddl)
    • multi-application access
    • many mature tools
    • but, sql is a full programming language and you have to learn how to use it
  • no sql promotes ...
    • programmers as dbas
    • no impedance mismatch
    • fast interfaces
    • fast development and deployment
    • but, man involve learning complex proprietary APIs
      • in some cases not easier than sql
  • main reason to use sql-relational databases
    • "immortal data"
    • your data has a life independent of this specific application implementation
    • important to be able to access data accurately and consistently forever
  • how DO i choose?
    • define the problem you're trying to solve
      • what is it that my application wants to do with this data and how does it want to do it?
      • i need a database for my blog, i need to add thousands of objects per second on a low-end device, etc.
    • from the definition you create you can define a database shopping list
      • define the features you ACTUALLY need
  • fit the database to the task
  • "I need a database for my blog"
    • use anything!
    • no open source databases that wouldn't support someone's individual blog, flat files would even work
  • "I need my database to unify several applications and keep them consistent"
    • high-end sql-relational database best choice for this
    • "PostgreSQL: It's not a database, it's a development platform"
  • "I need my application to be location-aware" -- geo applications
    • PostGIS - geographic relational database
    • queries across "contains" "near" "closest"
    • complex geometric map objects
    • couchdb spatial and spatialite are now available as well
  • "I need to store 1000s of event objects per second on a piece of embedded hardware"
    • db4object -- embedded key-value store
    • others: berkeleydb, redis, tokyocabinet, mongodb
    • db4object
      • project was german train system -- records data every few milliseconds
      • low-end embedded console computer
      • simple access in native programming language (java, .net)
  • "I need to access 100K objects per second over thousands of connections from the web"
    • memcached - distributed in-memory key-value non-persistent database
    • use: public website
    • typically supplements another database
    • alternatives: redis, kyototyrant, etc.
  • "i need to produce complex summary reports over 2tb of data"
    • luciddb - relational column-store database
    • for reporting and analysts
    • large quantities of data
    • complex olap and analytics
    • used along-side oltp running production apps
  • "I have 100s of govt documents I need to serve on the web and need to mine the data as cheaply as possible"
    • CouchDB
    • storing lots and lots of government documents that didn't have a consistent format (don't know content or structure)
    • used in combination with postgres to keep structured metadata
    • couchdb is also great for mobile apps
  • "I have a social application and I need to know who-knows-who-knows-who-knows-who"
    • surprisingly hard question to answer with a normal db
    • use a graph database -- neo4j is most popular open source one
    • social network website
    • 6 degrees of separation
    • "you may also like"
    • type and degrees of relationship
  • "I get 1000s of 30K bug reports per minute and I need to mine them for trends"
    • used on mozilla firefox crash reports
    • hadoop -- massively parallel datamine
    • hadoop + hbase
      • reports are then put into postgres for viewing
  • conclusion
    • database systems do better at different tasks
      • every database feature is a tradeoff
      • no database can do all things well
      • need to make tradeoff decisions when picking databases
    • relational vs non-relational doesn't matter
      • pick the database(s) for the project or task
Questions
  • how difficult to migrate from something like couchdb to postgres?
    • depends on how much data and in what form
    • since couchdb works with json that's pretty easy
    • if you want to take the document structure out of something like couchdb and put it into a relational model, the decomposition process will be complicated

Open Source Bridge - Thursday Keynote: Mayor Sam Adams on CivicApps

Sam Adams, Mayor of Portland
  • CivicApps for Greater Portland
    • interaction at last year's OSB kicked off regional effort to put Portland in the forefront of open source and open data
  • update on how Portland is leading in the area of open source
    • asked for city government to be an early client of open source work that is happening locally
    • also asked for financial assistance with open source projects
  • civicapps can change the relationship between the government and the people they serve
    • give people new tools to get at what they need
    • greater accountability
    • example: iphone app where people can take pictures of potholes--this gives the photo and the exact coordinates of the pothole
      • submitters get notification when the pothole they reported is fixed
      • idea of government reporting back directly to citizens is huge source of satisfaction
      • now the challenge is to provide this type of customer service on more platforms
  • civicapps has gotten a lot of national attention because it's regional--isn't specific to one city
    • portland isn't that unique--60K people who work in portland but don't live in portland
    • lots of other open data apps are specific to cities--not as realistic to the daily life of people
  • contests to develop apps around open data
    • area that needs a lot of work ("significant mystery to citizens") is the inner workings of the decision making process
      • e.g. city council issues being considered
      • city council agendas can have 60-150 items on them
      • mystery to citizens how the work actually gets done
      • need to make actual decision making easier to access and more understandable to citizens
  • have tried to surround these efforts around an economic development strategy
    • digital development is now one of the four targeted industries for economic assistance
    • investigated what local assets match growth trends regionally
  • portland's value proposition
    • not silicon valley and don't want to be
    • where portland can compete and win on a proportional basis is that there's a more diverse group of technology folks in portland
    • clearly is smaller, but especially around software, there's a greater diversity of different talents and approaches than anywhere else in the world
    • portland technologists are also very agile
    • passion in portland in these areas is huge--everyone involved tends to work together very well
  • how can portland's government help technologists succeed?
    • recession has hit some businesses, but software/technology has overall survived better than other industries
    • has been a lack of access to capital to bring best ideas to market
    • started portland seed business fund -- direct result of requests from software industry
    • http://mayorsamadams.com -- more details on this fund
    • portland development commission -- funded "Portland Ten"
      • bootcamp that works with 10 startups
      • access to capital, connection to very best thinkers from around the world
      • "tough love"--helping ten startups get to a million dollars a year in revenue
    • survey will be sent out to people participating in all these efforts--really need the feedback to make sure we stay on track
  • "I want you to take over the world from your base here in Portland"
  • 25 people involved with civicapps here today to help people hack on data and apps
Rick Nixon - project manager for civicapps
  • first ever Civic Code Day
  • lots of people here responsible for lots of data for the city, trimet, civicapps web site
  • role of govt: supply data
  • role of citizens: do something with the data
  • seeking to transfer data collection effort to citizens as well
    • open source the web site, number of other efforts going on

Open BlueDragon CFML Manual

The release of Open BlueDragon 1.3 not only added some amazing new features, but also marks the initial release of the Open BlueDragon CFML Manual. The intent of the OpenBD Manual is to serve as a handy reference as well as a learning tool, and not only is it available online but it's included in OpenBD itself so it's available wherever OpenBD 1.3 is installed.

There's a ton of content in the manual already but it's a work in progress, so keep your eye on the online version (as well as the version in the nightly builds) for updates. If you see any gaps in the manual and want to contribute, let us know!

Wednesday, June 2, 2010

Open Source Bridge - How Two Fools Made Themselves Indispensible From Their Basement Office

Chris Chiacchierini, Mason Bondi - Oregon College of Oriental Medicine
  • 50 staff, 250 students
  • university originally chose proprietary commercial cms for $15K
    • used for marketing site
    • only one of 5 people in the world using it
  • attended open source conference and learned about open source cmses
    • "holy shit, we just got snowed"
    • chose Joomla! for ease of use
  • didn't want to give up on open source cmses
    • went to library, asked if they'd be interested in putting up a site using oss cms
    • did this all under the radar, wanted to wait until it was too late to go back before anyone knew
  • through this a lot of institutional knowledge is being built, and since people had freedom they were starting to get more creative
    • class materials being put online
    • financial information about student loans
  • lots of people and departments were using the open source cms
  • once it was opened up to the students, it really started to take off
  • two parallel forces going
  • got call while he was on vacation -- marketing site got hacked
    • could now go to executive council with security concerns and propose switching over to the open source cms
    • was getting hacked every couple of days
  • executive council asked how much it would cost
    • couldn't believe cost would be $0
  • 4-5 departments already using the open source CMS loved it
  • what about the rest of the community?
    • every single office at the university has a stake
    • lots of departments only updated static information once a year
      • weren't interested in doing anything with the CMS, or are too busy
    • e.g. grades--wanted to put grades online
      • would be vastly more efficient than mailing, but hard to make people change
  • a project management cycle that works
    • opportunity -> needs analysis -> cost benefit analysis (proof of concept) -> project plan proposal -> approval -> design, build test -> implementation & training -> review
    • most often the people with the money aren't the ones who have the technological know-how so there's a lot of politics involved
    • a lot of the project management cycle is done in stealth mode
    • technical people tend to want to take the ball and run with it and tell everyone to get out of the way, but this doesn't work without a sponsor
    • need to get a sponsor at the executive level
      • often best to get the person who has the power to pull the rug out from under you to be the sponsor--they get part of the credit
    • need to sell the idea to a sponsor in the opportunity phase
    • still selling hard in the needs analysis phase
      • can do some of this in stealth--pick people you like and trust to get a little bit of insight into what the needs are
    • after this, set up a needs analysis meeting to ask for feedback--make it clear this is the one chance people have to offer feedback
      • better than going office to office--gets everyone in the same room
      • people see what the impact is outside of their own department
      • also lets people gripe if they want to
    • in cost/benefit phase have to make sure the proof of concept is in place
      • can't only look at cost as cost of the software--also involved is implementation, training, cost of support, etc.
    • project plan proposal--where people who gave feedback see what will actually be implemented
      • even though it won't be exactly what each person wanted, they'll see the bigger picture
    • approval -- doesn't only mean budget, but once something's approved there's no turning back
      • if you're not sure you can follow through, stop before approval
      • after approval you HAVE to deliver
    • implementation & training
      • second tier thing they implemented was putting course materials online using moodle (course management software)
      • no way could get everyone involved and get all the instructors to get all their materials online
        • students weren't excited at first either, but once they used it they won't go back
      • had a goal of rolling out three classes per quarter as a pilot
        • chose instructors that taught enough classes that most of the students would be involved
        • none of the courses are exclusively online but there are pieces of the course you can only get online
      • followed faculty around for three weeks, constant contact, asked for feedback from both instructors and students
        • important to get everyone comfortable
      • what they wanted to happen started to happen--students didn't want to deal with printed materials anymore
        • students started talking to faculty members whose courses weren't online
      • building things out slowly builds inertia, but always keep the costs in mind
      • as far as timing goes, as long as you can keep showing progress, specific deadlines aren't that important
      • if possible getting isolated focused time with people is ideal for getting things ramped up
        • can avoid ongoing and future training, also gets people thinking more creatively
    • review
      • constant cycle
      • some formal review, some informal
      • important to make incremental changes so the solution continues to meet the needs
  • before the cms project started, the IT guys were seen as a necessity
    • now people are looking at the IT department as something that can help with solutions
    • creating a lot of work for the department
  • through a successful project you can gain "street cred"
    • create a working relationship with other departments

Open Source Bridge - Being a Catalyst in Communities - The scientific facts about the open source way

Karsten Wade, Sr. Community Gardener, Red Hat
  • @quaid
  • FOSS advocate for 10 years
  • being a catalyst in communities
    • to be the catalyst in communities of customers, contributors, and partners creating better technology the open source way
      • word "in" is important
      • "in" could have been "over," or "of," or "for" ... but none of these work as well because it creates an imbalance
      • "in" suggests everyone being in it together
    • currently have a doctoral candidate in cultural anthropology at Fedora who's helping the project get better
      • learn about Fedora, learn how they do things
  • a lot of people think open source works the same way tom sawyer got his fence whitewashed
    • tom sawyer would rather go down to the swimming hole
    • friends come along, so tom acts like he LOVE painting the fence and his friends start painting the fence for him
  • model actually works a lot differently -- barn raising analogy is better
    • need to do a lot of work before you call people in to do the actual barn raising
      • build the foundation, do the preliminary work
      • doing the actually raising is a great thing to have the community do
    • can't call everyone in too early or too late
  • example of interaction in red hat community - http://lwn.net/Articles/83360
    • written by someone inside red hat projects to show what it looks like to the outside world
    • story of how fedora came about
    • early days of red hat, developer side was coming along pretty well
    • customers and people writing software for RH had issues with the speed at which new distributions came about
    • now, fedora is released every six months, and RHEL is a snapshot every 18 months with longer-term support
      • this model works well for businesses but didn't work as well for the community overall
    • always a challenge to strike a balance when a company needs to keep some information private but the project is open source
  • around fedora 2 timeframe the NSA contacted red hat to talk about a security-enhanced version of linux
    • trying to get out of vendor lockin with another vendor
      • took base OS and slapped security stuff on top, but couldn't upgrade, support, etc.
      • NSA understood the open source model and thought it would address their issues better
    • fedora core 2 added SELinux
      • gotten better over the years, was kind of rough early on
      • people turned it off originally--it's on by default and that irritated a lot of people
      • it's ok to disappoint people in OSS, but it's never OK to surprise people--you'll lose them and they won't come back
  • fedora 7
    • dropped the word "core" from the name
    • prior to fedora 7, all the build tools were behind red hat's firewall
    • in the meantime the community created a build process that was better than red hat's
    • fedora 7 and forward enabled anyone to do builds
  • virtualization
    • RHEL 5 timeframe, xen was the only technology really available at the time
    • xen is code outside the kernel so this made including it in RH more difficult
    • at the same time, kvm was being done and it was in the kernel, which made things more easy to integrate
    • libvirt abstracted the actual VM technology under the hood
      • this kind of went against the direction of the community but red hat gave a clear roadmap as to why they were doing what they were doing
    • red hat wound up buying the company that was working on kvm
  • POSSE - Professors' Open Source Summer Experience
    • week-long bootcamp to help teachers teach open source
    • people graduating don't have good experience with real-world projects by the time they graduate
    • red hat went to people who taught open source to learn about the challenges
      • wasn't a single location for educators to discuss teaching participation in open source
      • academic calendar very different from the open source calendar
      • success formula -- if educator is involved in an open source project and knows how to participate, they're much more likely to encourage their students
    • even with the projects, helps to teach students how best to interact with open source projects
    • have some real-world marketing classes to each people how to market open source projects
    • created http://teachingopensource.org
    • text book - Practical Open Source Software Explorations
    • first POSSE held in Raleigh in 2009
      • worked on mozilla, packaging, dealing with bugs, etc.
      • nothing too technically difficult but getting a feel for the process and how to get involved
    • cultural differences
      • did a POSSE in China and while in the US we're OK with more collective learning, professor not necessarily being an expert, etc. but it's different in China
    • idea of "productively lost"
    • contributing
      • e.g. wiki translations and contributions -- showing people how they can make a difference
  • communities of practice
    • "communities of practice are formed by people who engage in a process of collective learning in a shared domain of human endeavor: a tribe learning to survive, a band of artists seeking new forms of expression ... in a nutshell: communities of practice are groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly." Wenger, McDermott, Snyder - "Cultivating Communities of Practice"
    • entire body of social science work around how communities work
    • about more than number of users in the system or number of downloads
    • more about the overall health of the community
  • elements of a communities of practice
    • domain (what)
    • community (who)
      • the people who care about the domain of knowledge
      • need to allow people to be themselves and not try to force them to be who you want them to be
      • only reason to shut someone down is if they're actively doing something poisonous
    • practice (how)
      • turn knowledge into something that can be spread around
  • principles of communities of practice
    • design for evolution
    • open a dialogue between inside and outside people
      • people on the inside need to maintain openness and transparency
      • people on the outside need to feel they're being heard, and also know how they'd become one of the inside people
    • invite different levels of participation
      • every aspect is important
      • experts don't fall out of the sky, more likely they'll be created from within
    • develop public/private spaces
    • focus on value
    • combine familiarity & excitement
    • create a rhythm for the community
      • weekly meetings, release every six months, etc. -- create expectations
  • free <3 open <3 free
    • bottom line is that free software is a great brand that works for hackers, open source is a great brand that works for businesses
    • not trying to say one brand is better than the other
  • Book: The Open Source Way: Creating and Nurturing Communities of Contributors
  • http://quaid.fedorapeople.org/presentations

Open Source Bridge - The Rise of Hacker Spaces

Leigh Honeywell, Co-Founder, hacklab.to
  • what's a hacker space?
    • been around for thousands of years under different names
    • hacker spaces is latest incarnation of a community space
    • place to go to use the kinds of tools you might not have at your house
    • also connection with idea of "third space" (not work, not home)
    • instead of hacking alone at home, you hack with others who are either like-minded or have divergent interests
    • like a "permanent hallway"
  • history
    • model railroad clubs, HAM radio clubs, toastmasters, etc.
    • major influences around north america
      • the loft in Boston
      • new hack city in san francisco
    • in europe there's a 25 year history
      • tend to be more political in nature
  • international community
    • around 400
    • many in germany affiliated with the chaos computer congress
      • publish a quarterly magazine with 4000 subscribers
      • heavily involved with push against biometric ID in germany
  • 2007 - hackers on a plane
    • trip from defcon in las vegas to event in dusseldorf
    • this got people very enthusiastic about hackerspaces -- build! unite! multiply!
    • hackerspaces.org founded by people from vienna's metalab
      • metalab is considered a youth center--get funding from the government
  • Hacker Space Design Patterns - some highlights
  • examples
    • KC - cowtown computer congress
      • located in a mine underground
      • pioneered fundraising by raffle
    • DC - HacDC
      • located in a church
      • help improve technology for the church, do a lot with community wireless efforts
    • hacklab.to
      • founded july 2008
      • 35 members
      • about 200 people on public discussion list
      • 800 sq ft space in kensington market
  • hacklab.to infrastructure
  • projects
    • most of the interesting stuff going on in hackerpsaces is free software/open source
    • not all technology projects--sewing, etc.
    • lots of electronics hacking
    • roombas
    • laser cutter
    • newbie-oriented programming classes
  • lots of free hardware projects coming out of hackerspaces as well
    • e.g. discussions of building an open source laser cutter
    • arduino microcontroller -- came out of art college in italy
    • 3d printer (makerbot?)
  • what do hackerspaces mean for free software?
    • another backchannel
    • place for people to collaborate on free software in a place that's potentially higher bandwidth
    • more opportunity for cross-pollination
    • space where small user groups congregate or grow out of
    • can show up at hackerspace with question and even if you don't get the answer, you get a lot of ideas you wouldn't get elsewhere
  • challenges
    • institutional/organizational issues similar to the ones free software projects run into
    • politics
      • bootstrapping a business out of a hackerspace can be pretty perilous
    • who's in charge
    • where does the money come from?
      • unlike free software projects, there's overhead (rent) for hackerspaces
    • scaling
      • once you get a core, how do you get new members involved?
    • how do we recruit the next generation?
      • outreach to schools -- teachers love having people come in and talk to their students about what they're doing
  • "the future is already here, it's just not very evenly distributed" -- william gibson

Tuesday, June 1, 2010

Open Source Bridge - OAuth: an Open Specification for Web Services

John Jawed - jawed@php.net
  • talk is about oauth at a high level and the problems it tries to solve
  • won't get low-level into the spec, etc.
  • problem - rasmus has private photos on flickr but wants to print them out using kodak's online photo printing service?
  • before oauth: have to give kodak your flickr user name and password
    • too powerful
      • can change passwords, delete photos, delete account
  • e.g. has anyone had to share their twitter u/p with another site to gain access?
  • how about some identifier issued by flickr to kodak?
    • would allow someone else to access your account but only with certain permissions and only for a certain length of time
  • for a few years we've had ...
    • yahoo bbauth
    • google authsub
    • aol openauth
    • windows live id
    • flickr token auth
    • many others
  • equivalent but different model -- drove the creation of oauth
  • the oauth dance
    • kodak sends rasmus to flickr
    • flickr asks rasmus: kodak wants to access your private photos
    • rasmus: sure!
    • flickr gives kodak a token with a secret
    • kodak makes api requests to flickr using the token/secret
    • fin
  • behind the scenes ...
    • timestamps
    • nonces
    • tokens
    • secrets
    • refresh tokens
    • signatures -- "ssl bastardized"
      • handy if requests aren't over https
  • yahoo big on openid and oauth
    • internal implementers at yahoo found oauth difficult to use
    • php.net/oauth designed to address problems with working with oauth
      • open source (bsd)
      • mature
      • supports tlaking to oauth apis
      • supports creating oauth apis
      • available as a php extension
  • code sample -- using oauth in php
    • start the session
    • create a new oauth object using api key
    • get request token using endpoint, e.g. twitter's request token url
    • redirect user to authentication url
    • after authentication, get an access token for interaction with the target service, e.g. twitter
    • fetch method in the oauth object handles all the token negotiation for you
  • to use oauth you don't need to understand the spec, just need to understand the flow
    • get request token (throw-away token)
    • get access token (token used to sign requests)
  • netflix example
    • pretty similar to twitter, but also need to pass user id
  • financial services example - wepay (paypal for groups)
    • similar paradigm--get request token, authorize access, get access token, interact with service
    • wepay also includes a verifier in addition to the tokens
  • example with no user
    • sign request with secret and api key--consumer key and consumer secret
  • what about accepting oauth in your API?
    • create oauthprovider object in php
    • get consumer key from caller, make sure it's still valid
    • three handlers
      • $provider->is2LeggedEndpoint()
      • $provider->consumerHandler()
      • $provider->timestampNonceHandler()
    • google: rasmus oauth to see full example walkthrough
  • oauth for php supports
    • all oauth paramer types
    • hmac-sha1, rsa-sha1
    • 1.0a
    • oauth extensions: session, problem reporting
    • setting millisecond request timeouts
    • and much more
  • oauth 2.0
    • still a work in progress
    • same token paradigm
    • profile-based (web, client, etc.)
    • oauth 1.0 with session extension
    • oauth 1.0 implementations will be around for a long time
    • will be supported in pecl/oauth
    • problem oauth 2.0 tries to solve is that oauth 1.0 is very browser-focused, lots of requests/redirects
      • support for non-browser flow, native client flow, etc.

Open Source Bridge - Open Source and the Open Social Web

Evan Prodromou, StatusNet Inc. Communications Revolutions
  • email
    • ca. 1993
      • large consumer systems
      • university networks on internet
      • govt systems
      • proprietary systems inside corporations
      • x.400 - itu recommendation, govt mandated
      • ad-hoc bridges
      • bbses (fidonet)
    • ca. 1995
      • only 18 months later, almost entirely unified around internet email
      • hierarchical addressing - user@domain
      • bbs -> isp
      • aol opens up
      • bbses almost disappeared
      • open source via sendmail was an important catalyst
  • documents
    • ca. 1992
      • proprietary, complex internal systems
      • some file sharing bbses
      • aol, compuserve
      • some ftp systems
    • ca. 1997
      • web documents almost ubiquitous
      • hierarchical addressing
      • intranet/internet/extranet
      • http + html
      • open source via apache is an important catalyst
  • personal publishing
    • ca. 2001
      • "home page" on isp
      • geocities, tripod
      • frontier, blogger
      • rss confusion
      • personal vs. business
    • ca. 2005
      • hosted blogs ubiquitous
      • urls for identity
      • feed readers
      • podcasting (enclosures)
      • rss 1.0, 2.0, atom: more than we need!
      • personal and business -- line blurs
      • open source: moveable type, wordpress
  • what's the point?
    • revolutions happen quickly
    • unconnected islands--federated networks
    • commercial adoption drives need for control
    • open source implementation is key
  • why open source?
    • low or no-cost to install
    • bottom-up adoption
      • people with more tech skills than money, not the other way around
    • rapid innovation as things scale
      • open source adopts rapid innovation better than proprietary software
  • what is federation?
    • network of networks
    • open protocols
    • uniform namespace
    • hierarchical addressing
    • anyone can play
  • other federated networks
    • postal system (country + postal code + local addressing)
    • telephone systems
    • sms
    • tcp/ip
    • dns
  • what drives federation on the internet?
    • tcp/ip
    • dns
    • scale
    • globalism
  • what drives federation?
    • control
    • distrust
    • greed
  • metcalfe's law
    • value of network proportional to the square of the number of nodes
    • the more the merrier--much, much merrier
    • "value" is a little vague--depends on the particular network
  • metcalfe's law and federation
    • big networks more resistant to change initially
    • as network of networks gets bigger, puts pressure on the bigger networks to participate
  • social software: 2010
    • facebook: 400M+ users
    • twitter: 100M+ users
    • application-specific networks threatened: flickr, digg, youtube
    • national networks threatened: orkut, friendster, bebo, hi-5
    • niche networks threatened: linkedin, ning
    • social gaming
      • pressure on social gaming developers to use biggest platform
  • one vision
    • some networks become de facto substrate for internet
      • facebook: "open" social graph
      • twitter: social messaging, "real time"
    • "open" means "use our API"
    • "shoot the moon" approach is a doable vision--it does happen
      • skype for voice
      • google for search
  • another vision--federated vision
    • commercial adoption of social messaging
      • businesses looking to share outside their firewall
    • need to connect
    • threatened networks fight to survive
      • one way to survive is to adopt leading networks' social graph
      • another way to survive is through federation
    • open govt requires 100% engagement
      • pay taxes through facebook? twitter as only means to connect to my members of congress?
      • systems need to be open for engagement with citizens
    • business needs of providers
      • hard to run a business using the "we're the X layer of the internet" model
  • Social Software - 2012?
    • email like identity, either email or URL
    • distributed real-time follow
    • combination of small and large networks
      • combo of public/private
    • application-specific networks, e.g. social gaming, photo sharing, etc. will move towards open standards
  • why should hackers care?
    • most important parts of our life: family, friends, romance
      • being social is a huge part of being a human being
    • politics require open discourse
      • need to continue to push for openness
    • making software that matters
      • can have a very large impact
  • protocol suites
    • email has smtp, mime, etc.
    • web = http, html, css, etc.
    • blogging = web, rss, atom, etc.
    • social web = ???
      • what will the protocols be that make up this system?
    • everything is made up of a combination of standards
  • openid
    • http://openid.net
    • authentication
    • url for identity
    • devolving to a few identity providers (google, yahoo)
    • whitelist oriented
  • oauth
    • http://oauth.net
    • authorization
    • widely implemented
    • whitelist oriented (consumer keys)
    • not a lot of social parts
    • authentication (e.g. sign in with twitter)
  • PubSubHubbub
    • http://code.google.com/p/pubsubhubbub
    • real-time publishing ("Pu$H")
    • atom or rss-based
    • web hooks
    • great support: google buzz, posterous, tumblr, wordpress, livejournal, statusnet, cliqset ...
      • lots of people prepared to be publisher, not too many prepared to be subscriber
  • activitystreams
    • http://activitystrea.ms
    • represent social actions in atom with xml namespace extension
    • subject, verb, object
    • "evan published a photo"
    • powerful when combined with pubsubhubbub
    • can push activities to people who are interested across the web
  • salmon
  • webfinger
    • http://code.google.com/p/webfinger
    • email-like identity for the web (user@domain)
    • xml document format matches identity to urls (my photo service is x, my social messaging service is Y, my profile is Z ...)
    • lrdd uses urls instead of webfingers
  • portable contacts
    • http://portablecontacts.net
    • define social relationships
    • static social graph
    • user-controlled sharing of contact data
    • compare: xfn, foaf
  • ostatus
    • http://ostatus.org
    • combines various protocols -- first stake in the ground towards building social web systems
    • created by statusnet
    • webfinger + lrdd = discovery
    • push + activitystreams = follow
    • salmon + activitystreams = reply
    • activitystreams + poco = profile
  • xmpp
    • http://xmpp.org
    • originally developed for IM (Jabber)
    • distributed system with email-like identifiers
    • social relationships = buddy list
    • profile = vcard
    • supports publish-subscribe
    • not widely implemented, not http based
      • can be difficult to work with, but very nicely federated
  • what's missing?
    • privacy
    • client API
    • microapps
  • the open source enabler
    • who will be the apache of the open social web?
    • not sure yet
    • many contenders
      • statusnet is a good start
    • trying to work with others providing open social network code so things work well together
  • diaspora
    • http://joindiaspora.com
    • 4 students in nyc, 1 summer, $200K!
    • ostatus-like stack
    • ruby on rails
    • agplv3
    • no working version ... yet
    • very interested in using the stack of existing technologies
    • if they're able to pull this off, they'll be an important part of the federated social web
  • DiSo
    • http://www.diso-project.org
    • based on wordpress
    • chris messina, steve ivy
    • xfn
    • leading activitystreams
    • have had some problems getting traction
  • Elgg
    • http://elgg.org
    • most advanced general purpose social network
    • LAMP
      • any time you use something other than LAMP you're limiting the popularity
    • commercial hosting system http://elgg.com
    • some federation (push), more coming
    • lorea fork/branch leading the way with elgg federation
    • gplv2
  • gnu social
  • buddypress
    • http://buddypress.org
    • general purpose social network
    • automattic project, very nicely done
    • lamp
    • gplv2
    • very little federation
  • statusnet
    • http://status.net
    • microblogging server
    • lamp
    • agplv3
    • ostatus for federation
    • twitter-like api
    • plugin architecture
    • identi.ca + 25K other sites on the web, 1.5 million users
  • onesocialweb
    • http://onesocialweb.org
    • vodaphone project
    • uses xmpp as core protocol
    • java plugin for openfire server
    • one of the worst things that can happen now is competing standards
  • others
    • aroundme
    • appleseed
    • crabgrass
    • noserub
  • what next?
    • projects working together
    • integration testing
    • real-life usage
    • innovation
    • growth
  • how to help
    • hack
    • translate
    • theme
    • implement
    • use
    • spread

Open Source Bridge - Transparent, Collaborative, Participatory - Grass Roots Implementation of the Open Government Directive

Mark Frischmuth - founder of Democracy Labs
  • current system yielding polarizing candidates that yell at each other
  • idea was we shouldbe able to use technology to inform citizens to enable better decisions
  • Obama signed open govt directive first day in office
  • data.gov
    • 270,000 data sets online
    • hundreds of applications built on top of this data
    • movement towards democratization of data
  • still not as successful in facilitating collaboration
Jim ???
  • 3dna
  • idea: if internet can bring democracy to music, etc. should be able to bring democracy to democracy
  • whitehouse2 -- how would whitehouse be run if it was truly democratic
Travis ???
  • phd in CS at UW
  • computer-human interaction
  • interested in design of platforms for facilitating public communication
  • working with city of seattle
  • cto of flashvolunteer -- neighborhood-centric volunteerism
  • works on opensim platform
??? - Democracy Lab
  • interested in having conversations that are more structure
  • open democracy can dwindle into flame wars pretty quickly
  • values-based conversation mechanism helps keep the conversation structured
Three elements of open govt directive
  • transparency, collaboration, participation
Transparency
  • data.gov--lots of info out there, but no real increase in public trust--why not?
  • not really transparency
    • lot of datapoints
    • to get true transparency you need to get beyond the data points to the root causes
      • e.g. poverty in one area vs. another -- how was data produced?
  • collaboration and participation are missing
    • if you're not being heard as citizens, you won't trust what's going on
    • without transparency, collaboration, and participation (all three) it doesn't mean much
  • long term project--can't expect open govt initiative itself would have some tangible impact in overall trust
    • 5-10-15 year project
  • how does data that's available get brought up in public discourse?
    • can have a robust dataset with a few errors, and the errors become the story in the media
  • grassroots solutions trying to solve problems govt doesn't
    • do the grassroots efforts have their own agenda?
    • goal is to improve the discourse gradually--trying to make it better in a little way
    • have to convince people the data you're getting from grassroots organizations is a little better than you'd get elsewhere
    • site like whitehouse2 doesn't even work unless it's totally transparent and people can interact freely with the data
      • have to publish everything that everyone on the site does
      • have to help people formulate opinions for and against each topic -- all on a wiki
        • anyone can edit, but it's also tracked so the changes are transparent, so easy to see who isn't participating legitimately
    • relate open govt work with open source software
      • if open source projects aren't fully open, they don't work, e.g. android -- what's the agenda? is it REALLY open?
      • contrast with apache projects--can see the mailing lists and see why decisions were made
Participation
  • in order for this to work at all we need engaged citizens
    • expressing points of view, learning, engaging, etc.
  • what tools help this?
  • this is the core issue--lots of efforts targeting getting citizens engaged
    • hard part is getting people to care
    • important to give people a voice and allow them to share this with their friends
    • in some ways if one person says what someone is saying in a "better" way, this will get shared and spread around
      • efforts can get bogged down over "what's the best way to express this"
      • open systems allow the best message rise to the top
    • single biggest way to get people involved is to have meaningful results
      • people need to see how their efforts are impacting the political process
  • current way for people to participate is to vote, and they don't
    • people don't feel their voice is being heard
    • key to systems around open govt is to make people's voices heard
    • all kind of pointless if people aren't listening though
  • have to show impact
    • very easy to think your vote doesn't count
  • work at national scale?
    • go local where outcomes are more tangible--projects that get voted on you can see the results of this in your everyday life
  • what kinds of tools will best facilitate getting the citizens' information to decision makers?
    • get decision makers to sign up and interacting
      • important to show this happening
      • e.g. peer2patent--crowdsourcing to help USPTO see prior art
        • home page shows which people submitted specific prior art associated with specific patents
    • frustrating thing about "open for questions" that obama did a few months ago
      • marijuana question showed up multiple times -- couldn't figure out how to handle this so they haven't had one since
      • why not use FAQ type format?
Collaboration
  • political issues are adversarial by nature--how best to facilitate collaboration between adversarial parties?
  • point is to enable rational discussions
    • have to operate under the assumption that people want to have a rational discussion
  • need to improve online comment boards
    • introduction of second column into online discussion that would show summary bullet points
    • introduction of neutrality into comments which are typically very subjective
    • current comment board implementations drive away people who might otherwise participate, but overly prescriptive comment systems do the same thing
  • split the issue in two
    • how do you deal with jerks/trolls?
      • set up rules--e.g. no personal attacks
      • differentiate between attacks on ideas and attacks on people
        • this didn't really work
      • created public 4-step process to eliminate trolls/jerks
        • allow others to flag people/comments
        • escalating system of warning, losing karma, up to being expelled
      • interesting thing was nobody every got a second warning -- people played along or they left of their own accord
    • how do you deal with partisanship?
      • big problem is that people couldn't agree on what the goal was so there was no common ground to be found
  • through the application of better processes we can get more people involved without worrying about solving every problem involved
Questions
  • Portland working on releasing their data, currently have about 110 datasets. Most jurisdictions are willing to participate, but there's a lot of datasets that haven't been released. Need to prove to the govt organizations that releasing the data is worth their time. What's the major catalyst to prove to politicians to release more data?
    • again, have to make clear to people that they're having an impact
      • e.g. iPhone app that lets people submit where potholes are on bike trails
    • easy to solve the "read" side of the problem, harder to solve the "write" side
    • do developer contests to get people to show what can be done with the data
    • try to do something useful with what's out there
      • target datasets with a concrete application in mind--easier to work with people in government at that point
  • If an open source project starts out closed, then goes open, it tends never to get the full benefit of being truly open. Is government ever going to be open enough?
    • tends to be generational--if developers don't start with openness in mind, it doesn't ever get there
    • with turnover, change easier to happen
    • govt isn't going to be completely transparent and open until we put people in govt who got there via the open systems

Open Source Bridge - Activity Streams, Socialism, and the Future of Open Source

Chris Messina, Google Google
  • dataliberation.org
    • trying to make it easy for people to move their web data around
    • one of the challenges is figuring out where you can move your data
Generative Structures
  • example: rhizome
    • can self-generate from the root structure
  • jonathan zittrain - the future of the internet and how to stop it (book)
Activity Theory 101
  • promoted by russians from late 19th-century on (vygotsky, leontiev, engestrom)
  • vygostky
    • how to model work to measure outcomes
    • focus on tool mediation -- relationship of single actor to the object of their work
  • scandanavians (engestrom) expanded to include rules, community, roles, culture, social systems
    • can look at very small or very large communities
  • by putting things in relationship to this model we start to see how meaning and cultural understanding grows over time
    • is the work satisfying? do people pursue it of their own volition, or do they need to be coerced?
    • by understanding motivations we can get people to do things
  • social objects
    • people don't connect to each other, they connect through a shared object (jyri engestrom)
    • e.g. flickr -- comments, tagging, metadata, galleries -- photo is mediating artefact
    • flickr has granular settings so people can control what can and can't be done with their photos
  • rules often emerge organically
  • contrast flickr with facebook
    • when you change the rules and surprise people, they react--expectations were betrayed
Examples from Mozilla Community
  • spreadfirefox
  • steven garrity - blog post from 2004
    • called out for help to bring designers into the mozilla community
  • a lot of the imagery from mozilla early on was communist in nature
  • browser wars were at their most heated at this point
  • for mozilla, the "enemy" of IE served as a social object
  • firefox ships november 9, 2004
  • once the browser is out, how do you get regular people to know about it?
    • spreadfirefox born to serve this need
  • "Welcome to Spread Firefox. You are our marketing department ..."
    • "... wrestling control from a monopoly that has let [the web] stagnate ..."
    • "... focus our community's energy on very tangible, specific goals"
  • first goal: 1 million downloads in 10 days
    • threw in game mechanics--kept track of points for clicks on links on people's web site
  • little "take back the web" button changed the game
  • roles
    • volunteering page on spread firefox--articulated very specifically what people could do to help
      • specific tasks for developers, security people, designers, users, etc.
    • had specific teams (webapps, for the record, wordsmiths, ad team, cd bundling team, events, college reps)
      • college reps team--seed firefox to kids who will then install firefox on their friends and relatives' computers
  • rules
    • focus on points people could get when they put getfirefox buttons on their web site
    • mozilla continues this today with their design challenge -- earn badges by contributing
  • mediating artefacts
    • new firefox images based on download milestones
    • people took these and put them on their blogs, modified, made their own
    • mozilla creative collective group
    • finding an interesting balance between the commercial side of things vs. rewarding people who are contributing
  • community
    • blogs, user profiles, affiliates, forums, etc.
    • ad for the new york times
      • 10,000 people donated money
      • all the names incorporated into the ad
    • important to consider how to incorporate collective voice of contributors into the project
  • goals and outcomes
    • "promote openness, innovation, and opportunity on the web"
      • vague, but can evolve over time
    • mozilla drumbeat
      • next generation of spread firefox
      • promoting open web to a wider audience
      • "Mozilla Drumbeat is keeping the web open"
      • images of people used on drumbeat site helps people visualize -- feel they can get involved, put themselves in the photos, make a difference
  • we need to do more to mobilize ourselves into a coherent narrative, bring more people in
    • can't take open web for granted
  • ideas from mozilla apply to open source in general as well as the open web
  • how do we get the open source/open web message to a wider audience?
  • How did the facebook "like" button trump the get firefox button? why was that opportunity not capitalized upon?
  • orwell: "history is written by the winners"
Activity Streams
  • one piece of leverage not currently being tapped
  • how do we leverage the social web and social networking to our benefit
  • activity streams format
    • take all the feeds that already exist in the wild (e.g. rss)
    • when rss was invented, the idea was to provide as little information as possible to route people back to the source site
    • in 2010 rss is still used but we're representing a vastly different type of information when you take into account social networking sites
    • basic model: actor, verb, object
      • e.g. person share link, usera follows userb, developer fork project
    • two formats -- atom and json
    • pretty decent list of verbs and object types that model what's already happening on social sites
  • process
    • inherits a lot from the micrformats community
    • ask why
    • do your homework / document
    • propose
    • iterate / implement
    • interoperate
  • number of sites already publishing activity streams
    • github timeline
    • gitorius also publishes an activity stream, but not in the same way
    • stackoverflow has its own activity stream
      • has verbs already highlighted, but the data isn't available in a way that other sites can use
  • lots of common verbs, activities, and objects between sites that initially seem disparate
  • could for example take stackoverflow's reputation system and create a common way of spanning that across projects
    • could join a project and people would "know" you
    • combine with openid and oauth and things get interesting
Distributed Social Networks
  • people still using single siloed social networks
  • lots of efforts in open source world to solve this (status.net, diaspora, etc.) and they don't interoperate
  • want to create an inclusive model that applies to all these efforts
  • create system in which people can work on what they care about, but also collect around objects regardless of where they exist
  • connect individual to a shared outcome through involvement and engagement in a community, and you get a much greater outcome than someone could achieve on their own
  • this results in a changing of culture
  • if we can do this, we'll be writing the history of open source

Open Source Bridge - The Return of Command-Line Kung Fu

Hal Pomeranz (@Hal_Pomeranz)
  • Independent consultant
  • SANS faculty fellow
    • author, track lead for Sec506:Linux/Unix Security
    • instructor for SANS forensics classes
      • teach a lot of people *nix over the years, see people struggling
      • Windows people aren't happy that all the forensics are done from the Linux command line
  • blogs
Simple Output Redirections
  • output one way, errors another
  • two different output streams (stdout, stderr)
    • make >/tmp/build.log 2>/tmp/build.errors
  • getting stderr out of your way
  • output and errors together to the bit bucket
    • make distclean >/dev/null 2>&1
  • can also use >> to append instead of overwrite
Add /dev/tcp/... Goodness
  • command output to network (bash only, disabled in older versions of ubuntu)
    • df >/dev/tcp/foo.example.com/9999
    • "netcat without netcat"
    • useful for offloading data without adding any additional software
  • or a simple port checker
    • echo >/dev/tcp/127.0.0.1/22
    • echo >/dev/tcp/127.0.0.1/23
    • this throws error message if the port isn't open (returns nothing if it is open)
    • e.g. if/then/else on command line
      • echo >/dev/tcp/host/22 && echo live port || echo dead port
      • this still shows error output and redirecting output (2>/dev/null) doesn't fix this unless you use parens--parens cause things to happen in subshell
    • put for loop around this and you get a port scanner
    • for ((i=1; $1 < 1024; i++)); do (echo >/dev/tcp/host/$i) 2>/dev/null && echo $i/tcp live; done
      • this spawns a new subshell for each loop iteration
      • can do the output redirection on the entire loop so a new subshell doesn't get spawned each time
      • for ((i=1; $1 < 1024; i++)); do echo >/dev/tcp/host/$i && echo $i/tcp live; done 2>/dev/null
  • practical uses
    • security/crime incident responses
    • want to capture a bunch of information about the status of the local machine, but can't write anything to the local machine because that would taint the state of the machine from a forensics standpoint
    • need to fake out the script command to write the data to a remote system
Fun With FIFOs
  • first in first out or named pipe
  • can make a fifio using mkfifo
  • pipe that looks like a file
  • script command wants to write to a file
  • can use a fifo to fake out the script command--send output elsewhere
  • mkfifo /tmp/fifio
  • cat /tmp/fifo > /dev/tcp/host/port &
  • script -f /tmp/fifo
  • from this point on, everything run in the terminal on the source machine is sent over the wire to another machine
    • mirrors to the other machine keystroke by keystroke
  • when dealing with disk images, need to do string searches to hunt down evidence from disk images
  • can use strings command to search
  • disk images are getting larger and larger
  • strings would have to do two passes to find ASCII strings, then unicode strings
  • can use the fifo to split the strings command and do both passes at once
    • cat /tmp/fifo | strings -a -t d -e l | gzip > /tmp/strings.unicode.gz (this gets unicode strings)
    • tee command splits input (like a t-joint in plumbing)
    • e.g. cat diskimage.dd | tee /tmp/fifo | strings -a -t d | gzip > /tmp/strings.ascii.gz
    • can keep stacking fifos and tee commands to split as many times as you want
Kill! Kill! Kill!
  • kill processes by name
    • pkill sshd
  • or perhaps more selectively
    • pkill -P 1 sshd
      • -P is parent id--process ID 1 is init, so this would kill the sshd that was started at boot
  • kill processes by user
    • pkill -u user
  • killling process associated with a particular port--not built into kill and pkill
    • kill `lsof -t -i :22`
    • -t only outputs the PIDs
  • unmount that volume
    • kill `lsof -t /home` -- kills all processes under the /home mount point
Killing by Start Time is Hard
  • timestamps on /proc aren't related to proc start time
  • pkill can only kill oldest (-o) or newest (-n) proc
  • lsof has no options to select process start time
  • ps -eo pid, comm, start_time is useless
    • if something is more than a few months old, you just get a starting year
    • completely inconsistent
  • ps does let you get at a couple of different time values
  • if you list etime (elapsed time) from ps, still irregular output, but irregular in a sensible fashion
    • get either mm:ss, hh:mm:ss, or days-hh:mm:ss
    • need to canonicalize the format to get the fields in a consistent order
      • change all the hyphens and colons to whitespace so awk can deal with it better
    • ps -eo pid,comm,etime | sed 's/[-:]/ /g' | awk '{print $1, $2, $6, $5, $4, $3}'
      • awk will output null for fields that don't exist
      • will cause issues if there are hyphens or spaces in the command name
      • with everything in a canonical order can finally parse the time
    • ps -eo pid,comm,etime | sed 's/[-:]/ /g' | awk '{print $1, $2, $6, $5, $4, $3}' | awk '{print $1, $2, ($3 + $4 * 60 + $5 * 3600 + $6 * 86400}'
      • this gives pid, command name, and seconds of elapsed time
  • comment from audience cat /proc/1/stat -- 11th field is the start time as unix epoch (?) -- need to verify; states this in the docs
  • convert epoch time to normal date/time
    • date -d @epochtime
Got the touch
  • use touch (as root) to manipulate timestamps at will
    • touch -t 201001010000 /tmp/testing
    • stat /tmp/testing
    • this sets last access/modify to whatever date you give it, but at this point change date is the timestamp when touch was executed
  • aside -- !$ gives you the argument from the last executed command
  • can use touch to set date on something for comparison purposes
  • obviously can be used by the bad guys to modify timestamps on files
  • ls -lrt will sort ascending on last modified date so the most recently modified stuff is right above your command prompt
  • what about ctime?
    • could go into the inode with a hex editor and change things that way
    • can get specific tools to handle this
    • debugfs on ext file system
      • debugfs -w -R 'set_inode_field /tmp/testing ctime 201001012222' /dev/mapper/elk-root
      • if you run debugfs it will show the ctime as what it was set to with debugfs
      • if you run stat, it'll show the real time still because there's caching going on at the OS level
      • how do you flush the cache?
        • echo 2 >/proc/sys/vm/drop_caches (can pass in 1, 2, or 3 to flush file, directory, or both)
Stumper Question #1
  • searching a directory structure
  • want to find files containing a particular string
  • only want to look in ASCII text files
  • find /etc -type f | xargs file | grep ' text' | sed 's/:[^:]*$//' | xargs grep -l mystring
Stumper Question #2
  • two directories
  • each will have some files in common and some unique
  • matching files may have different names between directories
  • create a list of unique files from both dirs
  • would have to use checksums due to matching files not necessarily having the same name
  • find dir1 dir2 -type f | xargs md5sum | sort
    • shows which checksums are the same, but the lines would be different as far as the unique command is concerned
  • find dir1 dir2 -type f | xargs md5sum | sort -u -k1,1 | awk '{$1=""; print}'
    • sort can be used to selectively uniqueify on columns
Slides at http://www.deer-run.com/~hal/