Google Open Source Blog

Releasing Neatx, an Open Source NX Server

2009-07-07T12:38:00.000-07:00

We at Google have been looking at remote desktop technologies for quite a while. The good old X Window system can be used over the network, but it has issues with network latency and bandwidth. Neatx remedies some of these issues.

In 2003, NoMachine released a large portion of the source code of their NX product under the GPL licence. NX is a protocol compressing X requests and reducing round-trips. Although mostly Open Source, NoMachine's NX product contains one closed component, the NX server. It's the part connecting clients with the Open Source libraries doing the work.

A free implementation of an NX server based on NoMachine's libraries named FreeNX was published in 2004 by Fabian Franz. FreeNX's primary target is to replace the one closed component and is written in a mix of several thousand lines of BASH, Expect and C, making FreeNX difficult to maintain.

Last week, we released the source code of our own proof-of-concept implementation of an NX server, Neatx. Designed from scratch with flexibility and maintainability in mind, Neatx minimizes the number of involved processes and all code is split into several libraries. It is written in Python, with the exception of very few wrapper scripts in BASH and one program written in C for performance reasons. Neatx was also able to reuse some code from another Google Open Source project, Ganeti. The code still has some issues, but we're confident interested developers will be able to fix them.

Also, Neatx implements features not found in FreeNX, such as the drop-down menu for session control in rootless sessions. At the same time, not all of FreeNX's features are implemented in Neatx.

Michael Hanselmann gave a presentation at FISL 10 in Porto Alegre, Brazil describing our implementation and use of virtual workstations (slides: PDF, 200 KB).

More information and the code can be found at the Neatx code.google.com project. You can also send us questions and feedback on the Neatx discussion list. Happy hacking!

By Euan Guttridge, Stephen Shirley and Michael Hanselmann, Systems Administration Team

London Open Source Jam 13

2009-07-02T16:10:00.000-07:00

Here in London, we recently hosted London Open Source Jam 13 (Our 14th jam, as naturally we count from zero.) - unlucky for some, but not for us! We threw the floor open to talks of any kind and we had a bumper crop of lightning talks on a diverse range of topics. We learned how to make a wiki in 58 lines of python, why open source developers should care about open standards, some new approaches to database design, and a whole lot more.

Kai Hendry talked about Webconverger, a teeny weeny Linux Live CD designed for web kiosks, and his experience commercialising an open source project by offering services to go with it.

Zak Cohen shared his experiences using open source libraries in the games world in developing the award-winning game Climbactic. It turns out there's some great open source stuff out there, but sometimes paying for support is the only way to get the features you need.

Frederik Dohr and Mike Mahemoff spoke about TiddlyWeb, a generic RESTful store for structured data, and Scrumptious, a jQuery-based web app that allows people to annotate and comment on web pages, which uses TiddlyWeb for storage.

Other contributors included:

Simon Stewart - Testing Google Wave with WebDriver
Jon Skeet - The Dynamic Language Runtime and C# 4.0
Zaheda Bhorat - Open Standards
Ambikesh Jayal - Open source e-Learning middleware
Paul Walmsley - Bayesian data modelling
David Sheldon - Where's Java's CPAN?
Robert Rees - Bazaar Wiki
Rob McKinnon - Politics and representation
Ivan De Marino - Caching templated CSS
Matt Godbolt - Testing+Mobile
Nicolas Roard - Seaside and Smalltalk
Phil Dawes - New approaches to database server design

The OSJam website has more information about all the talks, and the photographs taken at the event are available on Picasa Web Albums.

We'll be planning the next OSJam for a couple of months' time - subscribe to the London Open Source Jam Group or keep any eye on the OSJam website (Atom feed) for more details.

By Malcolm Rowe and Matt Godbolt, Software Engineering Team

Google Update, regularly scheduled

2009-07-01T15:00:00.000-07:00

Client software products have unique challenges, with one of the important challenges being keeping the product up to date. Deploying critical security fixes in a timely and effective manner is one key way that we help keep users secure. Another benefit is that cool new features get deployed and adopted quickly. Google Update is the shared updating infrastructure used by Google Chrome, Google Earth and other Google software that runs on PCs to keep our products up to date on users' computers.

A couple of months ago we released the Google Update source code to give users and developers transparency into our update mechanism. One month later, we released Update Controls that allowed network administrators and advanced users the ability to control the installation and updating of Google products via Google Update.

We hope to help address concerns users have voiced around Google Update running as a continuous process. Until now, Google Update would always run in the background, functioning primarily as a reliable scheduler performing update checks at periodic time intervals. With today's release, Google Update now uses the Windows Task Scheduler to only run at periodic intervals. We have worked hard to ensure that automatic updates work just as reliably, and that our users are just as safe and secure as before.

There are a couple of details that we want to mention. First, in a very small number of cases when Google Update determines that the Windows Task Scheduler or Service mechanisms are not working as expected, we have added in fallback mechanisms that cause Google Update to begin running as a continuous process again to ensure users are still receiving updates to their Google software. Second, if you opted in to sending anonymous usage statistics and crash reports to Google for a particular Google application, we will run a process in the background called GoogleCrashHandler.exe. GoogleCrashHandler.exe is responsible for reporting crashes to Google when they occur in your Google product.

by S. Ganesh, Google Update Team

Introducing Apache Commons Math SimplexSolver

2009-07-01T10:00:00.000-07:00

SimplexSolver is an easy-to-use, object-oriented method of solving linear programming problems. We're happy to announce today that we've Open Sourced the code that runs the newly released Google Spreadsheets Solve Feature and made it a part of Apache Commons Math 2.0.

While numerous other libraries are available that run linear optimization problems, SimplexSolver is the first written in Java with a commercially-friendly license.

Let's say we have the following LP:

MIN -2x + y - 5
S.T.
x + 2y <= 6
3x + 2y <= 12
y >= 0

We could solve the problem in Java using the SimplexSolver:

// describe the optimization problem
LinearObjectiveFunction f = new LinearObjectiveFunction(new double[] { -2, 1 }, -5);
Collection constraints = new ArrayList();
constraints.add(new LinearConstraint(new double[] { 1, 2 }, Relationship.LEQ, 6));
constraints.add(new LinearConstraint(new double[] { 3, 2 }, Relationship.LEQ, 12));
constraints.add(new LinearConstraint(new double[] { 0, 1 }, Relationship.GEQ, 0));

// create and run the solver
RealPointValuePair solution = new SimplexSolver().optimize(f, constraints, GoalType.MINIMIZE, false);

// get the solution
double x = solution.getPoint()[0];
double y = solution.getPoint()[1];
double min = solution.getValue();

Looking at the LP problem and Java code side-by-side, we can see how easy it is to describe the problem in the Apache Commons Math API. More examples can be viewed in the SimplexSolverTest source code. Apache Commons Math 2.0 is not quite released yet, so for the time being if you want to use the SimplexSolver, you'll need to compile it from source.

So that's SimplexSolver: a clean, fast method of solving LP problems. We hope you like it as much as we do! Thanks to Luc Maisonobe and the rest of the Apache Team who helped integrate the SimplexSolver into the Apache codebase.

by Ben McCann, Software Engineering Team

Australia Goes Open

2009-06-30T14:30:00.000-07:00

Here in Google's shiny new Sydney office, we recently hosted the first hackfest for OpenAustralia.org. OpenAustralia takes the data from Australia's record of Federal Parliament speeches, the Australian Hansard, and makes it easy for people to follow topics they're interested in.

Back in 2004, Matthew Landauer and Katherine Szuminska found themselves at the launch of the UK site TheyWorkForYou.com, a website designed to allow the average person to quickly and easily search the United Kingdom's Hansard - the record of all happenings in Parliament. Their work blossomed and they were inspired to make a similar site happen in Australia, OpenAustralia.org.

Both websites give ordinary people, who often have no idea who their local representative is, let alone what their representative has been doing on their behalf, the ability to track topics they're interested in and find out exactly what their representatives are doing.

When Matthew emailed out asking for a location for a place to host a hackfest, we were very happy to lend a helping hand. The hackfest ran on Saturday the 13th of June at the newly opened Google offices in Sydney, Australia and was successful beyond anyone's expectations.

When originally announced, the event's original 25 attendee slots filled so fast that we ended up increasing the number to 50 (which also filled remarkably fast). More surprisingly, almost everyone turned up and we even had a number of attendees fly in from other Australian states. No one expected this level of enthusiasm from the community, and we were pleased to share in everyone's excitement.

Some cool outcomes include a tool for crowd sourcing transcription of the "register of member's interests", an API and datasource for mapping street addresses to representatives, a "FixMyStreet" iPhone App and numerous bugs on the site fixed. More information on these and the ongoing efforts of this community can be found on the Open Australia Development list.

Matthew was also gracious enough to give a tech talk for Googlers about some of the many challenges faced by creating such a site, e.g. dealing with Crown Copyright, problems with getting a clean source of data, and problems getting the data fixed when it's clearly in error. Enjoy!

By Tim Ansell, Technical Solutions Engineering Team

Guten LinuxTag!

2009-06-24T18:03:00.000-07:00

The Open Source Team's Cat Allman will be in Berlin, Germany this week to present at LinuxTag 2009. Cat's talk, Getting Started in Open Source: An Overview for Newbies, will take place on Friday, June 26th at 4 PM local time. This will be Cat's second time presenting at LinuxTag - in 2008 she gave a talk about Google Summer of Code™.

This year LinuxTag expects over 10,000 visitors from around the world- If you are there make sure to attend Cat's talk!

By Ellen Ko, Open Source Team

Chris DiBona and Leslie Hawthorn at FISL

2009-06-23T18:06:00.000-07:00

If you are in South America this week and you have been wanting to find out more about Google's Open Source activities or the future of Open Source and its communities, Chris DiBona and Leslie Hawthorn from the Google Open Source Programs Office will be speaking at the tenth annual Fórum Internacional Software Livre (FISL) in Porto Alegre, Brazil.

Leslie will be presenting tomorrow, June 24th, 1 PM local time about Google Summer of Code™, and she'll be staying afterward to meetup with Google Summer of Code participants who are in the area. On June 26th at 2 PM, Leslie will be presenting "Community Management Basics" and how to make FLOSS projects and communities welcoming for new contributors.

Chris' first talk, "Open Source: Then, Now and Tomorrow" takes place on June 26th at 3 PM local time. In addition to the past, present, and future of FOSS, Chris will discuss how Google uses and releases Open Source software. In Chris's second talk, "An Introduction to Android" at 5 PM on the 26th, he will explain the ideas and structure behind the Open Source mobile operating system Android.

All four talks are great opportunities to learn more about Google and Open Source. Come ask questions and get to know members of the Open Source Team!

by Ellen Ko, Open Source Team

Sojourning at SouthEast LinuxFest

2009-06-09T08:59:00.000-07:00

Fresh on the heels from speaking at Google Developer Day in Beijing and Tokyo, Chris DiBona will be heading to the SouthEast LinuxFest. This new conference promises to provide a welcoming environment for novice users of Open Source software, with a little something for everyone with talks on data warehousing, education and philanthropy. Should you find yourself in or around Clemson, South Carolina this Saturday, June 13th, please stop by to hear Chris discuss "Open Source: Then, Now and Tomorrow." As much as the Open Source Team enjoys connecting with Open Source developers and users worldwide, Chris is particularly looking forward attending this grassroots, community driven event and sharing his thoughts in this intimate setting.

We hope to see you there and, if you're attending, please do come by to say hello!

By Leslie Hawthorn, Open Source Team

Introducing Android Scripting Environment

2009-06-08T12:52:00.000-07:00

The Android Scripting Environment (ASE) brings scripting languages to Android by allowing you to edit and execute scripts and interactive interpreters directly on the Android device. These scripts have access to many of the APIs available to full-fledged Android applications, but with a greatly simplified interface that makes it easy to:

Handle intents
Start activities
Make phone calls
Send text messages
Scan bar codes
Poll location and sensor data
Use text-to-speech (TTS)
And more

Scripts can be run interactively in a terminal, started as a long running service, or started via Locale. Python, Lua and BeanShell are currently supported, and we're planning to add Ruby and JavaScript support, as well.

Scripts can be edited directly on the phone.

The script manager displays available scripts.

Scripts can be launched interactively or as background services.

Interactive terminals can be started for interpreters that support it.

Scripts can use the Android UI to get user input.

You may ask, why write scripts instead of real Android applications? Admittedly, Android's development environment makes life pretty easy, but you're tied to a computer to do your work. ASE lets you develop on the device itself using high-level scripting languages to try out your idea now, in the situation where you need it, quickly. Have a look at the following example Lua script to see for yourself:


--Placing the phone face down will disable the ringer. Turning it face up again will enable
--the ringer.
require "android"
android.startSensing()
android.sleep(1)  --Give the sensors a moment to come online.
silent = false
while true do
  s = android.readSensors()
  facedown = s.result and s.result.zforce and s.result.zforce > 9
  if facedown and not silent then
    android.vibrate()  --A short vibration to indicate we're in silent mode.
    android.setRingerSilent(true)
    silent = true
  elseif not facedown and silent then
    android.setRingerSilent(false)
    silent = false
  end
  android.sleep(1)
end

Here's another useful script, this time in Python.


"""Say chat messages aloud as they are received."""

import android, xmpp

_SERVER = 'talk.google.com', 5223

class SayChat(object):
  def __init__(self):
    self.droid = android.Android()
    username = self.droid.getInput('Username')['result']
    password = self.droid.getInput('Password')['result']
    jid = xmpp.protocol.JID(username)
    self.client = xmpp.Client(jid.getDomain(), debug=[])
    self.client.connect(server=_SERVER)
    self.client.RegisterHandler('message', self.message_cb)
    if not self.client:
      print 'Connection failed!'
      return
    auth = self.client.auth(jid.getNode(), password, 'botty')
    if not auth:
      print 'Authentication failed!'
      return
    self.client.sendInitPresence()

  def message_cb(self, session, message):
    jid = xmpp.protocol.JID(message.getFrom())
    username = jid.getNode()
    text = message.getBody()
    self.droid.speak('%s says %s' % (username, text))

  def run(self):
    try:
      while True:
        self.client.Process(1)
    except KeyboardInterrupt:
      pass

saychat = SayChat()
saychat.run()

These scripts demonstrates several of the available APIs available for both Lua and Python. It is intended to be run as a service and silences the ringer when the phone is placed face down. For some scripting languages, like BeanShell, it's possible to access Android's Java API directly. To simplify things, ASE provides the AndroidFacade class. For other languages, like Python and Lua, the API is made available via JSON RPC calls to a proxy. Naturally this means that only the part of the API which has been wrapped by the AndroidFacade and AndroidProxy are available to cross-compiled interpreters like Python and Lua. Thankfully, both AndroidFacade and AndroidProxy are simple to extend.

If you'd like to give ASE a try, it's not yet published to the Market, but will be soon. You can download the latest APK from our project page. Some sample scripts and documentation are also included there to help you get started. We always love to hear what you think, so please send us feedback or ask your questions in the ASE discussion group.

By Damon Kohler, Software Engineering Team

Getting Started in Free and Open Source

2009-06-05T16:27:00.000-07:00

The Open Source Programs Office's Cat Allman and Leslie Hawthorn will be presenting at a brand new conference, Open Source Bridge, in two weeks. Their talk, "Getting started in Free and Open Source" will be held on Wednesday, June 17 from 11:20am – 12:05 PM. Leslie and Cat will share their wisdom with Open Source newbies and those looking to attract or retain them. Their talk will cover the basics of:

Why you might want to get involved
What you can get out of participating
What you can contribute besides code
How to choose a project
How to get started
Etiquette of lists and other communication
Dos and don’ts of joining a community

The conference is held in Portland, Oregon and runs from June 17th to the 19th. It features over 80 talks in five tracks of Open Source topics and a 24-hour hacker lounge for code sprints. One of the most exciting aspects of Open Source Bridge is that it is entirely volunteer-run, and because this conference brings together developers from all different types of Open Source projects, the structure is designed to provide developers an opportunity to learn from people they might not connect with at other events.

We hope to see you there!

By Ellen Ko, Open Source Team

Improving Freenet's Performance

2009-05-29T10:46:00.000-07:00

The Free Network project is the community that creates and maintains Freenet, free software that allows you publish and obtain information on the Internet without fear of censorship by means of a decentralized, anonymous network. Since version 0.7 , the software has had built-in support for downloading and uploading large files. These are long-term downloads, which persist between restarts of the node. This support has improved performance and usability, but it has also meant that when lots of downloads are going on at the same time, Freenet uses a lot of memory, takes a long time to complete the startup process, and crashes if you queue too many downloads. By storing the current progress of uploads and downloads in db4o.com's open source object database (= a file on disk) rather than in memory, Freenet's memory usage can be greatly reduced, the end-user doesn't need to worry about running out of memory, we can have an unlimited number of uploads and tens of gigs of downloads, and so on.

To begin at the beginning, Freenet divides all files into 32KB blocks (called CHKs), which are each fetched and decrypted separately. Then we have a layer of redundancy, and various complexities surrounding putting files together and putting in-Freenet websites together, which makes up the client layer. Before the db4o branch, uploads were persistent, but downloads were restarted from scratch after every restart, pulling huge numbers of blocks from the datastore (on-disk cache). Worse, memory usage was rather large if you had any significant number of downloads on the queue.

The db4o project puts the client layer (persistent downloads and uploads) into a database (db4o). I had initially hoped that this would be a relatively quick project, which shows how much I knew about databases then! We decided to use db4o in a fairly low-level way, specifically to minimize memory usage. We had heard from testimonials that some embedded applications had done this, but unfortunately this is not really the way that db4o is usually used, which caused some complications. Overall, the project took one developer most of a year, the final diff was over 46K lines of code covering 320 files, and went well beyond its original remit, solving many long-standing problems in the process. New architecture was required for optimal performance, including using Bloom filters to identify blocks we are interested in, a queue of database jobs, major refactoring in many areas of the client layer, a new system for handling temporary files, etc.

The effort was well worth it. Our client layer overall has vastly improved and Freenet now

starts up quickly

resumes work on downloads and uploads almost instantly on startup

can have an almost unlimited number of downloads and uploads

doesn't need the user to worry about or configure the maximum memory usage

doesn't go into limbo with constant 100% CPU usage desperately trying to scrounge a few more bytes

can insert DVD-sized files and huge websites (or git/hg repositories) on relatively low end systems

uses fewer file handles

This project would not have happened without support from Google's Open Source Programs Office. It will be one of the most important changes in version 0.8 of Freenet when it is released later this year, and current work includes Bloom filter sharing, a new feature that should greatly improve performance both for popular and rare content. Google is also funding that project, watch this space!

By Matthew Toseland, Freenet Core Developer

Web Storage Portability Layer: A Common API for Web Storage

2009-05-28T14:33:00.000-07:00

As discussed in our Google Code Blog post on HTML5 for Gmail Mobile, Google's new version of Gmail for iPhone and Android-powered devices uses the Web Storage Portability Layer (WSPL) to let the same database code run on browsers that provide either Gears or HTML5 structured storage facilities. The WSPL consists of a collection of classes that provide asynchronous transactional access to both Gears and HTML5 databases and can be found on Project Hosting on Google Code.

There are five basic classes:

google.wspl.Statement - A parametrizable SQL statement class

google.wspl.Transaction - Used to execute one or more Statements with ACID properties

google.wspl.ResultSet - Arrays of JavaScript hash objects, where the hash key is the table column name

google.wspl.Database - A connection to the backing database, also provides transaction support

google.wspl.DatabaseFactory - Creates the appropriate HTML5 or Gears database implementation

Also included in the distribution is a simple note-taking application with a persistent database cache built using the WSPL library. This application (along with Gmail mobile for iPhone and Android-powered devices) is an example of the cache pattern for building offline web applications. In the cache pattern, we insert a browser-local cache into the web application to break the synchronous link between user actions in the browser and server-generated responses. Instead, as shown below, we have two data flows. First, entirely local to the device, contents flow from the cache to the UI while changes made by the user update the cache. In the second flow, the cache asynchronously forwards user changes to the web server and receives updates in response.

By using this architectural pattern, a web application can made tolerant of a flaky (or even absent) network connection!

We'll be available at the Developer Sandbox at Google I/O to discuss the cache pattern, HTML5 development and the WSPL library. Check it out! If you have questions or comments, please visit our discussion list.

By Robert Kroeger, Software Engineer - Mobile Team

Support for Mercurial Now Available for All Projects Hosted on Google Code

2009-05-28T12:27:00.000-07:00

You may recall that we recently asked for help from some early testers of Mercurial on Project Hosting on Google Code. As of today, all of our Project Hosting users can make use of this added functionality. For full details, check out the Google Code Blog. Better still, if you happen to be joining us at Google I/O, stop by the Mercurial on Big Table Tech Talk to learn more.

By Leslie Hawthorn, Open Source Team

2009 Google Summer of Code™ Celebration at Beijing LUG

2009-05-21T19:09:00.000-07:00

Newly elected BLUG President, Pockey Lam, handing over Google Summer of Code stickers

The Beijing Linux User Group (BLUG) invited all the Google Summer of Code students and mentors in China to a special BBQ lamb skewers night night (we call these "chuan'r nights" actually) on Saturday, May 9th. The BLUG has been supporting this fantastic initiative from Google as much as possible since 2007 by organizing conferences in universities and spreading the word online in Chinese websites.

This year we're very proud to have 2 mentors and at least 5 student members of our group selected for Google Summer of Code. Both mentor proposals are centered around building better Linux support for MIPS and the Chinese Loongson CPU. We also managed to get the two main hardware manufacturers (Dexxon/Emtec and Lemote) to donate free MIPS laptops to the selected students.

And to add icing on the cake, Google sent schwag for our little party, Lemote a few free T-shirts and the Beijing LUG distributed its 2009 edition of their own T-shirts for free as well! Definitely something not to be missed if you were in town (Beijing, China)!

Celebration announcement details here, location details here, and full picture gallery here.

By Frederic Muller, Beijing LUG Vice President

Happy Birthday, ICU!

2009-05-19T10:54:00.000-07:00

The ICU project is celebrating 10 years of being open source this month.

"ICU" in this case stands for International Components for Unicode - not to be confused with Intensive Care Unit or International Communist Union... It is the premier software internationalization library, appearing in everything from your Google Android phone or your iPod all the way up to IBM mainframes. It provides the Unicode support that all of these programs need for handling the languages of the world, from Arabic to Chinese to Vietnamese.

ICU originated back in an Apple/IBM/HP joint venture. That code was morphed into the core of Java internationalization for JDK 1.1.4 - a large portion of this code still exists in the java.text and java.util packages. At that time, it included pretty much just sorting, locale/message support, and formatting for dates, numbers and so on. (If you're interested in early history, see an older paper by Laura Werner - now at Google). The libraries were refined over time and ported back to C and C++; now there are also wrappers for other languages, such as PHP.

ICU's data comes from the Unicode Consortium's open source project for locale data - CLDR - and typically releases each new version right after CLDR does. CLDR 1.7 was just released Friday, May 8, with ICU 4.2 following on the very same day.

While ICU was around before Google, more recently Google has played a strong role in the development of ICU, and in providing major contributions to the Unicode CLDR project. ICU forms the foundation of our 40 language initiative, so we look forward to many successful future birthdays!

By Mark Davis, Internationalization team

Report from Day 2 of the Linux Storage and Filesystem Workshop, April 6-7th, 2009

2009-05-18T16:42:00.000-07:00

(The sequel to Report from Day 1)

The first official discussion of the day was around optimizing Solid State Disk performance, led by Matthew "willy" Wilcox (Intel). SSD behaviors and "modeling" that behavior are still of interest. File systems developers need to understand how the performance costs have shifted compared to regular disks. Matthew noted the Intel SSDs are adaptive and export a 512 byte sector illusion. Ric Wheeler (RedHat) also raised concerns about current use of fallocate() and how adding use of "TRIM" to fallocate() would affect "Thin Client Provisioning." Consensus was any form of over subscription of HW would cause problems and TRIM command might exacerbate those issues. But TRIM would have measurable positive impact for most users (of SSDs). Ted Ts'o expected ext4 to already properly issue TRIM at the right times.

Moving on, "4KB sector" hard disk support is mostly done and Martin Petersen did much of the recent work to prepare linux kernel for it. Performance issues are still lurking however when using FAT partition tables or anything else that might affect the alignment offset of a write. The
root cause of this disaster is the drives export a 512 byte logical sector (despite implementing 4k physical sectors). They export 512 byte sectors in order to boot from older BIOS and function with "legacy" operating systems. But in order to get good performance with FAT partition tables, HW vendors have "offset" the logical->physical block mapping so logical sector 63 (size of "track" in ancient BIOSs) is "well aligned" physcially. "Badly aligned" means a 4k write will require reading and writing parts of two 4K physical sectors and thus "burning" an extra rotation (must read data first and then write it out on the next rotation). Well, anyone using full disk or some other partition table (e.g. GPT) will learn the joys of unnecessary complexity when they demonstrate and try to explain two levels of disk performance for the same application binary. The only way to avoid this mess is if HDD vendors provide the means to directly use native 4k blocks.

The second issue with 512 byte block emulation is error handling. The problem here is performance will be horrid for sequential small writes to a single 4k block unless the intermediate writes are cached anyway and then written in one go when the last sector is written....if the 8th write fails, the OS will think the previous 7 sectors are fine and just rewrite the last sector again. Previous 7x512 bytes are gone. With Write Cache Enabled (WCE) turned on for most SATA drives, this problem already exists. The only thing new this speculation exposes is disk vendors have strong incentives to violate the intent of "WCE off" despite dire consequences.

The last presentation I want to mention was "Virtual Machine I/O". The challenge was how Block IO schedulers need to manage bandwidth in various topologies typically seen in Virtualized IO. Google's Naumann Rafique was one of the presenters with a focus on "Proportional IO" implementation. Hannes Reincke summarized the core problems nicely: IO scheduling is only needed when there is contention at the device level- keep the mechanism to enforce scheduling at that level. Different policies should be implemented at higher levels as needed.

Later on, when talking with the group in a "hacking session", I backed up my assertion that this is not a new problem by showing my copy of OLS 2004 schedule where IO prioritization was mentioned by Jens Axboe in his talk and in a BOF led by Werner Almsberger (http://abiss.sourceforge.net/). My advice was to solve and push the simplest piece first before confusing everyone with grand designs and huge patches.

And I'll close with my kudos to Linux Foundation staff to pulling this off smoothly! Really. It was nice to see a small event get handled so professionally and courteously.

By Grant Grundler, Platforms Kernel Team

Zurich Open Source Jam 7

2009-05-15T10:35:00.000-07:00

We did it again, and we are getting better at it. Last Thursday, May 7th we hosted the 7th Open Source Jam in Zurich. It was probably our largest event, with close to 50 participants!

It was a bit more than 3 hours of event, and 10 projects were presented: EVMS (Enterprise Volume Management Software) which promises to be a new model of volume management for Linux, My paint - very interesting software to create images from scratch, RTEMS - a real time operating system for multiprocessor systems, Solid State Drives, Monitoring Systems, Gurtle - an issue tracker integration for TortoiseSVN, RDKit - cheminformatics and machine learning software, a very nice overview on the development model and features of Drupal, SQL for Google App Engine, and OSGi, the dynamic module system for Java.

We would like to thank you all again for participating and sharing the interesting projects you are working on, and also invite you to subscribe to our Open Source Jam Zurich Group to stay informed about other events in Zurich.

Unfortunately, our official photographer was out of town, but we promise pictures for the next Open Source Jam!

By Fernanda G. Weiden, Systems Administration Team

Google Update Releases Update Controls

2009-05-14T15:00:00.000-07:00

Whenever we build out new products and features at Google, we try to ensure that we provide users with two key components: transparency and control. About a month ago we released the Google Update source code to give users and developers transparency into Google's update mechanism. Today we hope to fulfill this second component by providing advanced users the ability to control the installation and updating of Google products via Google Update. Thanks to automatic updates, most users should already have this version of Google Update.

The update policy is controlled via Windows Group Policy, allowing network administrators to apply policies to all computers on their domain and power users with administrative privileges to set the policy on individual machines. We provide an Administrative Template file that allows selection of policies using standard graphical user interfaces such as Group Policy Editor.

The new Group Policy support allows an administrator to specify which Google applications can be installed and how they should be updated. You can select from one of three update options: automatically, manually, or not at all. Administrators can also control how frequently Google Update checks for software updates.

Mac users have similar controls over Google Software Update. Mac users are able to change how often update checks occur or disable update checks all together. See the Managing updates in Google Software Update Help Center article for details.

We work hard to keep our users safe and secure when using our applications, and we believe that making sure users have the latest software available using automatic updates is a key component of that. However, we realize that there are situations where automatic updates may not be desirable so we wanted to provide the ability to control updates when necessary.

To get started, take a look at the Google Update for Enterprise documentation.

By David Dorwin, Google Update Team

Spreading the Summer Love in Chicago

2009-05-12T16:18:00.000-07:00

There are many ways to get the word out about Google Summer of Code™, and one of the most fun ways is the Google Summer of Code meetups that participants organize around the world. After a successful meetup last year for University of Chicago students, the University's ACM student chapter decided to organize another meetup this year, open to other universities in the Chicago area and again hosted by Google. Thus, on April 30th, around 70 students from The University of Chicago, Northwestern University, DePaul University, the Illinois Institute of Technology, and the University of Illinois at Chicago converged upon Google's offices in downtown Chicago, who happily allowed us to use their space, eat their food, and consume their caffeinated beverages.

Like last year, the event revolved around a series of lightning talks where Google Summer of Code students and mentors in Chicago talked about their upcoming work, and Google engineers talked about cool stuff they are involved in. Among the evening's highlights, there was mingling:

Whereas last year we were greeted by the Tower of Hanoi of Chinese Food, this year we were met by the Mexican Food Buffet of Awesomeness:

The talks took place in Google's offices on the 17th floor of a downtown building, with stunning views of the city:

(That tall black building is the John Hancock Center )

Our Google Summer of Code speakers were (left to right, top to bottom):

Derek Schaefer (DePaul University), who will be working for MySQL to improve the import functionality of phpMyAdmin
Chelsea Bingiel (University of Chicago), who will be adding support for the Atom Publishing Protocol in Plone .
Dev Ghosh (Northwestern University), who will be adding a new "mosaic mode" for Hugin/Panotools .
Dirk Eddelbuettel (R Foundation, mentor), who told us all about the cool Google Summer of Code projects the R Foundation has mentored and will mentor this year.

Not pictured are a couple other Google Summer of Code students from Chicago who couldn't make it to the meetup or did not give talks: Joe Doliner (University of Chicago, student for BRL-CAD), Caden Howell (DePaul University, student for the Electronic Frontier Foundation), Chandra Ramachandran (University of Illinois at Urbana-Champaign, student for the National Center for Supercomputing Applications ), and Ori Rawlings (Illinois Institute of Technology, student for the Natural User Interface Group).

Our Google speakers were (left to right) Nathaniel Manista, Jon Trowbridge, and Jacob Lee, who provided us with a steady supply of very amusing slides.

Oodles of thanks go to Google for hosting this event (specially to Jon Trowbridge, organizer extraordinaire) and congratulations to our Chicago-area students for making it into Google Summer of Code!

By Borja Sotomayor, Ph.D. Candidate, University of Chicago and Google Summer of Code™ Organization Administrator (Globus Alliance)

Photos by Anne Celestino, Lauren Ellsworth, and Borja Sotomayor.

Introducing WebDriver

2009-05-08T16:45:00.000-07:00

WebDriver is a clean, fast framework for automated testing of webapps. Why is it needed? And what problems does it solve that existing frameworks don't address?

For example, Selenium, a popular and well established testing framework is a wonderful tool that provides a handy unified interface that works with a large number of browsers, and allows you to write your tests in almost every language you can imagine (from Java or C# through PHP to Erlang!). It was one of the first Open Source projects to bring browser-based testing to the masses, and because it's written in JavaScript it's possible to quickly add support for new browsers that might be released

Like every large project, it's not perfect. Selenium is written in JavaScript which causes a significant weakness: browsers impose a pretty strict security model on any JavaScript that they execute in order to protect a user from malicious scripts. Examples of where this security model makes testing harder are when trying to upload a file (IE prevents JavaScript from changing the value of an INPUT file element) and when trying to navigate between domains (because of the single host origin policy problem).

Additionally, being a mature product, the API for Selenium RC has grown over time, and as it has done so it has become harder to understand how best to use it. For example, it's not immediately obvious whether you should be using "type" instead of "typeKeys" to enter text into a form control. Although it's a question of aesthetics, some find the large API intimidating and difficult to navigate.

WebDriver takes a different approach to solve the same problem as Selenium. Rather than being a JavaScript application running within the browser, it uses whichever mechanism is most appropriate to control the browser. For Firefox, this means that WebDriver is implemented as an extension. For IE, WebDriver makes use of IE's Automation controls. By changing the mechanism used to control the browser, we can circumvent the restrictions placed on the browser by the JavaScript security model. In those cases where automation through the browser isn't enough, WebDriver can make use of facilities offered by the Operating System. For example, on Windows we simulate typing at the OS level, which means we are more closely modeling how the user interacts with the browser, and that we can type into "file" input elements.

With the benefit of hindsight, we have developed a cleaner, Object-based API for WebDriver, rather than follow Selenium's dictionary-based approach. A typical example using WebDriver in Java looks like this:

// Create an instance of WebDriver backed by Firefox
WebDriver driver = new FirefoxDriver();

// Now go to the Google home page
driver.get("http://www.google.com");

// Find the search box, and (ummm...) search for something
WebElement searchBox = driver.findElement(By.name("q"));
searchBox.sendKeys("selenium");
searchBox.submit();

// And now display the title of the page
System.out.println("Title: " + driver.getTitle());

Looking at the two frameworks side-by-side, we found that the weaknesses of one are addressed by the strengths of the other. For example, whilst WebDriver's approach to supporting browsers requires a lot of work from the framework developers, Selenium can easily be extended. Conversely, Selenium always requires a real browser, yet WebDriver can make use of an implementation based on HtmlUnit which provides lightweight, super-fast browser emulation. Selenium has good support for many of the common situations you might want to test, but WebDriver's ability to step outside the JavaScript sandbox opens up some interesting possibilities.

These complementary capabilities explain why the two projects are merging: Selenium 2.0 will offer WebDriver's API alongside the traditional Selenium API, and we shall be merging the two implementations to offer a capable, flexible testing framework. One of the benefits of this approach is that there will be an implementation of WebDriver's cleaner APIs backed by the existing Selenium implementation. Although this won't solve the underlying limitations of Selenium's current JavaScript-based approach, it does mean that it becomes easier to test against a broader range of browsers. And the reverse is true; we'll also be emulating the existing Selenium APIs with WebDriver too. This means that teams can make the move to WebDriver's API (and Selenium 2) in a managed and considered way.

If you'd like to give WebDriver a try, it's as easy as downloading the zip files, unpacking them and putting the JARs on your CLASSPATH. For the Pythonistas out there, there's also a version of WebDriver for you, and a C# version is waiting in the wings. The project is hosted at http://webdriver.googlecode.com, and, like any project on Google Code, is Open Source (we're using the Apache 2 license) If you need help getting started, the project's wiki contains useful guides, and the WebDriver group is friendly and helpful (something which makes me feel very happy).

So that's WebDriver: a clean, fast framework for automated testing of webapps. We hope you like it as much as we do!

by Simon Stewart, Engineering Productivity Team

O BSDCanada!

2009-05-07T14:27:00.000-07:00

BSDCan 2009, an annual BSD conference at the University of Ottawa in Ontario, Canada will be held this year on May 8th and 9th, 2009. The Open Source Team's Leslie Hawthorn and Cat Allman will be there to mingle with the Open Source community and present a talk on Getting Started in Free and Open Source on May 8th at 11 AM local time. This talk is a fantastic introduction to the Open Source community for those who are new and want to get involved. In addition, Open Source veterans will discover insights into the concerns of newbies and learn ways to improve retention and make their projects more welcoming. By running projects such as Google Summer of Code™ and the Google Highly Open Participation™ Contest, Leslie and Cat have gathered a huge amount of experience working with Open Source newcomers, and they are excited to share their knowledge with the rest of the community.

This will mark Google's third year at BSDCan, with Brian 'Fitz' Fitzpatrick and Ben Collins-Sussman speaking there in 2007 and Leslie presenting in 2008. If you are in the area, make sure to attend this year's talk, and feel free to say hello or introduce yourself afterward!

By Ellen Ko, Open Source Team

Report from Day 1 of the Linux Storage and Filesystems Workshop, April 6-7, 2009

2009-04-30T17:13:00.001-07:00

My karma was apparently very good three weeks ago. In the last minute I secured an invitation to Linux Storage and Filesystems Workshop 2009 (LSF). This year the invitation-only workshop was hosted by the Linux Foundation on April 6-7 in San Francisco, CA. It was, as always, an intense 2 days of non-stop information exchange and decision making. I've attempted to summarize what I took away as the most interesting and important discussions of the first day. These are my opinions, and your mileage may vary.

The group of about 50 developers were nearly all present when Zach
Brown (Oracle)welcomed us and reminded folks about the ground rules of the event. Share the brownies. Wash your hands, ...just kidding. This event remains small so folks can participate and we were pretty "cozy" in the small conference room with 5 good sized round tables. First step was to turn off the projector. :)

Chris Mason and James Bottomley then did a great summary and "scoring" of promises made at LSF2008. IO stack was up first and had some good initial scores with high points for Power management, Request Based Multipath, BIO's TRIM/ERASE support, T10 DIF/DIX (complete) and FCoE (also complete). Chris Mason managed to nearly match that with 4/4 points for Barriers, BTRFS (upstream but not stable yet), IPV6 NFS, NFS RDMA.

The first problem/topic was how to cache device scanning in the kernel or how to properly export an API for device scanning. General problem is there are several methods the kernel exports info and it's very time consuming on large systems. This was followed by Async IO and Direct IO discussion led by Zach Brown (Oracle) and Jeffrey Moyer (Redhat). Zach has been the AIO maintainer "forever" and made it clear AIO was async in only a very few circumstances that happen to suit database developers.

Joe Eykholt gave a summary of "FC/SCSI Targets", how to get initiator *and* target mode support from one FC HBA at the same time. Interesting stuff. Nick Bellinger gave a concise summary of state of "LIO/iSCSI" code and the "tgt" driver.

I was interested in Tejun Heo's "libata status and issues" discussion. First we talked about the status of several patches: mvsas updates, "ATA Bus" transport class, SFF vs Native transport classes from my co-worker Gwendal Grignou, and a pile of power management patches from Kristen Accardi (Intel). Tejun then dove into the "Spurious Power Off" problem. The cause seems to be short loss of power from the PSU is causing massive FS corruption. He's documented 5 incidents so far. Additional symptoms are "clicking" sounds and START/STOP count increments (reported via SMART data). Tejun suspects the FS is issueing a FLUSH to all disks simultaneously. We further speculated that the drives might be in a low power (slower RPMpossibly) and suddenly all come to life. Currently no fix is available.

Some possible workarounds we considered:
- disable Write Cache Enable (and take a write perf hit on loads that are single threaded)
- disable power management.

He moved on to discuss ambiguities around libata/block layer data structures (e.g. hard_ vs w/o hard_ ) fields that have similar (but not the same) names.

One of the last issues was something I raised: Can we reduce the CPU utilization of the block layer? I was asking since several new flash technologies are under development and they are all capable of 200+ *thousand* IOPS. The answer was Jens Axboe was working on this already since about December 2008 had committed his initial results to his own git tree already. I just need to find the git tree and proper branch now. :)

That wraps up day one. I hope you find the information useful. If you want to read about day two, please leave a comment and if demand warrants it, I'll cover that in a future post.

By Grant Grundler, Platforms Kernel Team

Student Applications Open for 2009 OpenUsability Season of Usability

2009-04-29T10:13:00.000-07:00

Inspired by Google Summer of Code™, the OpenUsability Season of Usability is a series of sponsored student projects to encourage students of usability, user-interface design, and interaction design to get involved with Free/Libre/Open Source Software (FLOSS) projects. Students experience the interdisciplinary and collaborative development of user interface solutions in international software projects while getting into FLOSS development.

If you are a student of design, usability, human factors, or other HCI-related field and you are interested in working on an open source project, you could work with an experienced usability mentor on a fun and interesting design project! As a bonus for working 10-15 hours a week between June 1 and August 31, there is a $1000 USD internship stipend at the end of the project.

The OpenUsability Season of Usability will be supporting 10 students to work on 10 open source projects during the June 1 - August 31 2009 season.

* Amarok
* Drupal
* Gallery
* GeneMANIA
* GNOME
* Kadu
* KOrganizer
* OLM
* SemNotes
* Ubuntu

Student applications are due May 20 2009. See http://season.openusability.org/index.php/projects for more information on the projects, student requirements, and how to apply. Questions about the projects or application process may be directed to students@openusability.org.

by Celeste Lyn Paul, Season of Usability Mentor

Distributed Version Control for Project Hosting Users

2009-04-24T15:09:00.000-07:00

Love our Open Source project hosting service but wish it supported distributed version control? Pine no longer! You can find full details about Mercurial support for hosted projects on the Google Code Blog.

For those of you who are Git fans, you may want to check out these articles: Develop with Git on a Google Code Project and Exporting a Git Project to Google Code.

Happy hacking!

By Leslie Hawthorn, Open Source Team

Ed. Note: Post update to correct link.

Now Accepting Nominations for the 5th Annual Google-O'Reilly Open Source Awards

2009-04-22T12:01:00.000-07:00

The Google-O'Reilly Open Source Awards are back again for 2009! These awards recognize individual contributors who have demonstrated exceptional leadership, creativity, and collaboration in the development of Open Source Software. Past recipients for 2005-2008 include Angela Byron, Andrew Tridgell, Harald Welte, Julian Seward, Karl Fogel, Martin Dougiamas, Pamela Jones and Paul Vixie.

The nomination process is open to all, so please send your nominations to osawards@oreilly.com. Nominations should include the name of the recipient, any associated projects or organizations, suggested title for the award ("Best Hacker", "Best Community Builder", etc.), and a description of why you are nominating the individual. Google and O'Reilly employees are not eligible for the awards, though we thank you if you thought of us.

Nominations close on May 22, 2009. The awards will be presented during the kickoff ceremonies for OSCON 2009. We look forward to hearing from you and having your help to honor those community members who make Open Source that much better for all of us!

Leslie Hawthorn, Open Source Team