blog.mhartl | Michael Hartl's tech blog

Like most projects that use Rails 2.1, the Insoshi source code ships with a “secret” string (which lives in environment.rb) needed for the new cookie-based sessions. Recently, an alert observer noted that this raises a security issue in Insoshi sessions: the secret key is currently the same for all Insoshi installations, which opens the sessions up to attack (as noted in this discussion thread). This problem is not unique to Insoshi; it affects essentially any Rails application installed from source.

Part of the reason this problem isn’t more widely known is because projects generated using the rails script automatically receive a unique security string. The way we’ve fixed the secret string problem at Insoshi involves piggybacking on the mechanism Rails already has for generating such strings, by replacing the hard-coded string with a file read:

config/environment.rb

Before:

config.action_controller.session = {
    :session_key => '_instant_social_session',
    :secret      => '63143b62...8522327'
  }

After:

.
.
.
require File.join(File.dirname(__FILE__), 'boot')
require 'rails_generator/secret_key_generator'

Rails::Initializer.run do |config|
  .
  .
  .
  # Your secret key for verifying cookie session data integrity.
  # If you change this key, all old sessions will become invalid!
  # Make sure the secret is at least 30 characters and all random,
  # no regular words or you'll be exposed to dictionary attacks.
  secret_file = File.join(RAILS_ROOT, "secret")
  if File.exist?(secret_file)
    secret = File.read(secret_file)
  else
    secret = Rails::SecretKeyGenerator.new("insoshi").generate_secret
    File.open(secret_file, 'w') { |f| f.write(secret) }
  end
  config.action_controller.session = {
    :session_key => '_instant_social_session',
    :secret      => secret
  }
  .
  .
  .

(N.B. The session key _instant_social_session is a hint about the origins of the name Insoshi.) In place of a hard-coded string, the updated code uses the contents of a secret file, if it exists; otherwise, it makes a new string using the same machinery as the rails script (included with the line require 'rails_generator/secret_key_generator') and writes it to the secret file.

It’s important at this point to prevent our source code management tool from versioning the secret file, since the whole point of this exercise is to prevent the secret key from being distributed with the source code. Using Git, this is trivial; we just add ‘secret’ to our .gitignore file. (Note: if you are running an application on multiple servers, you should copy the same secret file to each one to ensure that sessions will work with a load-balancer.) Everyone using the Insoshi source code should pull from our GitHub repository to get the update.

Handling session expiration

Unfortunately, the above steps don’t completely solve our problem. The comments in environment.rb note that “If you change this key, all old sessions will become invalid!” That’s not quite accurate; the old sessions don’t merely become invalid: they actually raise an exception, so users with active sessions will be met with your application’s error page, and a CGI::Session::CookieStore::TamperedWithCookie exception will show up in your application’s log file. (The error page goes away if the user reloads the page in their browser, but there’s no way for them to know that.) Serving up error pages to all those users isn’t very friendly behavior, and we’d like to catch the exception and show the page they’re trying to access instead.

This isn’t as simple as it seems, because the exception gets raised deep inside the Rails internals. We can figure out where by running in development mode, where the stack trace look something like this:

CGI::Session::CookieStore::TamperedWithCookie in HomeController#index 

vendor/rails/actionpack/lib/action_controller/session/cookie_store.rb:144:in `unmarshal'
vendor/rails/actionpack/lib/action_controller/session/cookie_store.rb:101:in `restore'
/usr/local/lib/ruby/1.8/cgi/session.rb:304:in `[]'
vendor/rails/actionpack/lib/action_controller/cgi_process.rb:136:in `session'
vendor/rails/actionpack/lib/action_controller/cgi_process.rb:168:in `stale_session_check!'
vendor/rails/actionpack/lib/action_controller/cgi_process.rb:116:in `session'
.
.
.

To catch the exception, we need to override the default restore method in cookie_store.rb. To do that, we need to load our change before the application loads, and the easiest way to do this is with a plugin, which we can generate with a script:

$ script/generate plugin catch_cookie_exception

Once we edit a couple files, the solution is complete:

vendor/plugins/catch_cookie_exception/init.rb

require 'catch_cookie_exception'

vendor/plugins/catch_cookie_exception/lib/catch_cookie_exception.rb

require 'cgi'
require 'cgi/session'
class CGI::Session::CookieStore
  # Restore session data from the cookie.
  # This method overrides the one in
  # actionpack/lib/action_controller/session/cookie_store.rb
  # in order to handle the case of a "tampered" cookie more gracefully.
  # The issue is that changing the 'secret' in config/environment.rb
  # breaks all sessions in such a way that everyone gets an error page
  # the first time they revisit the site.  Catching the exception here
  # prevents this ugly behavior.
  # This is in a plugin so that it loads after Rails but before environment.rb.
  def restore
    @original = read_cookie
    @data = unmarshal(@original) || {}
  rescue CGI::Session::CookieStore::TamperedWithCookie
    logger = Logger.new("#{RAILS_ROOT}/log/#{RAILS_ENV}.log")
    logger.warn "Caught TamperedWithCookie exception on #{Time.now}"
    @data = {}
  end
end

Note that, since the exception could be the result of someone attacking the site by tampering with their cookies, we log the exception for future reference.

UPDATE: The catch_cookie_exception plugin is now available at GitHub.

Acknowledgments

Thanks again to Trevor Turk for alerting us to this issue.

I’m happy to announce the release of a new edge branch of the Insoshi social networking platform, which is fully updated with Rails 2.1 support. (The Insoshi demo site is currently running off this edge branch.) The result is a good example of upgrading a real-life application to Rails 2.1, so I thought a blog post detailing the steps might be useful.

Most of the changes here are quite generic—updates to widely used plugins, for example—and are not specific to Insoshi. I’ve been especially careful to include error messages when applicable, so that search engines can index them; I’m sure I’m not the only programmer who follows the “Google the error message” algorithm when debugging. I’ve also included the Git commands I used, since I know a lot of Rails developers are working to learn Git and I thought some examples might be helpful.

It’s important to note at the outset that you do not need to follow these steps yourself to upgrade Insoshi. Insoshi contributors, and anyone else who wants the edge version of Insoshi, should follow the instructions at the Insoshi wiki (if they haven’t already) and then issue the following commands:

$ git fetch origin
$ git branch --track edge origin/edge
$ git checkout edge

This way, you’ll get all these changes for free. (Once hosting support for Rails 2.1 is more widespread (especially at Heroku), we’ll merge the Rails 2.1 Insoshi edge into the master branch.)  UPDATE: The merge has happened, and now the Insoshi master branch runs Rails 2.1.

And now, on with our show. Here’s what it took to get Insoshi running under Rails 2.1.

Update RubyGems

Upgrading Insoshi to Rails 2.1 involves updating a bunch of plugins, many of which are now hosted at GitHub. Unfortunately, older versions of RubyGems can’t install directly from GitHub sources; for example, trying to install will_paginate with RubyGems 1.1.0 gives you this error message:

$ gem --version
1.1.0
$ sudo gem install mislav-will_paginate -s http://gems.github.com/
ERROR:  could not find mislav-will_paginate locally or in a repository

The solution is to update the system gems (including RubyGems) as follows:

$ sudo gem update --system
$ gem --version
1.2.0

Install Rails 2.1

Installing Rails 2.1 itself is the easiest step in the entire upgrade. (I think when people wonder, “How hard could upgrading to Rails 2.1 possibly be?”, they have mainly this step in mind.)

$ sudo gem update rails

Things now get a little trickier, since we want to freeze Rails 2.1 in vendor/rails to follow the best practice for production Rails apps. The first part goes smoothly:

$ git rm -r vendor/rails
$ git commit -a -m "Cleared out vendor/rails"

There’s a hitch with the second step. If you happen to have only Rails 2.1 (but not 2.0.2) on your machine, there’s a bootstrapping problem due to a variable in environment.rb:

$ rake rails:freeze:gems
Missing the Rails 2.0.2 gem. Please `gem install -v=2.0.2 rails`, update your
RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you
do have installed, or comment out RAILS_GEM_VERSION to use the latest version
installed.

The solution is to update environment.rb with the proper Rails gem version:

# Specifies gem version of Rails to use when vendor/rails is not present
RAILS_GEM_VERSION = '2.1.0' unless defined? RAILS_GEM_VERSION

Then the freeze works fine:

$ rake rails:freeze:gems
$ git add .
$ git commit -a -m "Updated vendor/rails to Rails 2.1"

Finally, following the advice froma Akita’s Rolling With Rails 2.1, we add a new file called config/initializers/new_defaults.rb:

# These settings change the behavior of Rails 2 apps and will be defaults
# for Rails 3. You can remove this initializer when Rails 3 is released.

# Only save the attributes that have changed since the record was loaded.
ActiveRecord::Base.partial_updates = true

# Include ActiveRecord class name as root for JSON serialized output.
ActiveRecord::Base.include_root_in_json = true

# Use ISO 8601 format for JSON serialized times and dates
ActiveSupport.use_standard_json_time_format = true

# Don't escape HTML entities in JSON, leave that for the #json_escape helper
# if you're including raw json in an HTML page.
ActiveSupport.escape_html_entities_in_json = false

$ git add config/initializers/new_defaults.rb
$ git commit -m "Added new defaults initializer"

Update RSpec for Rails 2.1

A necessary step in verifying that the Insoshi application is working under Rails 2.1 is to get the test suite to pass. Our tests are written using RSpec, but older versions of RSpec don’t work with Rails 2.1, so we can’t even run the test suite. D’oh!

Since the old plugins don’t work, first we remove them:

$ git rm -r vendor/plugins/rspec*
$ git commit -a -m "removed outdated RSpec plugins"

Then we need to install the most recent versions of the RSpec plugins from GitHub:

$ script/plugin install git://github.com/dchelimsky/rspec.git
$ script/plugin install git://github.com/dchelimsky/rspec-rails.git
$ script/generate rspec

In my case, I was careful not to overwrite spec_helper.rb when running script/generate rspec in the last step, as the current spec helper contains several custom modifications that I didn’t want to lose.

Then we need to add the changes:

$ git add .
$ git commit -a -m "Added latest RSpec plugins from GitHub"

By the way, running specs from the command line won’t work:

$ spec spec/models/person_spec.rb
Your RSpec on Rails plugin is incompatible with your installed RSpec.

RSpec          : 20080526202855
RSpec on Rails : 20080628203842

To fix this, you can use script/spec in place of spec, or you can upgrade to the latest RSpec gem using the RSpec source from GitHub:

$ cd ~/tmp
$ git clone git://github.com/dchelimsky/rspec.git
$ cd rspec
$ rake gem
$ sudo gem install pkg/rspec-1.4.1.gem

(All of Insoshi’s tests use RSpec, but if you use Test::Unit you should know that as of Rails 2.1 default tests can’t be run at the command line using the ruby executable; for tests generated by Rails 2.1, you now have to include the test directory explicitly using the -I test flag. For a hypothetical resource foobar, for example, you would get this:

$ script/generate scaffold foobar baz:string
$ ruby test/functional/foobars_controller_test.rb
test/functional/foobars_controller_test.rb:1:in `require':
no such file to load -- test_helper (LoadError)
        from test/functional/foobars_controller_test.rb:1

It passes if you tell ruby about the test directory:

$ ruby -I test test/functional/foobars_controller_test.rb
Loaded suite test/functional/foobars_controller_test
Started
.......
Finished in 0.24375 seconds.

7 tests, 13 assertions, 0 failures, 0 errors

(N.B. Using rake still works fine.) It’s unclear why the Rails Core team decided to make this change, but it’s a definite gotcha so I thought it deserved mention.)

With the new RSpec installed we can at least run the test suite, but unfortunately there are huge amounts of breakage. This is mainly due to several plugin incompatibilities and a slight Rails 2.1/Insoshi conflict (discussed below). Let’s get started fixing them.

Update obsolete helper specs

One source of breakage is the helper specs (in the spec/helpers/ directory). All these specs have obsolete code, resulting in errors such as

NoMethodError in 'TopicsHelper should include the TopicHelper'
undefined method `metaclass'

By going to a temp directory and running a sample rspec_scaffold with the updated RSpec to get a sample spec template file, you can discover that the line

included_modules = self.metaclass.send :included_modules

needs to be changed to

included_modules = (class << helper; self; end).send :included_modules

in each helper spec. You can use search-and-replace in your text editor to make all the changes, and then commit them:

$ git commit -a -m "Updated outdated spec helpers"

This won’t fix everything in the helper specs; there are still warning messages like this:

Modules will no longer be automatically included in RSpec version 1.1.4

This is because the ActivitiesHelper module is not explicitly included in the helper spec. The fix is to add the relevant include:

spec/helpers/activities_helper_spec.rb

require File.dirname(__FILE__) + '/../spec_helper'
include ActivitiesHelper
.
.
.

Update will_paginate for Rails 2.1

The old will_paginate plugin won’t work with Rails 2.1. You get tons of errors like

SystemStackError in 'PeopleController people pages should have a working show page'
stack level too deep

We need to remove the old plugin and install an update from GitHub. The current recommended method is to install it as a gem, and luckily Rails 2.1 has a slick new method for dealing with gem dependencies. In principle, we just need to add the following lines to environment.rb

Rails::Initializer.run do |config|
  .
  .
  .
  # Custom gem requirements
  config.gem 'mislav-will_paginate', :version => '~> 2.3.2',
                                     :lib => 'will_paginate',
                                     :source => 'http://gems.github.com'
end

This tells Rails that our application requires will_paginate version 2.3.2 or later, and that it can be found at GitHub. We can do the installation like this:

$ sudo rake gems:install

Unfortunately, in our case this won’t work, since will_paginate doesn’t work as a gem if the application has Rails in vendor/rails. (This gotcha is buried in the will_paginate wiki at GitHub.) We have to install will_paginate as a plugin after all:

$ git rm -r vendor/plugins/will_paginate
$ script/plugin install git://github.com/mislav/will_paginate.git
$ git add .
$ git commit -a -m "Updated will_paginate plugin"

(Unfortunately, this still didn’t fix the problem for Insoshi, because there were extra files in the lib/ directory:

$ git rm -r lib/will_paginate*
$ git commit -a -m "Removed will_paginate from lib"

I’m not sure how that happened, but it sure took a while to figure out…)

Update TextMate footnotes for Rails 2.1

Now that will_paginate is fixed (and the corresponding specs pass), you’d think the relevant pages would work. You’d be wrong. We were using the edge version of TextMate foototes, and it turns out that the footnotes-edge plugin breaks horribly in Rails 2.1. Basically, every page gives you something like this in the browser:

ActionController::RenderError in HomeController#index

You called render with invalid options : {:layout=>false, :action=>"index"}, nil

To fix this, we need to install an update from GitHub (are you seeing a pattern here?):

$ git rm -r vendor/plugins/footnotes-edge
$ script/plugin install git://github.com/drnic/rails-footnotes.git
$ mv vendor/plugins/rails-footnotes vendor/plugins/footnotes
$ git add .
$ git commit -a -m "Updated TextMate footnotes"

Update attachment_fu for Rails 2.1

We’re almost there. A few photo specs still fail, with messages like

NoMethodError in 'PhotosController when logged in should create photo'
undefined method `callbacks_for' for #<Photo:0x529a61c>

The solution is to update attachment_fu:

$ git rm -r vendor/plugins/attachment_fu/
$ script/plugin install http://svn.techno-weenie.net/projects/plugins/attachment_fu/
$ git add .
$ git commit -a -m "Updated attachment_fu"

Fix the broken verify action

This would seem to complete the update, but unfortunately this isn’t the end; there’s one more (Insoshi-specific) problem. Insoshi includes the option to verify the email addresses of new members, using a custom action called verify inside the People controller. Unfortunately, the specs that test the email verification fail with the error

No action responded to verify

This looked to be a general problem with custom actions in Rails 2.1 tests, but it turns out that the culprit is the word verify. For some reason, an action called verify causes problems in Rails 2.1, but only in tests. (This took a long time to figure out.) The (rather inelegant) fix is to rename the controller action to verify_email, and then add a line in the routes file so that old email verification links still work:

app/controllers/people_controller.rb

def verify_email
  .
  .
  .
end

config/routes.rb

map.resources :people, :member => { :verify_email => :get,
                                    :common_contacts => :get }
map.connect '/people/verify/:id', :controller => 'people',
                                  :action => 'verify_email'

Of course, we also need to change get :verify to get :verify_email in the spec file (spec/controllers/people_controller_spec.rb). Once that is done, all the specs pass, and the upgrade is complete:

$ rake spec
..............................................................................
..............................................................................
..............................................................................
..............................................................................
..............................

Finished in 12.337675 seconds

342 examples, 0 failures

Phew!

Postscript: Rails 2.1 migrations, schema_info, and schema_migration

One benefit of moving to Rails 2.1 is the new method for handling migrations. We’ve already run into instances with the Insoshi project where we needed to merge updates with conflicting migration numbers, and it’s really not any fun with Rails 2.0. There is a potential gotcha, though, since in order to perform the new migration cleverness Rails 2.1 uses a table called schema_migration in place of the old schema_info table. The change is supposed to happen automatically when you first migrate after installing Rails 2.1, but we ran into some difficulties…

When making a Rails 2.0 to 2.1 upgrade, you shouldn’t run into any problems if you start from a clean database and run

$ rake db:migrate

If you have an existing database (for instance, our demo site database), the migration should automatically convert the schema_info table used in 2.0 (which stores only a single integer value) to schema_migration (which has entries for all the migrations that have been performed).

While that worked for us in development, we ran into an issue on our staging server (where we test everything before installing it on our production servers): the expected table conversion didn’t happen. Instead, the migration tried to recreate the tables from scratch. We got around this by bootstrapping the conversion before the migration, as follows.

1. Create the schema_migrations table via SQL:

   $ mysql insoshi_production -u root
   mysql> CREATE TABLE schema_migrations (
          version VARCHAR(255) NOT NULL,
          UNIQUE KEY unique_schema_migrations (version)
          );
   

2. Find out the current migration version number:

   mysql> SELECT version FROM schema_info;
   +---------+
   | version |
   +---------+
   |      26 |
   +---------+
   1 row in set (0.07 sec)
   

3. Insert values from 1 to the current version as strings (i.e., as a VARCHAR array):

   mysql> INSERT INTO schema_migrations VALUES ('1'),('2'),('3'),('4'),('5'),
   ('6'),('7'),('8'),('9'),('10'),('11'),('12'),('13'),('14'),('15'),
   ('16'),('17'),('18'),('19'),('20'),('21'),('22'),('23'),('24'),
   ('25'),('26');
   

Of course, you shouldn’t have to do this—things should Just Work™—but then again, upgrading to Rails 2.1 wasn’t supposed to be difficult, either. :-)

Git is awesome at merging and branching, but what if you want to pull in just one patch from a single commit?

We ran into this recently with Insoshi at GitHub, where piotrj updated the README to be in RDoc format. Why not just merge in his changes?  Well, Piotr has also been working on image galleries, but in the mean time billsaysthis has picked up that torch and run with it.  As a result, Piotr’s image gallery changes would cause conflicts with the current master branch, and in any case we don’t want those changes just yet—we only want the RDoc-ified README for now. If only there were a way to use Git to cherry-pick just one commit…

Aha, git cherry-pick to the rescue!  Here are the steps for my particular case:

1. I use

    $ git fetch piotrj

   to fetch Piotr’s changes to my local machine. (I had already connected to his GitHub fork using the steps from the relevant Insoshi Git guide.)

2. Looking at GitHub for the commit label, I see that it’s 3b4257f0454fc31349a0505c9a883f691fe8889d. (I could also checkout Piotr’s branch locally and use git log to see the commits.) So all I need to do is switch to the master branch and cherry-pick the change:

   $ git checkout master
   $ git cherry-pick 3b4257f0454fc31349a0505c9a883f691fe8889d

   Note that I don’t have to reference Piotr’s branch explicitly; Git figures out the right branch to use from the commit label.

That’s it! Amazingly, I don’t even have to do a commit; Git adds Piotr’s message to my log automatically. After pushing the updated master branch to GitHub, the Insoshi README is noticeably improved.  Thanks to piotrj—and to git cherry-pick!