Hungry Machine

  I'm wondering what is the main set of features preventing folks
today from using mongrel-esi in production?  Is it :

* performance
* documentation
* stability

If it's performance, can someone do some load testing maybe provide
some numbers and even some target numbers?  I am working on an
improved concurrency model, that should help improve page performance
for pages with lots of esi:include tags

if it's documentation, what's the missing bits, and can any of you
help out by filling in the gaps?

if it's stability, can you provide some samples that fail?

-Todd

Problem

With high volume Rails applications, entities with unique constraints are expensive and error prone to create/update. ActsAsInsertOrUpdate helps solve that problem (if you're using MySQL), by leveraging the "INSERT ... ON DUPLICATE KEY UPDATE" functionality.

Scenario

Lets say you have a Person, and Entity, and a Rating. Each user can rate each entity only once, and if they re-rate the entity, it should update the value.

class Entity < ActiveRecord::Base
  has_many :ratings
end

class Person < ActiveRecord::Base
 has_many :ratings
end
  
class Rating < ActiveRecord::Base
 belongs_to :Person
 belongs_to :Entity
end  

Here is the table that back's Rating. Notice the Unique Key constraint on (entity_id, person_id).

CREATE TABLE `ratings` (
  `id` int(11) NOT NULL auto_increment,
  `rating` tinyint(4) default '0',
  `person_id` int(11) default NULL,
  `entity_id` int(11) default NULL,
  `created_at` datetime default NULL,
  `updated_at` datetime default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `index_ratings_on_entity_id_and_person_id` (`entity_id`,`person_id`),
)

Previously, the logic would be something like:

• 1) Check if a rating exists for the User + Entity
• 2) If so, update
• 3) If not, insert
• 4) rescue the insert in case there is a unqiue constraint error
• 5) retrieve the record (and/or update with the new rating)

If the table is MyISAM, Steps 1-5 aren't transactionally safe. If you're using InnoDB, and experience heavy volumes of traffic, you're prone to Deadlock's. This is even more of a concern is the unique entity is shared across multiple users, as seen with a recent client of ours.

Solution:

class Rating < ActiveRecord::Base
 belongs_to :Person
 belongs_to :Entity
 acts_as_insert_or_update :field_to_update => "rating"
end  

Now Steps 1-5 above become, just one. Rating.create(..)

In the background, ActsAsInsertOrUpdate overwrites the implementation of ActionRecord:Base#create, to leverage an often unsed feature of MySQL called INSERT ... ON DUPLICATE KEY UPDATE. As configured above, if a duplicate record is found for the unique constraint, the rating field will be updated with the new value.

Caution

This is a brute force hack on ActiveRecord::Base#create. Use at your own risk.

Code

Waiting for a rubyforge account. Will post more info soon.

One of the challenges with writing a Facebook or Bebo application is staying within a limit it gives you to respond with data before it shows the Application Did Not Respond page to a user. Having a content reach application calling external APIs, like Amazon or YouTube, with response times beyond your control, forces you to keep such calls short to allow extra time for processing. We usually wrap them in aggressive timeouts with a retry. As an example is this code from the Ruby Amazon E-Commerce REST Service API gem rewritten to limit a single call attempt to two seconds with one more retry.

module Amazon  
  class Ecs

    def self.send_request(opts)
      request_url = prepare_url(opts)

      res = Net::HTTP.get_response(URI::parse(request_url))
      unless res.kind_of? Net::HTTPSuccess
        raise Amazon::RequestError, "HTTP Response: #{res.code} #{res.message}"
      end
      Response.new(res.body)
    end

  end
end

module Amazon  
  class Ecs

    class EmptyResponse
      def items; []; end
      def total_pages; 0; end
    end

    def self.send_request(opts)

      res = timed_try(request_url, 2) do |url|

        uri = URI::parse(url)
        req = Net::HTTP.new(uri.host, uri.port)

        # Agressive timeouts
        req.open_timeout = 1
        req.read_timeout = 2

        req.start { |http| http.request_get(url) }

      end

      res.kind_of?(Net::HTTPSuccess) ? Response.new(res.body) : EmptyResponse.new

    end

private

     def timed_try(url, attempts, &block)

       attempt = 1
       begin
         block.call(url)
       rescue Timeout::Error
         if attempt >= attempts
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] gave up after attempt ##{ attempt } to get data from #{ url }"
           nil
         else
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] attempt ##{ attempt } timed out on getting data from #{ url }"
           attempt += 1
           retry
         end
       end

     end

  end
end

class ActionController::Routing::RouteSet
  def extract_request_environment(request)
    contents = request.body.read
    facebook_method_override = nil
    if contents =~ /fb_sig_request_method=([A-Z]+)/
      facebook_method_override = $1
    end
    request.body.string = contents
    { :method => facebook_method_override ? facebook_method_override.downcase.intern : request.method}
  end
end

ActionController::AbstractRequest.class_eval do
  def request_method_with_facebook_overrides
    @request_method ||= begin
      case 
        when parameters[:_method]
          parameters[:_method].downcase.to_sym
        when parameters[:fb_sig_request_method]
          parameters[:fb_sig_request_method].downcase.to_sym
        else
          request_method_without_facebook_overrides
      end
    end
  end
  alias_method_chain :request_method, :facebook_overrides
end

The challenge with hosting of multiple Rails-based Facebook applications is that the amount of users grow quickly. To address this problem we are using EC2 nodes that we can expand/shrink as the demand grows. The price/performance ratio isn’t quite what we first expected, so we are moving toward having a few dedicated boxes instead. Another problem that we add at least a couple of applications a week. On each box that hosts them, we need to reconfigure monit, haproxy, nginx, logrotate and nagios.

dedicated-1:

    description: "The dedicated box #1"
    ip: 64.233.167.99
    failover: dedicated-2

    apps:

        bookshelf:
            port: 5000
            instances: 20
            response: Book

        ljconnect:
            port: 6000
            instances: 7
            virtual: ljconnect.hungrymachine.com
            response: Journal  
                      

That definition would generate a monit config with 20 instances of the bookshelf application and 7 instances of the ljconnect application plus all other configurations (including nagios health checks expecting the response value) . It is all possible because we adopt a fixed application deployment file structure and port numbering conventions (via offsets) for all servers.

Developing applications for Facebook is a pain. The tunnel approach helps a lot to ease that pain but even then I prefer to start a FB app as a regular application, polish the logic, and then convert it to the Facebook one by adding FBML and such. At the early stages of the development I have the mocked parameter in config/facebook.yml set to true and keep this code in config/initializers/facebook.rb:

PERSON_PROFILE_URL = "http://www.facebook.com/profile.php"

FACEBOOK_CONFIG = YAML.load_file("#{RAILS_ROOT}/config/facebook.yml")[RAILS_ENV] || {}

if FACEBOOK_CONFIG['mocked']

  class Facebook::FBMLController

    require 'ostruct'
    FB_SESSION = OpenStruct.new(:session_user_id => 1, :session_key => "12345", :is_valid? => true)

    def fbsession; FB_SESSION; end

    def require_facebook_install; true; end

    def redirect_to(url); super; end

    def url_for(*params); super; end

  end

  module Facebook::Acts::FbUser
    module InstanceMethods
      def friends
        (self.class.find(:all) - [ self ]).collect(&:uid)
      end
    end
  end 

end

It mocks out just enough of Facebook on Rails functionality to use FBMLController and acts_as_fb_user from the beginning without Facebook backend.

Some facebook applications might have multiple entries. For example, a user might be adding an application (action – new) or replying to an invitation (action – reply, param – id). Since the UI for Facebook application configuration allows to provide only static Post-Add URL it might seem like there is no way to route users back to the original action if they tried to reach when the application has not been installed for them. Luckily, we have full control on the destination via the next paramater of the post install URL. All we need is to build a URL using the incoming call parameters with the exclusion of Facebook-specific ones.

This is an example for Facebook on Rails based code that might go to the application controller:

class ApplicationController < Facebook::FBMLController

protected

  before_filter :require_facebook_install 

  def require_facebook_install    
    if in_canvas? && !fbsession.is_valid?
      redirect_to fbsession.get_install_url(:next => url_for(post_install_params))
      false
    end
  end

  def post_install_params
    params.merge(:init => true).delete_if { |k, v| k.starts_with?('fb_sig') }
  end

end

Notice that the code sets the init parameter so it can be used to identify a post install call

Sometimes it is useful to do some action on a Facebook user right after your application has been installed by the user. For example, you might want to push some default FBML to user’s profile in case he does not complete the action you expect him to do after installation. Facebook application configuration allows to provide Post-Add URL to route users to the destination url after the application install. It could be a dedicated post_add action or, in case of a default action where you have some code in the controller and since Facebook limits amount of redirects you can use, it could be a parameter to the url, like &init=true, used to identify that it was a post-install action and execute on it.