Slow Responses from the BrainTree Ruby Gem? Try This Fix.

A few weeks ago I was tasked with trying to mitigate some timeout issues in a client’s Rails app making BrainTree calls. This was becoming more of a problem as the client’s users built up more & more history in BrainTree. Apparently you can’t paginate the results or ask BrainTree to exclude certain parts of the response via the API. So you can end up getting two years worth of transaction history that you don’t even care about attached to the piece of data that you do care about. As you’ll see, parsing that potentially big ball of XML can become a problem.

I started by outputting timestamps of the interactions with BrainTree to see if the slowness was on our side or theirs. For many calls it was slow on both ends. As an example, it might take 20 seconds for BrainTree to respond with the XML for the request and then another 28 seconds(!) for the BrainTree gem to parse that response. My client’s server was set to issue a timeout after 45 seconds, so you can see how this was a problem (besides the fact that we wouldn’t want the users to have to wait so long for a response).

As I dug a little deeper I discovered that the gem *should* use the speedy LibXML gem instead of the default REXML to do XML parsing. Unfortunately, we hadn’t installed LibXML. So I installed and configured LibXML but I still got the same results. So then I dug into the BrainTree gem’s code and discovered that there was a bug which was preventing it from finding LibXML.

The problem was a simple typo — the gem was looking for “::LibXml::XML” but it should have been checking for “::LibXML::XML”. See the difference? The ‘M’ and the ‘L’ need to be capitalized.

So I changed the gem’s code and ran my test again. This time it still took the same amount of time for BrainTree to send us the XML response but the parsing only took 2 seconds instead of 28 seconds.

I’ve submitted a pull request to BrainTree for this fix. You can see my commit here.

Calculating Standard Deviations in Ruby on Rails (and PostgreSQL)

I need to calculate some Bollinger Bands (BBs) for SwingTradeBot, which is built in Rails 4. Here’s a quick definition of Bollinger Bands:

Bollinger BandsĀ® are volatility bands placed above and below a moving average. Volatility is based on the standard deviation, which changes as volatility increases and decreases.

So I needed to do some standard deviation calculations. I found a few Ruby gems that allow you to do statistics but I quickly ran into issues with them. The general approach of the gems is to monkey patch Array and/or Enumerable, which can cause other issues. I was getting conflicts with ActiveRecord b/c of the monkey patches redefining “sum” and there was another conflict with a different gem that I tried. There are supposedly fixes for this stuff but it just felt dirty.

Then, as I often do, I wondered if I could just get the database to do the calculation for me. If so, it would be faster that way and I wouldn’t have to go monkey patching Ruby and/or clutter my app with my own standard deviation code. It was a pretty simple thing to have PostgreSQL do the calc for me. I just needed Rails to produce a query like this:

SELECT stddev_pop(close_price) FROM prices
WHERE (stock_id = 3313 and day_number > 195 and day_number <= 215)

Seems simple enough. So here's the Rails code to do just that:


result = Price.select('stddev_pop(close_price)').where("stock_id = #{stock_id} and day_number > #{day_number - 20} and day_number <= #{day_number}").load
#Note that I couldn't do ".first" on the line above b/c that creates an ORDER By clause that PostgreSQL complains about b/c the column being ordered by is not in the GROUP clause...
standard_deviation = result.first.stddev_pop
self.upper_bb = twenty_day_moving_average + (standard_deviation * 2)
self.lower_bb = twenty_day_moving_average - (standard_deviation * 2)

Done!

I’ve Finally Found a Rails 4.x Blogging Engine / Gem

I can’t believe how difficult it’s been to find a good solution for plugging a simple blog into an existing Rails app. I wanted to add a blog to SwingTradeBot, the new site I’m building but most answers to this question that I’ve found say to either use RefineryCMS or “roll your own. Well I tried Refinery and quickly ran into gem conflicts galore. As for rolling my own… I don’t have time for that — I’d rather use something that’s been thought through and is well suited to the task.

I was ready to give up and just roll my own when I found the Monologue gem. That looked really promising but then I ran into a Rails 4 compatibility issue. However, reading through the discussion thread on that issue I discovered that somebody had created the Blogo gem (plugin / engine).

It’s still early days with this gem but so far, so good for the most part. Installation and set-up went smoothly (in development mode). Here are some things I ran into after pushing to production (on Heroku):

  1. There’s a generator to create the admin user (rake blogo:create_user[user_name,user@email.com,password]) – that didn’t work in production. I didn’t find out until after creating the user manually in a Rails console that I needed to prepend ‘RAILS_ENV=production” to the rake command.
  2. The assets were missing. running “RAILS_ENV=production rake assets:precompile” fixed that.
  3. Note that for comment to appear you need to be signed up for Disqus and you need to enter your site’s short name into the Blogo config.
  4. There are some configuration options that I had to discover via digging through the code. See below for an example of what I’ve added to my config/application.rb

Here’s what’s in my config/application.rb:


Blogo.config.site_title = "SwingTradeBot Blog"
Blogo.config.site_subtitle = "Some clever subtitle..."
Blogo.config.keywords = 'stock trading, technical analysis, stock scanning'
Blogo.config.disqus_shortname = 'swingtradebot'
Blogo.config.twitter_username = 'swingtradebot'