Technology Musings

Deploying a Rails Application to Elastic Beanstalk

JB — Sat, 14 Jul 2018 05:22:00 +0000

I'm not a big AWS fan because it is a lot of work. However, sometimes a client requires it. In any case, if you are wanting to deploy to AWS Elastic Beanstalk using a Rails app, here are your step-by-step instructions!

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545; min-height: 14.0px} li.li1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545} span.s1 {color: #e4af0a} ol.ol1 {list-style-type: decimal}

Preparing a Rails app for EB deployment:

Install the elastic beanstalk tools:
From Python: sudo pip install awscli awsebcli --upgrade
From homebrew: brew install aws-elasticbeanstalk
AWS Config
Set ~/.aws/config as follows:

[default]
region = us-east-1

[profile eb-cli]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET

The name of the profile "eb-cli" will need to match future steps.
Create an AWS EB application in the EB GUI:
1. Create the application and give it a name
2. Create an environment (select "web server environment") and name it whatever you want
3. Select "Ruby" for the preconfigured platform
4. Leave the application as "sample application"
5. Hit "Create Environment" and wait for it to finish (this takes forever)
Enable load balancing so that you have more control (optional, but required for HTTPS I think):
1. Go to "configuration" then "Capacity"
2. Change "Single Instance" to "Load Balanced"
3. Reduce "max" to "1" to prevent additional instances from launching
4. Hit "apply" and "confirm" and then wait (this takes forever)
Change the instance type to "t2.small" at the lowest. If it is smaller than this, you will get strange errors when deploying because it is running out of RAM to do the deployment.
1. "Configuration" -> "Instances" -> "Modify" -> "Instance Type". Set to t2.small.
Add a data tier
1. "Configuration" -> "Database" -> "Modify"
2. Options:
  Engine: postgres
  Engine version: 9.6.8
  Username/password: set them and write them down, but you don't really need them
Configure Environment Variables
1. "Configuration" -> "Software" -> "Modify"
2. BUNDLE_WITHOUT: test:development
3. RACK_ENV: production
4. RAILS_SKIP_ASSET_COMPILATION: false
5. RAILS_SKIP_MIGRATIONS: false
6. RAILS_SERVE_STATIC_FILES: 1
7. SECRET_KEY_BASE: use "rake secret" to generate one (note that leaving this out can lead to some weird errors that seem senseless)
8. Any other configuration (check your .env file if you have one)
Modify your app to use your data tier. In config/database.yml, make sure that the adapter is postgresql for "default", then set production to be like the following:

production:
<<: *default
database: <%= ENV['RDS_DB_NAME'] %>
username: <%= ENV['RDS_USERNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
port: <%= ENV['RDS_PORT'] %>
In your app, Create a .elasticbeanstalk/config.yml file in your project's home directory
1. File contents:
  
  global:
  application_name: your-app-name
  default_region: us-east-1
  profile: eb-cli
  branch-defaults:
  master:
  environment: your-staging-environment
2. The region should probably match the region of the application (though I am not certain), the profile should match the one in ~/.aws/config, and the application_name should match the actual name of the application in elastic beanstalk.
3. "branch-defaults" allows you to set some parameters per-branch (based on the git branch). Here, we are saying that when using the master git branch, the environment we should push to is "your-staging-environment".
Create a file in your app .ebextensions/010_packages.config (loads packages we will probably need - may need others based on requirements):

packages:
yum:
git: []
gcc: []
libxml2-devel: []
libxslt-devel: []
patch: []
sqlite-devel: []
libffi-devel: []
postgresql96-devel: []
Make sure that everything in your repository is committed and ready to go. Then, run

eb deploy

and everything should work.
Add a CNAME for your environment in your DNS manager, whatever that is. It should CNAME to the name listed in your environment (i.e., my-environment-name.us-east-1.elasticbeanstalk.com) (optional - you can also just leave it with the generated name, but then you won't be able to add SSL).
Turn on SSL (optional)
1. Go to AWS Certificate Manager
2. Click "Request a Certificate" -> "Request a Public Certificate"
3. Enter in your domain name (the CNAME you created)
4. Choose "DNS Validation"
5. When it says "Pending Validation", open up the "domain" area and create the requested CNAME in your DNS in order to validate your domain. When complete, click "Continue"
6. Go to your EB environment page
7. Go to "Configuration" -> "Load Balancer" -> "Modify"
8. Click "Add Listener":
  Listener Port: 443
  Listener Protocol: HTTPS
  Instance Port: 80
  Instance Protocol: HTTP
  SSL Certificate: select the certificate previously generated
9. Note that after adding the listener, there is an "apply" button buried on the webpage.
10. You can now access the site using SSL on the CNAME you created.

Hot-Patching Ruby for TLS 1.2

JB — Fri, 02 Mar 2018 22:21:00 +0000

Authorize.net recently started denying TLS requests that were less than 1.2. I thought I was fine, but I had one server running CentOS 5. It was not fine. I had a very ancient Rails application on an ancient box unable to connect to Authorize.net.

Anyway, if you are in a similar situation, here is how I solved it. This may not be exact, but should hopefully get you far enough along to figure it out yourself:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Helvetica Neue'; color: #454545; min-height: 14.0px}

For authorize.net's TLS upgrade, it is based on your server's OpenSSL library version. To see if you are compatible, run the following command:

    openssl s_client -connect secure.authorize.net:443

If you get a one-line error message like this:


    6641:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:586:

Then you are NOT compatible. To fix, you need to either update your OpenSSL in Linux or get the latest OpenSSL from here - https://www.openssl.org/source/

For building, you may have to configure it something like this:


    export CFLAGS=-fPIC
    ./config shared
    make
    make install     # this one as root

I also copied the generated shared libs into /lib64. In theory, after updating, it would be best to rebuild ruby from source. However, absent that, there is a trick you can do:

Create a new file, I called mine ruby_openssl in the same directory as the old ruby. The file should have the following contents:

    #!/bin/sh
    export LD_PRELOAD=/lib64/libssl.so.1.0.0:/lib64/libcrypto.so.1.0.0
    exec /opt/ruby-1.8.7-p352/bin/ruby $*

The LD_PRELOAD should go to the shared files that were installed by installing OpenSSL, wherever you happened to install them.

Then, in your apache config, reference your new ruby_newopenssl wherever you were referencing ruby.

Calculating an Even Baseline

JB — Tue, 24 Oct 2017 02:49:00 +0000

So, I've been working in Xamarin for cross-platform phone development. I was using their AbsoluteLayout class to try to get a bar graph working using relative sizes (AbsoluteLayout with relative sizes gives everything an absolute position from 0, top or left of the screen, to 1, the bottom or right of the screen). The goal was to have the bottom of the bar graph halfway down the screen (to allow room for negative values to point down). The goal was to look like this:

The problem, though is getting the boxes to align at the center line. Now, part of the problem may be my inexperience with the platform. There may be a really easy way to do this. However, here is what I was running into:

I thought that Xamarin worked by aligning the your specified position to the middle of the box. Therefore, I started by calculated the box height, and then calculating the origin of the box by setting it above the 0.5 (i.e., 50%) line by half of the box height. Therefore, when it extended, it should extend down to the 0.5 line.

The problem, though, is that Xamarin doesn't align to the middle of the box. It varies the alignment by where in the positioning it is positioned. So, if you position a box at 0.2, not only is it positioned at 0.2, but the anchoring point is also 0.2 down the box!

Therefore, I needed a way to calculate where I needed to actually position the box to get it to rest on the 0.5 line.

The way I calculated it is this - let's call the y-position that you want the box to rest on as P. Let's call the y-position which you are going to set your box to as A (for the anchoring position). Let's call the height of the box itself H.

What we want to happen is for the bottom of the box to hit P.

Now, we can split up the measurements from the top of the box to P into two lengths - the length from the top to the anchoring point (which is actually just A) and the length from the anchoring point to the bottom of the box.

Since the anchoring point sits A down on the box, the length of the top half of the box is given by A*H. However, this is not the distance we want. We want the distance down the rest of the box. That will be (1 - A)*H.

So, our two distances are: A and (1 - A)*H. They should add together to be P. Therefore, our equation is:

A + (1 - A)*H = P

However, we want to solve this for A - we already know our box height (H), and we know what position we want this to extend to (P). Therefore, we have to solve for A:

A + (1 - A)*H = P
A + H - A*H = P
A - A*H = P - H
A*(1 - H) = P - H
A = (P - H) / (1 - H)

And this gives a perfect calculation for A. In my case, since P is 0.5, I just do A = (0.5 - H) / (1 - H) and it comes out perfectly.

It should be fairly simple to come up with similar calculations for top alignment and center alignment. Additionally, these work with right-to-left alignments as well.

Finding Derivatives without Limits or Differentials

JB — Thu, 11 Feb 2016 16:29:00 +0000

I always enjoy finding new ways of doing things. I was reading Henle/Kleinberg's Infinitesimal Calculus, which is a great book. It has, throughout the book, a main text and a side-text. The side-text is infinitely more interesting than the main text.

Anyway, on pages 65-66, it gives a method for finding the slope of y = x^2 at a specific point, let's say its at (2,4). The way you do it is kind of strange, but it seems to work. You start by defining a line to intercept the equation:

y = mx + b

Now, we don't know what m and b are, but we *do* know what x and y are. Therefore,

4 = 2m + b

We don't really care about b in the long run, so we want to find an equation for b to remove it from the mix:

b = 4 - 2m

Now our equation for the line becomes:

y = mx + 4 - 2m

Interestingly, we actually have a value for y - y = x^2, so this becomes:

x^2 = mx + 4 -2m

We can rearrange this into quadratic form with:

x^2 - mx + (2m - 4) = 0

Using the quadratic formula, we can do:

x = (m +- sqrt(m^2 - 4(2m - 4))) / 2

x = (m +- sqrt(m^2 - 8m + 16)) / 2

x = (m +- sqrt((m - 4)^2)) / 2

x = (m +- (m - 4)) / 2

Now, we want to take the "+" side of the "+-" since we are solving for "m". Otherwise, we will lose "m". Therefore, this becomes:

x = (2m - 4) / 2

x = m - 2

So, at the point we are looking at, what is the value of x? At (2, 4), x is 2. Therefore

2 = m - 2

m = 4

Therefore, the slope at (2, 4) is 4.

Interestingly, we can also generalize this to get the full derivative for y = x^2. To do this, we will introduce the variables p and q to be the specific x and y values at a specific point. Therefore, the slope at (p, q) for y = x^2 is:

y = mx + b, therefore q = pm + b, therefore b = q - pm

q = p^2, therefore b = p^2 - pm

y = mx + p^2 - pm

y = x^2, therefore:

x^2 = mx + p^2 - pm

x^2 - mx + (pm - p^2) = 0

x = (m +- sqrt(m^2 - 4(pm - p^2)))/2

x = (m +- sqrt(m^2 - 4pm + 4p^2))/2

x = (m +- sqrt((m - 2p)^2))/2

x = (m +- (m - 2p))/2

x = (2m - 2p) / 2

x = m - p

m = x + p

Since p = x, this gives

m = x + x = 2x, which is the derivative of x^2

I don't know how many classes of functions you can do this successfully for, but I imagine it should cover quadratics at least.

Ratios of Infinity

JB — Wed, 13 Jan 2016 03:49:00 +0000

A lot of people do a lot of thinking about whether or not infinities exist, and wonder how real numbers can add up to infinity.

Sometimes I wonder if that sort of thought might be the inverse of reality - perhaps the infinities are the foundational realities, and all finite numbers are merely infinities in ratio.

A Meaning for d^2y/d^2x?

JB — Fri, 01 Jan 2016 20:29:00 +0000

I have recently been randomly curious about random calculus-y things. Anyway, the notion for the second derivative of a function is d^2y/dx^2. This is the second differential of y divided by the first differential of x squared.

However, there is, technically, also a d^2x, though it doesn't get much attention. And, d^2y and d^2x can be put into ratio with each other, but I don't really know what it means. But it is an interesting operation nonetheless.

So, the derivative of an equation is dy/dx and the derivative of its inverse is dx/dy.

The second derivative of an equation is d^2y/dx^2 and the second derivative of its inverse is d^2x/dy^2.

Therefore, to get the d^2y/d^2x you just do:

(2nd Derivative of y wrt x / 2nd derivative of x wrt y) * (First derivative of the inverse)^2

Or:

(d^2y/dx^2) / (d^2x/dy^2) * (dx/dy)^2

I will post an example later when I have a good one.

The derivative of u^v

JB — Sun, 27 Dec 2015 06:45:00 +0000

I don't know why this isn't listed as a standard rule. The differential of the exponent function, u^v (u raised to the v power), is pretty basic, and you can use it to formulate the other exponent rules for differentiation. However, for some reason it seems to be left off of most rules for differentiation.

The basic rule is this:

d(u^v) = v*u^(v-1)*du + u^v*ln(u)*dv

This can be clearly seen to be a more general form of u^n, because if n is a constant, this becomes:

d(u^n) = n*u^(n - 1)*du + u^v*ln(u)*0

And that zero drops it to the rule we all know and love:

d(u^n) = n*u^(n - 1)*du

This can be derived as follows:

z = u^v
ln(z) = ln(u^v)

Using log rules, we get:

ln(z) = v*ln(u)

Now take the differential of both sides:

d(ln(z) = d(v*ln(u))

Differentiating both sides, we get:

dz/z = v*du/u + ln(u) * dv

Then we multiply by z:

dz = z*v*du/u + z * ln(u) * dv

Substituting in z = u^v:

dz = u^v * v * du / u + u^v * ln(u) * dv

Now, u^v/u simplifies to u^(v-1), giving us:

dz = v*u^(v-1)*du + u^v*ln(u)*dv

Since z = u^v, dz = d(u^v), so

d(u^v) = v*u^(v-1)*du + u^v*ln(u)*dv

Rails and Updated Postgres not Performing Migrations

JB — Tue, 22 Dec 2015 16:50:00 +0000

Apparently, there was a change to something in postgres 9.5, or maybe just my install of it. If you upgrade your Postgres database version and suddenly your rails app stops doing migrations, add the following line to config/database.yml database config to get it working again:

  schema_search_path: public

The error message i was getting was:

ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR:  relation "schema_migrations" already exists

Basically, sometimes Rails was finding the schema_migrations table, and sometimes it wasn't, but setting the schema_search_path fixed it.

I'm still not sure where exactly the problem came from, but setting the setting fixed it.

Explanation of Liebniz Notation of the Second Derivative

Sun, 13 Dec 2015 14:27:00 +0000

The notation for the second (and higher) derivatives in Liebniz notation has always troubled me. The second derivative is usually notated as d^y/dx^2. And I always wondered, why is the the 2 in relation to the d on the top, but in relation to the whole term on the bottom?

This puzzled me for a while, and I looked through at least 10 Calculus textbooks to find the answer, all to no avail. Finally, I put pen to paper and figured it out. The answer was straightforward, and I am sure that whoever invented the 2nd derivative Liebniz notation knew why they did it that way, it's just that every Calculus book since then seems to have forgotton.

Anyway, the usual notation for the derivative operation id d/dx. I eventually came to realize that this is not a single operation, but TWO operations in one. The first is to take the differential (not derivative) of the equation. For that, I mean, let's say you have the equation y = 2x. The differential of that equation is dy = 2xdx. Doing a differential instead of a derivative is powerful, because it allows a more natural approach to both implicit differentiation and multivariable differentiation.

Now, the derivative is simply the differential divided by dx. So, if our differential is dy = 2xdx, then when we divide by dx we get our normal notation dy/dx = 2x.

Now, let's leave off dividing by dx, and just look at the differential. What happens if we take a second differential?

In this case, we have to treat dy as a separate variable. So, if the differential of y is dy, what is the differential of dy? This is where the d^2y comes in. The 2 is by the "d" not because it is "d squared", but it is "d applied twice". So, if we take our initial differential dy = 2xdx and take another differential, what do we have?

By the product rule, we get:

d^2y = 2((x)(d^2x) + (dx)(dx))

Now, since we want the second derivative, we divide both sides by dx twice (once for the first derivative, and once for the second), which is the same as dividing by dx^2.

That gives us:

d^2y / dx^2 = 2((x)(d^2x) + (dx)(dx)) / dx^2

The (dx)(dx) simplifies to dx^2, and we can separate this out at the plus sign to give us:

d^2y / dx^2 = 2(x)(d^2x)/dx^2 + 2(dx^2)/dx^2

This further simplifies to:

d^2y / dx^2 = 2(x)(d^2x)/dx^2 + 2

Now, we know that the second derivative is 2, so that means that 2(x)(d^2x)/dx^2 must equal zero. But why?

Well, let's rewrite this term a little:

2x * (d^2x)/(dx^2)

d^2x/dx^2 is simply the second derivative of x with respect to itself!

To understand why this must be zero, first imagine the first derivative of x with respect to itself: dx/dx

This must always be 1! Another way of stating this is that "for every change in x, the corresponding change in x is equivalent", which seems obvious.

Now, 1 is a constant, so the second derivative must be zero! This zeroes out the whole term, leaving:

d^2y/dx^2 = 2

Thus, we have the notation of the derivative from first principles.

Another way to look at this is to use the quotient rule.

If we start with dy/dx, then take the differential, what do we get?

((dx)(d^2y) - (dy)(d^2x)) / dx^2

If we separate, we get:

(dx)(d^2y) / dx^2 - (dy)(d^2x)/dx^2

The first term simplifies to d^2y / dx. The second term is our second derivative of x with respect to itself again, so it becomes (dy)*0, giving us:

d^2y/dx - (dy)*0

Which gives us just

d^2y/dx

Now, the second step of the derivative is to divide by dx. Doing so gives us:

d^2y/dx^2

Which is our second derivative!

A General Solution for Line Integrals

JB — Sat, 14 Nov 2015 17:27:00 +0000

Update: I found that this is a standard formula, but it is referred to as an "arc length" formula, not a line integral. I knew it had to be somewhere, I was just searching the wrong terms.

I was playing around last night, and I came up with, what appears to be, a general solution for line integrals. I have looked around and I have not found this equation anywhere, although I am certain in must be well-known somewhere. In any case, I will give it here, because it is definitely not well-known on the Internet.

This is an equation for determining the line integral of y = f(x), where f(x) is a true function of x (i.e. it passes the vertical line test), and is differentiable on the portion of the line where you want to find the length of the line segment. Most line integrals work by adding in an auxiliary parameter, t, to the equation. This operates directly on the function itself.

The short version:

To get a line integral from x=a to x=b on y=f(x), we can use the following formula:

Evaluate this from x=a to x=b and you will have the length of the line. Note, the software I was using gives this as a partial derivative on the inside. I don't know enough to know if it is only limited to partials or not.

So, if you know the derivate, you can figure out (in theory) the line integral. Whether or not this actually helps you find a solution to the integral is another story.

Now, how does one arrive at that?

Well, first, remember, that an integral is an infinite sum of infinitesimals. A curve is simply an infinite sum of tiny, infinitesimal lines. So, all we need to do is find the length of each line segment and add it up.

How big are our line segments?

Well, if y=f(x), then the line segment is going to be the line between the points x, y and x + dx, y + dy. The distance formula states that the line length is sqrt( (x1 - x0)^2 + (y1 - y0)^2). The difference between x + dx and x is simply dx, and the difference between y + dy and y is simply dy. So, each of our infinitesimal line segments (dL) will be:

dL = sqrt(dx^2 + dy^2)

If we square both sides, we get the following:

dL^2 = dx^2 + dy^2

dL^2 - dx^2 = dy^2

Now we can divide both sides by dx^2 and get:

dL^2/dx^2 - 1 = dy^2/dx^2

dL^2/dx^2 = 1 + dy^2/dx^2

dL^2/dx^2 = 1 + (dy/dx)^2

Now we can square root both sides:

dL/dx = sqrt(1 + (dy/dx)^2)

dL = sqrt(1 + (dy/dx)^2) dx

And this is the final form of our equation. I tested it out with a few simple integrals (a line and the top half of a circle) and it seems to work as expected. And, as I said, if you have a line that loops back around, you still need to use the parameterized methods. But this is an easier way to setup line integrals where f(x) is a true function of x.

ActiveMerchant, Authorize.Net Root Certificates, and Linux SSL

JB — Thu, 28 May 2015 21:27:00 +0000

This week, Authorize.net is starting its rollout of new server certificates, signed by a new certificate authority - Entrust. If you are using ActiveMerchant, this could break your stuff! It may or may not affect newer versions of ActiveMerchant - I don't know, I only have sites using old versions (like 1.18). So, even if it breaks, the solution may be different depending on your version of ActiveMerchant. Still, I'll try to keep the info detailed enough that you can find the solution yourself.

First, the problem. This affects you if, when ActiveMerchant tries to connect to Authorize.net, you receive the error:

SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B:
  certificate verify failed

Note that you may not get this error when trying to connect to Authorize.net via manual Ruby code, but only via ActiveMerchant.

The reason for this is ActiveMerchant maintains its own list of Root CAs!

Therefore, to fix, you have to, (a) make sure that the new Root CAs for authorize.net are stored in your server's root CA list, and (b) make sure that ActiveMerchant is using your server's list rather than its own. For good measure, you can also copy ActiveMerchant's CA list to yours.

So, first, to add the Entrust root CAs to your Linux server (assuming it is a fairly recent CentOS-based distro), do the following as root:

Enable the trust infrastructure with the following command:update-ca-trust enable
Go to /etc/pki/ca-trust/source/anchors
Grab the .cer files from https://www.entrust.net/downloads/root_request.cfm (just right-click on the links, copy the URLs, and download them to the current directory with wget)
Go to wherever the gem files for your project are, and find the active_utils gem. In the lib/certs directory, there is a file called cacert.ca or something like that. Copy that to /etc/pki/ca-trust/source/anchors
Now run the following command to update your CA list:
update-ca-trust

Now you have all of the certs installed. Now we need to tell ActiveMerchant to use your own certs rather than its internal list. However, again, this is in the active_utils gem, not the activemerchant gem. Within the gem, find the file called lib/active_utils/common/connection.rb. Now look for a function called configure_ssl. Copy that function to your clipboard.

Now create a file in your own project called config/initializers/active_merchant_pem_fix.rb (the name doesn't matter as long as it is in config/initializers). In this file you need to have:

module ActiveMerchant
  class Connection
    # Paste the configure_ssl function you copied from lib/active_utils/common/connection.rb here.
  end
end

Now, there will be a line that says something like this:

        http.ca_file = File.dirname(__FILE__) + '/../../certs/cacert.pem'

This is the offending line!!! Comment that sucker out!!!

For my version of ActiveMerchant, this is what my file looks like:

   def configure_ssl(http)
      return unless endpoint.scheme == "https"

      http.use_ssl = true

      if verify_peer
        http.verify_mode = OpenSSL::SSL::VERIFY_PEER
        # http.ca_file = File.dirname(__FILE__) + '/../../certs/cacert.pem'
      else
        http.verify_mode = OpenSSL::SSL::VERIFY_NONE
      end
    end

Now that you have the updated root CA on your server, and ActiveMerchant is using your server's list rather than your own, you are all set!

Why You Shouldn't Put Validations on Your Model

JB — Wed, 11 Jun 2014 21:19:00 +0000

This post is about model validators in Ruby on Rails, although it probably applies to validators in most similar MVC systems that allow declarative model validation.

One of the biggest disasters, in my opinion, has been the trend for developers to put more and more validation in their models. Putting validation in the model always *sounds* like a good idea. "We want our objects to look like X, therefore, we will stick our validations in the place where we define the object."

However, this leads to many, many problems.

There are two problems, and the most general way of stating them is this:

life never works the way you expect it, but validating on the model assumes that it will.
while it is cleaner to validate on the model for a given configuration, when that validation needs to change, you don't always have enough information to understand the purpose of the validation in the first place (i.e., if a new developer is making the change)

Example of both -

Let's say that we require all users to enter their first name, last name, birthdate, and email. Okay, so we'll put validators on the model to make sure that this happens.

Let's say that six months go by, and your company signed an agreement with XYZ corporation to load in new users. However, XYZ corp didn't require that their users enter in their birthdate. However, you don't know that, and the bit of data you looked at, it looks like they are all there. So, you do a load through the database, and everything loads in cleanly, and you think that everything is good.

In fact, *Everyone* can view their records.

However, the next week, people start getting errors when they try to update their profile. Why? Because the people who don't have a birthdate entered are getting validation errors when they try to update something else.

So here is problem #1 - if an existing record doesn't match the validation, nothing can be saved, and this causes headaches because you are getting errors in places where they were unexpected. This doesn't just happen with dataloads. It happens every time you add a validator as well. So, every time you change policy, even if it seems benign, you have the potential of messing up your entire program, even if all of your tests succeed (because your tests will never have had the bad data in them, since it doesn't have the old version of the code!).

So, we go back and recode, and remove the validator from the model. Great, but let's say this is a new programmer. Now the programmer has to figure out where to put the validation in. Presumably, we still want new users to enter in their birthdate, even if we allow non-birthdate users that we load from outside. However, now, because we aren't validating on the model, we have to move it somewhere else. But where? Since the validator just floated out there declaratively, there is no direct link between the validator and *where* it was to be enforced. Now the new programmer has to go through and recode and retest all of the situations to figure out the proper place(s) for the validator. This is especially problematic if you have more than one path for entrance on your system. It's not so bad if it is a system you wrote yourself, but when having to maintain someone else's code, this is a problem.

Therefore, the policy I follow is this - only use model validation when a failure of the validator would lead to a loss of data *integrity*, not just cleanliness. The phone number field should be validated somewhere else. You should validate a uniqueness constraint only if uniqueness is actually required for the successful running of the code. Otherwise, validate in the controller. Now, you can make this easier by putting the code to validate in the model, but just don't hook it in to the global validation sequence. So, go ahead and define a method called "is_proper_record?" or something, but don't hook it up to the validation system - call it directly from your controller when you need it.

NOTE - if I don't get around to blogging about it, there are similar problems with State Machine, and a lot of other declarative-enforcement procedures in the model. Basically, they are elegant ways of coding, but *terrible* when you need to modify them, especially if the person modifying the code wasn't the one who built it. Untangling the interaction between declarative and manual processes leads to both delays (as the new developer tries to figure out the code) and bugs (when the new developer doesn't figure out the code perfectly the first time). When your processes are coded *as processes* and not declarations, it is more visible to future programmers, and therefore more maintainable.

New Library for Reading Pickle Files from Ruby

JB — Wed, 11 Jun 2014 04:18:00 +0000

This is announcing the release of "pickle-interpreter", a new ruby gem for reading Python pickle files in ruby and rails projects. The project is available at https://github.com/newmedio/pickle-interpreter.

It is called "pickle-interpreter" because Pickle is actually a mini-stack-based programming language, so this is an interpreter for the Pickle language.

New Book Integrating Engineering, Philosophy, and Theology

JB — Mon, 03 Mar 2014 05:07:00 +0000

I thought some of you, my faithful blog readers, might be interested to know about a project I just completed with some other people - a new book about integrating engineering, philosophy, and theology. It is titled Engineering and the Ultimate: An Interdisciplinary Investigation of Order and Design in Nature and Craft.

Creating a WSDL file for a RESTful Web Service

JB — Wed, 12 Feb 2014 22:45:00 +0000

NOTE - this is about XML-based REST services. There *might* be a way to write WSDL for JSON, but I don't know it yet.

Most of my work is done in RESTful environments. However, the Microsoft world seems to favor SOAP. REST is simple. It is so simple, that you don't even really need special APIs for the different services. You just need the URL and the documentation. However, SOAP is complex, and requires complex tooling. SOAP services are described using the WSDL format. Usually what happens is that a program reads the WSDL file and spits out stubs that you can code against directly.

In any case, the people that come from a SOAP environment don't realize that you can do cool stuff without tooling, and the idea that I could just give them a URL to hit and they could just hit it directly boggles their mind. So, if I have a handy-dandy REST service, they still want to ask me for the WSDL file. There's really no reason, but that's what they want. So, thankfully, WSDL 2 allows for us to create WSDL descriptions of RESTful services.

So, before I describe how WSDL works, let me first say that, if you are used to the simplicity of REST coding, WSDL will blow your mind. WSDL is built around the complexities of SOAP. There are a lot of things where you will say, "why did they put in that extra layer?" The answer is, "because the SOAP protocol was built by crack addicts."

WSDL is grouped into 4 parts - services, bindings, interfaces, and types.

An interface is a list of operations you might want to perform. This is, essentially, the API itself. Our API will be for retrieving form submissions to our website. The interface is a list of operations, where each operation is an individual API call. In our example, we will have a "list" and a "get" operation in our inteface. The interface relies on types in order to define the input and output of the operations.

A binding tells you how to make an API call for an operation. This is significantly simpler in REST than in SOAP, as you really only need the URL path and the method (this is the relative URL - the absolute path will be defined in the service). The binding will have a operation tag for each operation it binds.

A service defines where on the web you can go to in order to find the interfaces. It defines a set of endpoints, each of which tells the top-level URL of the resources, like http://www.example.com/api, and what interfaces they support and which bindings to use on them (telling both the interface and the binding seems redundant to me, but then again so does writing a WSDL file to begin with).

So, below, is a simple REST web service. I've excluded the "types" section, which I will discuss later.

<?xml version="1.0" ?>
<wsdl2:description
	xmlns:wsdl2="http://www.w3.org/ns/wsdl"
	xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl"
	xmlns:whttp="http://www.w3.org/ns/wsdl/http"
	xmlns:xs="http://www.w3.org/2001/XMLSchema"
	xmlns:myexample="http://www.example.com/api"
	targetNamespace="http://www.example.com/api"
	<wsdl2:types>
	  <xs:schema
			attributeFormDefault="unqualified"
			elementFormDefault="unqualified">
             <!-- omitted -->
          </xs:schema>
	</wsdl2:types>

	<wsdl2:interface name="FormSubmissionsInterface">
		<wsdl2:operation
			name="listFormSubmissionsOperation"
			pattern="http://www.w3.org/ns/wsdl/in-out"
			whttp:method="GET">
			<wsdl2:input element="formSubmissionSearchParameters" />
			<wsdl2:output element="response" />
		</wsdl2:operation>
		<wsdl2:operation
			name="getFormSubmissionOperation"
			pattern="http://www.w3.org/ns/wsdl/in-out"
			whttp:method="GET">
			<wsdl2:input element="formSubmissionIdentifier" />
			<wsdl2:output element="response" />
		</wsdl2:operation>
	</wsdl2:interface>

	<wsdl2:binding name="FormSubmissionBinding"
		interface="myexample:FormSubmissionBinding"
		type="http://www.w3.org/ns/wsdl/http">
		<wsdl2:operation
			ref="myexample:getFormSubmissionOperation"
			whttp:method="GET"
			whttp:location="form_submissions/{id}.xml" 
		/>
		<wsdl2:operation
			ref="myexample:listFormSubmissionsOperation"
			whttp:method="GET"
			whttp:location="form_submissions.xml"
		/>
	</wsdl2:binding>

	<wsdl2:service
		name="FormSubmissionsServiceSSL"
		interface="myexample:FormSubmissionsInterface">
		<wsdl2:endpoint
			name="FormSubmissionEndpoint"
			binding="myexample:FormSubmissionBinding"
			address="https://www.example.com/api"
		/>
	</wsdl2:service>
	<wsdl2:service
		name="FormSubmissionsService"
		interface="myexample:FormSubmissionsInterface">
		<wsdl2:endpoint
			name="FormSubmissionEndpoint"
			binding="myexample:FormSubmissionBinding"
			address="http://www.example.com/api"
		/>
	</wsdl2:service>
</wsdl2:description>

So, starting at the bottom, we have 2 services defined - an HTTP service and an HTTPS service. They both use the same binding and interface. So, they are called exactly the same way, buy have a different base URL. Depending on your client, the service name often corresponds to the main class name used to access the service.

The services implement the myexample:FormSubmissionsInterface. This interface defines two operations: listFormSubmissionsOperation and getFormSubmissionsOperation. Each one defines its input and output parameters. Now, the output parameter is just a regular XML element from your schema (defined in the "types" section). The input parameter is special. It, too, is defined in your types section, but, when bound to an HTTP endpoint, rather than being treated as an XML document, it will be treated as a combination of URL pieces and query string parameters. Here is mine:

                       <xs:element name="formSubmissionSearchParameters">
				<xs:complexType>
					<xs:all>
						<xs:element name="since" type="xs:dateTime" minOccurs="0" />
						<xs:element name="since_id" type="xs:integer" minOccurs="0" />
						<xs:element name="until" type="xs:dateTime" minOccurs="0" />
						<xs:element name="until_id" type="xs:integer" minOccurs="0" />
					</xs:all>
				</xs:complexType>
			</xs:element>

Each of the elements under xs:all actually become query-string parameters. WSDL treats it like an XML document, because of its SOAP roots. But, when we specify an HTTP binding, then this gets transmogrified from an XML document into query-string parameters. I have a separate type for the "get" operation, which defines the ID number of the form submission I want:

			<xs:element name="formSubmissionIdentifier">
				<xs:complexType>
					<xs:sequence>
						<xs:element name="id" type="xs:integer" />
					</xs:sequence>
				</xs:complexType>
			</xs:element>

As we will see, in the bindings section, we can move a named parameter into the URL itself, rather than just being sent as a query string.

The output is simply an XML document, whose structure is defined in the types section. I would need to teach you XSD documents to tell you how this works, but you can google it yourself. The one thing I will say is that because my result document does not use namespaces at all, in the xs:schema declaration I set elementFormDefault to "unqualified".

The binding is where we tell WSDL that we are using REST rather than SOAP. We tell it what interface we are binding, and the fact that we are using an HTTP binding. Then, for each operation, we define the HTTP method and relative path for the binding. We can substitute in our parameters into the URL path by saying {name-of-xml-input-element} in the location. Any parameters given which are not inserted into the URL path will be put in the query string. So, for instance

whttp:location="form_submissions/{id}.xml"

Says to take the "id" element and substitute it in the URL. The other location,

whttp:location="form_submissions.xml"

just puts all of the parameters in the query string.

So that's it! That's a WSDL file for a REST service!

I tested my service and WSDL using AXIS 2's WSDL2JAVA command:

wsdl2java.sh -uri path/to/wsdl -wv 2

Then I created a simple TestMe.java to make sure it ran right. The example also shows how to use HTTP Basic Authentication with the service:

import java.rmi.RemoteException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.math.*;

import org.apache.axis2.databinding.ADBBean;
import org.apache.axis2.Constants;
import org.apache.xmlbeans.XmlOptions;
import org.apache.xmlbeans.XmlValidationError;
import org.apache.axis2.client.*;


import com.example.www.FormSubmissionsServiceStub;
import com.example.www.FormSubmissionsServiceStub.*;

import org.apache.axis2.transport.http.*;

public class TestMe {
        public static void main(String[] args) throws RemoteException {
                /* Setup Service */
                FormSubmissionsServiceStub stub = new FormSubmissionsServiceStub();

                /* If you need authentication */
                HttpTransportProperties.Authenticator basicAuth = new HttpTransportProperties.Authenticator();
                basicAuth.setUsername("USERNAMEHERE");
                basicAuth.setPassword("PASSHERE");
                basicAuth.setPreemptiveAuthentication(true);
                final Options clientOptions = stub._getServiceClient().getOptions();
                clientOptions.setProperty(HTTPConstants.AUTHENTICATE, basicAuth);

                /* Setup Single Get */
                BigInteger subm_id = BigInteger.valueOf(500);
                FormSubmissionIdentifier myid = new FormSubmissionIdentifier();
                myid.setId(subm_id);
                System.out.println(TestMe.debugAxisObject(myid));

                /* Run Single Get */
                Response r = stub.getFormSubmissionOperation(myid);

                /* Setup Multi Get */
                FormSubmissionSearchParameters p = new FormSubmissionSearchParameters();
                p.setSince_id(BigInteger.valueOf(3200));
                System.out.println(TestMe.debugAxisObject(p));

                /* Run Multi Get */
                Response r2 = stub.listFormSubmissionsOperation(p);
        }
}

Note that "Response" is not a generic Java class, it is actually com.example.www.FormSubmissionsServiceStub.Response, because the name of the output element is "response".

Other Resources:

Getting my Inspiron 8000 to run Ubuntu 13.10

JB — Wed, 30 Oct 2013 19:17:00 +0000

This is for my own benefit if I need to reinstall again.

Links to helpful articles:

What I remember doing:

I installed using a text-mode install from the network installer
I booted into recovery mode root prompt (don't forget to remount / readwrite!)
I modified /etc/default/grub to add "nolapic nomodeset" to the default parameter
I made s imple xorg.conf file to say that Card0 should use vesa

Then everything seemed to work. It shows it in a small box, though, and that is better than nothing.

Uniqueness Methods for PostgreSQL

JB — Wed, 31 Jul 2013 05:13:00 +0000

Just finished packaging together a set of uniqueness functions for PostgreSQL, mostly focused on Cantor pairing. Here is the introduction from the documentation:

uniqueness-pg

A collection of functions such as cantor pairing which help make unique identifiers for PostgreSQL.

What is a Cantor Pair? Let's say that you have a table which is uniquely identified by two numeric foreign keys. However, let's say that you need a single, unique key to refer to the row externally. Probably a better approach is to create a new primary key, but, if you can't, you can use a Cantor Pairing function to create a single number from the two.

select cantorPair(23, 50);

-- yields 2751

select cantorUnpair(2751);

-- yields {23,50};

Remember, in Postgres, you can even create indexes on these functions.

create table cantortest(a int, b int, c text, primary key(a, b)); create index on cantortest(cantorPair(a, b)); insert into cantortest values (1, 5, 'hello'); insert into cantortest values (3, 18, 'there');

select cantorPair(3, 18); -- =>  (yields 249)

select * from cantortest where cantorPair(a, b) = 249; -- uses index on large enough tables, or if enable_seqscan set to no.

select * from cantortest where cantorUnpair(249)[1] = 3; -- parses 
number out into a foreign key that can be looked up on its own

You can use more than two values with an array and the cantorTuple function:

select cantorTuple(ARRAY[5, 16, 9, 25]);

-- yields 20643596022

select cantorUntuple(20643596022, 4); -- NOTE - the 4 tells how many components

-- yields {5,16,9,25}

Additional functions include:

cantorPairBI - the regular cantorPair uses DECIMALs, which can be slow. If you know the final value will fit in a BIGINT, this version should be a little bit faster.

cantorUnpairBI - same as above but for unpairing

uniquenessSpace(key, spaces, space_index) - this is useful for when you need to UNION tables which each have their own PKs, but you want all of the rows to also have their own PKs. It is a super-simple function (key * spaces + space_index), but it is helpful because people can better see what/why you are doing things. So, for instance, if we had an employees tables and a customers table, but wanted a people view that had a unique PK, we can do:

CREATE VIEW people AS 
     SELECT uniquenessSpace(employee_id, 2, 0), employee_name from employees
     UNION ALL
     SELECT uniquenessSpace(customer_id, 2, 1), customer_name from customers;

To retrieve the original ID, use uniquenessUnspace:

select * from employees where employee_id = uniquenessUnspace(person_id, 2, 0); -- use same arguments as used on uniquenessSpace

These are both horrid names, and I am open to renaming them.

I am currently working on another uniquenessSpace, which doesn't require the number of tables to be known, only the index. However, this function is problematic because it relies on exponentials. The current version doesn't work because I need a prime filter. However, the goal is to overcome the problem in the previous example that if the number of tables change (i.e. we add on an additional supplier table or something), then all of the generated keys have to change (because there are more tables, which affects the second parameter). In an ideal setting, you will be able to do:

CREATE VIEW people AS 
     SELECT uniquenessSpace(employee_id, 0), employee_name from employees
     UNION ALL
     SELECT uniquenessSpace(customer_id, 1), customer_name from customers;

And then add on additional tables as necessary, without ever having to worry about changing keys. This can work by generating a key which is p0^employee_id, and p1^customer_id, where p0 is the first prime (2) and p1 is the second prime (3). As you can see, this gets out of hand pretty quickly, but PostgreSQL is surprisingly accomodative to this. Anyway, this is probably a generally bad idea, but I thought it should be included for completeness. I should probably rename this function as well.

Knowing When to Start Over

JB — Thu, 27 Jun 2013 15:03:00 +0000

A friend of mine shared this following quip on FB:

Quinn's design tip o' the day: You can't iterate your way out of local maxima. In other words, no amount of horse breeding will produce a Model T.

Sooner or later, you just have to throw it all away and start over.

Adding a "Copy" context menu to a disabled UITextfield in iOS

JB — Wed, 29 May 2013 16:28:00 +0000

I have a project which has a lot of fields which, at any moment, may be enabled or disabled. However, even when they are disabled, I want to show a context menu. It turns out that no one has a solution, so I hacked one together for myself. This code is adapted from a working project, but I have not tested the version presented here, but let me know if you have problems.

The general problem you run into is this: if your UITextField is disabled, then it cannot receive touch events. Therefore, it cannot fire the menu, even if you try to do so explicitly using Gesture recognizers. The normal answer is to simply not allow any editing on the field. However, while this prevents you from editing the field, it still *draws* the textfield as active. We want it to *look* inactive as well, which you only get by disabling it.

So, what I did was to create a subclass of UITextField (JBTextField) which has a subview (a specialized subclass of UIView - JBTextFieldResponderView) that has a gesture recognizer attached. We override hitTest on JBTextField to do this: if JBTextField is disabled, then we delegate hitTest to our JBTextFieldResponderView subclass.

As I said, this is from another project, which had a whole bunch of other stuff mixed in. I have tried to separate out the code for you here, but I don't guarantee that it will work out-of-the-box.

@interface JBTextFieldResponderView : UIView
@property (strong, nonatomic) UIGestureRecognizer *contextGestureRecognizer;
@end

@interface JBTextField : UITextField
@property (strong, nonatomic) JBTextFieldResponderView *contextGestureRecognizerViewForDisabled;
@end

@implementation JBTextFieldResponderView
@synthesize contextGestureRecognizer = _contextGestureRecognizer;

-(id) initWithFrame:(CGRect)frame {
  self = [super initWithFrame:frame];

  // Add the gesture recognizer
  self.contextGestureRecognizer = [[UILongPressGestureRecognizer alloc] initWithTarget:self action:@selector(longPressGestureDidFire:)];
  [self addGestureRecognizer:self.contextGestureRecognizer];

  return self;
}

// First-level of response, filters out some noise
-(IBAction) longPressGestureDidFire:(id)sender {
  UILongPressGestureRecognizer *recognizer = sender;
  if(recognizer.state == UIGestureRecognizerStateBegan) { // Only fire once
    [self initiateContextMenu:sender];
  }
}

// Second level of response - actually trigger the menu
-(IBAction) initiateContextMenu:(id)sender {
  [self becomeFirstResponder]; // So the menu will be active.  We can't set the Text field to be first responder -- doesn't work if it is disabled
  UIMenuController *menu = [UIMenuController sharedMenuController];
  [menu setTarget:self.bounds inView:self];
  [menu setMenuVisible:YES animated:YES];
}

// The menu will automatically add a "Copy" command if it sees a "copy:" method.  
// See UIResponderStandardEditActions to see what other commands we can add through methods.
-(IBAction) copy:(id)sender {
  UIPasteboard *pb = [UIPasteboard generalPasteboard];
  UITextField *tf = (UITextField *)self.superview;
  [pb setString: tf.text];
}

// Normally a UIView doesn't want to become a first responder.  This forces the issue.
-(BOOL) canBecomeFirstResponder {
  return YES;
}

@end

@implementation JBTextField
@synthesize contextGestureRecognizerViewForDisabled = _contextGestureRecognizerViewForDisabled;

/* Add the recognizer view no matter which path this is initialized through */

-(id) initWithCoder:(NSCoder *)aDecoder {
  self = [super initWithCoder:aDecoder];
  [self addDisabledRecognizer];
  return self;
}

-(id) initWithFrame:(CGRect)frame {
  self = [super initWithFrame:frame];
  [self addDisabledRecognizer];
  return self;
}

// This creates the view and adds it at the end of the view chain so it doesn't interfere, but is still present */
-(void) addDisabledRecognizer {
  self.contextGestureViewForDisabled = [[JBTextFieldResponderView alloc] initWithFrame:self.bounds];
  self.contextGestureViewForDisabled.userInteractionEnabled = NO;
  [self addSubview:self.contextGestureViewForDisabled];
}

-(void) setEnabled:(BOOL) enabled {
  [super setEnabled:enabled];
  self.contextGestureViewForDisabled.userInteractionEnabled = !enabled;
}

// This is where the magic happens
-(UIView *) hitTest:(CGPoint)point withEvent:(UIEvent *)event {
  if(self.enabled) {
    // If we are enabled, respond normally
    return [super hitTest:point withEvent:event];
  } else {
    // If we are disabled, let our specialized view determine how to respond
    UIView *v = [self.contextGestureViewForDisabled hitTest:point withEvent:event];
  }
}

@end

Now, in interface builder, we can enabled copy on disabled fields just by setting the class to JBTextField, or, if built programmatically, by replacing [UITextField alloc] with [JBTextField alloc].

I hope this helps!

Additional Resources:

UIControl Class Reference
Event Handling Guide for iOS
Displaying and Managing the Edit Menu
UIResponderStandardEditActions Protocol Reference
hitTest vs pointInside
Custom Menu Items
UIMenuController Class Reference
UIResponder Class Reference
Here is a simpler solution using shouldChangeCharactersInRange, but this does not give the field a disabled "look"

My Beef with Model Validators in Rails

JB — Thu, 09 May 2013 15:06:00 +0000

When I first started using Rails, it was a dream come true. I estimate that I reduced the amount of code I was writing by 60-75%. This is probably still true. However, as I've gotten to know the platform better, there are some philosophical issues that I have with Rails. Sometimes it is more the Rails community and its conventions rather than rails itself, but nonetheless, it continually manifests itself.

My current beef - validators! Having written lots and lots of code that interacts with lots and lots of systems, I have to say that you should never, ever, ever put in model validators. Ever. Validation is a process issue, not a data one. In theory it should be a data issue, but it never ever works out that way.

Here's what happens. Let's say I have a telephone number validator for a model. It makes sure the numbers are in the format (918) 245-2234. If not, it gives a nice error message. However, now let's connect that same model to an external system. Let's now say that the author of the external system was not as picky about phone number formats. What happens now? Well, every record that gets migrated from that system that isn't formatted according to your desires will now get kicked out! And it might not even show up during testing! In addition, your pretty error messages mean practically nothing on system->system communication. So, what you have is the system randomly breaking down.

This causes even more problems in long-term maintenance. I've had many bugs pop up because someone added a validator that broke several processes. It worked in all of their tests, but they weren't thinking about all of the ways the system was used. For instance, we decided to lock down the characters used in our usernames a little more. The programmer decided to implement it as a validator. Well, the problem is that several users already had usernames that didn't match the validator. So what happened? Those records just stopped saving. Not just when there was someone who was at the terminal, where they could see the error message, but every job started erroring out because of users who had faulty names.

Because of these issues, I *always* do validation in my controller, and *never* in the model. It is true that there are probably workarounds for all of these issues, but the point is that they are *not* that exceptional! They are everywhere! What would be nice is to have validators which could be run optionally, in specific situations. So, a controller could declare that it was using :data_entry validations, and then the model would only run those validators. So, it would look something like this:

In the model:

validates_presence_of :email, :for => [:data_entry]

In the controller:

uses_validators :data_entry

One of the issues I have found with dynamic languages such as ruby is that errors arise when "magic" is sprinkled in unexpected places. Ruby magic is awesome, but when I do something simple and it won't save, it is sometimes hard to even locate where the error is. Explicitly declaring which validator set you are using allows programmers to more easily see what they are doing, and only apply magic code where it goes.

Someday I'll write a package like this --- when I get extra time :)