The Sleepless Geek

The beauty of xargs

2012-02-01T13:59:00.001-05:00

Every time we deploy our Rails application, our deployment tool creates a new folder based on the date and time, puts the right files in it, and when everything is ready to go, updates a symlink to point to the newest release.

Yesterday I found that I couldn't deploy because we were out of disk space. I needed to delete some of the old release folders.

When I sshed into the server and looked in the releases folder, it had a ton of folders named like this:

20120126191222
20120126193901
20120127153732
20120127171244
20120127204235
20120127204517
20120130172837
20120131152908
20120131160422

Here you see some from 2012, but they went back all the way to 2010. I wanted to delete everything older than 2012.

Enter the magic of xargs. It lets you take the output of one command and input each line of it as an argument to another command.

Here's what I did:

ls # shows me all the folders
ls | grep ^201[01] # shows only the ones starting with 2010 or 2011
ls | grep ^201[01] | xargs rm -rf # delete all those

Building up the command bit-by-bit lets me verify that I'm going to delete the right things. And xargs knocks it out.

Here's another useful example. Vim creates a temporary .swp files that sometimes don't get cleaned up. To find and delete them all out of a folder:

find . -name '*.swp' | xargs rm

Faster feedback

2011-12-22T06:25:00.000-05:00

In programming, the faster you get feedback from your code, the more productive you can be.

For example, suppose you're going to use the Javascript Date class in a web page to do some time calculations. Do you:

Write some code, save the file, reload the page, look at the results, and repeat?
Open a Javascript console (like Chome Developer Tools or Firebug) and experiment?

The second method gives you instant feedback. This means you can find the right approach more quickly and not lose focus in the meantime. The same is true when coding Ruby: if you want to see what Array#select does, irb will give you much faster feedback than a Rails app.

Other ways you can get feedback faster include:

To learn more about Git, create a temporary folder and a dummy project: mkdir ~/test; cd ~/test; git init; echo 'hello' > test.txt; git add .; git commit -m"first"; Now you can experiment: branch and merge, rebase, and whatever else you want to try without fear of screwing up a real project. When you're satisfied, just delete that folder.
If you're thinking of adding a command to one of your runtime configuration files, like .bashrc or .vimrc, run the command directly first and see what it does.
Take the time to make your automated tests run faster: you'll be more likely to run them often and less likely to lose focus when you do.

Testing a Rails 3.1 Engine's Javascript with Jasmine

2011-08-26T14:47:00.003-04:00

At work, we've got a Rails engine that provides some complex drop-in forms to various applications. The forms have associated Javascript which, on Rails 3.0, we had to copy and paste between applications. That's both annoying and fragile, since things can get out of sync.

To test our engine, we had created a dummy app with the Enginex gem, located in spec/dummy.

In upgrading the engine to 3.1 (currently on release candidate 6), we wanted to move the associated JS into the engine, so that they would be versioned together in a single gem.

That wasn't so hard, but we also needed our associated Jasmine tests to run. For that, we needed to precompile our Javascript before rake jasmine ran so that Jasmine could load and test it.

Precompile Problems

And that's where things got sticky. According to Ryan Biggs' documentation, the default matcher for precompiling files is:

[ /\w+\.(?!js|css).+/, /application.(css|js)$/ ]

Basically, besides application.js and application.css, any js (or css) files with multiple dots in it, whether it's foo.js.coffee or jquery.cookie.js, will get compiled to its own file.

We had two problems with that:

It didn't match our engine's manifest file, because it's not named application.js. (It's not named that because in the host application, we will already have an application.js and we don't want a conflict.)
It did match a bunch of other JS files which didn't need to be compiled separately. For example, the manifest requires jquery.cookie.js, so that file gets compiled and added into the manifest file. It doesn't need to also be compiled as jquery-aaff723a97d782d30e3fc2f0dee5d849.cookie.js; we don't plan to serve it separately.

To solve the first problem, in Engine.rb, we added:

initializer "myEngine.asset_pipeline" do |app|

  app.config.assets.precompile << 'myEngine-manifest.js'

end

To solve the second problem, we... did nothing. We don't care. We're letting it create extra compiled files that we don't need.

We could have changed the match regex to only match files with endings like in .js.something or .css.something, instead of the broad rule that will also match .min.js. But doing that in an engine might have broken a host application if it were compiling something we didn't foresee. Maybe at some point a host app will want to compile .csv.erb files or something; we don't want to preclude that.

So we chose "be unobtrusive to the host app" over "prevent unnecessary compilation".

(We also could have prevented jquery.cookie.js from being compiled separately by renaming it to jquery-cookie.js, but we think that's annoying.)

Jasmine Setup

Now that our myEngine-manifest.js was being compiled to myEngine-manifest-20309309d9309330309390.js, we needed to let Jasmine know to load that single file, containing all our compiled Javascript, rather than all the separate files it had to load previously. So, in jasmine.yml, we now have:

src_files:

  - spec/dummy/public/assets/myEngine-manifest-*.js

The * will match whatever hash fingerprint is added to the filename.

Now, before we can run rake jasmine, we needed to make sure everything was compiled appropriately. So we created this simple Rake task in the engine's Rakefile:

task :precompile_jasmine => ['assets:clean', 'assets:precompile', 'jasmine']

Engine Rake tasks don't necessarily need access to Rails tasks, so to get the task above to work, we had to put the following above it so that we'd have access to those asset-related Rails tasks:

load File.expand_path('../spec/dummy/Rakefile', FILE)

Turning on Sprockets

To get Sprockets working in our dummy application, we had to make two changes to spec/dummy/config/application.rb:

Below the other railties requirements, add require sprockets/railtie
Within the app settings block, add config.assets.enabled = true

You may not need to do this; it's a side-effect of the fact that we generated our engine with a version of the Enginex gem that preceded Rails 3.1.

Other setup

To get Jasmine working with Rails 3.1, make sure you've got a new enough version. Jasmine 1.0.2.1 worked for us.
You don't want to check in all your compiled assets, so be sure to add this to your .gitignore: spec/dummy/public/assets/*

Weirdness

Something weird about assets:precompile makes it, and any task that runs after it, run twice. This means that `rake jasmine` requires two interrupts to shut down properly in our current setup. (In a previous attempt, it tried to run again while it was running, and got an error because it couldn't re-bind to the same port.)

Addendum

There's no way I could have figured this out by myself, and I'm not sure that I'll be able to answer questions about it. Adam Hunter and I worked on it together, and he contributed more brainpower than I did. But after all the blind alleys and frustration, I was determined to write up what we did, and got Adam's help doing so. So: good luck replicating this. :)

Also, if you're looking to use Jasmine in a Rails 3.1 app with coffeescript, this blog post from Pivotal Labs may be more helpful. We got some ideas from it, too.

Sometimes you can be too DRY

2011-07-14T12:43:00.001-04:00

In a Rails app, my favorite idiom for describing what users can and cannot do goes like this:

# In view code, for example...
  link_to "Edit Widget", edit_widget_path if current_user.can_edit?(widget)

  # which calls this method
  class User
    def can_edit?(resource)
      resource.editable_by?(self)
    end
  end

 # ... which in turn calls this method
 class Widget
   def editable_by?(user)
     # whatever logic makes sense in this particular app
   end
 end

I first saw this approach a couple of years ago, when I was new to Rails and found Nick Kallen's post about from 2007. John Nunemaker has since created the Canable gem to make this even easier.

Defining these methods yourself is not that hard. But because they are all so similar, you might be tempted to use metaprogramming to shorten the code and make it more DRY.

This is a rare example where sacrificing a little bit of DRY can make the code a lot more readable and maintainable: readable because your eyes don't have to bounce around as much, and maintainable because you can edit each method separately.

  class User
     # Metaprogramming way
    {
      :create => 'creatable', 
      :read => 'readable', 
      :edit => 'editable', 
      :delete => 'deletable', 
      :assign => 'assignable'
    }.each do |verb, adjective|
      define_method "can_#{verb}?".to_sym do |resource|
        resource.send("#{adjective}_by?", self)
      end
    end

    # Long-form way: less DRY but easier 
    # to understand at a glance
    def can_create?(resource)
      resource.creatable_by?(self)
    end

    def can_read?(resource)
      resource.readable_by?(self)
    end
  
    def can_edit?(resource)
      resource.editable_by?(self)
    end

    def can_delete?(resource)
      resource.deletable_by?(self)
    end

    def can_assign?(resource)
      resource.assignable_by?(self)
    end
  end

Metaprogramming Danger Box

2011-06-12T17:52:00.004-04:00

In my last post, I showed how one can use the "execute around" pattern in Ruby to log any calls to a method, without cluttering the methods themselves with logging calls.

That gave us code like this:

def defuse_bomb
    with_log('defuse_bomb') {puts "Snip the red wire..."}
  end

Our with_log method took care of printing out some logging info before and after executing the block given to it.

But as Adam pointed out in the comments, this is still kind of ugly. We don't have all the actual logging code inside of the "defuse_bomb" method, but have had to modify it to use with_log. What if we could add logging more transparently?

We can! With some metaprogramming magic added to the Logging module, here's the way the DangerBox class looks now:

class DangerBox
    extend Logging

    add_logging_to(:defuse_bomb, :throw_error)

    def defuse_bomb
      puts "Snip the red wire..."
    end

    def throw_error
      puts 'I feel an error coming on...'
      raise 'Success: an error!'
    end

    # This method's output won't be logged
    def eat_poison
      puts "Mmmmm, poison!! Nom Nom Nom Nom!!!!"
    end

  end

It still works the same way as it did before:

d = DangerBox.new
  d.defuse_bomb
  # ==== Sun Jun 12 16:55:49 -0400 2011 ====
  # About to execute 'defuse_bomb'...
  # Snip the red wire...
  # Executed 'defuse_bomb' successfully.

  d.throw_error
  # ==== Sun Jun 12 16:55:49 -0400 2011 ====
  # About to execute 'throw_error'...
  # I feel an error coming on...
  # Error while executing 'throw_error': 'Success: an error!'!

  d.eat_poison
  # Mmmmm, poison!! Nom Nom Nom Nom!!!!

Adam, I see your "hmmmm" and raise you a "BOOYAH!" (Although I didn't write tests for my version of this, so you still win on that front.)

But how the heck did we get all this logging output? Obviously, it's something to do with the line that says:

add_logging_to(:defuse_bomb, :throw_error)

Well, yes: that and a spiffy new Logging module that redefines our methods on the fly to use logging. Here's the code:

module Logging
    # Keeps track of the methods we want to log
    def add_logging_to(*args)
      @logged_methods ||= []
      @logged_methods.concat(args)
    end

    # Anytime a method gets added to our class (like with "def"),
    # this will execute
    def method_added(method_name)
      if @logged_methods.include?(method_name)
        # A lock to prevent infinite recursion; since we're
        # going to add a method below, we don't want that
        # to make method_added fire again, which would
        # make it fire again, etc...
        unless @adding_with_metaprogramming
          @adding_with_metaprogramming = true
          stashed_method = "stashed_#{method_name}".to_sym
          # Creates a new method named, for example,
          # "stashed_defuse_bomb", which does the same
          # thing as "defuse_bomb"
          alias_method stashed_method, method_name
          # Now we redefine "defuse_bomb" to do this:
          define_method method_name do
            begin
              puts "==== #{Time.now} ===="
              puts "About to execute '#{method_name}'..."
              # Calls "stashed_defuse_bomb" to get the original version
              send(stashed_method)
              puts "Executed '#{method_name}' successfully."
            rescue Exception => e
              puts "Error while executing '#{method_name}': '#{e.message}'!"
            ensure
              puts "\n\n"
            end
          end
        end
        # Remove the lock
        @adding_with_metaprogramming = false
      end
    end
  end

Metaprogramming! It sounds terrifying, doesn't it? Like, if programs can write programs, next thing you know, the Terminator is going to be ripping through your door! But it's actually slightly less terrifying than that.

Anyway, I got the idea for the last post from reading one chapter in "Eloquent Ruby," and I got the idea for this post while lying in bed after reading another chapter. So: thumbs up, Eloquent Ruby! I hereby recommend you.

Using the Execute Around pattern in Ruby

2011-06-10T17:10:00.003-04:00

At my last job, I wrote a script that was run by a cron job overnight. The script was supposed to go through a MySQL table of scheduled database queries and, if they met some sanity checks, run them. This was so we could run slow queries like ALTER TABLE during the wee hours of the morning, when customers weren't hitting our database. It may sound odd to execute MySQL queries that had been stored in MySQL, but it worked. (Just don't drop the stored queries table!)

The first time my script ran, it failed, and I didn't know why. My boss wisely suggested that I add some logging: print messages to standard output. We were a PHP shop, so that meant echo statements; all this output was captured in a log file by the cron job, which ran my script like this:

query_runner.php > some_log_file.txt

I went crazy with the logging, announcing up front how many queries I had to run, my progress ("Running query 4 of 7..."), how long each query took, whether it had errors, etc. At the end I printed a nice summary.

All this made the log files very useful when something went wrong: I could tell which queries had and hadn't run, and generally why. I learned the value of logging, and that day, I became a man.

No wait, I'm confusing my stories. I just learned the value of logging, that's all.

Execute Around

Anywho, these days I'm coding Ruby, and I'm currently reading "Eloquent Ruby" by Russ Olsen. In one section, he shows a nice way of adding logging to any method you like, without cluttering up the internals of the method itself.

The secret is the "execute around" pattern: you pass your method, as a block, to another method, which can do things before and after executing it. In this case, the logging method will log a message, execute the block, log another message, and even catch and log any exceptions.

Here's an example:

module Logging
    def with_log(description, &block)
      begin
        puts "==== #{Time.now} ===="
        puts "About to execute '#{description}'..."
        block.call
        puts "Executed '#{description}' successfully."
      rescue Exception => e
        puts "Error while executing '#{description}': '#{e.message}'!"
      ensure
        puts "\n\n"
      end
    end
  end

  class DangerBox
    include Logging

    def defuse_bomb
      with_log('defuse_bomb') {puts "Snip the red wire..."}
    end

    def throw_error
      with_log('throw_error') do 
        puts 'I feel an error coming on...'
        raise 'Success: an error!'
      end
    end

  end

Here's what it looks like in action:

d = DangerBox.new
  d.defuse_bomb
  # ==== Fri Jun 10 16:41:54 -0400 2011 ====
  # About to execute 'defuse_bomb'...
  # Snip the red wire...
  # Executed 'defuse_bomb' successfully.

  d.throw_error
  # ==== Fri Jun 10 16:41:54 -0400 2011 ====
  # About to execute 'throw_error'...
  # I feel an error coming on...
  # Error while executing 'throw_error': 'Success: an error!'!

Lovely! All the logging and error handling is done in one place, and the methods that use it can be much cleaner.

Understanding Rack applications

2011-06-09T17:19:00.001-04:00

I'm starting to understand Ruby's Rack application architecture, and it's really cool. It's a simple concept, but very powerful, because of the ways it lets you stack up different components with different responsibilities.

First, the basics. According to the spec:

A Rack application is an Ruby object (not a class) that responds to call. It
takes exactly one argument, the environment and returns an Array of exactly
three values: the status, the headers, and the body.

So each component in a rack stack is like a rung in a ladder. An incoming request is stashed inside a hash called the environment, which includes info about the request itself and othert things (see the spec). That environment is passed down the ladder from one component to the next until it gets to the app at the end. The app receives the environment, generates a response, and returns it. Since the app was called by the final rack component, it hands it response back to that component. That component then returns the response to the component which called it, etc, effectively passing it back up the ladder, until it's returned to the user.

Got that? Requests go down the ladder, responses come back up.

Of course, in that description, each component is just passing the environment down and then passing the response back up. If that's all a component did, it would be useless. To be useful, it will have to actually something. Specifically, a given component will do (at least) one of three things:

Screen the requests. For example, if the component is for authorization, it verifies that the user has entered a valid login. If it's an invalid login, it will return a "failed login" response and never call the next component. If it's a valid login, it will pass the request to the next component - maybe a Rails app.
Modify the environment before passing it on to the next component. Our authorization component might do this in addition to screening: if there's a valid login, it might set a user object in the environment before passing that to the next component. With this setup, the next component can assume that the user object will always be set whenever it gets a request.
Modify the response after getting it back from the next component. A silly example would be to translate the response into Japanese.

Notice that each component needs to know what the next component in the chain is, so that it knows who to call. That's why whatever you give to Rack::Builder#use needs to have an initialize method that accepts the next component, in addition to a call method that accepts the env.

The application at the end, however, doesn't need to call anybody else; it just needs to take the env as it then stands and generate a response. So whatever you pass to Rack::Builder::run doesn't need an initialize method; it just needs to respond to call(env). Even a lambda can do that.

Some pseudocode examples of call methods:

# A component with this call method would be useless.
# Still, it's worth remembering that this:
def call(env)
  @next_component.call(env)
end

# ... is Ruby shorthand for this:
def call(env)
  result = @next_component.call(env)
  return result
end

# Instead, we can do something useful on the way in:
def call(env)
  valid_user = contains_valid_login?(env)

  if valid_user
    add_current_user_object_to(env)
    # Next component can rely on having a current_user object
    @next_component.call(env)

  else
    redirect_to_login_page_with_error
  end
end

# Or we can do something useful on the way out:
def call(env)
  result = @next_component.call(env)
  result.translate_to_russian!
end

Ruby's Set class

2011-06-03T09:22:00.001-04:00

Yesterday at work, we ran into an interesting problem. We're creating the new version of an application and discarding the old, ugly code. But we need to migrate some data: the old system has (let's say) widgets, and the new system has widgets, too. The old system uses 5 different databases (see how ugly?) with weird row schemas, but it does reliably have widget color, size, and shapes. The new system uses one database and has a nice row schema, but it also has widget color, size, and shapes.

We need to know: which widgets are only in the old system? Which widgets are only in the new? Which are in both?

Enter Sets

Asking these questions tripped a switch in my mind. "I know about this!", I thought. "This is a job for sets! And Ruby has a Set class."

I'd never used them yet, but sets are made for this kind of thing. Sets are often illustrated with Venn diagrams: overlapping circles, where you ask "which things are only in the left circle? What's in the overlap?", etc.

For instance:

A set is a list of items where no item is repeated. If you have more than one set, you can compare them and answer the kinds of questions we've been asking. Here's a demo I just threw together:

require 'set'

  def sets_demo

    # Sets ignore duplicate values
    game_words = Set.new(['duck','duck','duck','goose'])
    puts "Unique game words                 : #{game_words}\n\n"
    #=>   Unique game words                 : goose, duck
     

    # Here are two sets with one thing in common
    fast  = Set.new(['bullet', 'cheetah'])
    round = Set.new(['bullet', 'beach ball'])

    # All the ways we can compare them
    puts "Round                             : #{round}"
    #=>   Round                             : bullet, beach ball
    
    puts "Fast                              : #{fast}"
    #=>   Fast                              : cheetah, bullet
    puts ''


    puts "Round and Fast (&)                : #{(fast & round)}"
    #=>   Round and Fast (&)                : bullet
    #
    puts "Round but not Fast (-)            : #{(round - fast)}"
    #=>   Round but not Fast (-)            : beach ball
    
    puts "Fast but not Round (-)            : #{(fast - round)}"
    #=>   Fast but not Round (-)            : cheetah

    puts "Round OR Fast (|)                 : #{(round | fast)}"
    #=>   Round OR Fast (|)                 : cheetah, bullet, beach ball

    puts "Round OR Fast, but NOT both (XOR) : #{((round | fast) - (fast & round))}"
    #=>   Round OR Fast, but NOT both (XOR) : cheetah, beach ball

  end

  # Formatting the way the sets print
  class Set
    def to_s
      to_a.join(', ')
    end
  end

  sets_demo

Got it?

In my examples, the items in the sets were strings, but they could be anything. In our case at work, we used hashes: a widget was represented by a hash containing its color, shape and size. So, we just had to:

Connect to each of the databases in the old system, getting all the widgets, creating a hash for each one, and dropping each into an old_system_widgets set (which automatically ignores duplicates)
Connect to the new system's database and make a similar set of its widgets
Do the kinds of set operations illustrated above

Voila! Now we knew which widgets were new and which ones still needed to be migrated to the new system.

In conclusion: sets are swell!

Hmmm. That's a pretty weak ending.

Ruby's String#unpack and Array#pack and Unicode

2011-05-04T06:37:00.002-04:00

At work, I'm exporting some data for use by a system that doesn't understand Unicode. I ran across an Obie Fernandez post explaining his approach.

One method he showed was this:

def to_ascii_iconv
    converter = Iconv.new('ASCII//IGNORE//TRANSLIT', 'UTF-8') 
    converter.iconv(self).unpack('U*').select{ |cp| cp < 127 }.pack('U*')
  end

Essentially, this means that Iconv will convert the characters that it can from Unicode to ASCII. For example, á will be converted to 'a. (I don't know where to find a comprehensive list of these transliterations.) After the transliteration step, any remaining non-ASCII characters - for example, Japanese characters, which have no ASCII equivalent - are discarded using the pack and unpack methods.To see how this works, try this in irb:

"abcdefg".unpack('U*')

Your string is converted into an array of numbers representing their Unicode values, as we requested by using "U*". (I'm still a bit fuzzy on exactly how Unicode values work, but let's keep rolling.)Ruby's Array#pack method can do the opposite conversion: numbers to string values.

(1..127).to_a.pack('U*')

There you should see a string of all the legal ASCII values, which, apparently, are all in the 1 to 127 range.Knowing this, it's easy to see how you can throw away non-ASCII values in a string:

some_string.unpack('U*').select{|character_value| character_value < 127}.pack('U*')

And that's what Obie's code does after its initial conversions with Iconv.Now, if you're curious, you might want to see what Unicode values some other numbers map to. No problem: just change the range value from our earlier example, and write the resulting values to a file (in my case, the Unicode characters don't show correctly in irb):

string = (1..300).to_a.pack('U*')
  File.open('unicodes_hooray.txt','w'){|f| f.write(string)}

Open that up in an editor that can display Unicode to see what you got.

Authorized_keys and SCP

2011-03-31T12:11:00.001-04:00

I have SSH access to a machine, and I want to add my public key to the `authorized_keys` file so I don't have to type a password to log in. A friend taught me a neat command for doing this:

% cat ~/.ssh/id_rsa.pub | ssh user@server "cat >> .ssh/authorized_keys"

Meaning:

Output the contents of my public key
Send that as standard input to this ssh command
The ssh command runs `cat` on the server
Since `cat` on the server is called without arguments, it uses standard input - in this case, the contents of my public key file
My public key gets added to `authorized_keys`

Added my key to `authorized_keys` has another benefit, besides not having to type the password when I SSH in: since I'm automatically recognized as a user, I can use tab completion for scp commmands. (Note: I'm using the zsh shell; remote tab-completion doesn't work for me in bash.)

In other words, if there's a file called "foo.txt" in my home directory on that server, and I want to copy it to my local machine, I can start type this:

% scp user@server:fo

... and hit tab, and it will auto-complete "foo.txt". That makes using scp a lot less tedious.

Using git cherry-pick

2011-03-31T10:15:00.002-04:00

One Git command that I've found to be really useful lately is "git cherry-pick". It lets you take a single commit from one branch and apply it to another.

For example, say you're working on your "add_blinking_lights" feature branch, and you find a bug in the "rotate_missile_chamber" method. You fix it and commit to the branch between a couple of other feature commits.

It's going to be a while before you merge this feature back into your master branch, but you do want that bugfix to go live on the next deploy. So, on the feature branch, you do this:

cyborg[add_blinking_lights]% git log -3
e9dd159 Added tests for blink rate
a8e358c Fixed bug in rotate_missile_chamber
a7d3302 Added tests for lights turning on

OK, so you know the hash for the bugfix. Now you do "git checkout master."

On the master branch you run "git cherry-pick a8e358c". Here's what that command does:

Look at that commit on the feature branch
See what changes it introduced
Create a new commit on the master branch that introduces the same changes

Notice that it creates a new commit on the master branch. If, on master, you run "git log", you'll see a different hash for the same commit message. Why?

This is because of how Git models what a commit is. A commit is a complete snapshot of the whole repository, and the hash for a given commit reflects the state of every file in the whole directory - it is a hash of all their hashes.

So clearly, since master branch doesn't have all of the commits from the feature branch, a complete snapshot of it at the time the bugfix is applied will generate a different hash than a complete snapshot of the feature branch at the time the bugfix is applied there. Thus, different hashes.

But when you do merge the feature branch into master, that won't matter; the hashes for the individual file where you made the bugfix will be the same, because their contents will be the same, so there will be nothing to update on master for that file.

Of course, another way of approaching this problem would have been to switch to master before committing the bugfix in the first place, then merge master into the feature branch, or cherry-pick from master to the feature, or rebase the feature on master. But the method above is simple, and if you did already commit the bugfix on your branch, is the best way to get that commit into production.

Where cliche is a mandate

2011-03-10T20:35:00.000-05:00

A recruiting company that spams me has the slogan "Where Commitment is a Passion."

What does that even mean?

"We're always looking for stuff to commit to. I just committed to eating a whole jar of pickles! Jimmy over there committed to sing everything he says! Linda committed to paying the mortgage for the zoo! We're c-c-crazy for commitment!"

Short Rubyzip tutorial

2011-03-07T15:30:00.003-05:00

Rubyzip is a handy tool for creating zip files with a Ruby script. But to me, the documentation doesn't make it immediately obvious how to use the thing. This quick tutorial should get you started.

Say you're submitting some pictures of kitchen gadgets to Strange Cooks Weekly. You need to zip up your images before sending them, and you're contractually obligated to use Ruby to do it. Here's how.

require 'rubygems'
require 'zip/zip'

puts "Zipping files!"

file_path = "Users/me/Pictures/kitchen_implements"
file_list = ['toast_mitten.png', 'gravy_jug.png', 'biscuit_juicer.png']

zipfile_name = "/Users/me/Desktop/someFile.zip"

Zip::ZipFile.open(zipfile_name, Zip::ZipFile::CREATE) do |zipfile|
  file_list.each do |filename|
    # Two arguments:
    # - The name of the file as it will appear in the archive
    # - The original file, including the path to find it
    zipfile.add(filename, file_path + '/' + filename)
  end
end

Note that if the zip file already exists, this script will try to add stuff to it. If you try to add a file that's already in the archive, it will complain.

Using Ruby for SCP file transfers

2011-03-04T13:43:00.001-05:00

Today I needed to write a Ruby script to fetch a file using SCP. I thought it would be nice if it also displayed a "percent complete" counter as it went. This is what I came up with.

require 'rubygems'
require 'net/scp'
puts "Fetching file"

# Establish the SSH session
ssh = Net::SSH.start("IP Address", "username on server", :password => "user's password on server", :port => 12345)

# Use that session to generate an SCP object
scp = ssh.scp

# Download the file and run the code block each time a new chuck of data is received
scp.download!("path/to/file/on/server/fileName", "/Users/me/Desktop/") do |ch, name, received, total|

  # Calculate percentage complete and format as a two-digit percentage
  percentage = format('%.2f', received.to_f / total.to_f * 100) + '%'

  # Print on top of (replace) the same line in the terminal
  # - Pad with spaces to make sure nothing remains from the previous output
  # - Add a carriage return without a line feed so the line doesn't move down
  print "Saving to #{name}: Received #{received} of #{total} bytes" + " (#{percentage})               \r"

  # Print the output immediately - don't wait until the buffer fills up
  STDOUT.flush
end
puts "Fetch complete!"

Best error message ever

2011-02-26T10:34:00.000-05:00

A while back, I sent a text message and got this error back from AT&T.

Wow... thanks. I... didn't know you cared?

Apparently they just append the text of the message (this one was to my wife) to the end of the error, without saying "original message follows" or anything like that.

In case anyone wants to try getting their own amusing messages, I believe this happened because I mistakenly sent to a land line.

Watching your log for a specific value

2011-02-23T17:05:00.001-05:00

If you use Unix, you probably know about piping one command's output through another. You may also know that

tail -f somefile

will show you whatever gets added to the end of a file in real time.

While working with Rails, I do

tail -f log/development.log

all the time to watch how Rails processes my page requests. Today, I was looking for a specific error message to show up - a MissingTemplate error that a user had experienced**, and I was trying to reproduce it.

Since I didn't care about anything that was being logged at that moment other than the error I was looking for, it occurred to me to do this:

tail -f log/development.log | grep 'MissingTemplate'

As I poked around our site, I saw nothing. Until I encountered the error, and BAM - that one line popped into my terminal. Error case located!

Just one more way that chaining simple tools together can be very powerful.

**Actually, the user didn't see the MissingTemplate error; they got a friendly error message, as they should. But we got an email from Hoptoad with the actual error.

PHP: now running in the browser!

2011-01-11T16:01:00.000-05:00

Got this email from a recruiter just now:

Hope you are doing well. We have an urgent requirement for
PHP Front End Developer . Please go through the below mentioned job opportunity and let me know your availability for the same.

PHP Front End Developer!? Wow, those zany Facebook developers must be at it again. First they got PHP to compile, now they've got it running in the browser!

IE: Still the biggest yak to shave

2011-01-06T12:32:00.000-05:00

My conclusion after a day and a half of fixing IE-only Javascript bugs.

And this is IE8, mind you.

Come on, Microsoft. Just use Webkit, already.

Introducing Bindler

2010-12-28T22:02:00.000-05:00

Rails. Hobos. Bundler. Bindles.

I awoke one day last week with this comic idea percolating in my brain. Too bad I'm lousy at drawing.

Chaining jQuery Pseudoselectors

2010-12-06T09:02:00.001-05:00

I just finished a very simple Javascript tweak to an app. Here was the requirement: "When the user opens our sign-in modal, we should focus the on the first input." Whatever code I wrote, I would put it a callback when the modal was finished opening.

This was my first solution: target the id of the first input, and focus on it.

$('input#user_email').focus();

But I didn't like that, because it's not very flexible. What if we add another input to the beginning of the form? We'd have to change the javascript, too.

After a little tinkering with jQuery's psuedoselectors, I came up with this:

$('#sign-in-modal-form').find('input:not(:hidden):first').focus();

Besides being more flexible to future changes, I think this is still pretty readable: select the first input that isn't hidden and focus on it.

Seeing where Ruby methods come from

2010-08-22T12:15:00.002-04:00

When it comes to reflection - asking an object or class to tell you about itself - Ruby makes a lot of sense. If you want to know what methods an object has, for example, you just ask it:

someobject.methods

.

Any array has lots of handy, built-in methods, which it gets from the Kernel module (mixed in by every object), the Enumerable module, and Array itself. (For example, somearray.shuffle returns a randomized version of the array.)

To see what comes from where, we'll make an array, and ask it: "array, please get me an alphabetized list of your methods, go through that list, and print a statement saying where each one was defined." (Thanks to several respondents on Stackoverflow for showing me this.)

The resulting list presents all kinds of opportunities for exploration. (For example, "tainted?" is very interesting.)

irb(main):001:0> a = Array.new
=> []
irb(main):002:0> a.methods.sort.collect {|m| puts "#{m} defined by #{a.method(m).owner}"}
& defined by Array
* defined by Array
+ defined by Array
- defined by Array
<< defined by Array
<=> defined by Array
== defined by Array
=== defined by Kernel
=~ defined by Kernel
[] defined by Array
[]= defined by Array
__id__ defined by Kernel
__send__ defined by Kernel
all? defined by Enumerable
any? defined by Enumerable
assoc defined by Array
at defined by Array
choice defined by Array
class defined by Kernel
clear defined by Array
clone defined by Kernel
collect defined by Array
collect! defined by Array
combination defined by Array
compact defined by Array
compact! defined by Array
concat defined by Array
count defined by Array
cycle defined by Array
delete defined by Array
delete_at defined by Array
delete_if defined by Array
detect defined by Enumerable
display defined by Kernel
drop defined by Array
drop_while defined by Array
dup defined by Kernel
each defined by Array
each_cons defined by Enumerable
each_index defined by Array
each_slice defined by Enumerable
each_with_index defined by Enumerable
empty? defined by Array
entries defined by Enumerable
enum_cons defined by Enumerable
enum_for defined by Kernel
enum_slice defined by Enumerable
enum_with_index defined by Enumerable
eql? defined by Array
equal? defined by Kernel
extend defined by Kernel
fetch defined by Array
fill defined by Array
find defined by Enumerable
find_all defined by Enumerable
find_index defined by Array
first defined by Array
flatten defined by Array
flatten! defined by Array
freeze defined by Kernel
frozen? defined by Array
grep defined by Enumerable
group_by defined by Enumerable
hash defined by Array
id defined by Kernel
include? defined by Array
index defined by Array
indexes defined by Array
indices defined by Array
inject defined by Enumerable
insert defined by Array
inspect defined by Array
instance_eval defined by Kernel
instance_exec defined by Kernel
instance_of? defined by Kernel
instance_variable_defined? defined by Kernel
instance_variable_get defined by Kernel
instance_variable_set defined by Kernel
instance_variables defined by Kernel
is_a? defined by Kernel
join defined by Array
kind_of? defined by Kernel
last defined by Array
length defined by Array
map defined by Array
map! defined by Array
max defined by Enumerable
max_by defined by Enumerable
member? defined by Enumerable
method defined by Kernel
methods defined by Kernel
min defined by Enumerable
min_by defined by Enumerable
minmax defined by Enumerable
minmax_by defined by Enumerable
nil? defined by Kernel
nitems defined by Array
none? defined by Enumerable
object_id defined by Kernel
one? defined by Enumerable
pack defined by Array
partition defined by Enumerable
permutation defined by Array
pop defined by Array
private_methods defined by Kernel
product defined by Array
protected_methods defined by Kernel
public_methods defined by Kernel
push defined by Array
rassoc defined by Array
reduce defined by Enumerable
reject defined by Array
reject! defined by Array
replace defined by Array
respond_to? defined by Kernel
reverse defined by Array
reverse! defined by Array
reverse_each defined by Array
rindex defined by Array
select defined by Array
send defined by Kernel
shift defined by Array
shuffle defined by Array
shuffle! defined by Array
singleton_methods defined by Kernel
size defined by Array
slice defined by Array
slice! defined by Array
sort defined by Array
sort! defined by Array
sort_by defined by Enumerable
taint defined by Kernel
tainted? defined by Kernel
take defined by Array
take_while defined by Array
tap defined by Kernel
to_a defined by Array
to_ary defined by Array
to_enum defined by Kernel
to_s defined by Array
transpose defined by Array
type defined by Kernel
uniq defined by Array
uniq! defined by Array
unshift defined by Array
untaint defined by Kernel
values_at defined by Array
zip defined by Array
| defined by Array

Visualizing our example setup

2010-08-15T08:33:00.000-04:00

Some people reading my post on setting up Apache, PHP, MySQL, Filezilla Server and PHPMyAdmin have seemed confused about how the whole setup works, or what the pieces are for. I put together this diagram a while back for a coworker and thought it might be helpful to readers here, too. Click the image for a larger version, suitable for zooming or printing out.

The basic idea that I want to convey is that web traffic has two sides: a client and a server. All conversation between them passes over a network. When someone visits a web page, their client, the browser, says "hey, send me your web page, please!" And on the other end of the network, a server gets the request and responds with the web page.

What's a server?

Before we go further, let's clear up some confusion about terms. Specifically, what is a server?

People use the term "server" to mean different things. In some cases, we mean "the program that responds to requests." With that meaning in mind, we'd call Apache a web server, MySQL a database server, and Filezilla Server an FTP server. In other cases, we mean "the computer that the server program(s) run on." With that in mind, we might say "our server sits on a rack in the server room."

In some setups, MySQL might be running on one server machine and Apache on another. In complex setups, both will be running on multiple machines.

The tutorial was all about setting up the various pieces of server software. If you set them up on your own computer, then visit your site from that same computer, then you refer to the server you're connecting to as "localhost" - local in the sense that it's on the same machine. In that case, the top part (the server) and the bottom part (the client) are really the same computer, and the network request never travels outside the computer you're working on. But of course most web sites you visit are served from elsewhere, and the network request might travel around the world.

Now, to look at the components we set up. Let's start by looking at the right side of the diagram, the HTTP side.

HTTP

This is the conversation between a browser and the server program - in our case, Apache. HTTP is the protocol they use to communicate. This just means that each of them expects requests and responses to be formatted a certain way - with headers, a body, and various sub-components of each.

When a user requests a URL, it's up to Apache to decide what to send back. In basic cases, if the user asks for "http://servername/foo.html", Apache will look for foo.html, read it, and send back its contents. If you ask for "bar.html" and no such file exists, it will tell you so. But you could have Apache do anything you like. For example, it could send back the same message no matter what URL was requested. Or it could map URLs that match certain patterns to certain responses, without there being a one-to-one relationship between requests and files on the server. (Ruby on Rails has a complex "routes" system that does this.)

Now imagine that the browser has requested a page, and Apache sends it back. What exactly does it send back? Just the HTML (in an HTTP "envelope", of course). The browser then looks at the HTML and says, "oh, I see that this lists some images, and a CSS file, and a Javascript file." It's then up to the browser (and the user's settings) whether it will ask for those other resources or not.

If the browser requests, say, an image on the page, the server sends it back, too. As far as the server is concerned, each request is totally separate. You can imagine the conversation like this:

Browser: I'd like this URL, please.
Server: Oh, hi, stranger! OK, here's some HTML.
Browser: (Hmmm, there's an image listed.) Server, please send me the image at this URL.
Server: Oh, hi, stranger! OK, here's that image.

Notice that the server doesn't "remember" the browser's previous request. In fact, it might even be a different server responding to the second request. This shows that HTTP is "stateless" - meaning that each request is separate, and doesn't depend on any previous ones. (This gets more complicated when you have logins and cookies and sessions, etc, but at bottom, it's the way HTTP works.)

In our example setup, when you ask for an HTML file, Apache just finds it and sends back the contents "raw" - without changing them. The same thing is true for CSS and Javascript files. But PHP is different.

Adding PHP

In our setup, we configured Apache to treat requests for .php files differently. When it sees that type of request, it asks php.exe to process the request and tell it how to respond.

PHP.exe will look at the file and execute any of the PHP instructions, like including headers from another file, running functions, pulling information from classes or objects, etc.

If the script requires it to, PHP will talk with a database server (in our case, MySQL) to get some of the information it needs to complete the request. For example, maybe it has to ask MySQL for a list of products, then output image tags and product descriptions for a catalog.

When it has finished running the script, PHP.exe gives the complete web page output to Apache, which sends it back to the browser, which can then ask for the images if it pleases.

One more thing to notice: as far as the browser is concerned, PHP and MySQL don't exist. It asks for a URL, and it gets back HTML (or an image, or other resource). It doesn't know or care how that gets put together. When it asks for "http://somesite.com/foo.php", it doesn't "know" that "foo.php" is a PHP script (and in fact, depending on how the server is configured, it might not be). The browser could ask for "foo.jpg" and get back HTML and it wouldn't be surprised, as long as the HTTP headers in the server's response said "what I'm sending you back is HTML."

In our setup, "foo.php" gets run when the browser requests a URL and the URL path is the file path to that script. But this is just a convenience for us, which makes it easy to see how the process works. We could make it more complicated, and as long as the browser could make a request and get a response it understood, it wouldn't know or care.

FTP

So where does Filezilla Server come in? If your server is on localhost, it may not. The basic purpose for Filezilla is to let you move files between the client and server computers. If they're the same computer, you can just copy the files there directly. But if not, you'll have to do it over the network, which is what FTP (File Transfer Protocol) is for.

Looking at the left side of the diagram, you'll see that there's another client-server relationship. This time, Filezilla Server is the server, and an FTP client (which could be Filezilla Client) is the client. If Filezilla Server is configured to use the same folder of files as Apache, you'll be able to change the content of your web site using this FTP conversation. To add or update a script, you'll upload it via FTP. The next time Apache gets a request for it, it will find it, get PHP to process it and send the results to the browser.

Quick Ruby Experiment

2010-08-07T12:04:00.002-04:00

Here's an interesting Ruby experiment:

Class.ancestors # [Class, Module, Object, Kernel]

Module.ancestors # [Module, Object, Kernel]

Object.ancestors # [Object, Kernel]

Kernel.ancestors # uninitialized constant Kernel

Ruby is to Esperanto as PHP is to English

2010-08-07T10:33:00.000-04:00

In 1887, L.L. Zamenhof published a book called "Unua Libro," describing a brand-new language that he hoped would change the world. A language anyone could learn to speak. One without political affiliation, without a messy history. And, I have to imagine he thought: finally, one that made sense.

English speakers know that our language is a mess - comedy routines have been built around it.

Here's Brian Regan on learning plurals in school.

So she asked this kid who knew everything. Irwin. “Irwin, what’s the plural for ox?”

“Ox. Oxen. The farmer used his oxen.”

“Brian?”

“What?”

“Brian, what’s the plural for box?”

“Boxen. I bought 2 boxen of doughnuts.”

“No, Brian, no. Let’s try another one. Irwin, what’s the plural for goose?”

“Geese. I saw a flock of geese.”

“Brian?”

[Exasperated laughing]“Wha-a-at?”

“What’s the plural for moose?”

“Moosen! I saw a flock of MOOSEN! There were many of ‘em. Many much moosen. Out in the woods…in the wood-es…in the woodsen. The meese want the food in the woodesen…food is the eatenesen…the meese want the food in the woodesenes…food in the woodesenes.”

We learn English rules, like "to make a past tense, add 'ed' to a verb: sailed, repeated, succeeded, constructed, cleaned." But many common verbs don't follow this. I ate, not eated; I ran, not runned; I went, not goed; I slept, not sleeped.

Other languages are the same. Spanish, for example, has tons of irregular verbs. Like many languages, it also has genders for all its nouns. Mountains are feminine. Trees are masculine. Why? Nobody knows. Mark Twain had hilarious complaints about German, which adds another gender called "neuter," among other complications.

Every noun has a gender, and there is no sense or system in the distribution; so the gender of each must be learned separately and by heart. There is no other way. To do this one has to have a memory like a memorandum-book. In German, a young lady has no sex, while a turnip has. Think what overwrought reverence that shows for the turnip, and what callous disrespect for the girl. See how it looks in print -- I translate this from a conversation in one of the best of the German Sunday-school books:

Gretchen: "Wilhelm, where is the turnip?"
Wilhelm: "She has gone to the kitchen."
Gretchen: "Where is the accomplished and beautiful English maiden?"
Wilhelm: "It has gone to the opera."

The reason for all this messiness is simple: nobody planned this. Languages are organic, evolving things. We forget words, make up new ones, pronounce things lazily, and the mistakes become normal. The rules change, and the exceptions pile up.

But not in Esperanto. Esperanto was designed to make sense.

Esperanto is much easier to learn than other languages because:

The different letters are always pronounced in the same way and every letter in a word is pronounced. Therefore there are no difficulties with spelling and pronunciation, as one knows that the penultimate syllable is always stressed.
The grammar is simple, logical and without exceptions. The grammatical exceptions are often what make it so difficult to learn a new language.
Most of the words in Esperanto are international and are found in languages around the world.
It is easy to make new words with prefixes or suffixes. Thus, if one learns one word, ten or more usually come as part of the package.

And while I'm sure it has its peculiarities, it's fundamentally different from English or Russian or Chinese or Swahili, which basically became what they are by chance.

PHP is like English

PHP is a very useful language for web development. But it's a bit haphazard, like English. Just look at some of its string functions:

count_chars() should have a counterpart called count_words(), right? But it doesn't. The counterpart is str_word_count() .
strip_tags() is named with words separated by underscores. stripslashes() has no underscores.
hebrev() will "convert logical Hebrew text to visual text." I don't know what that means, but really: a whole function for this, in the global namespace? Could something similar be done with Korean or Arabic or Portuguese or Tagalo? If so, would we create korev() and arabiv() and portugv()? Wouldn't it make more sense to have something like langv('languageName') and be done with it? For that matter, why include such a specific function in the base langauge, when 99.9% of users won't need it and those who do could use a library?

This haphazard design reflect's PHP's history: once called "Personal Home Page/Forms Interpreter," it was made with one vision and has been amended and revised and rewritten over and over. I get the impression that somebody said, "hey, I want to add a function for messing with Hebrew." And the PHP team said, "sure, knock yourself out. We'll put it in there."

Now don't get me wrong. I'm not smart enough to design a language myself. And these days, you can write solid, test-driven, object-oriented code in PHP. But fundamentally, it feels like a language that happened, not one that was made.

Ruby is like Esperanto

Ruby, on the other hand, was designed by Yukihiro "Matz" Matsumoto, a Japanese computer scientist who wanted to draw on the strengths of Perl and Python, and above all, to write a language for human beings. As Matz wrote in the foreword to Hal Fulton's The Ruby Way:

...Programming languages are ways to express human thought... Machines do not care whether programs are structured well; they just execute them bit by bit. Structured programming is not for machines, but for humans... So to design a human-oriented langauge, Ruby, I followed the Principle of Least Surprise. I consider that everything that surprises me less is good. As a result I feel a natural feeling, even a kind of joy, when programming in Ruby.

The language feels almost philosophical. It starts with principles: everything is an object. This has deep implications: classes are objects. True and false are objects. "Class" itself is an object, of the class "Class." Odd, yes, but not haphazard. Where PHP seems sloppy, Ruby seems mysterious, like real life. How can light be both a particle and a wave? How can "nil" be an object? We don't know, but we suspect there are satisfying reasons if we dig deep enough.

This logical consistency generally extends to the particulars of the language. Want to know the number of items in an array, or the number of characters in a string? .length is your method, in either case. (PHP employs count() and strlen(), respectively.)

If anything can be converted to a string, you can rely on .to_s to do it; to_a and to_i will likewise convert to an array or integer, if possible. There are some unexpected or duplicate methods: str.to_str is the same as str.to_s, but Array doesn't have .to_str. But on the whole, it seems easier to guess what a method will be called than in PHP.

Ruby, like Esperanto, is quirky. It considers 0 to be true, for example, unlike any other language I know about. But when I encounter its quirks, I assume that they're by design; a necessary condition for some beautiful aspect of the language. In PHP, I think, "somebody didn't think that through."

But perhaps I should stop there. Who am I to judge? I'm still a beginner in this field, with plenty to learn in both PHP and Ruby. Judgment can wait.

For now, I should go back to my studies. I should open my copy of "The Well-Grounded Rubyist" with a curious and open mind. And with the enjoyable expectation that, if I read and think and tinker and practice and ask questions, it will all make sense in the end.

AT&T U-Verse

2010-06-19T15:59:00.002-04:00

I just learned AT&T will probably roll out "U-Verse" fiber service in our area soon and went to look at the price and speeds.

Their naming scheme is pretty funny: Pro, Elite, Max, Max Plus, and Max Turbo. It makes me imagine this conversation.

Me: I'm not a heavy bandwidth user. What's your base package?
Them: That would be Pro.
Me: As in Professional?
Them: Yes.
Me: So if I were a professional internet user, I'd want the worst connection.
Them: No, you'd probably want at least Max.
Me: There is more than Max?
Them: Oh yes, there are two more levels past Max. But you probably don't need that. Why don't you try Elite?
Me: How good is that?
Them: It's the second-to-slowest one.
Me: I see. Is your marketing department staffed by native English speakers?

I don't need high speed, but I like where this is going. Maybe I should hold off until they offer Max Xtreme Warp Nuclear.

Cache busting in PHP: Part 2

2010-03-25T21:48:00.006-04:00

In my previous post, I showed how to borrow a technique from Ruby on Rails for busting the browser cache for a particular file.

If you haven't read that, please check it out and come back. It's OK, I'll wait here. I've got a snack.

Improving on cachedFile()

Back? OK, well I've made some improvements to cachedFile() and thought I'd share them. Here are the new capabilities:

1) The function now extracts the file type from the extension
2) It handles images, and specifies their dimensions for faster and smoother rendering by the browser
3) It caches all the information it calculates about a file for faster performance on subsequent requests

#1 and #2 are pretty straightforward: you can use the function like this:

cachedFile('foo.png');
cachedFile('subdirectory/bar.png','class="buz"')

...and it outputs something like this:

I put a cache in your cache so you can cache while you cache

But what about #3? What's this caching business? How can we add caching to a caching function?

Let's back up a bit.

First off, cachedFile() was a bit of a misnomer. This function is really for BUSTING a cache.

1) First, we configured our web server to tell the browser "you can cache these types of files for a whole year - don't ask for them again."
2) Second, we made sure that the browser saw each filename as the combination of the ACTUAL filename, like 'foo.png', and the file's time stamp, resulting in 'foo.png?1241452378' (or something like that). Those numbers represent the last time the file was changed; they're the same time stamp you see on any file on your computer.
3) Third, since the time stamp is automatically pulled from the file, we verified that we can update the file, which will update the time stamp, which will trick the browser into thinking it's never seen that file before, and therefore requesting it again.

The end result: the browser asks for a file once, then never again (at least for a year) - until the moment you change the file. As soon as it's updated, the browser asks for a new copy; until then, it uses the one it cached.

So, instead of cachedFile(), we could have called the function browserCacheBuster(). (But we won't, because I think that sounds cheesy.)

Now, this is all great, but the server is doing a bit of work for each file. Like before, each time you ask our function for a file, it has to go and determine the time stamp. In addition, my new features mean that for image files, it has to compute the width and height of the image.

This is all very fast in human terms, but how will it scale? What if you're using cachedFile() to spit out the same image tag several hundred times on the same page?

In that case, it might be nice to remember what you calculated last time. "Foo.png? Oh yeah, I remember him. I wrote down his dimensions and time stamp right here. No need to calculate them again."[1]

Memoization

To make this happen, we're going to use a design pattern called memoization. It works like this:

1) Before you calculate a result or pull it from the database, see if you've already got that result stored in a cache
2) If not, figure out your result and store it in your cache. If so, skip this step.
3) Now you've verified that you've got it in your cache, so return it from there.

For a given input, the first time the function runs, it will check the cache, find nothing, calculate a result from the input, store the result in cache, and return. Every time after that, it will just check the cache, find a result for that input, and return.

Does it matter?

But is there any point in doing this? Are we prematurely optimizing? Maybe. Let's see how much performance gain this really gets us.

I did a little not-very-scientific testing: added some caching to cachedFile(), called it from a loop a few hundred times, and timed the results using PHP's microtime(). I tried this with js, css, and image files, and did five or ten iterations of each.

Not a great sample size, but here's what I found: for .js files, having a cache made the function 2.72 times faster. For .css files, it made it 3.18 times faster. But for image files, having a cache made the function 119.63 times faster!

Clearly, computing those image dimensions is a bit expensive for the server, and we don't want to do it more than necessary.[2] Caching cuts the workload considerably.

Enough talk - code time

OK, let's see how our function looks with these changes. (The cache is stored in a global variable so it will persist between function calls. To offset this minor sin, I have labeled it clearly and awkwardly to prevent accidental meddling from elsewhere.)

$GLOBAL_cachedFile_cache = null;
function cachedFile($name, $attr=null){
 global $GLOBAL_cachedFile_cache;
 if (!isset($GLOBAL_cachedFile_cache[$name])){
  $root = $_SERVER['DOCUMENT_ROOT'];
  $filetype = substr($name,strripos($name,'.')+1);

  /* Configuration options */
  $imgpath = '/images/';
  $csspath = '/stylesheets/';
  $jspath = '/scripts/';

  switch ($filetype){
   case 'css':
    $output = '<link rel="stylesheet" type="text/css" href="/includes/';
    $output .= $name;
    $output .= '?' . filemtime($root . $csspath . $name) . '" ';
    if($attr){
     $output .= $attr . ' ';
    }
    $output .= '/>' . "\n";
    break;
   case 'js':
    $output = '<script type="text/javascript" src="/includes/';
    $output .= $name;
    $output .= '?' . filemtime($root . $jspath . $name) . '"';
    $output .= '</script>' . "\n";
    break;
   case 'jpg':
   case 'gif':
   case 'png':
    //This code will get run in any of the three cases above
    $output = '<img src="' . $imgpath . $name;
    $output .= '?' . filemtime($root . $imgpath . $name) . '"';
    $imgsize = getimagesize($root . $imgpath . $name);
    $output .= ' ' . $imgsize[3];
    if($attr){
     $output .= ' ' . $attr;
    }
    $output .= ' />';
    break;
  }
  $GLOBAL_cachedFile_cache[$name] = $output;
 }
 echo $GLOBAL_cachedFile_cache[$name];
}

Magnanimousness

What's that? Want to use this code somewhere? Well, sure. No, you don't have to thank me, or license it, or anything. Just name your kid after me or send me a solid gold pickle.

Humility

And of course, perhaps I did something very stupid here. Well, that's what comments are for.

[1]You might worry if this will create problems. After all, if we cache the time stamp, won't we miss the fact that the file has been updated and defeat our purpose? No worries: the cache only lasts as long as the page script is running. So if you update a file while a user is loading the page, they won't see it. But on the next reload, they will.

[2]In fact, it would be reasonable not to do it at all; there are lots of factors in how fast a site performs and seems, but how quickly it renders is certainly one of them. This is meant to help with that, but costs processor speed. You'll have to decide what works best for your site.

Rails caching and cache busting in PHP

2010-03-20T13:32:00.009-04:00

Ever wondered how to use browser caching to speed up your page loads?

I was working on a Rails project recently, and noticed something interesting in the documentation:

Using asset timestamps

By default, Rails appends asset‘s timestamps to all asset paths[1]. This allows you to set a cache-expiration date for the asset far into the future, but still be able to instantly invalidate it by simply updating the file (and hence updating the timestamp, which then updates the URL as the timestamp is part of that, which in turn busts the cache).

It‘s the responsibility of the web server you use to set the far-future expiration date on cache assets that you need to take advantage of this feature. Here‘s an example for Apache:

# Asset Expiration
ExpiresActive On
<filesmatch "\.(ico|gif|jpe?g|png|js|css)$">
ExpiresDefault "access plus 1 year"
</FilesMatch>

As I explained on Stackoverflow (more on that in a moment):

If you look at a the source for a Rails page, you'll see what they mean: the path to a stylesheet might be "/stylesheets/scaffold.css?1268228124", where the numbers at the end are the timestamp when the file was last updated.

So it should work like this:

1. The browser says 'give me this page'
2. The server says 'here, and by the way, this stylesheet called scaffold.css?1268228124 can be cached for a year - it's not gonna change.'
3. On reloads, the browser says 'I'm not asking for that css file, because my local copy is still good.'
4. A month later, you edit and save the file, which changes the timestamp, which means that the file is no longer called scaffold.css?1268228124 because the numbers change.
5. When the browser sees that, it says 'I've never seen that file! Give me a copy, please.' The cache is 'busted.'

Bringing it to PHP

Clever! Now how can we borrow that idea in a PHP app?

The first step, of course, is to set the server to tell browsers 'cache these files.' The example config above worked for me[2].

The second step is to append timestamps to your filenames. Here's a first-pass attempt at that:

That basically works - the time stamp is appended to the file name. But it's not nearly as streamlined as the Rails way, for a couple of reasons.

1) You have to type the file name twice - don't repeat yourself!
2) Come to think of it, all your stylesheet links are going to be the same format. Why keep typing in the boilerplate stuff?

In Rails, you'd just do this:

<%= stylesheet_link_tag 'main' %>

Slick! Helper tags like these take a lot of the drudgery out of HTML when you're using Rails.

A loose aproximation in PHP could be generalized to handle different file types. For example, you might write a function like this:

...which could then be used like this:

cachedFile('css','jquery-ui-1.7.1.custom.css');
cachedFile('css','main.css','title="Default"');
cachedFile('js','jquery-1.4.min.js');

Notice that this function assumes something - that your javascript files and stylesheets will always be in a particular folder. That's part of Rails' "convention over configuration" mentality: if you always do something the same way, you only have to specify it once.

Now, there's still room for improvement. For example, the type could be extracted from the filename, so that's one less argument to pass in. And more file types could be added. But this function already accomplishes several good things:

1) It gets your files to be cached by the browser and to bust the cache when necessary
2) It cuts down on code repetition
3) Naming the function cachedFile makes its purpose obvious

Now - how can you verify that this is working? I had the same question myself.

As Andy on Stackoverflow pointed out, you can load your page in Firefox, use the Firebug add-on, and look in the "Net" panel as you load the page. For any file that's cached, you should see a status message of 304 Not Modified. For anything that's pulled from the server, you should see 200 OK.

Try it:

1) Load the page to request everything once
2) Reload it to verify that things are being cached
3) Make a trivial change to a cached file, so that its timestamp will change
4) Reload the page again and verify that it was requested
5) Reload the page one last time and verify that it's cached again
6) Set up an elaborate Rube Goldberg machine to pat yourself on the back

(Step 6 is optional.)

Great ideas are worth borrowing

One reason that Rails has become so popular is that it codifies a lot of clever ideas and best practices into easy-to-use shortcuts. You can make a whole app with Rails without ever realizing that it's pulling the trick shown here on your behalf.

You don't have to use Rails, but if you see a great idea, it's always worth asking: "can I borrow this?"

Now go bust come caches!

[1]There's a danger here: notice that the Rails docs say all asset paths. If you set Apache to tell the browser to cache all images, style sheets and scripts for a year, and you only use a cache busting strategy for some of those things, then your visitors won't see updated versions of the others unless they clear their own browser cache manually or do a hard refresh with Ctrl+F5.

[2]I put this information into Apache's main config file, httpd.conf. If you're using a web host, they probably don't give you access to that, but they may have configured Apache to look for .htaccess files in your project folders. If so, you can set caching rules there.

It's not magic

2010-03-01T11:42:00.001-05:00

A while back, I had a small epiphany. I'd been asked to create a web form that could send emails with attachments. I already had forms that sent email, but attachments? What were they? In my mind, attachments were the little icons above the email. I had no idea how they were created or sent.

At the same time, I was reading The Code Book, an entertaining and fascinating look at cryptography - the art of sending scrambled messages, to be unscrambled only by the intended recipient.

The simplest, most brain-dead kind of encryption is a Caesar cipher, where you take all the letters in a message and shift them by the same amount. For example, with a Caesar cipher of 1, this:

HELLO WORLD

becomes this:

IFMMP XPSME

A modern cryptanalyst would pee his pants laughing if you used this method for anything serious. But one thing about it works: you can encode, and you can decode, as long as you know the rules.

Now, the Code Book traces the development of encryption methods so complex they'll make your head spin, but all of them are systematic: whatever is encoded can be decoded. You just need to know how.

This applies to any method of encoding information, even if it's not cryptography. For example, Morse Code encodes letters as electrical pulses - not to hide the message, but just to transmit it. How cool - you can encode actual language as beeps!

Which shows us another important idea: you can encode anything as anything else. You can encode the weather forecast with colored socks. You can encode the Constitution with duck calls. As long as there are consistent rules for encoding and decoding, it will work.

Going back to my email attachments, I soon discovered how attachments work: the file data is encoded as text in your email. It ends up looking like gibberish, but fortunately, no human has to read it, because the email program does that for you.

How does it know which parts of the text are text and which parts are files? You tell it. Say you want to attach an image. First, you choose some arbitrary string of characters which will probably never occur in an actual email text. Let's say it's "Woo_hamburgers_for_mayor_in_2050_yeeeeeha." Whenever you use that phrase, it means "I'm about to put in some different content." Each section of content also gets a label about what "MIME type" it is, like "image/jpg" or "application/pdf". Your email text goes in one section, and your image data goes in another, after being encoded as text.

(In PHP, you can encode the data as text like this: base64_encode($fileContents).**)

On the other end, when someone opens your email, their program is smart enough not to show all the scrambled-looking letters, but instead to say, 'hey, he said this part was an image - let's show it like that.' And it gets decoded. The little icons show up. It works!

The main thing is, it's not magic. For me, this turned on a light in my head. I stood next to a fax machine, and pointed my finger at it. "I know what you're doing with your crazy noises," I said. "You're encoding image data as sound!"

And that's how computers work, all the way down. Image information is encoded as characters, which are encoded as ones and zeros, which are encoded as magnetic charges on a disk platter, which are transmitted as electrical pulses in a circuit board.

It's hard to understand. It's hard to actually believe sometimes that everything I'm doing on screen can be represented as ones and zeros. But there's no magic. And if there's no magic, there's nothing to be scared of. If I work hard enough, I can understand a little piece of it - enough to get something done.

----
**For a detailed walkthrough of my attachment code, see this Stackoverflow post.)

The Bing Button

2010-03-01T11:17:00.000-05:00

From a NY Times article about upcoming Windows 7 phones:

In addition, Microsoft is requiring phone makers to keep basic elements of its user interface, including a physical button to start Web searches on Bing.

Microsoft. Listen. Nobody wants a button for Bing, or Google either, for that matter. This violates three principles about what I want in a smartphone:

Customizable. If a button goes to a web site, or opens a calculator, or gives me a voice prompt, and I can't change that, I'm going to be frustrated.
Neutral. It's my phone, not yours. I'll use Bing or Google or Jeeves or Big Larry's Virus-Laden Search Emporium if I want to. Don't force your product on me.
General purpose. I don't have a "word processor" key on my computer keyboard. I use on-screen menus for that. There are a million programs I could install, and a billion web sites I could visit. Smartphones are smartphones precisely because they share this characteristic. My flip phone has a single calendar program, take it or leave it. With a smartphone, I could install or write my own, or use one on the web. Having a button that does one thing makes this less like a smartphone and more like a calculator - a single-purpose device. I want a browser, not a Binger.

If you can't put your customers' desires above your need to cross-brand, you're going to make lousy products. And your market share will continue to drop.

Q&A with Michael Gundlach, creator of AdBlock for Google Chrome

2010-02-02T15:05:00.008-05:00

On Monday (2-1-2010), AdBlock for Google Chrome became the most popular extension for the browser, with about 250,000 users as of this writing. That achievement came after nearly two months of frenzied development by its creator, Michael Gundlach, during which he was interviewed by the NY Times, got rave reviews from users, and saved the world from a giant comet. But how?

Lucky for me, Michael also happens to be my close friend and coding mentor, so I’ve got him on speed dial. Today during lunch, we chatted about AdBlock, including how agile development and editor mastery helped him achieve so much in so little time.

How did you get started making AdBlock for Chrome?

"I was reading the Google blog at work and noticed that they said extensions were available in the beta channel... When I searched for Adblock on the extensions page, I saw that it didn't exist. I was like, 'oh that's too bad,' and went back to coding. About two hours later, I went, 'what am I doing at work!?' I jumped up and went home and read a tutorial, and posted a dummy version of AdBlock, and immediately started getting user feedback. They were rating it terribly, because it was just a ‘hello world.’ So based on the comments, I updated it so it would do one thing: block ads on ESPN. Then I just went as fast as I could to get a functional version that actually blocked everything that Easylist blocked... and then I kind of didn't stop coding all through December."

What’s your development background?

"I started coding when I was nine, when my brother gave me a computer with BASIC on it and showed me how to use it… I got in trouble in high school for installing the Pascal compiler on one of the physics computers because I wasn't paying attention. They charged me with hacking into the computer system when I was just trying to make a choose-your-own adventure for my mom for her birthday… I never stopped because I can't stand do anything but program." (Note: Michael went on to work for some well-known companies, including Google.)

What's the process for writing an extension? Was it hard to learn?

"It was great - the tutorial was really freaking simple, and you just make some Javascript files that you want to run on each page. Chrome’s extensions were based on the GreaseMonkey extension in Firefox, where you just run some Javascript on a random web page... My 20% project at Google was a Greasemonkey script for internal use, so it felt very natural." (Note: Michael recently heard from a current Googler that the update system in Chrome is based on an innovation in Michael’s Greasemonkey script that allowed it to update itself automatically.)

You and I talk a lot, so I know that you've used agile development throughout this. How do you think of agile development, and why did you choose to use it?

"It's better for your psyche to have something working quickly and see that it does work... I wanted to get something out there right then, and the system supported me making tiny changes and shipping them quickly... If I had done the waterfall method and waited until the extension was complete, somebody else would have already done it and I wouldn’t have been able to catch up… Agile development keeps you focused on the task, and helps you always have something usable for the user so you get feedback."

"Agile development keeps you focused on the task, and helps you always have something usable for the user so you get feedback."

What are some of the decisions you made that helped you move quickly?

"I knew it needed to have filter subscriptions, but at first it was simpler to just copy and past the entire Easylist into a file and use Unix regex tools to turn that into Javascript code [an array]… I didn't have any updating, but that was a quick and dirty way to have something that covered lots of websites at once. The next week, I had people reported ads that weren’t being blocked because I wasn’t updating Easylist, so I had to patch manually while I was trying to get filter subscriptions written.”

"When I put new features in, I tried to put them as opt-in experiments, using the Google Labs approach, where users that are geeky enough can try it out and give me feedback... Before that, I updated one night and went to bed, and the next morning I found people complaining in my comments that I blocked all images on all web sites. I kept getting those complaints for three days until everybody’s browser had gotten the version that fixed it, even though I had released 10 updates in that time. So now I do stuff in experiments first.”

Many developers feel like we can't publish something unless it's perfect. But clearly that attitude would have stopped you from succeeding here. How do you decide if something is "good enough for now?"

"It's a balance, but I think the idea that if I put something in there that's not perfect, it will hurt my reputation, that’s false, because that's what Microsoft has been doing forever. If you can put something out that's not perfect and get feedback, you're ahead of the person who hasn't released anything at all… I just look for the absolute simplest thing I can do that adds some functionality, then do that and release it… There are still a lot of things I want to do with AdBlock, but it's been useful for thousands of people even thought it's not perfect."

What did you learn during this process?

"This project has 500 times more users than anything I've ever built personally. I'm seeing how successful Agile can be if I just listen to users and do the thing that people are shouting for most. So I've validated the agile approach... combined with not ever sleeping."

"Also, I need to weigh how loud the user is with what percentage of the user base they represent. People were clamoring for a long time that I get a change log up and post my source code. I could have spent 4-5 hours doing that, but that would have been time I wasn’t blocking ads and fixing bugs, and most people don’t care about a change log. I actually wrote a priority list down – responding to user requests quickly was first, because people liked my fast customer service, then fixing ad reports, then fixing bug reports, then feature requests, and way down the list was administrative stuff like a change log. The list got more formalized when I finally got some breathing room, when the ads were being blocked and most of the bugs were fixed, so I created a Google Code project, where I can mark bugs with low or high priority and people can see my change log."

You used object orientation pretty heavily in your Javascript on this project. How did that help you?

"What makes it most useful for me in Javascript is that it simplifies the code; it's one level of complexity that's hidden. If an object has two methods exposed, I only have to remember those two methods, where with procedural code there are a bunch of methods in the global namespace that I have to keep up with. It's harder to do in Javascript because the syntax isn't as friendly as in something like Ruby, but once you learn the syntax, it's fine, and it makes your JS feel SO much nicer."

I had the chance to actually see you working for a bit, and the main thing that stood out to me was your fluid use of vim. It's almost like you're playing an instrument: you don't seem to think about the commands. You're just typing at a thousand miles an hour, and fully formatted code flows out and morphs on the screen, while split windows appear and disappear as needed. How did you attain such editor mastery, and how important is that to your career?

It's freaking wonderful to be fluent in one editor... I just think ‘I want an IF statement’ and the braces appear.

"I got it because I've been using Vim for maybe 21 years – my brother got me a Unix account at the University of Georgia and showed me vitutor and said ‘use this.’… I don't know nearly all there is to know about Vi, but I know the bits I need to... It's freaking wonderful to be fluent in one editor, because my thoughts get to flow straight from my brain onto the page, and I don't have to worry about what my fingers are doing in the meantime. I don't have to think at the level of how to put the braces in anymore, I just think ‘I want an IF statement’ and the braces appear. Maybe I'm manually indenting them and getting the formatting right, but it's such lower brain stem work that I can ignore that and think about the code. It's like my fingers are plugged into the computer. It's not 100% - sometimes I get things wrong. But because I've been using it so long, and because Vi rocks so hard, I don't have to think much about it."

So being really fluent in an editor is crucial?

"Absolutely. Vi or Emacs, whichever one, I don't care, learn it. The exception is if you're going to spend all your life doing Microsoft .NET programming, then learn Visual Studio and learn everything about it.”

“One nice thing about Vi is that it can be used for text editing as well as code, so every time you write a notepad file, you're improving your Vi skills… Get a good .vimrc, that's important... When I try to use Vi on a machine that I don't control, it's painful again. Syntax highlighting and stuff like that don’t work, and I have to stop thinking about my code and think about my editor.”

Projects like AdBlock for Chrome are not just fun for you – they’re part of your vision for your career. Why?

"I've always had the idea that when I can retire from having to work for money, I would still be doing all the same stuff I do now, it would just be for free. Instead, I am fortunate enough to find some hours that I can do this now, and holy crap, I'm the author of a successful open source project, which was one of those goals for retirement... I’m looking for a job now that will let me work from home full time, and possibly less than full time, so I can use some of those hours I would have spent working for pay working on AdBlock."

How is the AdBlock community coming?

"We've already got three official contributors who triage bug reports and figure out how to block ads, and two offers of translation work. So if I ever want to leave, I think can hand it off. That's what open source is so awesome for. If I want to move on to another project, it will be when the community is so strong that AdBlock is running itself. But I don't see that happening anytime soon, because I'm still really excited about making this thing great.”

Windows development toolbar

2010-02-01T09:57:00.002-05:00

At work, I develop on a Windows machine, and need access to various config and log files, as well as my main code trunk and a branch. I got tired of digging through layers of folders to find everything, and set up a quick way to access everything. Here's how.

(Note: these screen shots are from Windows XP, but I've confirmed that this works on Windows 7, and it probably works on Windows Vista as well.)

First, create a folder on your computer (anywhere you like) and create a bunch of shortcuts in it to everything you might need.

Second, right-click the menu bar and choose Toolbars > New toolbar. Select the folder you just created.

Finally, click the little arrows next to the toolbar name. It will pop up with all your shortcuts. Hooray!

Final tip - if you'd like to see the full date and time on your menu bar, right-click it again, deselect "lock the toolbar," and drag the toolbar edge up to make it taller. When the full date and time appears there, right-click again and re-lock it.

Ta-da! No more double-clicking to see today's date on the calendar.

Setting up Apache, PHP, MySQL, PHPMyAdmin, and Filezilla Server on Windows XP

2010-01-18T13:31:00.027-05:00

Need to set up a Windows web server? You're in luck! Today we're going to set up more server components than you can shake a frozen iguana at! (Or, for Latin-prejudiced grammar Nazis, "more server components than at which you can shake a frozen iguana!")

The Disclaimer

I just finished figuring all of this out myself. That's good, because it means I know how to talk to a beginner. But I don't pretend to be an expert. If anybody spots any glaring security holes or other blunders, let me know; otherwise, just be aware that these are only the basics, coming from a relative n00b in all things server-related.

The Goal

The goal today is to set up a web server on Windows where you can upload your PHP code, serve it with Apache, tie it in to a MySQL database, and administer MySQL with PHPMyAdmin.

(Update August 15, 2010 - see "Visualizing our example setup" for help understanding how these pieces work together.)

Let's start by setting up Apache.

A Bunch of Downloads

Here's what you need to download (along with the version numbers I'm using):

Apache Web Server (v 2.2.14) - get the Win32 binary (msi installer) from the official site. You can decide between the one that comes with SSL and the one that comes without - either will work for this tutorial.
PHP (v 5.3.1)- we'll download the Windows binary here. This is the PHP interpreter; when Apache needs to serve a PHP script, it will hand it off to the PHP interpreter to deal with the PHP code before it sends HTML to the client. Get the zip file labeled "VC6 x86 Thread Safe".
MySQL (5.1.42) - Go here and get the "Windows (x86, 32-bit), MSI Installer" - (I got the file named "mysql-5.1.42-win32.msi" - not sure how this is different from the 'mysql-essential' version.)
PHPMyAdmin (3.2.4) - get the zip file here.
phpmyadmin.sql - tables for some of PHPMyAdmin's advanced features (more on this later)
FileZilla Server (0.9.34) - download exe file here.

Setting up Apache

Run the executable you downloaded above. Other than setting your network and server name (I used 'localhost') and your administrator email address, you can just click "Next" through the whole installation. Choose "Typical" when asked for a setup type.

When setup is done, Apache is installed as a service on your machine. (If you right-click My Computer and choose Manage, then click > Services and Applications > Services, you can see all the services running on your machine.)

You can verify that this was successful by opening a browser and going to http://localhost. You should see this:

...which is the file index.html, located in Apache's htdocs folder.

Now, if you look in your task bar, you should see an icon that looks like this:

That's a status indicator that says "Apache is running!" You can click it to stop Apache:

...and you should see the status change.

Installing PHP

Extract the PHP zip file to your c:\ drive and rename the folder from "c:\php-5.3.1-Win32-VC6-x86" (or whatever) to simply "c:\php."

Now, to keep things clean so that you don't have to drop anything PHP-related into the Windows folder, you'll need to add c:\php to the Windows system path. Essentially, that means that when Apache says "I need to run php.exe," Windows will know that c:\php is one of the places where it can look for that.

To add something to the Windows path, right-click My Computer and choose Properties, then go to the Advanced tab.

Click on the "Environment Variables" button near the bottom, then scroll down in the bottom panel until you see "Path."

Click to highlight it, then click "Edit."

Go to the end of that line and add (without the quotes) "C:\php;", then click OK. Restart your machine so that this change will take effect.

Configuring PHP

Once you've rebooted, browse to c:\php. Copy either php.ini-production or php.ini-development, depending on the purpose of this server setup[1], to simply c:\php.ini. Then copy that file to c:\php.ini-original before you start making any changes, so you can always go back and see what you did by diffing your current config with the original one. (WinMerge is a good Windows-based program to do that.)

Now that you've got a backup, open php.ini in a text editor and make the following changes:

Optionally, set short_open_tag = On. This is not considered best practice, but it allows you to start PHP code blocks with just <? instead of <?PHP. It's convenient, and may be needed if your existing code uses tags like this.
Set extension_dir = "C:/php/ext"
Uncomment the 'extension=' line(s) for your preferred database adapter(s), if you've got existing code. For this example, uncomment extension=php_mysqli.dll (to match the config we'll use for PHPMyAdmin later on).
Set up your time zone to be used in any PHP date functions. For example, I set date.timezone = "America/New_York". (I know, it's weird to use a string like that. See the manual for more.)

Configuring Apache

Now browse to the Apache directory, and go to the "conf" subdirectory. Here, the main configuration file is called "httpd.conf". Just as before, we want to have a backup of it before we make any changes. Fortunately, Apache comes with backup config files, located in the "original" subfolder. Leave those alone for future comparison with your modified versions.

Now, open up httpd.conf in a text editor and let's make some changes.

Set "DocumentRoot" to point to the folder where you want to keep your web pages and scripts. (The default setting points to Apache's 'htdocs' subfolder.) For example, you could set it to DocumentRoot 'C:/www'.
Set <directory> to point to the same folder. Like this: <directory 'C/www'>
Set DirectoryIndex index.php index.html. This controls which file will be displayed if someone requests a folder. For example, if they browse to www.yoursite.com/foo, with this setting, the server will first look in the foo folder for 'index.php.' If it finds it, it will display it; otherwise, it will proceed to look for 'index.html.' You can add more alternatives to the end of the line or change the order as you like.
Set EnableSendfile Off to prevent some page requests from hanging. (This is a Windows-specific configuration that took me a few days to to learn about)
Finally, add the following lines to the end of the file

#Enable PHP processing
LoadModule php5_module "c:/php/php5apache2_2.dll"
#.php files should be processed as PHP
AddType application/x-httpd-php .php
#Where to find php.ini
PHPiniDir "C:\php"

Whew! That's it for Apache and PHP. Restart Apache, and create a simple page called 'index.html' in your designated web root folder (like "C:\www", or whatever you chose). Now if you visit http://localhost, you should see it.

Now create a simple php script called 'index.php'. With the DirectoryIndex setting I used above, it will now take precedence over 'index.html' and be displayed when you reload.

Installing MySQL

This is the easiest part yet. Run the installer and use all the default options.

Note: The "Typical" installation option puts the database data in "C:\Documents and Settings\All Users\Application Data\MySQL\MySQL Server 5.1". If you ever need to completely remove MySQL and reinstall, if you don't delete this folder along with the program's folder, you'll reinstall and open your database to find that it already has data in it. Spooky.

Setting Up PHPMyAdmin

This is also pretty simple. For the most basic case, all you need to do is this:

Unzip the phpmyadmin file into your web root folder
Rename its folder to simply 'phpmyadmin' (so that you can browse to 'localhost/phpmyadmin' to see it)
Create a config file in that folder, called config.inc.php

A very simple config file (taken from the docs) looks like this:

<?php
$cfg['blowfish_secret'] = 'gtqw0282bg7130jhga7d'; // change to random string of your choice
$i=0;
$i++;
$cfg['Servers'][$i]['auth_type'] = 'cookie';
?>

That's it! PHPMyAdmin should work.

Making PHPMyAdmin better

If you've got previous experience with XAMPP, you'll be accustomed to some nice little configuration features for PHPMyAdmin. For example, you can bookmark queries, and have your foreign keys display as hyperlinks to the associated record.

Fortunately, we can all get those features working, too. First, add the following lines to config.inc.php:

$cfg['Servers'][$i]['pmadb'] = 'phpmyadmin';
$cfg['Servers'][$i]['bookmarktable'] = 'pma_bookmark';
$cfg['Servers'][$i]['relation'] = 'pma_relation';
$cfg['Servers'][$i]['table_info'] = 'pma_table_info';
$cfg['Servers'][$i]['table_coords'] = 'pma_table_coords';
$cfg['Servers'][$i]['pdf_pages'] = 'pma_pdf_pages';
$cfg['Servers'][$i]['column_info'] = 'pma_column_info';
$cfg['Servers'][$i]['history'] = 'pma_history';
$cfg['Servers'][$i]['designer_coords'] = 'pma_designer_coords';

All of these little features depend on having a database called 'phpmyadmin', with tables named 'pma_bookmark,' etc.

Take a look at the sql file I provided. If you create the database 'phpmyadmin' and then import this file, it will set up the tables you need for the fancy features.

One last tweak that you can take or leave: I hate having too much pagination on my large tables, so I set PHPMyAdmin to display 500 records at a time by adding this line to the config file:

/*Personal Tweaks */
$cfg['MaxRows'] = 500; /* Show 500 rows of a table by default */

Setting up FileZilla Server

Once again, run through the installation process, accepting all the default settings. When you're done, Filezilla Server should launch, and you'll have a status window / control panel that looks like this:

Now you need to set up at least one user. Start by clicking "Edit > Users."

Take a look at this image, then I'll explain it a bit.

Over on the right, you've got a list of users. Here, I've created two. Everything to the left of there applies to the currently-highlighted user. So your first step is to create one, and after that you can manage settings.

At a minimum, you need to:

Enable the account
Set a password (well, you COULD skip this, but don't)
Click on "shared folders" to the left and add at least one folder that this user will have access to, as well as what type of access: read-only, write, etc. A developer will probably need full access to the web root folder.

You can tweak all kinds of other settings, like connection timeout and bandwidth limits, etc etc, but this takes care of the basic setup.

As long as the machine this is running on is online and doesn't have port 21 blocked in its firewall, you should be able to get on another computer, open an FTP client, put in your new server's IP address and the username and password you just created in Filezilla Server, and connect. You should see the folder you gave that user access to.

Done!

That's it! If you have any questions, read the documentation for that component, or check on serverfault - it's like stackoverflow, but for system administrators.

[1] The difference between the development and production versions of this file are mostly related to error reporting and performance profiling (although having error reporting turned on for your production server could expose sensitive security information). Some of the errors and warnings the development version will give you include stuff like "this isn't best practice" or "this code will be deprecated in future versions of PHP." They are great things to learn from your development setup, but you don't want your production server wasting its effort noticing them.
If you're curious to learn more, compare the two files. A diff program like Winmerge (which is free) will let you quickly see the differences between the two, and reading the documentation about those settings will teach you what they're for.

Firefox Offline Mode

2010-01-14T12:47:00.000-05:00

You know what's horrible? Firefox's "Offline Mode."

I've been tinkering with Apache and with some web pages, and I keep being surprised by weird stuff, like:

- I don't see my page edits when I reload
- A page displays just fine, even though the server is no longer installed

Then I realize that Firefox, whose whole job in life is to get online, is offline. It has, invisibly and without my asking, decided not to reload the page, even though I'm pressing Control+F5, which means "no for real, reload everything on this page, throw away your cache, I want to see the most current stuff possible."

I don't know what this feature is for, but I need a way to turn it off.

Adding fields to a web form: a job for closures!

2010-01-14T12:36:00.004-05:00

In my last post, I explained how closures work. Via their magic, I can make a function to do this:

nextNumber(); //returns 1
nextNumber(); //returns 2
nextNumber(); //returns 3

The most concise way to create this in Javascript would be:

var nextNumber = (function(){var i=1; return function() {return i++;}})();

Of course, that's not very readable. See my previous post for a step-by-step introduction to closures.

Putting closures to use

One place where this trick is really handy is when you want to dynamically add fields to a web form. You need each new field to have a unique name and ID, but pulling new IDs from a database is overkill.

Say you've got a form where users can order Viking helmets. For each helmet they order, they need to specify a color and size.

Suppose your inputs look like this:

<label for="helmet1color">Helmet 1 Color</label>
<input id="helmet1color" name="helmet[1][color]"/>
<label for="helmet1size">helmet 1 Size</label>
<input id="helmet1size" name="helmet[1][size]"/>

Ideally, you'd like to add as many new inputs as a customer wants. The more helmets they order, the sooner you can afford to buy that beet farm!

So you make a Javascript function to generate new inputs:

function newHelmetInputs(i){
var inputSet = '';            
inputSet += '<label for="helmet'+ i +'color">Helmet '+ i +' Color</label>'
inputSet += '<input id="helmet'+ i +'color" name="helmet['+ i +'][color]"/>'
inputSet += '<label for="helmet'+ i +'size">Helmet '+ i +' Size</label>'
inputSet += '<input id="helmet'+ i +'size" name="helmet['+ i +'][size]"/>'
return inputSet;
}

But of course you've got to tell it whether you need inputs for helmet #2 or #15. Which is where a closure function comes in: it can return a new number every time it's called. Like this:

i = nextNumber();
newInputs = newHelmetInputs(i);

Slap those inputs onto the page, and you're done! Well, almost.

Completing the Picture

To show exactly how this works, I've got to get into some specifics. For my form, I'm doing the following:

1) Using jQuery on the page
2) Validating the form prior to submission with jQuery's Validate plugin
3) Processing the form submission with PHP

With that in mind, here's how I'd add the new inputs: just after the set of helmet inputs, I'd add button a link or button that says "More Pointy Helmets!", and use the following jQuery to capture the click event:

$('#addhelmet').click(function(){
var i = nextNumber();
var newInputs = newHelmetInputs(i);
$(this).before(newInputs);
('#helmet' + i + 'size').rules('add', {digits: true});
});

That takes care of adding the inputs themselves, and adding a validation rule to make sure the size is given in digits.

Finally, you may have noticed that the names I gave the inputs are written like arrays. If you're using PHP, you can process those as arrays after the form is submitted. This means that no matter how many fields are added, you'll handle them the same way. For example, if you're generating an email that lists all the helmets in this order, you could do this:

foreach ($_POST['helmet'] as $key=>$val){
  $emailbody .= 'Helmet ' . $key . ' Color: ' . $val['color'];
  $emailbody .= 'Helmet ' . $key . ' Size: ' . $val['size'];
  $emailbody .= '
"
}

Voila! Whether the user wants a solitary helmet or enough to outfit his entire shuffleboard league, this form can handle it.

So what are these closure thingys?

2009-12-21T08:31:00.002-05:00

A couple of nights ago, I dreamed about closures. I was explaining them to my friend Michael. I think I did an OK job, but since Google still can't search my dreams (glare!), I thought I'd post something here.

What are closures?

Closures are a way to let a function have persistent, private variables - that is, variables that only one function knows about, where it can keep track of info from previous times that it was run.

Let's break this down.

Private variables

Giving a function a private variable is easy: just make a local variable inside it. For instance, in Javascript:

function annoyingAlert(){
   var x = "I'm private! ";
   x += "Nobody can see me outside this function.";
   alert(x);
   x = "Yep, still private.";
}

Although x can't be changed from outside our function, it is very short-lived: each time the function finishes, x gets thrown away. When the function runs again, it is re-created. The second string it gets assigned will never be seen in the alert.

Persistent Variables

These are also pretty easy. Just make a... (sinister music) global variable.

nastyGlobalX = 1;
function annoyingAlert(){
   var alertString = "I can count! ";
   alertString += "I'm currently at " + nastyGlobalX;
   alert(alertString);
   nastyGlobalX++;
}

Now you've got persistence - each time you call annoyingAlert(), you'll get a new number. But since we're using a global variable, it could get changed by a different function. Our counting would be ruined!

Enter closures

This is where closures come in. They work like this:

1. Make an outer, "function maker" function.
2. Declare a local variable inside it.
3. Declare an inner function inside the outer one.
4. Use the outer function's variable inside the inner function.
5. Have the outer function return the inner function
6. Run the function, and assign its return value to a variable

For example:

function functionMaker(){
   var x = 1;
   function innerFunction(){
     alert(x);
     x++;
   }
   return innerFunction;
}

var counterFunction = functionMaker();
counterFunction();
counterFunction();

See! Magic!

Wait, you look confused. OK, let's back up.

A Closer Look

OK, look at the example above. What gets assigned to the variable called 'counterFunction?'

At first glance, you might say it's the functionMaker function. But it's not - we said 'functionMaker()', with parentheses, which means 'run that function and put its return value here.' When functionMaker runs, it ends by returning innerFuction. So counterFunction = innerFunction.

The magic happens at step 5. When functionMaker returns innerFunction, its life is over. Like a used tissue, it has served its purpose and will be thrown away. That includes any local variables that it contained.

But wait! Its innerFuction can't be thrown away, because we've assigned it to a variable, counterFunction. We still need it! And counterFunction() uses the local variable that functionMaker() declared (which was called 'x'). So that variable can't be thrown away either.

But where is that variable? It's in the twilight zone. From counterFunction()'s perspective, it was declared in the parent scope and persists from one run to the next, just as though it were a global variable.

But from any other function's perspective, it's a local variable, declared inside functionMaker(). They can't touch it or see it.

Yeeha! Now we've got a variable that's both private and persistent. It's a place for counterFunction to keep its record, safe from meddling and from garbage collection.

What's it good for?

But... so what? What can we do with that?

Well, a closure could be used to generate, for example, a function called newUserID(). Every time you call that function, you'd get a new ID. But in most situations, you'd want to take care of something like that in a database.

A closure would only help if you need a variable to stick around while your program is running, but no longer.

Next post, I'll show an example where that's exactly what you'd need.

Hello World

2009-12-04T02:52:00.000-05:00

You have a blog. Your mom has a blog. Your cat has a blog. Various inanimate objects in your house have blogs.

Me too! I want to have a blog! I like the way it sounds. "Blog." Like a big robot with heavy fists. His job is to crush cars - never people! A trustworthy robot.

Oh, Blog. You and I will have many adventures together.