Mar 062017
 

This is to fix build, slug, and caching problems related to asset compilation during slug compilation.

Everyone needs a little spring cleaning, right? No, really. Sometimes when I switch around buildpacks or change up assets I run into a strange asset cache problem. Here’s the five magic commands to purge your slug compile process, from less-invasive to more-invasive. When you have a nasty asset cache problem, usually #4 is the one you need.

To use the Heroku repo plugin, I think you’ll want to install it following these instructions.
Note that the Heroku plugins are installed LOCALLY on your machine not on your environment. They are simply shell scripts that perform a series of actions on a remote Heroku environment.

Use with caution. After each one, re-push your branch to Heroku.

1.

heroku run rake tmp:cache:clear -a appname

(then push your branch to Heroku)

2.

heroku run rake assets:clobber -a appname

(Rails 4+, for Rails < 3, use heroku run rake assets:clean instead)
(then push your branch to Heroku)

3.

heroku repo:gc -a appname

This will run a git gc -agressive against the applications repo.

(then push your branch to Heroku)

4.

heroku repo:purge_cache -a appname

This will delete the contents of the build cache stored in the repository. This is done inside a run process on the application.
(then push your branch to Heroku)

5.

heroku repo:reset -a appname

This will fully empty the remote repository.
(then push your branch to Heroku)

Feb 202017
 

Site speed can be said to be the number one issue facing web developers today.

Whether it’s this KISS Metrics block post, another KISS Metrics block post, study after study show that delivering your content fast, fast, fast is make-or-break factor in today’s web economony. That’s why it’s so important that your images are optimized for the web.

Photoshop and other tools export notoriously large files -- well over 1 MB. This is unacceptable in today’s world, where 33% of mobile users in the US are on 3G connections.

If you’re on Rails using Paperlcip, I’ve got a great solution to explore for you today: Image-Optim. You can automagically compress all your images, inside the Rails pipeline and also the ones you upload with Paperclip. On Heroku, you’ll need to use two special buildpacks to make this work. As well, because Heroku uses an ephemeral file system, Paperclip needs to be configured to use an AWS bucket as its storage.

First, refer to my blog post from last year, about how to add the ImageMagick buildpack to your Cedar-14 herokubuikd.

The instructions above will direct you to do add this buildpack first:

heroku buildpacks:add -i 1 https://github.com/jasonfb/heroku-buildpack-cedar14-imagemagick704

Then add another buildpack to your Heroku environment

heroku buildpacks:add -i 2 https://github.com/bobbus/image-optim-buildpack

(you’ll note here you are using the index flag to put this buildpack into position 2 because you already should have the Imaegmagick buildpack at position 1)

You should now have 3 buildpacks, which can be check with heroku buildpacks like so:

$ heroku buildpacks -a your-heroku-app
=== your-heroku-app Buildpack URLs
1. https://github.com/jasonfb/heroku-buildpack-cedar14-imagemagick704
2. https://github.com/bobbus/image-optim-buildpack
3. heroku/ruby

Then add to your Gemfile these 4 gems, (for the sake of this post I will assume you already have gem ‘paperclip’ in your Gemfile).

gem ‘paperclip-optimizer’
gem ‘image_optim’
gem ‘image_optim_rails’
gem ‘image_optim_pack’

To get this working on Heroku, you’ll actually need to work through a few more steps: database setup, AWS. For the lazy, check out the example which you can find at the end of the blog post.

Here’s my has_attached_file. In this example, I’m creating only two styles: a thumbnail, and an optimized version.

Notice that I’ve turned off lossless compression, in other words, allow_lossy: true

With this safeguard on (allow_lossy: false, which is default), I’m usually able to only get an image down to about 75% of its original size.

A large 909KB file was only reduced down to 730 KB; whereas Optimizilla was able to get it down to a whopping 189 KB.

With the safety guard switched off allow_lossy: true, I get much better results but much worse quality.

1st Example
Here, I define a thumb and a optimized.

has_attached_file :attachment, {
 styles: {
  :thumb => ‘125×100>’,
  :optimized => ‘%’
 },
 processors: [:thumbnail, :paperclip_optimizer],
 paperclip_optimizer: {
  nice: 19,
  jpegoptim: { strip: :all, max_quality: 10, allow_lossy: true },
  jpegrecompress: {quality: 1},
  jpegtran: {progressive: true},
  optipng: { level: 2 },
  pngout: { strategy: 1}
 },
 convert_options: { :all => ‘-auto-orient +profile “exif”‘ },
 s3_headers: { ‘Cache-Control’ => ‘max-age=31536000’}
}

2nd Example
Here, I define a thumb and a large.

Remember, when configured together the whole thing looks like this, see the “Per style setting” on this paperlclip-optimizer doc:

(this is an example that mimics the paperclip-optimizer docs)

 has_attached_file :avatar,
          processors: [:thumbnail, :paperclip_optimizer],
          paperclip_optimizer: {

          },
          styles: {
           thumb: { geometry: “100×100>” },
           large: {
            geometry: “%”,
            paperclip_optimizer: {
             paperclip_optimizer:
{
              jpegrecompress: { allow_lossy: true, quality: 4}},
              jpegoptim: { allow_lossy: true, strip: :all, max_quality: 75 }
            }
           }
          }

The Magic Sauce

The docs say you should have allow_lossy set to its default, which is is false. Using this setting this way means your images come out with no quality loss. In my tests, I’ve found that this setting should be turned on, overriding the default.

I recommend paying attention to two important settings
jpegoptim max_quality – 0 through 4, with 4 being best quality
jpegrecompress quality – 0 through 100%, with 100% being best quality

In my tests, I’ve found that the following are acceptable for production websites with high-quality images.

Option A
jpegoptim max_quality quality: 4; jpegrecompress quality: 80
this yields 20-40% compress images of the uncompressed JPGS

Option B
jpegoptim max_quality quality: 3; jpegrecompress quality: 60
this yields 10-20% compress images of the uncompressed JPGS

As far as I can tell, jpegoptim max_quality setting appears to have very little effect on the file size, where as the jpegrecompress quality setting has the most dramatic effect, especially on larger files. The values for jpegrecompress quality are 0-4, with 0 being the least quality (most savings) and 4 being the best quality. With a settle of 4, you can’t perceive any quality loss, but you don’t get the benefit of extremely optimized files. I recommend a setting of 3, which is barely noticeable in terms of quality loss but a significant boost in file size.

Test App

I threw together a test demo here. It lets you upload your own JPGs and see how they compress. It’s important to examine your own files, weighing the quality loss with the file size gain (that is, speed gain in having smaller file sizes).

https://image-optim-paperclip-exmp-41.herokuapp.com/assets

You can read the source of this demo app on Github.

Please note this Heroku (production) app is configured with a few extra goodies:

AWS setup for a basic Amazon S3 bucket
Postgres setup for Heroku

This app is configured to use an Amazon S3 bucket called jasonfb-example1. Because I pay for this bucket, please do not abuse. This demo app is provided for developer testing purposes only; I reserve the right to delete any images uploaded for any reason, including copyright infringement or simply lack-of-space. Please do not upload any inappropriate photos or photos you do not own.

You can hit the “Destroy” button on any image you upload.

The jpegoptim max_quality and the jpegrecompress max_quality

You’ll notice my example app here creates 5 different versions, using the same jpegoptim setting (jpegoptim: { allow_lossy: true, strip: :all, max_quality: 75 }, but 5 different quality settings on the jpegrecompress setting (be sure to note the jpegrecompress takes a quality parameter of 0-4; the jpegoptim setting takes a max_quality setting of 0-100)

In my example app I’ve split the settings for jpegrecompress and jpegoptim into a global setting and a per-style setting. Its setup differs from the examples above.

In my sample app, I’ve set the jpegoptim max_quality setting to 75 and created five different jpegrecompress settings: 0, 1, 2, 3, and 4, named:

optimized_compress_0
optimized_compress_1
optimized_compress_2
optimized_compress_3
optimized_compress_4

(you’ll see these in the has_attached_file in app/models/asset.rb)

So go ahead, upload a color-rich un-optimized image. In my experiments, I found that quality settings 4, 3, 2, and 1 yield approximately the same file size, with only a small dip in file size when you went down to 0.

However, the noticeable loss in quality begins to happen even at quality setting 3, so it seems to me why not use quality setting 4. You will be baking in an automatic guard against very large un-optimized images coming into your app. You’ll need to play around with these two settings.

Important Addendum (2017-03-09)

I am adding an important addendum to this post. After switching around my buildpacks on Heroku, I ran into a strange Sprockets error:

undefined method `dependency_digest’ for #<Sprockets::StaticAsset:0x007fefb93d0d28>

The only way I found to fix this was to purge my assets in slug compilation. This will mean your 1st push after purging will take an extra long time to slug compile.

If you run into that error, do this before you push to your environment:

heroku repo:purge_cache -a appname

Also see this Stack overflow post. I corresponded with the maintainer of Sprockets regarding this issue, and he suggested later versions of Sprockets may have addressed this issue (we are on Rails 4.1 with Sprockets 2.12.4).

Feb 082015
 

First you need to know how to connect to your Heroku application using bash. You can then use the du -h command to get a read on how big the files inside your slug are.

heroku run bash -a yourappname

Once you’ve done that, here’s the magic command that will show you how big each folder in your slug is:

du -h --max-depth=1

Max-depth of 1 will give you 1 level deep output of how big each of those root folders are. You can change the max-depth to see more granular introspection of each folder. (However, realistically, it is probably more useful to cd down to some subfolders and run du -h --max-depth=1 again.)

Some observations I have made:

1. Obviously you basically want to get as much out of your assets/ folder as you possibly can. Almost everything I do these days is tied to some kind of managed content, so I can safely put almost all the images and videos in my app into a Paperclip-backed data model. Using S3, Paperclip stores all the different sized versions it will render remotely, so all that stuff stays out of the slug.

2. Beware of Gem bloat. If you don’t use it, you probably want to get it out of there.

3. Generally speaking just keep large files out of your app. If you’re needing a lot of larger data files re-think your architecture to work in a more distributed, background job kind of way that doesn’t require the files to be in the slug. (Remember, worker dynos can always download from or upload to S3 and storage on S3 is much cheaper than keeping things in your slug.)

4. Use .slugignore (well documented so I won’t go into it here.)

5. I found that using gems directly from Rubygems seems to save a lot of space compared to using gems which point to a Github repo.

You can recognize a gem pulled from its Rubygems version because it usually has few or no parameters, or only a version parameter:

(pulling from Rubygems)

gem ‘spree’, ‘2.1.12’

It turns out that because Spree can be a little lazy about building the Gems, they recommend you just point your Gemfile to the github repo. (To be fair, they maintain a lot of bugfixes across several branches, and pointing to a branch on Github makes much more sense if you have software that is retroactively fixed like Spree)

(Pointing to a github repo)

gem ‘spree’, git: ‘https://github.com/spree/spree.git’, branch: ‘2-1-stable’

The difference here is that Bundler can just pull a compiled gemfile in the first example, whereas it has to download the repo and turn it into a Gem in the second example. In my app, the latter (pulling from Rubygems) saves me a whopping 70 MB compressed in my slug. Although I haven’t confirmed, I suspect this is because Bundler is actually very efficient at packaging up a Gem and excludes all the support files you don’t want when you create the gem, whereas the ‘lazy’ way of just pointing to a Git repo has the side-effect of including all the files in the repo in your finished slug.

Aug 252014
 

[This is an excerpt from an email I answered on the Rails-Talk list today]

That’s actually pretty hard, but with a bit of work you will be able to pull that off.

You have two basic strategies:

1) Make the user upload the image file somewhere else (like amazon S3 bucket) using their own client (like an FTP client), then copy & paste the URL into your WYSIWYG editor (wrapped inside an image tag) – most editors let you do that easily.

2) provide an interface for the user to upload the image from their computer to your website (I see now why you were going down the paperclip route). Then make some kind of interface that allows them to drop a string-style reference to the image into the WYSIWYG editor. In the app I’m currently working on, we have a special macro inside the editor so you use something like this:

[IMAGE:123]

When output, this macro actually replaces the block of text with the image with id 123. (While editing, the editor doesn’t actually see the image, they only see the macro)

If you’re on Heroku, or designing for scale, you have some special considerations when creating a web app that accepts large file uploads. Check out this article here which explains how it is done.

In particular, see the note on this page that says:

Large files uploads in single-threaded, non-evented environments (such as Rails) block your applications web dynos and can cause request timeouts and H11, H12 errors. For files larger than 4mb the direct upload method should be used instead.

If your images files are large (they say larger than 4 MB, but I would even say larger than 500K), you need to do direct upload to S3. This is documented here.

As you can see, this is actually a complicated can of worms (which is why you should strongly consider if option #1 above is better for you since it is much easier and quicker to implement)

You could probably write an uploader using method #2 described above that works with TinyMCE and inserts some kind of high-level macro or the actual image tag using javascript. But you definitely would have to get your hands dirty with javascript.

If you want to go with Method #2, I strongly recommend that you DO NOT do pass-through uploading on Heroku. Although it will work for very small files, at scale you will create long running-request bottlenecks that will affect other users of your app – people who aren’t even using the upload tool will see slow performance. The s3_direct_upload gem (below) is one solution to this problem (it is an implementation of what the Heroku article discusses when it says “Direct upload”)

see also:
Direct Uploads to S3 with Rails & Paperclip
s3_direct_upload gem

As an alternative do “building this yourself,” you might consider this Rails engine-style gem calls Comfortable Mexican Sofa, which serve as a fully-fledged content management system (or “CMS,” you will take notice its cheeky name)
Comfortable Mexican Sofa

Dec 042012
 

What you may miss off-the-bat is that Heroku’s copy of your app is actually a git repository. It’s like github’s repository of your code, but it is used for the special purpose of deploying the app.

When you do a deploy to Heroku, you are pushing only your latest commits to the master branch of the Heroku repository for your app.

Type more .git/config at the command line of one of your apps, you will see a git remote for the heroku repository

[remote “heroku_abc_app”]
  url = git@heroku.com:abc-app.git
  fetch = +refs/heads/*:refs/remotes/heroku/*

In my case, the name of my heroku app is abc-app (made up) but I’ve attached it to a git identifier (just something I type at the command line) called heroku_abc_app

To deploy this app, I would use:

git push heroku_abc_app master:master

This is saying I want to put my local master branch (the first “master”) to the master branch of the remote repository (the second “master” after the colon) to the repository which is identified by heroku_abc_app (defined in .git/config), in my case that repository happens to be at git@heroku.com:abc-app.git

You can also push a different branch to Heroku, like this

git push heroku_abc_app some_branch:master

This is saying you want to push some_branch onto the master branch of your your app. Be careful – if you push a branch and then try to push another branch (or master) onto a non-downstream git timeline, you will get rejected. You can easily fix this with --force at the end of your command.

Although Heroku’s git repository acts just like a real git repository, most of us use github as the authoritative source for our app’s code and the Heroku git setup only for deploying.

Oct 032011
 

Ran into an annoying chicken/egg involving deploying to Heroku with a shared database. It seems that Heroku isn’t happy unless you have gem ‘pg’ in your Gemfile. If you are using the shared database and fail to have gem ‘pg’ in your Gemfile, you app actually deploys OK on Heroku but then generates an Application Error when you call a page. If you check the heroku logs you see something like this:

Continue reading »