Implement custom blocks to be inserted in nginx config. #142

berkes · 2014-11-13T21:12:31Z

This is a go at #97

houndci-bot · 2014-11-13T21:12:34Z

vendor/cookbooks/rails/recipes/default.rb

@@ -103,9 +103,11 @@
      end
    end

+    custom_configuration = app_info["nginx_custom"].reject{ |k,v| v.nil? || v.empty? }


Unused block argument - k. If it's necessary, use _ or _k as an argument name to indicate that it won't be used.
Line is too long. [86/80]
Space missing after comma.
Space missing to the left of {.

houndci-bot · 2014-11-21T14:15:38Z

vendor/cookbooks/rails/recipes/default.rb

+      variables(
+        name: app,
+        domain_names: app_info["domain_names"],
+        enable_ssl: File.exists?("#{applications_root}/#{app}/shared/config/certificate.crt"),


Line is too long. [94/80]

Fixing this involves a lot more then just wrapping the line.

Proper fix would be to either introduce some helper modules and call methods such as certificate_exists_for?(app).

The next best thing, and more chef-ish, is to make these blocks into LWRPs. That allows for a lot of cleaning up.

The next best thing is to extract all these into variables, but that really does not make the code more readable.

Alll this is is a much larger, and different task than what I'm trying to achieve in this PR,

Yeah that's fine. I think I'm going to ignore the Hound line length setting and set it to 150 or something like that.

Also: agree with you on refactoring this into nice cheffy libraries and LWRPs. But let's first see what the best approach is right now.

michiels · 2014-11-24T13:13:41Z

One suggestion from my part would be to make the custom_nginx subkeys a bit more verbose. So instead of before call it before_server so you know its outside of the server block.

Do you have a use case for these generic before and after blocks? Otherwise, I'd remove them for now and only focus on the things we'd want to put inside the server blocks.

berkes · 2014-11-24T14:18:29Z

Do you have a use case for these generic before and after blocks? Otherwise, I'd remove them for now and only focus on the things we'd want to put inside the server blocks.

One thing I've used them for in the past is to allow custom server {} blocks. Those server-blocks can then contain:

Redirect domains, a list of domains that the application does not listen to, but who redirect to a canonical domain on which the domain does listen.
WWW- non www- redirection. In reality this is technical equivalent of above.
Provide alternative backends. E.g. during a migration we'd provide a v1 backend and a v2 backend; then on e.g. a whitelist of IP-addresses, set certain requests to the new "v2" backend.
Provide a (temporary) server-block that disallows access, provides a basic-auth request or serves a simple, temporary launching-soon html-page.

The fist two could best be actual features in the configuration json, something like redirect_domains: [] instead of providing a hack where people generate custom server blocks to deal with this. But for now, it is the simplest way to achieve such redirect-domains.

The alternative-backends is mostly a hack to work around a (temporary) case which, arguably, should be dealt with in the app itself. But when you can add arbitrary server blocks, you can solve it this way too.

michiels · 2014-11-24T14:33:09Z

Thanks this makes sense. Especially for the redirect stuff. I would agree with also providing "redirect_domains" but I've also noted that are so much edge cases you might want to cover that sometimes just providing the nginx snippet yourself is much better.

Let's experiment with this approach. I'll test it and merge it in! Allright?

berkes · 2014-11-24T14:34:21Z

You don't want to wait untill I've renamed them to before_server and after_server? Because I can do that, but probably not before thursday.

michiels · 2014-11-24T14:35:43Z

@berkes If you like I can add a commit to your branch by pulling it from your remote. Would that be allright?

berkes · 2014-11-24T14:36:53Z

yea, sure!

michiels · 2014-11-25T10:04:51Z

Is there a way to merge this PRs on GitHub? I would like to retain this PR to do the changes :)

jvanbaarsen · 2014-11-25T10:21:48Z

@michiels Not sure what you mean? You can push the "Merge Pull Request" button?

michiels · 2014-11-25T10:29:45Z

I want to collaborate with @berkes on his branch by adding a commit myself. But the only way to do that is create a new branch in this repository and make a new PR of that, losing this discussion and PR in the process.

michiels · 2014-11-25T10:56:08Z

What do you guys think of some documentation? I'm a bit afraid with cluttering the README with all kinds of "advanced" features. Maybe just use the Wiki pages to explain features like these?

berkes · 2014-12-19T09:54:17Z

Documentation.

Below should go into
https://github.com/intercity/chef-repo/wiki/sample_host.json-explained#applications
when this PR is merged.

nginx_custom

The nginx_custom directive can be used to inject custom nginx
configuration at various point in the vhost configuration file.

  "active_applications": {
    "<appname>_<stage>": {
      ...
      "nginx_custom": {
        "before" => "",
        "server_main" => "",
        "server_app" => "",
        "server_ssl" => "",
        "server_ssl_app" => "",
        "upstream" => "",
        "after" => "",
       }

The template is the best reference where these will go. Below is a simplified
example for the unicorn-vhost (with upstream) and ssl enabled:

<%= @custom_configuration["before"] %>

server {
  listen 80;
  server_name example.com;
  root /u/app/example_com/current/public;

  try_files $uri/index.html $uri.html $uri @app;

  location @app {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;

    proxy_pass http://example_com;
    <%= @custom_configuration["server_app"] %>
  }
  <%= @custom_configuration["server_main"] %>
}

server {
  listen 443 ssl;

  ssl_certificate /u/apps/example_com/shared/config/certificate.crt;
  ssl_certificate_key /u/apps/example_com/shared/config/certificate.key;

  server_name example.com;

  root /u/apps/example_com/current/public;

  location / {
    try_files $uri @app;
  }

  location @app {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto https;
    proxy_redirect off;

    proxy_pass http://explained;
    <%= @custom_configuration["ssl_app"] %>
  }
  <%= @custom_configuration["ssl_main"] %>
}

upstream example_com {
  server unix:/u/apps/example_com/shared/tmp/sockets/unicorn.sock;
  <%= @custom_configuration["upstream"] %>
}

<%= @custom_configuration["after"] %>

Typical uses of the custom places:

before: extra server blocks to handle redirects, migrate domains or
handle things like "www to non-www redirection"
server_app: config options such as options for passenger or the
proxy.
server_main: include extra configs, such as redirects.
ssl_app and ssl_main: same as server_app and server_main, but for
SSL. Note: if you set these, but don't have SSL-certificates
configured, this block won't be rendered at all; the custom config will
not be rendered at all.
upstream: directives such as server for load-balancing or fallback
servers.
after: extra server blocks to handle fallthrough situations or
define custom upstream blocks for fallback (e.g.
offline-during-migration) situations.

Example

Given a file with redirects in
/u/apps/example_com/shared/config/nginx_rewrites.conf containing:

rewrite ^/tags/large-bags$ http://example.com/t/bags break;
rewrite ^/content/about$ http://example.com/t/about-us break;

You can now include that file in the vhost with (note the trailing ;):

  "active_applications": {
    "<appname>_<stage>": {
      ...
      "nginx_custom": {
        "server_main": "include /u/apps/example_com/shared/config/nginx_rewrites.conf;"
       }

Resulting in a vhost:

server {
  listen 80;
  server_name example.com;
  root /u/app/example_com/current/public;

  try_files $uri/index.html $uri.html $uri @app;

  location @app {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;

    proxy_pass http://example_com;
  }
  include /u/apps/example_com/shared/config/nginx_rewrites.conf;
}

You now have a file where you can maintain custom redirects for
backwards compatibility of an API or for SEO-purposes.

Troubleshooting

The contents of the custom_conf will be placed in there literally. So,
if you add bogus code, forget a trailing ; or do something else that
is invalid, the vhost won't be valid anymore and the reload will fail.
On the server, you can investigate /var/logs/nginx/error.log which
contains the error.
Nginx will be reloaded (not stopped+started or restarted) which means
the old configuration, the one before the error was introduced and a
reload attempted, will remain active: the server should remain up.
For anything but the most simple directives it is advised to test these
on a test-server; e.g. the Vagrant that ships with chef-repo.

Signed-off-by: Jeroen van Baarsen <[email protected]>

@conradwt

Found by @conradwt. intercity#79 (comment)

This fixes intercity#150.

houndci-bot · 2014-12-24T09:03:43Z

vendor/cookbooks/backups/metadata.rb

@@ -1,7 +1,7 @@
 name             "backups"
 maintainer       "Firmhouse"
 maintainer_email "[email protected]"
-license          "Apache 2.0"
+license          "MIT"


Put one space between the method name and the first argument.

berkes · 2014-12-24T09:05:35Z

Fixing the mergecomflicts within this PR will introduce a lot of needless commits. I'll rebase this on master and on @michiels branch and then recreate a new PR.

jvanbaarsen · 2014-12-24T09:06:20Z

@berkes Ok thanks!

Implement custom blocks to be inserted in nginx config.

fee6a17

houndci-bot reviewed Nov 13, 2014
View reviewed changes

Houdci: fix suggestions to bring code in line with hound guidelines

d416d9c

houndci-bot reviewed Nov 21, 2014
View reviewed changes

Move custom-config builder into helper

4da8b69

berkes force-pushed the 97-custom-config-nginx branch from aedf3d0 to 4da8b69 Compare November 21, 2014 17:07

michiels mentioned this pull request Nov 25, 2014

Implement custom blocks to be inserted in nginx config. #147

Closed

jvanbaarsen and others added 5 commits December 23, 2014 21:16

Moved all the licensing to MIT

05e6907

Signed-off-by: Jeroen van Baarsen <[email protected]>

Rake should be part of the gems.

eadc686

Added a custom .hound.yml that increases our line length.

be74db7

Typo in sample json, forgot a comma

b0bfcfc

Found by @conradwt. intercity#79 (comment)

Add "name" attributes to metadata.

8834d59

This fixes intercity#150.

berkes added 3 commits December 23, 2014 21:19

Add ability to define Redirect Domains.

e3831b4

Houdci: fix suggestions to bring code in line with hound guidelines

a2f5d31

Move custom-config builder into helper

85ac32d

houndci-bot reviewed Dec 24, 2014
View reviewed changes

berkes closed this Dec 24, 2014

berkes mentioned this pull request Dec 27, 2014

Feature/97 custom config nginx rebased #155

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement custom blocks to be inserted in nginx config. #142

Implement custom blocks to be inserted in nginx config. #142

berkes commented Nov 13, 2014

houndci-bot Nov 13, 2014

houndci-bot Nov 21, 2014

berkes Nov 21, 2014

michiels Nov 24, 2014

michiels commented Nov 24, 2014

berkes commented Nov 24, 2014

michiels commented Nov 24, 2014

berkes commented Nov 24, 2014

michiels commented Nov 24, 2014

berkes commented Nov 24, 2014

michiels commented Nov 25, 2014

jvanbaarsen commented Nov 25, 2014

michiels commented Nov 25, 2014

michiels commented Nov 25, 2014

berkes commented Dec 19, 2014

houndci-bot Dec 24, 2014

berkes commented Dec 24, 2014

jvanbaarsen commented Dec 24, 2014

Implement custom blocks to be inserted in nginx config. #142

Implement custom blocks to be inserted in nginx config. #142

Conversation

berkes commented Nov 13, 2014

houndci-bot Nov 13, 2014

Choose a reason for hiding this comment

houndci-bot Nov 21, 2014

Choose a reason for hiding this comment

berkes Nov 21, 2014

Choose a reason for hiding this comment

michiels Nov 24, 2014

Choose a reason for hiding this comment

michiels commented Nov 24, 2014

berkes commented Nov 24, 2014

michiels commented Nov 24, 2014

berkes commented Nov 24, 2014

michiels commented Nov 24, 2014

berkes commented Nov 24, 2014

michiels commented Nov 25, 2014

jvanbaarsen commented Nov 25, 2014

michiels commented Nov 25, 2014

michiels commented Nov 25, 2014

berkes commented Dec 19, 2014

nginx_custom

Typical uses of the custom places:

Example

Troubleshooting

houndci-bot Dec 24, 2014

Choose a reason for hiding this comment

berkes commented Dec 24, 2014

jvanbaarsen commented Dec 24, 2014