Skip to content

amiri/EmpireLogistics

BranchesTags

Repository files navigation

SITE

https://www.empirelogistics.org/

CONTRIBUTOR OVERVIEW

There's a lot going on here.

Application

Here are the layers of the application, from bottom to top:

  1. PostgreSQL

    Database

  2. Catalyst

    Backend Application, serving such URLs as

     /admin
 /blog
 /details

    et cetera.

  3. TileStache

    Backend Application, serving such URLs as

     /tiles/ports/16/15401/27089.json
 /tiles/warehouses/15/15021/89.json

    et cetera.

  4. uwsgi

    Backend Application Server, serving the above 2 backend applications at specified ports.

  5. nginx

    Frontend Web Server, serving the two uwsgi applications from their specified ports to the internet.

Code

Here is how the code is laid out in this directory.

  1. lib/

    The Catalyst application lives here.

  2. bin/

    Lots of data-mangling scripts live here.

  3. data/

    When you run make chef, or some other Makefile targets, data gets downloaded from various sources and it is stored here for use by the scripts.

  4. etc/

    • Our chef configuration lives here
    • As does some static, unchanging data in etc/data.
    • And our database changes are managed out of etc/schema by a tool call sqitch. This is for when we need to add a new column to a PostgreSQL table or add a new table entirely.
    • There are also some configuration files in here. That's a traditional use of the etc/ directory.

  5. local/

    "Sandboxed" versions of perl and python libraries, custom to this application, live here.

  6. node_modules

    We use a tool called Grunt to manage css and javascript assets, namely to combine and compress them. The node package manager downloads the plugins we use and stores them here. Grunt's configuration file is

     Gruntfile.js

    The npm configuration is

     package.json

  7. root/

    • CSS

      Our static javascript and css assets live here. The main css file for the site is

        root/css/el.css

    • Javascript

      The main js file for the site is

        root/templates/js/el.tt

      The main javascript file is, you will notice, a template. That is because we need to interpolate some variables, depending on what environment the application is running in. Namely, the tile server runs on a different port on a sandbox versus the real webserver.

    • Templates

      We use Template::Toolkit for templating. These file are in

        root/templates

      The main template for the site is

        root/templates/wrapper.tt

INSTALLATION

BARE METAL

If you are running a Debian-derived distribution, preferably Ubuntu 13.10, Saucy Salamander, you can run the following on the command line:

make sandbox

The "sandbox" target in the Makefile does all the hard work. It configures the server, imports and populates all the data, and it will overwrite existing data if you run it again. This is dangerous in production, of course, but it's OK for a sandbox.

If you want to break it down, you can build the parts individually:

make prereqs
make chef
make database
make sqitch

Now you should have a populated PostgreSQL database with a lot of data in place. You also have nginx, PostgreSQL, and uwsgi installed and everything running.

VAGRANT/VIRTUALBOX

This installation is more complicated. Here is the Mac installation:

  1. Clone the repository.

    • Go to https://mac.github.com and download https://central.github.com/mac/latest

    • Install it.

    • Clone the repository to your Desktop, e.g., /Users/You/Desktop OR

    • Open a Terminal: /Applications/Utilities/Terminal.app

    • Type the following commands:

        cd ~/Desktop
  git clone https://github.com/amiri/EmpireLogistics.git

  2. Install X11

  3. Install VirtualBox

  4. Install a base box for Virtualbox.

    • Leave the Terminal where it is, open a web browser and go to: https://cloud-images.ubuntu.com/vagrant/saucy/current/

    • Choose a vagrant base box image. I have chosen the amd64-vagrant image in the command below. Your Mac is most likely a 64-bit machine, so that's the one you probably want, too.

    • Run the following command in the Terminal:

            vagrant box add base http://cloud-images.ubuntu.com/vagrant/saucy/current/saucy-server-cloudimg-amd64-vagrant-disk1.box
      vagrant up

You should see some setup information as this is working. You may have to answer some questions. The defaults should be fine. Once the setup is done, you can ssh to the machine. * Complications & Troubleshooting

  There may be some complications here with the filesystem for the Vagrant shared
  folders. You will probably need to use the NFS filesystem. See this page
  for details on how to edit your Vagrantfile in this directory.

          https://docs.vagrantup.com/v2/synced-folders/nfs.html

  There may also be some complications with the size of your vagrant virtual machine's
  hard drive size. See this page for details on how to convert the VMDK to a VDI and
  resize that VDI.

          http://www.ifusio.com/blog/resize-your-sda1-disk-of-your-vagrant-virtualbox-vm

  1. SSH to the Virtual Machine

         vagrant ssh -- -A

  2. cd to the shared folder

    Now you are inside the virtual machine. Run the following commands:

     cd /vagrant

    Now you are in a shared directory, the directory of the repository.

     ls

    These are the files in the repository.

  3. Install the application

    • Run the installation commands:

        make prereqs && make all

  4. Install a browser to the VM

    • Open up another Terminal: /Applications/Utilities/Terminal.app

    • Run the following commands:

        cd ~/Desktop/EmpireLogistics
  vagrant ssh -- -A
  cd /vagrant
  sudo apt-get install chromium-browser
  chromium-browser --disable-accelerated-compositing --blacklist-accelerated-compositing

  5. Test your installation

    At the last command of the step above, you should now have a Chromium browser window open.

    Now you are running a browser inside the virtual machine. Visit http://localhost and you will see the application running.

    The idea here is that now you have 2 Terminal windows open. One is running chromium, and the other is sitting at the /vagrant directory inside the virtual machine. You can open up a text editor on your Mac and edit files inside the repo directory. When you are done editing, you can reinstall the application and check it.

    Reinstallation, again, is make all, which you will run inside the first Terminal window. It takes a much shorter time each time after the first.

DATA OVERVIEW

We have all the rail, warehouse, and port data importing now. Remaining is original data from the current site, which must be imported manually.

We have three major data layers:

  • rail
  • ports
  • warehouses

In North America:

  • USA
  • Mexico
  • Canada

North American port data comes from:

North American rail data comes from:

North American warehouse data comes from:

The shapefiles are imported directly into a PostGIS-enabled database. Other data sources are imported into the same database after being prepared/mangled/reformatted. Once all the data is in the database, we make new tables in that database by joining and selecting from the raw data tables. This produces the set of tables that our application, finally, consumes.

The visualization of each layer is a separate Leaflet TileLayer, rendered by d3, which is faster than Leaflet options now.

ARCHITECTURE

We run two uwsgi app servers behind nginx:

  • TileStache (http://tilestache.org/)

    uwsgi serves a TileStache app for the vector tiles.

  • EmpireLogistics

    uwsgi serves a PSGI app for the main application.

The database server is PostgreSQL 9.3 with PostGIS extensions.

We cache vector tiles on disk, where they are served directly, speedily, from nginx.

About

EmpireLogistics app, data, html

Resources

Readme
Activity

Stars

3 stars

Watchers

4 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Languages