Calvin's Maps

if (err) return callback(err);

Wed, 12 Jun 2013 20:08:59 -0400

Modern Projections

Sun, 26 May 2013 12:13:00 -0400

I was really excited by Jason Davies work on reprojected tile maps, this got me wondering, what projection would I want to use for a tile map, my answer: none of them.

Most map projections are from an era of paper maps not zoomable web maps, which means that they are all geared to a specific zoom level. For instance, when people say that mercator is a ‘terrible projection’ what they are actually saying is that mercator is a ‘terrible projection for polar regions and for zoom levels that feature both equatorial and polar regions.’ In fact when zoomed in to a regional level mercator is going to be a pretty great projection for the vast majority of places with permanent human settlements. There are already projections created for specific areas which can make those local areas more accurately then standard mercator.

Meaning that if I wasn’t going to an all day music festival today I’d be looking into making a map:

That at low zoom levels (entire earth) uses an equal area projection that doesn’t screw up shapes too bad, (one of these?)
When zoomed in switch to a Lambert Conformal Conic with the reference parallels dynamically calculated based on the current map view, why lcc? because most maps tend to be wider then they are tall, for a tall map, i.e one for a smart phone, you might use transverse mercator. LCC and TM might not be the best choices at polar and equatorial latitudes, so a few more projections might be in order there.

Hospital Prices

Fri, 10 May 2013 08:30:24 -0400

Hospital Prices:

Comparison of the different prices of procedures looking at the differences in prices at different hospitals optionally by state and also looking at the difference between the amount charged and the amount insurances payed.

Demographic changes in D3

Thu, 09 May 2013 20:34:20 -0400

I haven’t done much with D3 lately so I threw together this map of population, households, and employment forecasts. I made it in CoffeeScript and it really reminded me why I love CoffeeScript so much.

Updates to Communist.js

Tue, 07 May 2013 08:30:25 -0400

I did a lot of work on web workers in April leading up to entering the Mozilla Dev Derby (as an aside my favorite one, besides mine, has got to be Zen Photon Garden), I’ve taken what I’ve learned made a few updates to communist.

Dropped node, it just didn’t make sense for communist to be used server side, none of the problems you have are they same, the final nail in the coffin is that I couldn’t think of a reason I my self would ever use it server side.
Communist now has it’s own domain http://communistjs.com/ this has nothing to do with what I just said, but it’s true.
Having the various worker types build on top of each other, while nice from a theoretical point of view ended up resulting in triple layers of function calls.
I don’t think I had been describing it very well so I simplified the explanation of the API with less talk about the messy inner parts.
Call the workers in the same context to allow the use of “this” to persist values across calls. No more “_db” nonsense.
Split up the build into more logical files and added phantomjs headless testing on the build stage and also have started using sauce labs to test it.
With the help of sauce labs, it now works in IE10, Opera, and Safari >5.
Added a few demos

Check it out

State of the Web Worker (api)

Wed, 01 May 2013 17:38:01 -0400

Quick write up I did on web workers you can view the raw gist if it doesn’t show up below.

https://gist.github.com/calvinmetcalf/5495466

ASM (updated)

Sat, 27 Apr 2013 09:23:00 -0400

I was talking to Ryan Westphal the other day and he was saying how he’d been playing round with ASM.js and you could hand code it fairly easily and it was FAST. I’ve been playing around with a crazy fractal generating map so I figured that would be as good a place as any to try out ASM.js.

The code for the different fractal types was already stored in an object so switching it to be an asmjs style module wasn’t too bad, code before:

https://gist.github.com/calvinmetcalf/5473009

code after:

https://gist.github.com/calvinmetcalf/5473022

I was able to smooth out the bugs as firefox versions that support it will give you the line numbers that prevent your asm code from compiling, after I finally got rid of the last of the errors primarily static type N00B mistakes like pow(y,3) but also asmjs ones like trying to initialize a variable to undefined or to an expression and ones related to not reading the allowed operators closely (no logical and). So I fired it up and nothing happend.

Then I realized that ASM seems to be turned off if you open up the debug console. So close tools and hard refresh and … nothing, it’s actually slower in chrome too.

After some intense microbenchmarks and help from John-David Dalton and Vyacheslav Egorov on twitter I was able to figure out that

Replacing the logical and with a binary and (&&->&) was what was killing chrome perf
Despite what Firefox tells you DO NOT USE THE FUNCTION CONSTRUCTOR
ASM doesn’t optimize all code perf.

I have a feeling that there is an issue with the construction of my loop that’s preventing it from optimizing. More details to come, oh and the current fractal demo did I mention that all of this is inside a web worker?

Update: I was planing more updates sooner but then Ubuntu started updating to Raring and my laptop started overheating when I did anything

Update 2: I created a perf with the diag example from the spec and found that adding the words “use asm” cause it to go approximately 1/3 the speed. But for some reason when firefox nightly tries to post the results it gets a 413 so hears a screen shot.

Update 3: This is the perf that Ryan was talking about when he said asm was fast. This is the non-recursive version that I made, the asm version is now 206,350x faster, but the non-asm one is 1,654,225x faster giving it a commanding lead. I will admit I am not a C coder, I can make some pretty optimized JavaScript but this lower level stuff I may be way off on, feel free to give me a shout on twitter with any suggestions.

Let's make a map with Leaflet and d3

Mon, 11 Feb 2013 08:30:46 -0500

I made this map on Friday, today I wrote a detailed post on how I used leaflet and d3 to make it, actual post is over on github because tumblr sucks for code.

I was planning on writing this on earlier, but then I got 30 inches of snow

(not me)

lost power for 20 hours, and brewed bathrobe beer.

(me, and yes I’m aware it looks like poop)

Let's build a Pouchdb app we aren't ashamed of & backbone.js explained without words that start with C,M,or V

Wed, 06 Feb 2013 08:30:38 -0500

The previous post, while it showed the basics of pouch produced an app that is frankly not ready for prime time. So in this post we refactor the app and turn it into something less embarrassing. We are going to use the fantastic library backbone.js, written by the authors of a little language that turns into JavaScript, to tidy the app up, which will require an explanation of backbone.

I’ve been aware of backbone for quite a while but my reaction every time someone tries to explain it has always been “yes but what does it do?” I blame this on the proliferation of libraries and frame works built around the letters C, M, and V which all use terms starting with those letters in similar but divergent ways, so for the rest of this post we will be using no words that start with those letters. If don’t care about backbone you can go right to the updated app or read about how I improved it.

The first basic part of backbone we will refer to as an “item template” though backbone uses a term that rhymes with “bottle.” You can think of this as representing a blueprint for a thing: in SQL it might be equivalent to a row, in PouchDB and MongoDB this might be a document, in Neo4J it might be an edge or node. When you add data you actually make a new instance of your item template (aka use the key word “new”). Your item template can define tests that data needs to pass to be added and ways to serialized or transform your data for export, and can fire events when items are added, removed, updated etc.

The next portion we will refer to as a group of items, but backbone uses a term which rhymes with “selection.” A group is really what it is and in SQL it would be like a table and in PouchDB it would be like a DB. You use a group to not only to attache some events, like watching for updates to a specific attribute across all items, but as an object you can use all of underscores functional tools on. You can run map,reduce,filter,pluck,each and more on your group to easily get just the data you care about.

The third and final of the essential parts of backbone is what I’m going to refer to as a supervisor but backbone refers to with a word that rhymes with “queue”. If this was that show about the extravagant houses of of famous artists then this would be the bedroom. I call it a supervisor because it supervises a specific element of the page and it is the only thing that touches its element. And how do you get the supervisor to know when to do something? By having it listen for updates to a group and events in the dom inside its element. This is the gist of backbone, its key usefulness is segregating the logic so that like is grouped with like. So if you want to get rid of a side bar all if the logic related to it is in one supervisor. Now enough introduction go look at the updating the pouchDB app with backbone.

Lets make an app: PouchDB

Tue, 29 Jan 2013 08:30:36 -0500

This is part 3 in a series, you can (and should) checkout part 1 and part two, we’ll wait.

So we’re going to make an app, first remember that class we made last time, I had to make a few changes to get it to work, so overall it looks like this

https://gist.github.com/4648986

Why you ask? A couple of indentations errors (that nobody caught) but mostly the stop and start methods because I noticed that your browser doesn’t always get the message whey your back on again after going offline, those methods allow you to stop the continuous replication and start it again to get things to sync.

We’re going to make an app that allows you to write notes that others can see and edit, and should work when you go on and off line, this may seem trivial, but checkout @axemclion’s conference app which does something very similar but in a useful way.

So originally I was going to make this in backbone.js but instead I ~~discovered /r/cats~~ decided to do it slightly lover level to demonstrate the principles involved. The structure of the app is going to be as follows;

We have an object called ‘obj’ which has all the documents we care about from the pouch, our on change function will update/add/delete the document from obj.
we have a function called ‘update’ which takes obj and transforms it into an object call ‘views’.
We have a set of mustache templates, we also have a div with id ‘body’.
views is given to the mustache template to generate html, div#body is emptied and then we append the mustache html (I originally was doing this to the body tag before I realized this caused issues with the script tags.)
Besides the contents of obj the only thing that effects the function update is what the current location.hash is.
The event structure is that when a document is changed, pouch updates obj and then calls update(), when a link is clicked, the handler preventsDefault(), changes the location.hash, calls update()

That’s pretty much it. The first gotcha I noticed when doing this was that since I was replacing all the content rapidly:

https://gist.github.com/4654646

I would not be shocked to learn that this caused performance issues with a lot of events like this added. I’d also not be shocked to learn the opposite, so I’d probably test, and this was on hacker news this morning.

So onto the actual stuff, first we have the html, notice all it really does is load 1 css file and a bunch of scripts

https://gist.github.com/4660033

next we have our mustache templates, notice the use of CoffeeScripts triple quoted strings, use of partials, and my disdain for tabs and readability when it’s not forced upon me by CoffeeScript

https://gist.github.com/4660100

next we have a few misc function and the big update one, this does a couple that i’d consider Wrong and want to refactor out if this was for a real project, notable iterating through every object every time something changes, this is not going to scale to more docs very well. We also delete everything and rebuild the page from scratch every time anything changes if I was redoing this I’d break up the page into sections that could be re build individually.

https://gist.github.com/4660123

Lastly we have the jquery on load which mostly just sets up a bunch of events and the on database change which just makes sure it isn’t a design document and then replaces the document unless its been deleted in which case it’s deleted.

https://gist.github.com/4660159

Put that all together, and what do you get, THIS DEMO! feel free to leave a note, etc.

There you have it, next time I’ll probably do something related to this new development and leaflet. Feel free to follow me on twitter.

And with that I leave you with this video (omg nsfw) which was going to be basis of an explanation of how binary trees work until I remembered @daleharvey was Scottish.

In which we explain how to use PouchDB

Mon, 14 Jan 2013 08:30:41 -0500

This is a follow up to my previous PouchDB post, so if you can’t read that you probably are going to want to.

PouchDB is local storage primarily using IndexedDB, but with a WebSQL fallback, even if you arn’t doing crazy replications PouchDB may still be a easier to use because, do you understand IndexedDB? Because I don’t. Pouch on the other hand has an API based on CouchDB’s API which in turn is based on HTTP. Your DB stores documents, which is just JSON but with two special fields (note the underscores):

_id: this you can add your self, but if not it’s added for you, this is the documents key, if you want to find it you’ll need this.
_rev: this is the documents revision, this is what couch has instead of transactions, if you want to modify a document or delete it, you need the _rev of the version you want to change, that way if someone else modifies it first it won’t match.
There are actually 7 more, but we you can safely ignore them for now.

You update Couch via http and all the methods are based on HTTP verbs, Pouch’s methods are the same (though it should be pointed out that HTTP verbs are traditionally written in all caps but are not case sensitive, but JavaScript methods are case sensitive and lowercase):

POST: create a new new document and assigns it a random _id, use this if you don’t care what the key is (i.e. your using it programmatically).
PUT: Use this to either update a document, in which case you need to supply the _id and _rev, or to create a document with your chosen _id, in which case you don’t need _rev.
GET: This is for getting a document, you need the _id obviously.
DELETE/REMOVE: This is one deviation between Couch and Pouch as “delete” is a reserved word in JavaScript, don’t forget you need the _rev. This actually just adds _deleted:true which in Couch means that next time the database is compacted previous version can be removed, note you can’t remove it entirely because if you did you wouldn’t be able to tell other databases so when you replicated.

Now onto Pouch specific things: the first thing to know is that like IndexedDB Pouch is asynchronous. , compare this which is synchronous JavaScript code

https://gist.github.com/4507762

with this async code

https://gist.github.com/4507817

The reason for this async set up is so as not to block the execution, unless you are using workers (which you likely aren’t, see my previous post) all of your script is in one thread, but the IndexedDB is in another thread and when you ask it to get you something instead of putting you on hold it just asks you for a number to call you when it’s found it. Pouch follows the Node.js convention of the first argument given to any callback will be an error argument which will be null/undefined/false if there is not an error meaning you can use if(!err) to check for an error and be the last argument you give to the function (the callback that is), hopefully this makes more sense:

https://gist.github.com/4519828

Ok now for some basic Pouch usage which we will demonstrate by building a sample app …sort of, this post is really long so instead were going to create a class that will serve as the core of the app we will make next time, for serious.

Note unlike the previous code, this code is going to be in CoffeeScript, when I’m not writing directly for the browser this is what I tend to use for a variety of reasons which I go into excessive detail about at the end of this previous post.

So the first thing we’re going to need is a constructor function, it’s going to take a url as its first argument and implement the pattern from my previous post.The second argument its going to take is an on change function, this is the function that will be called every time a document is changed either remotely or locally.

https://gist.github.com/4520740

We want to be able to

Add a new document
Get a document
Delete one

https://gist.github.com/4520785

Which put together looks like

https://gist.github.com/4520837

And there you have a basic PouchDB class next time we’ll make a working app for real, just have to figure out what I was originally going to show something based on leaflet like my leaflet.pouch but that might require too much knowledge of how leaflet works, feel free to hit me up on twitter with suggestions.

PouchDB: an intro

Mon, 07 Jan 2013 08:30:28 -0500

So I’ve been hacking around with PouchDB a bit lately and I’m going to explain why I think it is so awesome.

Before we discus PouchDB we’re going to need to talk about CouchDB which Pouch is based on. CouchDB is an open source nosql document database written in Erlang that grew out of Lotus Notes of all things. I’m personally a fan of CouchDB but of it’s features the ones that matters here are, data is stored in JSON, you use JavaScript to query the data, you communicate with it entirely through a RESTful interface, and something called “Master-Master Replication,” now to put that in terms my cat would understand, this means that two databases can sync without one being master and the other slave but as equals, and this done in a way such that replication can be done in real time or sporadically. Think of email on your cellphone, you can have it so that it tries to sync every x minutes which just fails if it has no reception, or you can set it up so then when you have service you get push notification and when you loose reception you just get all of the emails when you get reception again. This is similar to how Couch can work, though it can be much more complex as well. If you want to know more about Couch then CouchDB: The Definitive Guide is a good place to start.

So one of the issues with CouchDB is that Erlang…well lets just say people have mixed feelings about it, which lead to pretty quickly, CouchDB compatible Databases, Big Couch from Cloudant which you can cluster, TouchDB is a version written in Objective-C targeting embedded apps, and then we have PouchDB.

PouchDB is written in JavaScript and targets the browser, but also works with Node.js. It has a number of different back ends that it can use so at the very basic you can run this (notice the Node.js style async callbacks, they always are function which take as a first argument an err object which is undefined unless there is an error):

https://gist.github.com/4469112

Which lets you access and modify a CouchDB on the same origin like so.

The benefits of this setup is that it works on browsers as old as Internet Explorer 8, but on the other hand this only works with an internet connection.

We can also set up one locally like this:

https://gist.github.com/4469674

This uses either IndexedDB or Web SQL depending on whats available and works on on all browsers except Internet Explorer versions besides 10. This is nice because we can create a large local storage with the same API whether on Chrome, Firefox, or mobile Safari, and the API is a whole lot easier then the IndexedDB API.

Now the neat thing we can do is combine the local and remote data sources with:

https://gist.github.com/4469426

Which sets a pouch db up that syncs to a local database (either IndexedDB or web SQL) which syncs to a remote CouchDB, like so:

If the browser looses it’s internet connection it can then still work with the local database, and it then can sync when it gets internet access again. If the browser doesn’t support IndexedDB or webSQL it just creates a direct connection to the remote Couch.

You can checkout Pouch on:

Next time, example apps.

On Communist.js improved

Wed, 02 Jan 2013 19:41:18 -0500

Two quick notes about Communist.js a library for working with web workers I wrote about in a previous post.

I totally blanked and forgot to put in a license, it’s MIT btw and that’s now reflected.
Added a default function if you don’t specify it that makes it more RPC like, so if you invoke it without an argument, then you can use the method add which takes 2 parameters a name and a function(e.g. var comrade = communist();comrade.add(“name”,function(){};), then when you can do this with multiple functions, the main difference is that the parameters for send are now name of function to evoke, params for function, callback e.g. comrade.send(“name”,param1,param2,…,cb);.

That is all for now, comeback next time for, Communist.js in the wild as I use it for something, or PouchDB, whichever I bother to write first.

On web workers and Communist.js

Sun, 30 Dec 2012 10:19:26 -0500

On web workers and Communist.js:

Alright so this isn’t actually about maps, it’s about JavaScript, which I end up using to make maps most of the time anyway, so that’s why it’s related. I’ve been working on library to help make JavaScript web workers easier to use.

First off, the idea with web workers is to be be able to do stuff on a webpage without blocking. On a web page everything is done in the same thread, so if something your doing take a while, it prevents everything else form working and can cause the page to freeze, in this example you can see that by hitting the red button, or running it in Internet Explorer versions older then 10. Hitting the green button executes that long script as a worker and thus the time keeps ticking.

The two main issues that seem to be keeping workers from being more widely used are:

Internet Explorer
Having to load the worker script as a separate file.

First, Internet Explorer doesn’t support workers until version 10, the one that just came out and nobody uses. This means that your almost definitely going to have people viewing on clients that don’t support workers.

You have to load the worker script from a separate file, now I can see the reasoning behind this restriction as it keeps the 2 contexts separated. But it introduces a couple of issues,

Poly fills are much trickier (though I then found this polyfill right after I wrote that).
Can’t use them with CDN hosted scripts, the worker script has to obey the same origin policy of the html page that it’s run on, but the script that’s calling it does not.

So my idea to deal with this is a library I’ve been writing called Communist.js (why the title, because it cares about the workers). It works around the pattern of write a function, create a new Communist with that function, then send the communist data and a callback, and the communist executes the function in the other thread and calls the callback with the result. If the client doesn’t support web workers it gets executed in thread by a function I call Socialist (Can you tell I wrote for a humor magazine in college?).

Under the hood, Communist is turning the function into a string, templating it with some code to handle message passing, and turning that string into a blob file, which we are then creating a blob URL out of the blob which we are using to make the worker. This is based on the “inline worker” example on the MDN page about using worker, but updated to take into account blob builder being deprecated. This method does have some drawbacks, notably you will get an error if your function references a variable defined outside the function.

Now Socialist ended up giving me much more grief, as I ended up having to recreated certain parts of the worker global object that acted subtly different there as opposed to when executed in the regular context. Notably setInterval which calls it’s it’s function with window as the context even when it’s called from a function with a different context.

So it still needs some work, but don’t hesitate to fork it on Github.

P.S. you may notice I decided to write it in CoffeeScript. Now why would I do that? My growing appreciation for CoffeeScript really deserves its own blog post but quickly:

Splats… are really helpful. This is a features where you can write your function to take (arrgs…, finalArg) and you get an array of values called arrgs and a variable called finalArg, you can also do (one, two, rest…) or you can execute with an array like func(params…). While I know this is just syntactic sugar for stuff you can do in JavaScript, it’s really nice not to have to worry about that when you do it a bunch.
Indentation, my code is so much neater when it isn’t optional.
Prototype as ::. I just think it looks cool.
Class keyword, this makes much of the prototype mess much easier (and ironically means I never got to use :: in this library).

the @avidgeo folks upon learning the osgeo summer code sprint was going to be in boston

Fri, 26 Oct 2012 08:30:30 -0400

the @avidgeo folks upon learning the osgeo summer code sprint was going to be in boston

CouchDB Presentation

Fri, 19 Oct 2012 17:36:09 -0400

CouchDB Presentation:

I gave a presentation at avid geo yesterday about couchdb and what it’s good (and not good) for.

Density

Wed, 17 Oct 2012 09:59:54 -0400

Density:

Quick link to a map i made of ma population density by 2010 census blocks

Street types by voter

Tue, 02 Oct 2012 08:16:50 -0400

Totally forgot that there is another source of street types, the voter record and yes I do have a copy of the 2010 voter record, all 3883893 records, so street types by how many people were registered to vote at them in 2010

But the voter file, now that has some other stats we can work off of, like party

That N stands for unenrolled FWI and the reason it’s so popular is that unenrolled voters can vote in any primary, but enrolled voters can only vote in their own. This does help by telling us that anyone listed as being a member in a party must care that much more, so, how do the parties stack up.

Much more likely to live on a street and less likely to live on anything else, Lane even falls bellow the cut off (4%), so how about Republicans.

In which we see the reverse, this most likely has to do with the suburban nature of MA republican voters vs the Urban Democratic voter now for unenrolled?

An very similar breakdown to the overall suggesting my follow up should be based on which primaries they tended to vote in. Now what about green party?

Un suprisingly they are like democrats, but more so. Sadly I forgot to write a view for the Libritarian party, and the Pirate party wasn’t set up in 2010. For now, thats all.

Full List

Thu, 06 Sep 2012 09:52:39 -0400

Haven’t had as much time as I’d like today but here is the full list of things that roads are called in MA, there are over 300 and some are weird

road lengths

Wed, 05 Sep 2012 11:47:11 -0400

so after reading bostonography’s awesome post on Boston road types, was inspired to do some figures for road vs street vs lane but across the commonwealth. First we can look at by count.

you’ll notice “street” is the top followed by “road” and “avenue”.

But if we look by length (in km) then “road” is first followed by “street” and “avenue” still in third.

Implying that their are less roads then streets but streets are longer. You’ll notice also Highway makes the list by length, this includes both roads commonly called highway, like McGrath Highway in Somerville (or Quincy) and roads not like Yankee Division Highway (aka 128). A similar story with Turnpike though I’m pretty sure that there is only such a high count due to me counting a road that crosses a town boarder as two roads for the count. More data when I have time to crunch the data.