Calvin's Maps

Hospital Prices

Demographic changes in D3

I haven’t done much with D3 lately so I threw together this map of population, households, and employment forecasts. I made it in CoffeeScript and it really reminded me why I love CoffeeScript so much.

Updates to Communist.js

I did a lot of work on web workers in April leading up to entering the Mozilla Dev Derby (as an aside my favorite one, besides mine, has got to be Zen Photon Garden), I’ve taken what I’ve learned made a few updates to communist.

1. Dropped node, it just didn’t make sense for communist to be used server side, none of the problems you have are they same, the final nail in the coffin is that I couldn’t think of a reason I my self would ever use it server side. 
2. Communist now has it’s own domain http://communistjs.com/ this has nothing to do with what I just said, but it’s true.
3. Having the various worker types build on top of each other, while nice from a theoretical point of view ended up resulting in triple layers of function calls. 
4. I don’t think I had been describing it very well so I simplified the explanation of the API with less talk about the messy inner parts.
5. Call the workers in the same context to allow the use of “this” to persist values across calls. No more “_db” nonsense.
6. Split up the build into more logical files and added phantomjs headless testing on the build stage and also have started using sauce labs to test it.
7. With the help of sauce labs, it now works in IE10, Opera, and Safari >5.
8. Added a few demos

Check it out

State of the Web Worker (api)

Quick write up I did on web workers you can view the raw gist if it doesn’t show up below.

ASM (updated)

I was talking to Ryan Westphal the other day and he was saying how he’d been playing round with ASM.js and you could hand code it fairly easily and it was FAST. I’ve been playing around with a crazy fractal generating map so I figured that would be as good a place as any to try out ASM.js.

The code for the different fractal types was already stored in an object so switching it to be an asmjs style module wasn’t too bad, code before:

code after:

I was able to smooth out the bugs as firefox versions that support it will give you the line numbers that prevent your asm code from compiling, after I finally got rid of the last of the errors primarily static type N00B mistakes like pow(y,3) but also asmjs ones like trying to initialize a variable to undefined or to an expression and ones related to not reading the allowed operators closely (no logical and). So I fired it up and nothing happend.  

Then I realized that ASM seems to be turned off if you open up the debug console.  So close tools and hard refresh and … nothing, it’s actually slower in chrome too. 

After some intense microbenchmarks and help from John-David Dalton and Vyacheslav Egorov on twitter I was able to figure out that

• Replacing the logical and with a binary and (&&->&) was what was killing chrome perf
• Despite what Firefox tells you DO NOT USE THE FUNCTION CONSTRUCTOR
• ASM doesn’t optimize all code perf.

I have a feeling that there is an issue with the construction of my loop that’s preventing it from optimizing.  More details to come, oh and the current fractal demo did I mention that all of this is inside a web worker?

Update: I was planing more updates sooner but then Ubuntu started updating to Raring and my laptop started overheating when I did anything

Update 2: I created a perf with the diag example from the spec and found that adding the words “use asm” cause it to go approximately 1/3 the speed.  But for some reason when firefox nightly tries to post the results it gets a 413 so hears a screen shot. 

Update 3: This is the perf that Ryan was talking about when he said asm was fast. This is the non-recursive version that I made, the asm version is now 206,350x faster, but the non-asm one is 1,654,225x faster giving it a commanding lead.  I will admit I am not a C coder, I can make some pretty optimized JavaScript but this lower level stuff I may be way off on, feel free to give me a shout on twitter with any suggestions. 

Let’s make a map with Leaflet and d3

I made this map on Friday, today I wrote a detailed post on how I used  leaflet and d3 to make it, actual post is over on github because tumblr sucks for code.

I was planning on writing this on earlier, but then I got 30 inches of snow

(not me)

lost power for 20 hours, and brewed bathrobe beer.

(me, and yes I’m aware it looks like poop)

Let’s build a Pouchdb app we aren’t ashamed of & backbone.js explained without words that start with C,M,or V

The previous post, while it showed the basics of pouch produced an app that is frankly not ready for prime time. So in this post we refactor the app and turn it into something less embarrassing. We are going to use the fantastic library backbone.js, written by the authors of a little language that turns into JavaScript, to tidy the app up, which will require an explanation of backbone. 

I’ve been aware of backbone for quite a while but my reaction every time someone tries to explain it has always been “yes but what does it do?” I blame this on the proliferation of libraries and frame works built around the letters C, M, and V which all use terms starting with those letters in similar but divergent ways, so for the rest of this post we will be using no words that start with those letters. If don’t care about backbone you can go right to the updated app or read about how I improved it.

The first basic part of backbone we will refer to as an “item template” though backbone uses a term that rhymes with “bottle.” You can think of this as representing a blueprint for a thing: in SQL it might be equivalent to a row, in PouchDB and MongoDB this might be a document, in Neo4J it might be an edge or node. When you add data you actually make a new instance of your item template (aka use the key word “new”).  Your item template can define tests that data needs to pass to be added and ways to serialized or transform your data for export, and can fire events when items are added, removed, updated etc.

The next portion we will refer to as a group of items, but backbone uses a term which rhymes with “selection.” A group is really what it is and in SQL it would be like a table and in PouchDB it would be like a DB.  You use a group to not only to attache some events, like watching for updates to a specific attribute across all items, but as an object you can use all of underscores functional tools on. You can run map,reduce,filter,pluck,each and more on your group to easily get just the data you care about.

The third and final of the essential parts of backbone is what I’m going to refer to as a supervisor but backbone refers to with a word that rhymes with “queue”. If this was that show about the extravagant houses of of famous artists then this would be the bedroom. I call it a supervisor because it supervises a specific element of the page and it is the only thing that touches its element. And how do you get the supervisor to know when to do something? By having it listen for updates to a group and events in the dom inside its element. This is the gist of backbone, its key usefulness is segregating the logic so that like is grouped with like. So if you want to get rid of a side bar all if the logic related to it is in one supervisor. Now enough introduction go look at the updating the pouchDB app with backbone.

Lets make an app: PouchDB

This is part 3 in a series, you can (and should) checkout part 1 and part two, we’ll wait. 

So we’re going to make an app, first remember that class we made last time, I had to make a few changes to get it to work, so overall it looks like this

Why you ask? A couple of indentations errors (that nobody caught) but mostly the stop and start methods because I noticed that your browser doesn’t always get the message whey your back on again after going offline, those methods allow you to stop the continuous replication and start it again to get things to sync.

We’re going to make an app that allows you to write notes that others can see and edit, and should work when you go on and off line, this may seem trivial, but checkout @axemclion’s conference app which does something very similar but in a useful way. 

So originally I was going to make this in backbone.js but instead I discovered /r/cats decided to do it slightly lover level to demonstrate the principles involved.  The structure of the app is going to be as follows;

1. We have an object called ‘obj’ which has all the documents we care about from the pouch, our on change function will update/add/delete the document from obj.
2. we have a function called ‘update’ which takes obj and transforms it into an object call ‘views’.
3. We have a set of mustache templates, we also have a div with id ‘body’.
4. views is given to the mustache template to generate html, div#body is emptied and then we append the mustache html (I originally was doing this to the body tag before I realized this caused issues with the script tags.)
5. Besides the contents of obj the only thing that effects the function update is what the current location.hash is.
6. The event structure is that when a document is changed, pouch updates obj and then calls update(), when a link is clicked, the handler preventsDefault(), changes the location.hash, calls update()

That’s pretty much it.  The first gotcha I noticed when doing this was that since I was replacing all the content rapidly:

I would not be shocked to learn that this caused performance issues with a lot of events like this added.  I’d also not be shocked to learn the opposite, so I’d probably test, and this was on hacker news this morning.

So onto the actual stuff, first we have the html, notice all it really does is load 1 css file and a bunch of scripts

next we have our mustache templates, notice the use of CoffeeScripts triple quoted strings, use of partials, and my disdain for tabs and readability when it’s not forced upon me by CoffeeScript

next we have a few misc function and the big update one, this does a couple that i’d consider Wrong and want to refactor out if this was for a real project, notable iterating through every object every time something changes, this is not going to scale to more docs very well. We also delete everything and rebuild the page from scratch every time anything changes if I was redoing this I’d break up the page into sections that could be re build individually. 

Lastly we have the jquery on load which mostly just sets up a bunch of events and the on database change which just makes sure it isn’t a design document and then replaces the document unless its been deleted in which case it’s deleted. 

Put that all together, and what do you get, THIS DEMO! feel free to leave a note, etc. 

There you have it, next time I’ll probably do something related to this new development and leaflet. Feel free to follow me on twitter.

And with that I leave you with this video (omg nsfw) which was going to be basis of an explanation of how binary trees work until I remembered @daleharvey was Scottish.

In which we explain how to use PouchDB

This is a follow up to my previous PouchDB post, so if you can’t read that you probably are going to want to. 

PouchDB is local storage primarily using IndexedDB, but with a WebSQL fallback, even if you arn’t doing crazy replications PouchDB may still be a easier to use because, do you understand IndexedDB? Because I don’t. Pouch on the other hand has an API based on CouchDB’s API which in turn is based on HTTP.  Your DB stores documents, which is just JSON but with two special fields (note the underscores):

• _id: this you can add your self, but if not it’s added for you, this is the documents key, if you want to find it you’ll need this.
• _rev: this is the documents revision, this is what couch has instead of transactions, if you want to modify a document or delete it, you need the _rev of the version you want to change, that way if someone else modifies it first it won’t match.
• There are actually 7 more, but we you can safely ignore them for now.

You update Couch via http and all the methods are based on HTTP verbs, Pouch’s methods are the same (though it should be pointed out that HTTP verbs are traditionally written in all caps but are not case sensitive, but JavaScript methods are case sensitive and lowercase):

• POST: create a new new document and assigns it a random _id, use this if you don’t care what the key is (i.e. your using it programmatically).
• PUT: Use this to either update a document, in which case you need to supply the _id and _rev, or to create a document with your chosen _id, in which case you don’t need _rev. 
• GET:  This is for getting a document, you need the _id obviously. 
• DELETE/REMOVE: This is one deviation between Couch and Pouch as “delete” is a reserved word in JavaScript, don’t forget you need the _rev.   This actually just adds _deleted:true which in Couch means that next time the database is compacted previous version can be removed, note you can’t remove it entirely because if you did you wouldn’t be able to tell other databases so when you replicated. 

Now onto Pouch specific things: the first thing to know is that like IndexedDB Pouch is asynchronous. , compare this which is synchronous JavaScript code

with this async code

The reason for this async set up is so as not to block the execution, unless you are using workers (which you likely aren’t, see my previous post) all of your script is in one thread, but the IndexedDB is in another thread and when you ask it to get you something instead of putting you on hold it just asks you for a number to call you when it’s found it. Pouch follows the Node.js convention of the first argument given to any callback will be an error argument which will be null/undefined/false if there is not an error meaning you can use if(!err) to check for an error and be the last argument you give to the function (the callback that is), hopefully this makes more sense:

Ok now for some basic Pouch usage which we will demonstrate by building a sample app …sort of, this post is really long so instead were going to create a class that will serve as the core of the app we will make next time, for serious.

Note unlike the previous code, this code is going to be in CoffeeScript, when I’m not writing directly for the browser this is what I tend to use for a variety of reasons which I go into excessive detail about at the end of this previous post.

So the first thing we’re going to need is a constructor function, it’s going to take a url as its first argument and implement the pattern from my previous post.The second argument its going to take is an on change function, this is the function that will be called every time a document is changed either remotely or locally.

We want to be able to

• Add a new document
• Get a document
• Delete one

Which put together looks like

And there you have a basic PouchDB class next time we’ll make a working app for real, just have to figure out what I was originally going to show something based on leaflet like my leaflet.pouch but that might require too much knowledge of how leaflet works, feel free to hit me up on twitter with suggestions.

PouchDB: an intro

So I’ve been hacking around with PouchDB a bit lately and I’m going to explain why I think it is so awesome.

Before we discus PouchDB we’re going to need to talk about CouchDB which Pouch is based on. CouchDB is an open source nosql document database written in Erlang that grew out of Lotus Notes of all things. I’m personally a fan of CouchDB but of it’s features the ones that matters here are, data is stored in JSON, you use JavaScript to query the data, you communicate with it entirely through a RESTful interface, and something called “Master-Master Replication,” now to put that in terms my cat would understand, this means that two databases can sync without one being master and the other slave but as equals, and this done in a way such that replication can be done in real time or sporadically. Think of email on your cellphone, you can have it so that it tries to sync every x minutes which just fails if it has no reception, or you can set it up so then when you have service you get push notification and when you loose reception you just get all of the emails when you get reception again. This is similar to how Couch can work, though it can be much more complex as well. If you want to know more about Couch then CouchDB: The Definitive Guide is a good place to start.

So one of the issues with CouchDB is that Erlang…well lets just say people have mixed feelings about it, which lead to pretty quickly, CouchDB compatible Databases, Big Couch from Cloudant which you can cluster, TouchDB is a version written in Objective-C targeting embedded apps, and then we have PouchDB.

PouchDB is written in JavaScript and targets the browser, but also works with Node.js. It has a number of different back ends that it can use so at the very basic you can run this (notice the Node.js style async callbacks, they always are function which take as a first argument an err object which is undefined unless there is an error):

Which lets you access and modify a CouchDB on the same origin like so.

The benefits of this setup is that it works on browsers as old as Internet Explorer 8, but on the other hand this only works with an internet connection.

We can also set up one locally like this:

This uses either IndexedDB or Web SQL depending on whats available and works on on all browsers except Internet Explorer versions besides 10. This is nice because we can create a large local storage with the same API whether on Chrome, Firefox, or mobile Safari, and the API is a whole lot easier then the IndexedDB API.

Now the neat thing we can do is combine the local and remote data sources with:

Which sets a pouch db up that syncs to a local database (either IndexedDB or web SQL) which syncs to a remote CouchDB, like so:

If the browser looses it’s internet connection it can then still work with the local database, and it then can sync when it gets internet access again. If the browser doesn’t support IndexedDB or webSQL it just creates a direct connection to the remote Couch.

You can checkout Pouch on:

• It’s website
• Github
• Twitter

Next time, example apps.