{"id":43,"date":"2013-12-23T23:10:45","date_gmt":"2013-12-23T22:10:45","guid":{"rendered":"http:\/\/spintag.de\/?p=43"},"modified":"2013-12-23T23:10:45","modified_gmt":"2013-12-23T22:10:45","slug":"d3-js-scene-controlled-by-remote-data-using-node-js","status":"publish","type":"post","link":"https:\/\/spintag.de\/?p=43","title":{"rendered":"d3.js scene controlled by remote data using node.js and socket.io"},"content":{"rendered":"<p>\t\t\t\t<a href=\"http:\/\/d3js.org\/\">d3.js<\/a> is cool. So is <a href=\"http:\/\/nodejs.org\/\">node.js<\/a>. Hence, doing something with both technologies is almost a logical consequence.<\/p>\n<p>Start the <a href=\"http:\/\/spintag.de:7777\/\">Demo<\/a>\u00a0<strong>at least in two browser instances\/devices<\/strong> (tested with Firefox, Chrome, IE9+, iPad, iPhone). Then click to create and to burst bubbles. Drag the bubbles around and try the action buttons to force larger changes to the d3.js scene.<\/p>\n<p><a href=\"http:\/\/spintag.de:7777\/\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-64\" src=\"http:\/\/spintag.de\/wp-content\/uploads\/2017\/09\/circles.png\" alt=\"\" width=\"977\" height=\"603\" \/><\/a><\/p>\n<p>This demo maintains a pool of \u201ccircles\u201d on the server and implements the distributed MVC pattern. The <em>Model<\/em> is kept on the server. Each instance of the browser keeps a copy of the <em>View<\/em> and the <em>Controller<\/em>. Furthermore, a lightweight version of the model is used for data presentation and is processed by the d3 engine. The <em>node.js<\/em> based server is responsible for synchronization of all models via push messages. All user interaction triggers a change on the server model first. This holds true even for the initiating client. An example scenario follows:<\/p>\n<ol>\n<li>user clicks on a bubble<\/li>\n<li>the event handler reads the &#8222;id&#8220; of the SVG-circle element and calls the server side function <em>remove(id)<\/em><\/li>\n<li>the server updates the model<\/li>\n<li>the server calls the<em> removeData(id)<\/em> function for each connected client (including the initiating one)<\/li>\n<li>the data structure in the client is updated (in this example the object with the specified id is removed)<\/li>\n<li>d3.js scene is updated<\/li>\n<\/ol>\n<p>The static files (index.html, CSS, templates and Javascript files) are served using <em>express.js<\/em>. The <a href=\"http:\/\/socket.io\/\">socket.io<\/a> library is used as a convenience layer on top of <em>websockets<\/em> and provides a reliable permanent server\/client connection.<\/p>\n<p>Remarks regarding performance issues:<\/p>\n<ul>\n<li>the good news: node.js and websockets are blazingly fast, playing with the demo for some minutes in three browser instances generates a CPU usage time for the node.js process of about 5 seconds, i.e. there is plenty of room for more serious stuff<\/li>\n<li>advice: use SVG opacity styles instead of HTML opacity styles whenever possible, this reduces the SVG render time and enables larger SVG scenes (10 fold improvement in Firefox)<\/li>\n<\/ul>\n<p>The source code for the server application is as follows:<\/p>\n<pre class=\"striped:false lang:js decode:true\">\/\/ utility library, see: http:\/\/underscorejs.org\/\nvar _ = require('underscore');\n\n\/\/ initialize web server framework\nvar express = require('express');\nvar app = express();\nvar http = require('http');\nhttp.globalAgent.maxSockets = 500;\nvar server = http.createServer(app);\n\nvar logFile = require('fs').createWriteStream('.\/circle.log', {flags: 'a'});\napp.use(express.logger({stream: logFile}));\napp.use(express.compress());\napp.use(express.static(__dirname + '\/public'));\n\n\/\/ websocket\nvar io = require('socket.io').listen(server, {log: false});\n\n\/\/the circle model\nvar model = require('.\/model').model;\nmodel.populate();\n\n\/\/ listen on connection event using namespace 'circle'\nio.of('\/circle').on('connection', function (socket) {\n    var address = socket.handshake.address;\n    logFile.write(\"New connection from \" + address.address + '\\n');\n    logFile.write('send circle data: ' + model.getSize() + ' entries' + '\\n');\n\n    \/\/ any connecting client gets the complete model data\n    socket.emit('update', {ts: Date.now(), circles: model.getData()});\n\n    var behaviour = {\n\t\t'add': add,\n\t\t'remove': remove,\n\t\t'resize': resize,\n\t\t'moveStart': moveStart,\n\t\t'move': move,\n\t\t'moveEnd': moveEnd\n\t};\n    \/\/ map events to functions\n    _.map(behaviour, function(func, eventName) {\n        socket.on(eventName, _.bind(func, socket));\n    });\n});\n\nfunction add(attr) {\n    logFile.write('add circle: x=' + attr.x + ', y=' + attr.y + '\\n');\n    var entry = model.add(attr);\n    io.of('\/circle').emit('add', {ts: Date.now(), entry: entry});\n}\n\nfunction remove(id) {\n    logFile.write('remove circle: id=' + id + '\\n');\n    id = model.remove(id);\n    if (id != -1)\n        io.of('\/circle').emit('remove', {ts: Date.now(), id: id});\n}\n\nfunction resize(info) {\n    var entry = model.resize(info.id, info.delta);\n    if (entry)\n    \tio.of('\/circle').emit('resize', {ts: Date.now(), entry: entry});\n}\n\nfunction moveStart(id) {\n    logFile.write('moveStart circle: id=' + id + '\\n');\n    \/\/socket.broadcast.emit('circles moveStart', {ts: Date.now(), id: id});\n    io.of('\/circle').emit('moveStart', {ts: Date.now(), id: id});\n}\n\nfunction moveEnd(entry) {\n    logFile.write('moveEnd circle: id=' + entry.id + ', x=' + entry.x + ', y=' + entry.y + '\\n');\n    model.move(entry);\n    io.of('\/circle').emit('moveEnd', {ts: Date.now(), entry: entry});\n}\n\nfunction move(entry) {\n    io.of('\/circle').emit('move', {ts: Date.now(), entry: entry});\n}\n\n\/\/the job controller\nvar jobs = require('.\/jobs').jobs;\n\/\/ prevent \"flooding\" the websocket connection with jobs update events\njobs.onChange = _.throttle(function () {\n    io.of('\/jobs').emit('update', jobs.getData());\n}, 333);\n\n\/\/listen on connection event using namespace 'jobs'\nio.of('\/jobs').on('connection', function (socket) {\n    logFile.write('send job list\\n');\n\n    \/\/ any connecting client gets the complete job data\n    socket.emit('update', jobs.getData());\n\n    var behaviour = {\n   \t\t'shuffle': shuffle,\n   \t\t'stream': stream,\n   \t\t'reset': reset\n    };\n    \/\/ map events to functions\n    _.map(behaviour, function(func, eventName) {\n        socket.on(eventName, _.bind(func, socket));\n    });\n});\n\nfunction shuffle() {\n    jobs.start('shuffle', function (job) {\n        model.shuffle();\n        io.of('\/circle').emit('update', {ts: Date.now(), circles: model.getData()});\n        setTimeout(function () {\n            job.complete();\n            logFile.write('shuffle completed' + '\\n');\n        }, 1000);\n    });\n    logFile.write('shuffle started' + '\\n');\n}\n\nfunction stream() {\n    var count = 100;\n    function streamEvent (job) {\n        io.of('\/circle').emit('add', {ts: Date.now(), entry: model.add(0, 0)});\n        io.of('\/circle').emit('remove', {ts: Date.now(), id: model.removeRandom()});\n        if (--count) {\n            setTimeout(function(){streamEvent(job);}, 150);\n            job.running(100 - count);\n        } else {\n            job.complete();\n            logFile.write('stream completed' + '\\n');\n        }\n    }\n    jobs.start('stream', streamEvent);\n    logFile.write('stream started' + '\\n');\n}\n\nfunction reset() {\n    jobs.start('reset', function (job) {\n        model.populate();\n        io.of('\/circle').emit('update', {ts: Date.now(), circles: model.getData()});\n        setTimeout(function () { job.complete(); }, 600);\n        logFile.write('reset' + '\\n');\n    });\n}\n\n\/\/ start the server\nserver.listen(7777, function () {\n    logFile.write('circle demo started on port ' + server.address().port + '\\n');\n});<\/pre>\n<p>The server model:<\/p>\n<pre class=\"brush:js \">var _ = require('underscore');\n\n\/\/ the circle model\nexports.model = {\n    INIT_CIRCLES: 20,\n    SIZE: 450,\n    COLORS: [\"red\",\"orange\",\"blue\",\"green\"],\n    nextId: 1,\n    data: [],\n    random: function(pool) {\n        if (_.isFinite(pool)) {\n            return (Math.random()*pool)|0;\n        }\n        if (_.isArray(pool)) {\n            return pool[Math.random()*pool.length|0];\n        }\n        return Math.random();\n    },\n    getData: function() {\n        return this.data;\n    },\n    getSize: function() {\n        return this.data.length;\n    },\n    getColorStatistics: function() {\n        var colors = _.object(this.COLORS, [0, 0, 0, 0]);\n\n        for (var i=0; i&lt; this.data.length; i++) {\n            colors[this.data[i].color] += 1;\n        }\n\n        return colors;\n    },\n    add: function(attr) {\n        var entry = {\n            id: this.nextId++,\n            x: attr.x || this.SIZE - 30 - this.random(this.SIZE - 60),\n            y: attr.y || this.SIZE - 30 - this.random(this.SIZE - 60),\n            r: 30 + this.random(35),\n            color: this.random(this.COLORS),\n            opacity: (30 + this.random(50))\/100\n        };\n        this.data.push(entry);\n         return entry;\n    },\n    resize: function(id, delta) {\n    \tvar factor = (delta &gt; 0) ? 1.03 : 1\/1.03;\n        for (var i=0; i&lt; this.data.length; i++) {\n        \tvar d = this.data[i];\n            if (d.id == id) {\n            \tvar old = d.r;\n                d.r *= factor;\n                d.r = Math.min(150, Math.max(20, d.r));\n                if (d.r == old)\n                \treturn null;\n                else\n                \treturn d;\n            }\n        }\n    },\n    move: function(entry) {\n        for (var i=0; i&lt; this.data.length; i++) {\n        \tvar d = this.data[i];\n            if (d.id == entry.id) {\n                d.x = entry.x;\n                d.y = entry.y;\n                \/\/ move to end, emulates highest z-index\n                this.data.splice(i, 1);\n                this.data.push(d);\n                return;\n            }\n        }\n    },\n    remove: function(id) {\n        for (var i=0; i&lt; this.data.length; i++) {\n            if (this.data[i].id == id) {\n                this.data.splice(i, 1);\n                return id;\n            }\n        }\n        return -1;\n    },\n    removeRandom: function() {\n        if (this.data.length) {\n            var i = this.random(this.data.length);\n            var id = this.data[i].id;\n            this.data.splice(i, 1);\n            return id;\n        }\n        return -1;\n    },\n    populate: function() {\n        this.data = [];\n        for (var i=0; i&lt; this.INIT_CIRCLES; i++) {\n            this.add({x: 0, y: 0});\n        }\n    },\n    shuffle: function() {\n        for (var i=0; i&lt; this.data.length; i++) {\n            this.data[i].x = this.SIZE - 30 - this.random(this.SIZE - 60);\n            this.data[i].y = this.SIZE - 30 - this.random(this.SIZE - 60);\n        }\n    }\n};\n\n\/\/TODO: normalize size and position data, get rid of pixel related information<\/pre>\n<p>The client code:<\/p>\n<pre class=\"brush:js\">var CircleController = can.Control({\n\n\tisMobile: navigator.userAgent.match(\/Mobile\/i),\n\n\tinit: function () {\n\t\tvar me = this;\n\t\tthis.eventCount = 0;\n\n\t\t\/\/ an array of circle objects\n\t\tthis.data = [];\n\n\t\tvar fragment = can.view('\/templates\/circle.ejs');\n\t\tthis.element.html(fragment);\t\t\t\n\n\t\tthis.setupDrawingArea();\n\t\tthis.setupWebsocket();\n\n\t\tvar throttleResize = _.throttle(function (info) { me.socket.emit('resize', info); }, 25);\n\n\t\tthis.element.bind('mousewheel', function(event, delta, deltaX, deltaY) {\n\t\t    if (event.target.nodeName == 'circle')\n\t\t    \tthrottleResize({id: event.target.getAttribute('id'), delta: delta});\n\t\t    event.preventDefault();\n\t\t});\n\t},\n\n\tsetupDrawingArea: function () {\n\t\tvar me = this;\n\t    \/\/ setup drawing area\n\t    this.element.css(\"width\", \"450px\").css(\"height\", \"450px\");\n\t    this.scene = d3.select(this.element.find(\"svg\")[0])\n\t        .attr(\"width\", 500).attr(\"height\", 500)\n\t        .on(\"click\", function() {\n\t            var target = d3.event.target;\n\t            if (target.nodeName == 'circle') {\n\t                var id = d3.select(target).attr(\"id\");\n\t                me.socket.emit('remove', id);\n\t            } else {\n\t                var pos = d3.mouse(this);\n\t                me.socket.emit('add', {x: pos[0]|0, y: pos[1]|0});\n\t            }\n\t        });\n\t},\n\n\tenter: function () {\n\t\tthis.dragging = false;\n\t\tvar me = this;\n\n\t\tfunction dragStart() {\n\t\t\tme.dragging = true;\n\t\t\tvar circle = d3.select(this);\n\t\t\tvar id = circle.attr(\"id\");\n\t\t\tme.socket.emit('moveStart', id);\n\t\t};\n\n\t\tfunction dragMove() {\n\t\t\tvar circle = d3.select(this);\n\t\t\tvar ev = d3.event;\n\t\t\tcircle.attr(\"cx\", ev.dx + parseInt(circle.attr(\"cx\")));\n\t\t\tcircle.attr(\"cy\", ev.dy + parseInt(circle.attr(\"cy\")));\n\t\t\tvar id = circle.attr(\"id\");\n\t\t\tvar x = circle.attr('cx');\n\t\t\tvar y = circle.attr('cy');\n\t\t\tme.socket.emit('move', {id: id, x: x, y: y});\n\t\t};\n\n\t\tfunction dragEnd() {\n\t\t\tme.dragging = false;\n\t\t\tvar circle = d3.select(this);\n\t\t\tvar id = circle.attr(\"id\");\n\t\t\tvar x = Math.min(450, Math.max(0, circle.attr('cx')));\n\t\t\tvar y = Math.min(450, Math.max(0, circle.attr('cy')));\n\t\t\tme.socket.emit('moveEnd', {id: id, x: x, y: y});\n\t\t};\n\n\t\t\/\/ macro function\n\t\tfunction P(propName) { return function(d) { return d[propName]; }; }\n\n\t\t\/\/ the key function binds the data entries to the DOM elements using the id\n\t\tvar circles = this.scene.selectAll(\"circle.default\").data(this.data, P(\"id\"));\n\n\t\tcircles.enter() \/\/ the enter set, a set of new data entries without DOM equivalent\n\t\t.append(\"circle\")\n\t\t.call(d3.behavior.drag()\n\t\t\t\t.on('dragstart', dragStart)\n\t\t\t\t.on('drag', dragMove)\n\t\t\t\t.on('dragend', dragEnd))\n\t\t\t\t.attr(\"id\", P(\"id\"))\n\t\t\t\t.attr(\"class\", \"default\")\n\t\t\t\t.attr(\"r\", 1)\n\t\t\t\t.attr(\"cx\", P(\"x\"))\n\t\t\t\t.attr(\"cy\", P(\"y\"))\n\t\t\t\t.style(\"fill\", function(d) { return me.isMobile ? d.color : \"url(#gradient_\" + d.color + \")\"; })\n\t\t\t\t.style(\"fill-opacity\", P(\"opacity\"))\n\t\t\t\t.transition().duration(500)\n\t\t\t\t.attr(\"r\", P(\"r\"));\n\t},\n\n\texit: function () {\n\t\t\/\/ macro function\n\t\tfunction P(propName) { return function(d) { return d[propName]; }; }\n\n\t\tvar circles = this.scene.selectAll(\"circle.default\").data(this.data, P(\"id\"));\n\n\t\tcircles.exit() \/\/ the exit set, a set of superfluous DOM nodes without data equivalent\n\t\t.attr(\"class\", null)\n\t\t.style(\"stroke-opacity\", 0)\n\t\t.transition()\n\t\t.attr(\"r\", function(d) { return 1.8*d.r; })\n\t\t.style(\"fill-opacity\", 0.05)\n\t\t.each(\"end\", function () { d3.select(this).remove(); });\n\t},\n\n\tupdate: function (dur) {\n\t\t\/\/ macro function\n\t\tfunction P(propName) { return function(d) { return d[propName]; }; }\n\n\t\tvar circles = this.scene.selectAll(\"circle.default\").data(this.data, P(\"id\"));\n\n\t\tcircles \/\/ the update set, all DOM nodes with a data binding\n\t\t.transition().duration(dur)\n\t\t.attr(\"r\", P(\"r\"))\n\t\t.attr(\"cx\", P(\"x\"))\n\t\t.attr(\"cy\", P(\"y\"));\n\t},\n\n\t\/\/\n\t\/\/ the payload called as event from server\n\t\/\/\n\tsetData: function (ts, circles) {\n\t\tthis.data = JSON.parse(JSON.stringify(circles));\n\t\tthis.exit();\n\t\tthis.enter();\n\t\tthis.update(800);\n\t\tthis.log(ts, this.data.length + \" circles updated\");\n\t},\n\n\taddData: function (ts, entry) {\n\t\tthis.data.push(JSON.parse(JSON.stringify(entry)));\n\t\tthis.enter();\n\t\tthis.log(ts, \"circle added, id=\" + entry.id);\n\t},\n\n\ttoForeground: function (d3Node) {\n\t\tvar node = d3Node[0][0];\n\t\tif (!node)\n\t\t\treturn;\n\t\tnode.parentNode.appendChild(node);\n\t},\n\n\tmoveStart: function (ts, id) {\n\t\tif (this.dragging)\n\t\t\treturn;\n\t\tvar circle = this.scene.select('circle[id=\"' + id + '\"]');\n\t\tcircle.style(\"stroke\", \"white\");\n\t\tthis.toForeground(circle);\n\t},\n\n\tmoveEnd: function (ts, entry) {\n\t\tthis.log(ts, \"circle moved, id=\" + entry.id);\n\t\tfor (var i=0; i &lt; this.data.length; i++) {\n\t\t\tif (this.data[i].id == entry.id) {\n\t\t\t\tthis.data[i].x = entry.x;\n\t\t\t\tthis.data[i].y = entry.y;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tvar circle = this.scene.select('circle[id=\"' + entry.id + '\"]');\n\t\tcircle.style(\"stroke\", null);\n\t\tthis.toForeground(circle);\n\t\tthis.update(300);\n\t},\n\n\tmove: function (ts, entry) {\n\t\tif (this.dragging)\n\t\t\treturn;\n\t\tthis.log(ts, \"circle moving, id=\" + entry.id);\n\t\tvar circle = this.scene.select('circle[id=\"' + entry.id+ '\"]');\n\t\tcircle.attr(\"cx\", entry.x).attr(\"cy\", entry.y);\n\t},\n\n\tresize: function (ts, entry) {\n\t\tthis.log(ts, \"circle resize, id=\" + entry.id);\n\t\tfor (var i=0; i &lt; this.data.length; i++) {\n\t\t\tif (this.data[i].id == entry.id) {\n\t\t\t\tthis.data[i].r = entry.r;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tvar circle = this.scene.select('circle[id=\"' + entry.id+ '\"]');\n\t\tcircle.attr(\"r\", entry.r);\n\t},\n\n\tremoveData: function (ts, id) {\n\t\tfor (var i=0; i &lt; this.data.length; i++) {\n\t\t\tif (this.data[i].id == id) {\n\t\t\t\tthis.data.splice(i, 1);\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tthis.exit();\n\t\tthis.log(ts, \"circle removed, id=\" + id);\n\t},\n\n\tformat: d3.time.format(\"%X\"),\n\n\tlog: function (ts, msg) {\n        var fragment = this.eventCount + \" circle events, \" + msg + ' @' + this.format(new Date(ts)) + ' +' + ts%1000 + 'ms';\n        $('#log').html(fragment);\n\t},\n\n\tsetupWebsocket: function () {\n\t\tvar me = this;\n\n\t    \/\/ map events to functions\n\t    var behaviour = {\n\t        'update': function (data) {\n\t            me.eventCount++;\n\t            me.setData(data.ts, data.circles);\n\t        },\n\t        'add': function (data) {\n\t            me.eventCount++;\n\t            me.addData(data.ts, data.entry);\n\t        },\n\t        'resize': function (data) {\n\t            me.eventCount++;\n\t            me.resize(data.ts, data.entry);\n\t        },\n\t        'remove': function (data) {\n\t            me.eventCount++;\n\t            me.removeData(data.ts, data.id);\n\t        },\n\t        'moveStart': function (data) {\n\t        \tme.eventCount++;\n\t        \tme.moveStart(data.ts, data.id);\n\t        },\n\t        'move': function (data) {\n\t        \tme.eventCount++;\n\t        \tme.move(data.ts, data.entry);\n\t        },\n\t        'moveEnd': function (data) {\n\t            me.eventCount++;\n\t            me.moveEnd(data.ts, data.entry);\n\t        }\n\t    };\n\n\t    \/\/ connect with websocket server using namespace: '\/circle'\n\t    this.socket = io.connect('\/circle'); \n\n    \t_.map(behaviour, function(func, eventName) {\n    \t\tme.socket.on(eventName, func);\n    \t});\n\t }\n});<\/pre>\n<p><a href=\"http:\/\/spintag.de\/wp-content\/uploads\/2017\/03\/circle.tar\">complete source code as tar file<\/a>\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"<p>d3.js is cool. So is node.js. Hence, doing something with both technologies is almost a logical consequence. Start the Demo\u00a0at least in two browser instances\/devices (tested with Firefox, Chrome, IE9+, iPad, iPhone). Then click to create and to burst bubbles. Drag the bubbles around and try the action buttons to force larger changes to the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,5,6],"tags":[9,11],"class_list":["post-43","post","type-post","status-publish","format-standard","hentry","category-d3-js","category-node-js","category-socket-io","tag-javascript","tag-websockets","entry"],"_links":{"self":[{"href":"https:\/\/spintag.de\/index.php?rest_route=\/wp\/v2\/posts\/43","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/spintag.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/spintag.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/spintag.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/spintag.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=43"}],"version-history":[{"count":0,"href":"https:\/\/spintag.de\/index.php?rest_route=\/wp\/v2\/posts\/43\/revisions"}],"wp:attachment":[{"href":"https:\/\/spintag.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=43"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/spintag.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=43"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/spintag.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=43"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}