Let me start this blog with a little example. Assume, Sachin have a leak in a water pipe in his garden. He takes a bucket and a some sealing material to fix the problem. After a while, he see s that the leak is much bigger that and he needs a specialist  to bring bigger tools. Meanwhile, he still uses the bucket to drain the water. After a while, he notices that a massive underground stream has opened and he needs to handle millions of liters of water every second. 

He does n't just need new buckets, but a completely new approach to looking at the problem just because the volume and velocity of water has grown. To prevent the town from flooding, maybe he needs his government to build a massive dam that requires an enormous civil engineering expertise and an elaborate control system. To make things worse, everywhere water is gushing out from nowhere and everyone is scared with the variety.

Welcome to Big Data.

Key elements of Big Data:

1. There are over 600 million tweets every day that is flowing every second which tells about the High Volume & Velocity
2. Next  need to understand what each tweet means - where is it from, what kind of a person is tweeting, is it trustworthy or not which tells about the High Variety
3. Identify the sentiment - is this person talking negative about iPhone or positive? which describes about the High Complexity
4. And finally need to have a way to quantify the sentiment and track it in real time which tells about High Variability

Traditional architecture of any Big Data solution would look something like below,

It has been 6 years since i entered into programming field, and 18 years since i started using a computer, Everyone things i am a computer geek. Some times in my mind sounds come that Is that I am really a developer or just a good nethead?.

It's because there has not been a single day i coded without using Google search and Stack Overflow.

My Experience

When I was 15, I wrote my first program.  That was a long time ago now.  And it was in pascal language. During my university days i had more interest on gaming and animation rather than programming. When i got my first job, i struggled to code in c# with the dot net platform  during the initial days. With the help of google and stack overflow, now i would rate myself 8 on c# and i have experience of various open source technologies. But still i felt i was a better googler , not a good programmer. 

What made me to think I am a really bad programmer?

(i) Choosing workarounds over doing the right thing.

(ii)Used Ctr+c and Ctr+v more than normal keys

(iii)When things went wrong, i asked who is at fault rather than what the problem was. **

Mistakes to avoid to become a better programmer

**

In the year 2016, i started to avoid above practices and i would say programming is the first step to solving problems using technology. My tips that i followed during the year to be a better programmer as follows,

• Every day find a small challenge that can  be done in an hour.

• Read code. There is a plethora of freely available code for applications. There are tons of free projects by others on github.

• Make small projects to build experience.  Make it an open source project and if you can encourage collaboration if your project is compelling enough.

• Try programming for a day without googling. continue it for two days, maybe a week. See how it feels.

• Go to Meetups, Workshops, meet with others who feel the same way you do about technology.

anyone can become a good developer if he/she is passioned about it and practice a lot, preferably daily."In order to remain at the same level you have to spend at least two hours daily programming.". There will be many programmers out there who would think the same! What do you think?

There has been plenty of tutorials and blogs on how to configure webpy application with apache and mod_wsgi, but none of them turned into successful one. After 2 days of research i have found the solution and decided to write a blog on the same. Hope it will be useful for others.

In the future, I hope to update this post to also include a complete list of steps for getting setup with python’s webpy over lighttpd.

1. Install web.py

1.1. Install webpy with apt-get

sudo apt-get install python-webpy

1.2. Install webpy using easy_install using python setuptools 

1.2.1. Install python setuptools (easy_install)

# 1.2.1.1. Using apt-get:

sudo apt-get install python-setuptools
# 1.2.1.2. Manually retrieving easy_install from the web using wget

wget http://peak.telecommunity.com/dist/ez\_setup.py
sudo python ez_setup.py

# 1.2.2. Now get the web.py egg using python’s easy_install
# This will put the python ‘web’ module in your python environment path

sudo easy_install web.py

1.3. Install webpy straight from git

# Or, get webpy straight from git

git clone git://github.com/webpy/webpy.git ln -s `pwd`/webpy/web .

2. Write Your Web.py App
Choose a directory where you would like your web.py python application to live. If my username is ‘mek’ and I want to name my project ‘project’, I might make a directory /home/sajee/project.

2.1. Make a directory for your web.py app to live
# Replace the word project in the path below with your desired project name

mkdir ~/project
cd ~/project # move into the project directory you have created

2.2. Create your application file using web.py
# this will create our application file ~/project/main.py

touch main.py
2.3. Open your application with your favourite editor

# Substitute “emacs -nw” with an editor of your choice: vim, nano, etc

emacs -nw main.py

2.4. Paste the following in your app file and save

import os
import sys
import web

app_path = os.path.dirname(file)
sys.path.append(app_path)

if app_path: # Apache
os.chdir(app_path)
else: # CherryPy
app_path = os.getcwd()

urls = (
'/(.*)', 'hello'
)
  
**3\. Install Apache2**  
  
3.1. Install apache and wsgi dependencies  
  
sudo aptitude install apache2 apache2.2-common apache2-mpm-prefork apache2-utils libexpat1 ssl-cer  
\# I like to also install python-dev (optional) to make sure I have  
\# python’s latest support files  
  
sudo apt-get install python-dev  
3.2. Install apache mod\_wsgi and enable mod\_wsgi + mod\_rewrite  
  
sudo aptitude install libapache2-mod-wsgi  
sudo a2enmod mod-wsgi;sudo a2enmod rewrite  
Need help troubleshooting your apache/mod\_wsgi installation?  
  
**4\. Configure Apache2 With Your App**  
  
In the following steps, replace ‘project’ with the name of your project  
  
4.1. Make Apache Directories for your project  
  
sudo mkdir /var/www/project  
sudo mkdir /var/www/project/production  
sudo mkdir /var/www/project/logs  
sudo mkdir /var/www/project/public\_html  
4.2. Create Symlinks  
Creating symlinks to your project files is an important covention as, if there is a problem with one of your code bases, you can simply change your symlink to a stable codebase without having to modify your apache configuration.  
  
ln -s ~/project/ production  
ln -s ~/project/static public\_html # If you created the static directory in step 2.4. 
4.3. Replace you /etc/apache2/sites-available/default with:  
  
  

ServerAdmin [email protected] /var/www/project.com/public_html/ErrorLog /var/www/project.com/logs/error.logCustomLog /var/www/project.com/logs/access.log combinedWSGIScriptAlias / /var/www/project.com/production/main.pyAlias /static /var/www/project.com/public_htmlAddType text/html .pyWSGIDaemonProcess www-data threads=15WSGIProcessGroup www-dataOrder deny,allowAllow from allOptions +FollowSymLinksOptions -Indexes

  
  
4.4. Change the group and owner of files requiring write access to apache’s www-data  
Careful in this step to only change the group and owner of directories or files that will require write access.  
  
sudo chgrp -R www-data  
sudo chown -R www-data  
  
**5.Try to run!**  
  
sudo /etc/init.d/apache2 restart # Open your browser and visit the url: http://localhost or 127.0.01  
  
You will see Hello World on the browser.

I've been a .Net developer since the beta days of .Net 3.0, Now i find myself doing less and less coding related with .Net related stuffs. However, the new strategy from Microsoft  encouraged all the developers including me to once again start doing some .Net work from time to time. One of the highlighting tool among them was the Visual Studio Code.

Sublime text has been my favorite text editor all these time.When i downloaded it and looked at the first time my impression was nothing more than a plain editor with very little added value. Before VS code I have tried all popular editors - Sublime, Atom, Brackets etc. After trying it for few weeks now I feel developers  have everything they need .

Some of the highlights of VS Code are as follows,

• Default integrated git system is really awesome.
• Powerful debugging option.
• Very smart code completion.
• Huge list of Languages Support
• Multi panel for side by side editing
• Always-On IntelliSense
• Debugging support
• Peek Information like grammer correction
• Command Palette

Choice of editor is a personal preference. If you like an lightweight IDE environment that's cross platform, you might enjoy VS Code. Give it a try, you will definitely love it.

It's been exactly 2 years since i started to learn Angular and it's sad that i dint write even a single blog on the same. Finally decided to start a series on the same topic. AngularJS is a JavaScript MVC Framework that integrates two-way data binding, web services, and build web components. There are enough number of blogs and tutorials to follow on the same.

The current product which i am working is a data visualization tool which is built on AngularJS  and has many visualization  been integrated with D3.js.

In this blog, will be describing how to build a directive using d3.js and angular.

Directive is very powerful feature of AngularJS. It easily wired up with controller, html and do the DOM manipulations.

Building a Decomposition Force directed d3 directive:

 App.directive('forceGraph', function() {  
   return {  
     restrict: 'EA',  
     transclude: true,  
     scope: {  
       chartData: '='  
     },  
     controller: 'hierarchySummaryCtrl',  
     link: function(scope, elem, attrs) {  
       var svg;  
       elem.bind("onmouseover", function(event) {  
         scope.svg = svg;  
         console.log("hierarchy svg", scope.svg);  
         scope.$apply();  
       });  
       scope.$watch('chartData', function(newValue, oldValue) {  
         if (newValue) {  
           scope.draw(newValue.data,newValue.id);  
         }  
       });  
       scope.draw = function(rootData,divID) {  
         var width = 400,  
           height = 320,  
           root;  
         var force = d3.layout.force()  
           .linkDistance(80)  
           .charge(-120)  
           .gravity(.05)  
           .size([width, height])  
           .on("tick", tick);  
         var divid = "#" + divID;  
         d3.select(divid).selectAll("*").remove();  
         svg = d3.select(divid)  
           .append("svg").attr("viewBox", "0 0 400 400")  
           .attr("width", '100%')  
           .attr("height", '100%');  
         var link = svg.selectAll(".link"),  
           node = svg.selectAll(".node");  
         root = rootData;  
         update();  
         console.log(svg);  
         scope.setSvg(svg[0][0].innerHTML);        
         function update() {  
           console.log(nodes)  
           var nodes = flatten(root),           
           links = d3.layout.tree().links(nodes);          
           var nodes = flatten(rootData),           
           links = d3.layout.tree().links(nodes);             
           force.nodes(nodes)  
             .links(links)  
             .start();  
           // Update links.  
           link = link.data(links, function(d) {  
             return d.target.id;  
           });  
           link.exit().remove();  
           link.enter().insert("line", ".node")  
             .attr("class", "link");  
           // Update nodes.  
           node = node.data(nodes, function(d) {  
             return d.id;  
           });  
           node.exit().remove();  
           var nodeEnter = node.enter().append("g")  
             .attr("class", "node")  
             .on("click", click)  
             .call(force.drag);  
           nodeEnter.append("circle")  
             .attr("r", function(d) {  
               return Math.sqrt(d.size) / 5 || 4.5;  
             });  
           nodeEnter.append("text")  
             .attr("dy", ".25em")  
             .text(function(d) {  
               return d.name + ", Count: " + d.size;  
             });  
           node.select("circle")  
             .style("fill", color);  
         }  
         function tick() {  
           link.attr("x1", function(d) {  
               return d.source.x;  
             })  
             .attr("y1", function(d) {  
               return d.source.y;  
             })  
             .attr("x2", function(d) {  
               return d.target.x;  
             })  
             .attr("y2", function(d) {  
               return d.target.y;  
             });  
           node.attr("transform", function(d) {  
             return "translate(" + d.x + "," + d.y + ")";  
           });  
         }  
         function color(d) {  
           return d._children ? "#FFEB3B" // collapsed package  
             :  
             d.children ? "#F44336" // expanded package  
             :  
             "#D32F2F"; // leaf node  
         }  
         // Toggle children on click.  
         function click(d) {  
           if (d3.event.defaultPrevented) return; // ignore drag  
           if (d.children) {  
             d._children = d.children;  
             d.children = null;  
           } else {  
             d.children = d._children;  
             d._children = null;  
           }  
           update();  
         }  
         // Returns a list of all nodes under the root.  
         function flatten(root) {  
           var nodes = [],  
             i = 0;  
           function recurse(node) {  
             if (node.children) node.children.forEach(recurse);  
             if (!node.id) node.id = ++i;  
             nodes.push(node);  
           }  
           recurse(root);  
           return nodes;  
         }  
       };  
     }  
   };  
 });  

My Repository With the sample