Do some good with what you learn and what you know they said. So I am trying to do just that. When the latest Panama Papers got released I was in the midst of learning about the vast field of data science. Then a thought crossed my mind "Why not do something with the data?'. And so began this small project. It seemed daunting task for my noob mind but then again whats the point of learning if you cannot experiment and go out of your comfort zone. So I set out to do 'something' with the data set at hand. First look at the data set and I was like, 'This is an experiment which is way out of my league'. But then again everything is to someone who is just starting out, so I said 'Onwards and to god knows where'. 
So what did I do? 
Sunday morning breakfast ->Download data set -> Load it up in R -> Check all the data and find relevant stuff -> Make a copy of the relevant data and ditch the rest -> Google about Geocoding -> R-Blogger to the rescue -> Trying to fit the code out of the box -> Of course it did not work -> Lot of head bashing for hours -> Lunch break -> Read through the code -> More head bashing -> Google for more geocoding stuff to find alternatives -> Intuition says R-Blogger code will work -> Back at the code again -> Hair pulling -> Cursing -> Voila ! Something worked -> 10% there -> Turns out there were no column headers in the intermediary file being used in the code -> Duh! Fix it! Re-wire the code and put in some customization which fit fits the data -> RUN ! -> Runs perfectly till 708th row and then BOOM! Error !! -> Look at the data and scratch head -> Do some mumbo jumbo (not exactly but some much needed cleaning of the data) -> Lot of iterations to get the needed output and fixing the errors or customizing the code -> There was some green tea time in between, forgot where -> After a lot of wishing , willing and cursing I make a markdown to knit and html -> KNIT ! -> 'Over the querying limit' -> Need I say more ? 
Next day it worked and here we are. So this some summary of all the Indian addresses present in the latest release of Panama Papers.
The source : Panama Papers
Total addresses : 151054
Redacted: 15
Missing : 4
Indian Addresses : 828
Correctly geocoded : 627
Wrongly geocoded : 38 + 163 unsearchable, due to wrong spellings or incorrect addresses.
The Visualization is here. Along with some flow of how the code ended up there. The page contains two maps. 
The first one is a heatmap which clearly shows the concentration of the addresses spread across the country. And the results are as expected, the majority being centered around the metros.
The second one provides a more in depth view of the addresses as it can be used to see particular areas of interests with their addresses mentioned along with it. The second map shows 657 Indian addresses which is more than the number shown by R. This is due to the fact that I used batchgeo.com to process the addresses file externally to get a different visualization. Now what has happened is the site has taken the nearest or the city point for some addresses other than the 627 which were correctly coded. There are still 14 addresses which are pointed out on the map and are in the wrong area. You can see the addresses by clicking on them and observe, which will give you a picture as to why that happened.
The final geocoded file is also included in the project folder at Github
Thanks to r-blogger for the geocoding script. 



View Indian Addresses in a full screen map
To do :
Fix the missing 163 values
Correct the 38 wrongly geocoded addresses.
Create a better connected visualization which shows which address was connected to which bank and where.

Comments

  1. vinayakMay 17, 2016 at 6:01 PM

    This is great! Love what you did with what you learnt.

    ReplyDelete

Post a Comment

Popular posts from this blog

Penning down this piece just to blurt out some things that are plaguing my mind since long. It’s a mixture of a lot of things, basis of which is related to our LFC and a subject close to every supporter’s heart (supposing everyone likes to learn about our vast history) and the other core is formed by what’s going on in the world since few years now. Yes you must’ve guessed about it, I’m talking of a day etched deep within the memories of every Merseysider. Some who were affected directly are still feeling the pain of an irrecoverable loss. And then there are others who can feel their agony and anguish at what happened on that wretched day of the 4 th month of 1989. A day which should be remembered by everyone alike, for every fan or supporter is family to us in the Red. I was born 3 months after the incident but I have tears whenever I read, hear or see videos about the day. Many say that people outside Merseyside cannot feel the pain, I agree with you but only partly. I say partl...
Read more
Image
Life can be hard and uncompromising or even downright painful. But sometimes it gives you happiness mixed with anxiety just like eating a chocolate with a sour coating, it tastes sour at the onset all the while making you want to spit it out but you hang on because you know the sweet part is coming.  And when it does, nothing can be as tastier. Thinking of chocolates reminds me of why I am writing this, our beloved Liverpool FC. These last few years (especially under the one whose face  resembles an owl, isn't it?) have been downright sour but we knew the sweet part would come. All these years we had that feeling of disbelief and even disgust at times. Nobody knew when this would go away but we all knew it was somewhere there lurking behind the dark clouds. Now the days of owls and whiskey are over, the days of Chihuahuas and laughter are here. The sweet in the chocolate has finally broken past the layers and layers of sourness; we have finally managed to wade...
Read more