Pithy NotesJekyll2022-11-29T16:26:46+00:00https://www.diyd2.in/Sudhanhttps://www.diyd2.in/sudhan@diyd2.inhttps://www.diyd2.in/blog/usb-isp-avr-firmware2022-11-06T00:00:00+00:002022-11-06T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<h1 id="introduction---getting-usb-isp-programmer-clones-to-work-on-macos">Introduction - Getting USB ISP Programmer (clones) to work on MacOS</h1>
<p>What started as an attempt to do a bit TinyML led to a day long exercise in getting a USB-ISP V2.0 programmer to work. These are cheap programmer boards used to flash programs to ATmega, ATtiny microcontrollers from Atmel. These micro-controllers usually very easy to use since most folks use them via Boards like Arduino UNO, Nano, Pro Micro and the like that have USB-to-Serial built-in.</p>
<p>If you need to program the chips and use the bare minimum components, you will have to move away from the Arduino boards to the bare-essentials.</p>
<p>BUT, to do that, you will need to somehow flash firmware to the chips. Enter: USB-ISP Programmers.</p>
<p>I bought a very cheap one for around 4 USD only to find out that it does not work (on Mac Monterey atleast).
Rather than return the device, I tried to figure it out and benefited from the lessons from many folks who had already made the journey.</p>
<p>To save time for others:</p>
<ul>
<li>Summarized the symptoms</li>
<li>Modified and compiled the source code</li>
<li>Written a small Readme section for folks to follow</li>
</ul>
<p>You can check out the repository at https://github.com/p2c2e/usbasp.2011-05-28-zhifengsoft-USB-ISP-Clone.git</p>
<p>Device that I bought: https://robocraze.com/products/usb-isp-programmer-version-2-0
Image: <img src="https://cdn.shopify.com/s/files/1/0262/6564/9240/products/HatchnHack_Makerspace_HNH_Cart_Components-46_72fc1b49-2b22-429e-bd43-a8e6d2adb7b3_800x.jpg?v=1648799456" alt="How the device looks like" /></p>
<p><a href="https://www.diyd2.in/blog/usb-isp-avr-firmware/">Rabbithole from TinyML to Arduino to USB ISP Firmware flashing</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on November 06, 2022.</p>
https://www.diyd2.in/articles/truenas-mirror2022-02-12T00:00:00+00:002022-02-12T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<h2 id="the-context">The Context</h2>
<p>Recently I decided to replace my old custom-build desktop with a newer setup. You may wonder, who uses a desktop anymore? I have been using mine to install licensed software that would be used by everyone in the house (like Adobe, Ableton Live etc.) as well as a local backup of important files. Usually I keep a desktop ‘alive’ by upgrading components over-time and on-average they are with me for 5-10 years each.</p>
<p>Old-setup - Intel Core i3, Gigabyte MB, SSD boot + 2x 4TB Seagate HDDs, 8GB RAM</p>
<p>New-setup - AMD Ryzen 7 5800x, MSI MB with 6x SATA, 16G NVMe, SSD Boot, Old HDDs, 16GB RAM</p>
<p>First hurdle to cross : Was the migration of the Windows 10 Pro license from the old SSD to the newer/larger SSD. Thankfully, this was done with no issues. I ensured that the older system license was Digitally activated AND linked to my MS account. Then I created a Restore USB for safety. I had an older copy of Seagate Disktools and I used it to clone the SSD to the newer SSD.
On using the cloned Disk to boot, I was impressed how resilient Windows 10 was. With a different MB, CPU and monitor, it managed to boot and subsequently download all the newer drivers etc. As expected, Windows 10 reported that it wasn’t able to activate windows on the new machine. I used “Change Product Key” and re-entered my key from the earlier machine - and voila - it worked!</p>
<h2 id="the-core-of-this-post---what-do-i-do-with-the-older-machine">The core of this post - What do I do with the older machine?</h2>
<p>I decided to try my hand at setting up a local NAS system. Any 4 drive NAS enclosure (without disk) was costing 50,000 INR or more. Hence this was worth a shot.
After some research, I decided on a few things:</p>
<ol>
<li>Software : There was FreeNAS, TrueNAS and Unraid. Found out that FreeNAS is now TrueNAS Core - and still Free. Unraid is very easy to setup but limited in config options. Settled on <a href="https://www.truenas.com/download-truenas-core/">TrueNAS Core</a></li>
<li>Hardware : I wanted to understand how to handle upgrade/degrades of disks. So, started by installing TrueNAS on the older SSD + ONE HDD of smaller size (2TB)</li>
</ol>
<p>With 1 HDD, there is not much safety and as part of the initial setup, that a single disk ‘stripe’ was risky and I had to select “Force”. This gave me the 1.8Tb of storage. No redundancy. After setting up the share and testing on a Mac, I decided to add another disk.</p>
<h2 id="adding-the-second-disk---of-a-differenthigher-capacity">Adding the Second Disk - of a different/higher capacity</h2>
<p>Adding a second disk - I had a choice of striping - and hence still do redundancy (since I would get larger storage + more speed, but data spread over both drives) OR do a mirror.
I decided to go the Mirror route. After much head-banging I found that there is no way in the WebUI to add a Second Disk to an existing pool with Mirror mode. Hence we needed to get into the shell to do stuff. See here for interesting info on <a href="https://icesquare.com/wordpress/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped/">performance impact of the choices</a>. Mirroring limits the storage space to the min size amongst all the disks AND slows down performance - but has high reliability.</p>
<p>Steps to setup a Second Disk with Mirroring in an existing Pool</p>
<p>Basically, there are 4 steps</p>
<ol>
<li>Physically install the new drive</li>
<li>Wipe the drive from the WebUI</li>
<li>Partition the drive as appropriate</li>
<li>Attach the new partitions to the existing ZFS pool and wait for Resilvering to complete (hopefully successful)</li>
</ol>
<p>I am pasting the 3rd and 4th steps below, in the hopes that it will help others.</p>
<p>In my setup, <em>ada0</em> was the first HDD, <em>ada1</em> was the Boot SSD and the newly added drive was <em>ada2</em>. <em>pool01</em> is the name of the storage/data pool with the single ada0 drive - it had two partition - swap and the actual data. I realized that I don’t need the swap</p>
<p>(3)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@truenas[~]#gpart add -i 1 -t freebsd-zfs /dev/ada2
</code></pre></div></div>
<p>(If you want a swap partition as well.. you would do two steps: - subsequent commands will have to reference the right ada2p?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpart add -i 1 -b 128 -t freebsd-swap -s 2g /dev/ada2 # 2G for swap
gpart add -i 2 -t freebsd-zfs /dev/ada2 # For data
</code></pre></div></div>
<p>)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@truenas[~]# glabel status
Name Status Components
gptid/45ae1680-8a9b-11ec-b2ce-408d5ceadf9f N/A ada0p2
gptid/890e9a43-8ac7-11ec-8bb9-408d5ceadf9f N/A ada1p1
gptid/45a1f4c3-8a9b-11ec-b2ce-408d5ceadf9f N/A ada0p1
gptid/785eca5d-8bc9-11ec-8e07-408d5ceadf9f N/A ada2p1
</code></pre></div></div>
<p>(4)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@truenas[~]# zpool attach pool01 /dev/gptid/45ae1680-8a9b-11ec-b2ce-408d5ceadf9f /dev/gptid/785eca5d-8bc9-11ec-8e07-408d5ceadf9f
root@truenas[~]# zpool status
pool: boot-pool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
pool: pool01
state: ONLINE
scan: resilvered 1.22G in 00:00:11 with 0 errors on Fri Feb 11 22:39:06 2022
config:
NAME STATE READ WRITE CKSUM
pool01 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/45ae1680-8a9b-11ec-b2ce-408d5ceadf9f ONLINE 0 0 0
gptid/785eca5d-8bc9-11ec-8e07-408d5ceadf9f ONLINE 0 0 0
</code></pre></div></div>
<p>Notice the “scan:” line above in the status. It shows “resilvered” - i.e. the data has been mirrored. Also, you can see the mirror-0 in the pool. It is also worth noting that the first drive was 2TB and the second drive was 4TB. Since it is setup for mirroring, the capacity is the lowest of all the drives - in our case, it is 2TB. As part of testing, I had 1.22G worth of data, which is mentioend in the scan line.</p>
<h2 id="adding-a-third-drive---just-for-kicks">Adding a third drive - just for kicks</h2>
<p>I was curious to see if I can add an external drive to the setup. I have a few 1TB Passport USB drives lying around. I tried the above flow to get it added to the same pool01.
As is to be expected, I could not attach the drive to the pool, since it is smaller than the smallest drive currently in the pool. i.e. We can mirror all the existing data (though I had not used the storage space yet).</p>
<h2 id="what-is-next">What is next?</h2>
<p>I want to randomly pull out/unplug the drives to see how the UI responds and whether I am able to import and enable disks etc. This has to be done for both the Mirror setup as well as - I would like to try atleast the RAIDZ1 and RAIDZ2. For this, I plan to get a few more 4TB HDDs. Let us see…</p>
<p><a href="https://www.diyd2.in/articles/truenas-mirror/">Setting up a second disk in TrueNAS mirror mode</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on February 12, 2022.</p>
https://www.diyd2.in/articles/octoprint-resend2022-01-01T00:00:00+00:002022-01-01T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<p><img src="/images/adiyogi_pic.jpg" alt="Adiyogi Lithophane" /></p>
<p>Every once in a while, I get carried away with a new hobby. The latest one in the series is 3D printing. I have been looking at this space since 2015 and neither the cost nor the interest level were enough for me to take the plunge. Then things changed in Dec-2021. What triggered this is when a small plastic part broke that required a replacement of the entire product + my search for a right-size enclosure of a DIY power supply.</p>
<p>I bought an Ender-3-Max. One of the many mods that I have chosen to do is install and run an Octoprint server on a Raspberry PI 3B+ ; If you are into 3D Printing this is a definite upgrade/mod you want to do - it will pay dividends within days. See <a href="https://octoprint.org/">Octoprint Site</a> for more details.</p>
<p>The point of this post is a bit different. Many (esp. low cost) printers when connected to a Raspberry PI suffer from a ‘backpower’ issues. That is, power to the motherboard / display on the printer if fed from RPi (in addition to the regular/proper power supply). This has potential to damage both the Pi as well as the printer.</p>
<p>One way this manifests itself is through the high amount of “Resend”s that the RPi needs to do since the messages get garbled. I used to have as much as 10% or more resends - at some point, Octoprint issues warnings about this.</p>
<p>I was hoping for a simple software fix of disabling USB power output from RPi - Alas, it looks like the support for addressing individual USB ports varies greatly between different Pi models. Folks have suggested all kinds of hardware solutions including ‘duct-taping’ the 5V power ping in the USB plug to powering the Raspberry from the power supply of the printer (this last idea does not solve the problem of backpowering - but instaead make sure the supply shared the same GND).</p>
<p>After many tries, what worked for me were two changes:</p>
<ol>
<li>Cut the Power and GND wires of a perfectly good USB cable and use it for connecting the RPi to the printer PLUS</li>
<li>If the USB cable has a metal shield (good ones do), connect the shielding back - you will see three solders in picture below.. red/white that are disconnected are the power and ground</li>
<li>Add a RFI Suppressor on the power cable feeding into the RPi</li>
<li>Just to be safe, run the power cable for the RPi about a foot away from the motherboard + display section of the printer.</li>
</ol>
<p><img src="/images/octoprint_usb_wire.jpg" alt="USB rewire" /></p>
<p>For what cables to cut in USB - See <a href="https://en.wikipedia.org/wiki/USB">Wikipedia</a> - the pinout diagrams on the right of the page. Essentially, you need to keep the center two (D+/-) cables intact and cut the two that on the outer edges.</p>
<p>Result: Voila - Almost no resends during even long prints…
<img src="/images/octoprint-resend-screen.png" alt="Low Resends" /></p>
<p>Since the last month I have had the printer, I have made multiple mods that I will try to share my lessons from:</p>
<ul>
<li>Installed a CRTouch Automatic Bed Level sensor</li>
<li>Upgraded the latest Marlin firmware (Steps are @ <a href="https://github.com/p2c2e/Ender3Max-CRTouch">GitHub</a>)
Will share more details on these experiments and more</li>
</ul>
<p>Useful links:
<a href="https://www.creality.com/goods-detail/creality-ender-3-max-3d-printer">Ender 3 Max</a><br />
<a href="https://octoprint.org/">Octoprint</a> <br />
<a href="https://marlinfw.org/meta/download/">Marlin</a></p>
<p><a href="https://www.diyd2.in/articles/octoprint-resend/">3D Printing : Octoprint woes</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on January 01, 2022.</p>
https://www.diyd2.in/articles/pita2020-11-28T00:00:00+00:002020-11-28T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<p>If you are like me, you have been furiously (for whatever reason) typing away much of your life - resulting in pain in the arms, wrist or fingers (Think RSI and Carpal Tunnel etc.). Whether you are a programmer or a “busy” emailer, you will encounter this issue at some point.</p>
<p>I have tried all remedies - addressing internal (hand, wrist posture, breaks) as well as external (standup desks, ‘ergonomic’ keyboards, mechanical long-travel keyboards, vertical/horizontal/handheld mice) etc. with limited success. This includes being one of the first testers of <a href="https://twiddler.tekgear.com/">Twiddler in late 90s</a> and flirting with the <a href="https://en.wikipedia.org/wiki/Dvorak_keyboard_layout">Dvorak layout</a></p>
<p>While posture and the like do help - if you are consistent/diligent, it is much easier to buy/install things than change behaviour :)</p>
<p>One of the things I have considered is voice dictation software - this is suitable for typing plain English as well as handling most of the app/window management - but falls way short of what is needed for programming. There is also the social impact of being seen saying things like ‘var a colon bool equals …’ and so on.</p>
<p>I shudder to think - if someone who is physically able - is finding working in a tech job so hard, how much harder it would be for someone who is visually, physically impaired..</p>
<p>Anyways, one of the findings has been that I get forearm issues with use of mice (esp. if they are small for the hand). And I have taken some steps to avoid using them - and thus avoid movement of the hand b/w keyboards and mouse.</p>
<p>If you are using a Mac AND you are a productivity fanatic / have ergonomics issues - you may want to consider the below tips.</p>
<p>Master Shortcuts:
<a href="https://shortcutworld.com/">Shortcut World</a><br />
<a href="https://blog.jetbrains.com/idea/2020/03/top-15-intellij-idea-shortcuts/">Top Intellij Shortcuts</a> <br />
<a href="https://www.shortcutfoo.com/">Shortcut Foo</a> <br />
<br />
Useful software:<br />
<a href="https://rectangleapp.com/">Rectangle</a> - Keyboard friendly Window manager <br />
<a href="https://shortcatapp.com/">Shortcat</a> - Any visible input button / control on any app <br />
<a href="https://chrome.google.com/webstore/detail/vimium/dbepggeogbaibhgnhhndojpepiihcmeb?hl=en">Vimium (Chrome)</a> - using Vim like commands to navigate chrome webpages<br />
<a href="https://www.alfredapp.com/">Alfred</a> - spotlight replacement with broader search<br />
<a href="https://mediaatelier.com/CheatSheet/">CheatSheet</a> - Find out what keyboard shortcuts are available on any app by holding down Cmd for an extra sec…<br />
And if you are user of <a href="https://shortcutworld.com/IntelliJ-IDEA/mac/IntelliJ_Shortcuts">IntelliJ IDE user</a> <br />
<a href="https://otter.ai/">Otter Transcription</a> - STT transcription app</p>
<p>While these tips may not completely solve your PITA, I hope it makes some difference.</p>
<p><a href="https://www.diyd2.in/articles/pita/">PITA - Pain in the Arms + Wrists, Fingers - RSI / Carpal Tunnel</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on November 28, 2020.</p>
https://www.diyd2.in/articles/freemind-mindmeister2019-01-04T00:00:00+00:002019-01-04T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<p>I used to be a subscriber for <a href="http://mindmeister.com">MindMeister</a> Pro (strongly recommend it, if you can afford it). In addition to the cloud storage part of it, the mindmaps look cleaner/slicker than what you would get in FreeMind + there is the Presentation Mode that I love.</p>
<p>As I alluded earlier, I no longer subscribe to MindMeister. Which means, I am out of luck with the already created mind maps + my ability to edit/view mindmaps shared by others.</p>
<p>The Free version of MindMeister still allows you to export to it’s own format (.mind)</p>
<p>Yesterday, I wrote a small piece of GoLang code to convert bi-directionally between .mind <> .mm (<a href="http://freemind.sourceforge.net/wiki/index.php/Main_Page">Freemind</a> format).</p>
<p>Keep in mind:</p>
<ul>
<li>There is data loss during conversion. Only the raw mindmap details - relationships and the text move over</li>
<li>Embedded images, links to media etc. will be lost</li>
</ul>
<p>Usage:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mind2mm -h
</code></pre></div></div>
<p>(for help information)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mind2mm -in <filename.mind> -out <filename.mm>
</code></pre></div></div>
<p>(converts from .mind to .mm format)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mind2mm -j2m=false -in <filename.mm> -out <filename.mind>
</code></pre></div></div>
<p>(converts from .mm to .mind format)</p>
<p>Code is <a href="https://github.com/p2c2e/mindmeister-to-freemind">in GitHub</a>
Links to binaries:</p>
<ul>
<li><a href="/assets/files/mind2mm.exe">Windows binary</a></li>
<li><a href="/assets/files/mind2mm">MacOS Binary</a></li>
</ul>
<p><a href="https://www.diyd2.in/articles/freemind-mindmeister/">Utility to convert between Freemind (.mm) and MindMeister (.mind) formats</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on January 04, 2019.</p>
https://www.diyd2.in/blog/graphdb-hr-analytics2017-09-20T00:00:00+00:002017-09-20T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<h1 id="introduction">Introduction</h1>
<p>As part of ongoing efforts towards learning (Sharpen the Saw!), I have been playing around with Graph Databases and Machine Learning
in the context of HR / People development.</p>
<p>In case of Graph Databases, I picked Neo4J to do some experiments - partly because it is quite easy to setup, is free etc.</p>
<p>Sources / types of data include:</p>
<ul>
<li>People records (Joining, Total Expr, Promotions, Titles, City)</li>
<li>Skill data (raw skills information, grouping, certifications, social endorsements)</li>
<li>Team data (name, members, skills required in general)</li>
<li>Performance data - ratings, feedback, coding behaviours</li>
<li>Internal Knowledge management / Expert Forums - to seek out skills</li>
</ul>
<h1 id="rough-schema">Rough Schema</h1>
<p>Here is roughly how the relationship are setup - for lack of a Schema.</p>
<p><img src="/images/graph_schema.png?=250x" alt="Rough Schema" /></p>
<h1 id="the-interesting-part">The Interesting part:</h1>
<p>Obviously, I am doing this for the insights. Here are a some interesting questions I was able to answer:</p>
<ol>
<li>Find all Junior managers managing teams remotely - they may need support to be successful</li>
<li>If a Person knows “X” Technology, what are technologies are they likely to knows</li>
<li>Which people know “X” technology, but are part of teams that don’t require that skill? Sub-optimal use</li>
<li>Find employees “similar” to a given person - This helps in mitigating key-person risk, succession planning</li>
<li>Find the “Best Fit” team for a given person</li>
<li>Given open-positions in the organization, who can be tapped internally? Job mobility.</li>
<li>What are some pre-requisites for a Certification?</li>
<li>If I want to work in Team X or for a particular function (say: business analysis), what certifications are useful?</li>
</ol>
<h1 id="is-this-really-easy-to-implement">Is this really easy to implement?</h1>
<p>If I can whip up some code to quickly answer these questions, why is it that larger organiztions are not using such approaches already?</p>
<p>Answer is simple - This is not a technology problem. Getting quality data requires connecting disparate systems that hold useful information. And for the most part, they are unreliable and “point-in-time”.</p>
<p>Sometimes, it is possible to infer information - For example: what skills are expected in a particular team? We could get this from the team management or infer from the most common skills in the team.</p>
<h1 id="some-ideas-for-implementation">Some ideas for implementation</h1>
<ul>
<li>Using internal and external social platforms to capture and validate skills and endorsements</li>
<li>Gamification to collect and cleanse data</li>
<li>Make these insights part and parcel of day to day team and individual activities, feedback sessions, internal mobility, hiring, learning etc.</li>
</ul>
<p><a href="https://www.diyd2.in/blog/graphdb-hr-analytics/">People insights using a Graph Database</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on September 20, 2017.</p>
https://www.diyd2.in/blog/cgit-lighttpd-docker2016-08-08T00:00:00+00:002016-08-08T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<p>When cloning and trying out public Git repos, I tend to keep them all under one root folder (git_public). While navigating and viewing files is a bit cumbersome, it is even more so if we want to view contents from different branches etc.</p>
<p>I had earlier suggested and implemented cgit in one of my work stints (more than 5+ years back). Since it is a light-weight solution, I decided to try it again on my laptop.</p>
<p>The code and the Docker image for this blog post illustrates a couple of things of technnical interest.</p>
<ol>
<li>How to create a lightweight image using Alpine as the base image</li>
<li>How to make docker images useful with minimal tinkering - by remapping ports. data-volumes and config files</li>
</ol>
<p>The entire repo is located on <a href="https://github.com/p2c2e/docker-lighttpd-cgit">github.com</a>. The ready-to-use docker image is at <a href="https://hub.docker.com/r/sudhan/lighttpd-cgit/">Docker Hub</a></p>
<p>The usage instructions for the ‘sudhan/lighttpd-cgit’ image are available on the README. Try it out and let me know.</p>
<p>Cgit repo is hosted on itself at <a href="https://git.zx2c4.com/cgit/about/">https://git.zx2c4.com/cgit/about/</a>. It will give you a fair idea of what to expect.
<img src="/images/Cgit Viewer Example.png" alt="Cgit website" /></p>
<p><a href="https://www.diyd2.in/blog/cgit-lighttpd-docker/">CGIT web viewer for Git - Using Docker/Alpine/Lighttpd </a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on August 08, 2016.</p>
https://www.diyd2.in/blog/userscripts2016-07-29T00:00:00+00:002016-07-29T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<p>Recently, someone asked me why and when I learnt Javascript. In fact, one of the things I have been using Javascript for a long time is something called ‘userscripts’. I realize that many folks still haven’t heard about this cool utility - hence this post.</p>
<h1 id="firefox-to-chrome---greasemonkey-to-tampermonkey">Firefox to Chrome - GreaseMonkey to TamperMonkey</h1>
<p>For the past few years, I have whole heartedly adopted Chrome as my main browser. Prior to that, I was a Firefox fanboy primarily due to the extensions.</p>
<p>One particular extension that I used frequently was “<a href="https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/">GreaseMonkey</a>”. Until I found that an equivalent for that called <a href="http://tampermonkey.net/">TamperMonkey</a> was available on Chrome, I was reluctant to switch-over.</p>
<p>For those who have not heard of GreaseMonkey, it is addon that allows you to run your own Javascript code within the context of any page from the net. This allows you to modify the page contents and post/pre process to your heart’s content. There are thousands of free user contributed scripts available online. So, the question then is, why would anyone want to modify webpages that they normally browse.</p>
<p>Here are 3 examples where I used GM scripts:</p>
<ol>
<li>Generate QIF (Quicken Interchange Format) of my bank account statements</li>
<li>Enable sections of page / form-elements that are otherwise locked down</li>
<li>Add new informational links between sites.</li>
</ol>
<p>I am sharing more details and code below for those who might want to learn more.</p>
<p>NOTE: For those wanting to run some simple processing of webpages, the ideal solution would be ‘bookmarklets’.</p>
<p>For the userscripts cited below, visit my <a href="https://github.com/p2c2e/tampermonkey-scripts">github.com repo here</a>. The README in the repo allows you to install the scripts. You will need to have either GreaseMonkey or TamperMonkey depending on the browser you use. Users of other browsers: Sorry!</p>
<h1 id="qif-bank-details">QIF bank details</h1>
<p>The userscripts scrape the online banking page when we are visiting and adds a textarea to the bottom of the page with the QIF formatted details. This can be copied to a file and imported into a compatible accounting system.</p>
<p>In my case, I used to use GnuCash for a long time. With multiple bank accounts and Credit card statements, this came in very handy. This use case was also the reason I started delving into jQuery.</p>
<p>If you have accounts with ICICI or Standard Chartered, you can use the scripts in my github repo as-is.</p>
<h1 id="enhancing-pages-with-additional-information">Enhancing pages with additional information</h1>
<p>FlipKart is a popular Amazon alternative marketplace based in India. On occasion, the prices of books on Flipkart could be lower. When browsing through their bookstore, I invariably want to read the reviews and the price and Amazon.</p>
<p>To allow this, I have a user script that scans the page for ISBN-like numbers and adds a link to Amazon website.
The code is available on github. For illustration, I am including the code below. It scans the page for 10 digit numbers that are present within specific elements (to avoid clobbering numbers in scripts etc). It then replaces each occurence with a link to the amazon search page with the given ISBN.</p>
<p>You can install the script by clicking <a href="https://github.com/p2c2e/tampermonkey-scripts/raw/master/LinkISBN.user.js">here</a></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">isbnRegex</span> <span class="o">=</span> <span class="sr">/</span><span class="se">\b([</span><span class="sr">0-9</span><span class="se">]{10,10})\b</span><span class="sr">/ig</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">allowedParents</span> <span class="o">=</span> <span class="p">[</span>
<span class="dl">"</span><span class="s2">abbr</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">acronym</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">address</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">applet</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">b</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">bdo</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">big</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">blockquote</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">body</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">caption</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">center</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">cite</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">code</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">dd</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">del</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">div</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">dfn</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">dt</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">em</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">fieldset</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">font</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">form</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">h1</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">h2</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">h3</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">h4</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">h5</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">h6</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">i</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">iframe</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">ins</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">kdb</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">li</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">object</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">pre</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">p</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">q</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">samp</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">small</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">span</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">strike</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">s</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">strong</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">sub</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">sup</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">td</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">th</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">tt</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">u</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">var</span><span class="dl">"</span>
<span class="p">];</span>
<span class="kd">var</span> <span class="nx">xpath</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">//text()[(parent::</span><span class="dl">"</span> <span class="o">+</span> <span class="nx">allowedParents</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="dl">"</span><span class="s2"> or parent::</span><span class="dl">"</span><span class="p">)</span> <span class="o">+</span> <span class="dl">"</span><span class="s2">)]</span><span class="dl">"</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">candidates</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">evaluate</span><span class="p">(</span><span class="nx">xpath</span><span class="p">,</span> <span class="nb">document</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="nx">XPathResult</span><span class="p">.</span><span class="nx">UNORDERED_NODE_SNAPSHOT_TYPE</span><span class="p">,</span> <span class="kc">null</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">amznPrefix</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">http://www.amazon.com/gp/search/ref=sr_adv_b/?field-isbn=</span><span class="dl">'</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">cand</span> <span class="o">=</span> <span class="kc">null</span><span class="p">,</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="p">(</span><span class="nx">cand</span> <span class="o">=</span> <span class="nx">candidates</span><span class="p">.</span><span class="nx">snapshotItem</span><span class="p">(</span><span class="nx">i</span><span class="p">));</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">isbnRegex</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">cand</span><span class="p">.</span><span class="nx">nodeValue</span><span class="p">))</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">span</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="dl">"</span><span class="s2">span</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">source</span> <span class="o">=</span> <span class="nx">cand</span><span class="p">.</span><span class="nx">nodeValue</span><span class="p">;</span>
<span class="nx">cand</span><span class="p">.</span><span class="nx">parentNode</span><span class="p">.</span><span class="nx">replaceChild</span><span class="p">(</span><span class="nx">span</span><span class="p">,</span> <span class="nx">cand</span><span class="p">);</span>
<span class="nx">isbnRegex</span><span class="p">.</span><span class="nx">lastIndex</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">match</span> <span class="o">=</span> <span class="kc">null</span><span class="p">,</span> <span class="nx">lastLastIndex</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="p">(</span><span class="nx">match</span> <span class="o">=</span> <span class="nx">isbnRegex</span><span class="p">.</span><span class="nx">exec</span><span class="p">(</span><span class="nx">source</span><span class="p">));</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">span</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">createTextNode</span><span class="p">(</span><span class="nx">source</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="nx">lastLastIndex</span><span class="p">,</span> <span class="nx">match</span><span class="p">.</span><span class="nx">index</span><span class="p">)));</span>
<span class="kd">var</span> <span class="nx">a</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="dl">"</span><span class="s2">a</span><span class="dl">"</span><span class="p">);</span>
<span class="nx">a</span><span class="p">.</span><span class="nx">setAttribute</span><span class="p">(</span><span class="dl">"</span><span class="s2">href</span><span class="dl">"</span><span class="p">,</span> <span class="nx">amznPrefix</span><span class="o">+</span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
<span class="nx">a</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">createTextNode</span><span class="p">(</span><span class="nx">match</span><span class="p">[</span><span class="mi">0</span><span class="p">]));</span>
<span class="nx">span</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">a</span><span class="p">);</span>
<span class="nx">lastLastIndex</span> <span class="o">=</span> <span class="nx">isbnRegex</span><span class="p">.</span><span class="nx">lastIndex</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">span</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">createTextNode</span><span class="p">(</span><span class="nx">source</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="nx">lastLastIndex</span><span class="p">)));</span>
<span class="nx">span</span><span class="p">.</span><span class="nx">normalize</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">})();</span>
</code></pre></div></div>
<p><a href="https://www.diyd2.in/blog/userscripts/">Userscripts in the browser</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on July 29, 2016.</p>
https://www.diyd2.in/blog/trading-12016-07-19T00:00:00+00:002016-07-19T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<p>Due to the nature of my work, I haven’t been able to trade in the stock market much for the 5 years or so. I have been a long-term
investor and hence I did not mind holding on to the stocks from long back.</p>
<p>Past month or two, I have started dabbling again in stocks - and this time, I decided to do a good measure of short-term
and even speculative/intraday trading. I am happy to say that I am in good profitable territory so far :)</p>
<h1 id="in-uncharted-waters">In UnCharted Waters</h1>
<p>After a fair bit of prodding from a friend, I decided to base my intra-day activities on Technical Analysis.
I am personally not fully convinced about the ‘logic’ behind many strategies relying solely on technical analysis.
I believe that Technical Charts tell a tale - though belatedly.</p>
<p>Much of my technical analysis studies have been using freely available (online and offline) charting tools
in conjunction with tools available with the broker trading apps. Unfortunately, none of the free (and very few of the commercial tools) support back-testing strategies. Hence, I started evaluating commercial options.</p>
<p>In this post, I will describe how to setup Amibroker, a popular (non-free) tool for charting and back-testing. It comes with it’s own Amibroker Function Language (AFL) for programming.</p>
<p>I plan to document my experience in a series of posts following this one. Where applicable, I will share some code snippets and files as well.</p>
<h1 id="the-free-tools">The Free Tools</h1>
<p>For those who don’t want to be bothered with installing new software, you can try out the free tools listed here. Obviously, this list is by no means exhaustive.</p>
<h2 id="online-tools">Online Tools</h2>
<ul>
<li><a href="http://economictimes.indiatimes.com/markets">Economic Times markets</a>
<ul>
<li>site contains robust tools with plenty of indicators. Based on ChartIQ (see apps section)</li>
<li><a href="http://economictimes.indiatimes.com/markets/technical-charts?symbol=NSE%2520INDEX&exchange=NSE&entity=index&periodicity=day">Here is a link to a sample NSE INDEX chart</a> with typical
indicators that I like to use</li>
</ul>
</li>
<li><a href="http://www.moneycontrol.com/technicals/">MoneyControl</a> - More than just charting, the site has good news, articles etc.</li>
<li><a href="http://techpaisa.com/stock/nifty/">Techpaisa</a>
<ul>
<li>The site has some good screeners as well as verbal summary of different indicators</li>
</ul>
</li>
<li><a href="http://www.bigpaisa.com/custom-technical-screener?date=1&exchange=1&stock_index=2&newhigh=All&priceaction=All&gapper=All&close=All&volume=All&volumeaction=All&avgvolumeaction=All&pricecrossover=All&emacrossover=All&macd=All&adx10=All&adx14=All&di10=All&di14=All&psar=All&rsi=3&bband=1&candlestick_patterns_3=All&candlestick_patterns_4=All&candlestick_patterns_2=All&candlestick_patterns_1=All&candlestick_patterns=All">Bigpaisa</a>
<ul>
<li>Above link gives us the list of stocks that are probably overbought + crossing the Upper Bollinger Band</li>
</ul>
</li>
<li><a href="http://www.marketcalls.in/charts">MarketCalls</a> Excellent site in general. You will inevitably end here if decide to get into AFL programming</li>
</ul>
<h2 id="apps">Apps</h2>
<ul>
<li><a href="https://itunes.apple.com/in/app/technician-real-time-stock/id570491345?mt=8">Technician / ChartIQ</a> Free App, easy to use.</li>
<li><a href="https://itunes.apple.com/us/app/chartiq-practice-trading-simulator/id517702104?mt=8">ChartIQ PTS</a> Also from ChartIQ. This is a very cool app that allows us to practice our TA skills. Well worth the install.</li>
</ul>
<h1 id="amibroker---getting-started">Amibroker - Getting started</h1>
<p><a href="http://www.amibroker.com/">Amibroker</a> has been around for quite a while. And it has a solid reputation amongst Technicians. It does
cost quite a bit, especially since many of the features are available in the free tools provided by most
brokerage firms. That said, speed and back-testing were two reasons I decided to try this one out.</p>
<p>You can follow along with some help from files I have <a href="https://github.com/p2c2e/amibroker-playground">uploaded to github</a></p>
<ul>
<li>Download the trial version of the app from <a href="http://www.amibroker.com/download.html">here</a>. The trial version does not allow
saving of any data, though settings seem to be preserved.</li>
<li>Getting your data
Since I don’t want to spend money for data providers right away <em>and</em> I am interested in EOD data alone,
I decided to configure Amibroker as follows:
<ol>
<li>Create a New Database for our EOD data using File > New > Database
<ul>
<li>Choose a suitable new directory (create one) and ensure the settings are ‘local database’ and end-of-day data.</li>
<li>Get some data from NSE.
As usual, I used my scripts to download the bhavcopy data from nseindia.com. Since the file contains more than just the equities segment, I filtered out all the ‘non-EQ’ rows from the files.
Keep all these files in a different temporary folder (NOTE: NOT the one you created from within Amibroker). I have uploaded a zip file with csv data in github.</li>
<li>Edit the import.types file found at “C:\Program Files\AmiBroker\Formats”
Add a new line as follows:</li>
</ul>
</li>
</ol>
<p>BhavCopy format (<em>.csv)|</em>.csv|BhavCopy.format</p>
<ul>
<li>Copy the new BhavCopy.format file from here into “C:\Program Files\AmiBroker\Formats” folder</li>
<li>From within Amibroker, Choose File > Import ASCII and select and import all the csv files from Step-3
<img src="/images/Import_ASCII.png?=200x" alt="Import ASCII screen with the CSV files" /></li>
<li>After a few seconds, you will see have all the symbols loaded into Amibroker. You can search and load a specific ticker for charting using F3
<img src="/images/amibroker_screen.png?=250x" alt="Sample screen with default charts for Bajaj Auto" /></li>
<li>Since the trial version of Amibroker does not support saving of the database, you will need to redo the “Import ASCII” step everytime you launch the app.</li>
</ul>
</li>
</ul>
<p><a href="https://www.diyd2.in/blog/trading-1/">Technical Trading - Part 1 - Tools</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on July 19, 2016.</p>
https://www.diyd2.in/blog/paradoxes2016-07-19T00:00:00+00:002016-07-19T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<p>When joining college, I decided to do Physics as my Masters major due to my love of Astronomy and Astrophysics. Something about gazing at
the stars and putting ourselves in perspective was always thrilling. It is another matter that the ‘course work’ was not all blue skies!</p>
<p>Similarly, the reason for taking an active interest in Computer Science was manifold - the instant gratification of seeing your program
obey your instructions, the level playing field that it provides regardless of the discipline of study etc. One of the arcane reasons
is the abundance of myths, anecdotes, pseudo “laws” surrounding computer science. I am listing a few below that you might like:</p>
<h1 id="buridans-principle-or-more-fondly-dilemma-of-buridans-ass">Buridan’s Principle or more fondly, Dilemma of <a href="https://en.wikipedia.org/wiki/Buridan%27s_ass">Buridan’s Ass</a></h1>
<p>The story is about a Donkey that dies of starvation because he is equidistant from both hay and water and cannot make up his mind which
way to go. Though originally used as an analogy by Aristotle, it has since been used to illustrate similar non-sensical and seemingly
intractable problems. See the Wiki link for more history.</p>
<p>In the context of computers, Leslie Lamport published a report in 1984. <a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/buridan.pdf">You can find it here</a>.</p>
<p>From my understanding of it, it talks about decision systems taking more than expected time due to ‘indecision points’, where two equally
possible states are making it hard to decide, requiring a need for arbiters.
It has been variously extended to explain thread dead-locks due to resource contention etc.</p>
<h1 id="pigeon-hole-principle">Pigeon hole principle</h1>
<p>This seemingly innocuous principle states that : If there are say 10 pigeons and 9 pigeon holes (each fits only one pigeon), there will always be 1 pigeon left-over after the pigeons are accommodated.</p>
<p>This is a principle from Discrete Mathematics. Though not directly relatable to computer science, it is very useful
in proving algorithm bounds, limits on solutions (‘no more than X satisfy some criteria’) etc.</p>
<p>In the realm of computers, some examples of use include:</p>
<ul>
<li>Sudokus where we can solve some boxes by ruling out others</li>
<li>Bounds on compression algorithms etc. : No lossless compression algorithm can reduce the sizes of all n-bit files, because there are 2^n different n-bit files and only 2^(n−1) possible files of length less than n bits. Or in other words, there is always a few files for which any lossless algorithm will likely result in the new file with same / larger size!</li>
</ul>
<h1 id="paxos-another-leslie-lamport-paper-from-microsoft-research">Paxos (another Leslie Lamport paper from Microsoft Research)</h1>
<p>Rarely have I come across research papers that read like novellas. “<a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/lamport-paxos.pdf">The Part Time Parliament</a>” is refreshing to read.
Highly recommended for leisure reading as well.</p>
<p>The Paxos problem and the corresponding algorithm address challenges in building fault-tolerant distributed systems. The story is about legislators that are free to come / leave the parliament at will. But, for some work to get done, there is a minimum quorum required. It is also applicable for work distribution, election of a node for some work etc.</p>
<p><a href="https://www.diyd2.in/blog/paradoxes/">Interesting Stories in Computer Science</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on July 19, 2016.</p>
https://www.diyd2.in/blog/minikube2016-07-07T00:00:00+00:002016-07-07T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<p>One of the problems with testing out cloud platforms is, we need to be connected to the net all the time. So, sooner or later, we learn to test using solutions like Vagrant as the cluster provider. Even starting a 2 node cluster takes quite a bit of time using this approach.</p>
<p>Good news is the recent addition of ‘Minikube’ - a single node cluster to test out simple services, pods and replication controllers. Note that any attempts to scale the services will probably result in ‘fit failure on node (minikubevm): PodFitsHostPorts’ errors due to it being a ‘single-node cluster’.</p>
<p>The minikube code is on <a href="https://github.com/kubernetes/minikubehttps://github.com/kubernetes/minikube">github</a></p>
<p>The Getting started guide for kubernetes is <a href="http://kubernetes.io/docs/getting-started-guides/minikube/">here</a></p>
<p>Definitely worth a try.</p>
<p><a href="https://www.diyd2.in/blog/minikube/">Minikube - a quick alternative to testing k8s services</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on July 07, 2016.</p>
https://www.diyd2.in/articles/nse-india-stock-chart-in-r2016-04-29T00:00:00+00:002016-04-29T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<ul id="markdown-toc">
<li><a href="#general-file-format" id="markdown-toc-general-file-format">General file format</a></li>
<li><a href="#getting-from-daily-stock-details-to-a-timeseries-not-literally" id="markdown-toc-getting-from-daily-stock-details-to-a-timeseries-not-literally">Getting from daily stock details to a timeseries (not literally)</a></li>
<li><a href="#the-charting-part" id="markdown-toc-the-charting-part">The charting part</a></li>
</ul>
<p>As part of my efforts at learning R, I stumbled upon the idea of charting Indian stocks listed on NSE. I already had some piece of code to get the stock information - written as part of pricing my portfolio account within GNUCash. GNUCash has the facility to write our own Perl modules to access different financial data.</p>
<p>In the case of National Stock Exchange, the historical market data (Open/Close/H/L/Volume etc.) are available freely on their website at <a href="http://www.nseindia.com/">www.nseindia.com</a>. These market figures are called ‘bhav’ (value in Hindi) and the files themselves were called bhavcopy.</p>
<p>NOTE: If you are interested in following long, I have a few sample csv files and a R-Markdown file with the code on <a href="https://github.com/p2c2e/nse-bhav-in-r">github.com</a></p>
<h1 id="general-file-format">General file format</h1>
<p>Each file for a given date is available as a zipped csv file. The zip file contain csv files named like cm23OCT2015bhav.csv</p>
<p>As an example, the contents are like below:</p>
<figure class="highlight"><pre><code class="language-csv" data-lang="csv">SYMBOL,SERIES,OPEN,HIGH,LOW,CLOSE,LAST,PREVCLOSE,TOTTRDQTY,TOTTRDVAL,TIMESTAMP,TOTALTRADES,ISIN,
20MICRONS,EQ,34.55,38.3,33.95,35.6,35.55,34.55,27313,965326.6,31-MAR-2015,307,INE144J01027,
3IINFOTECH,EQ,5.95,6.05,5.7,5.8,5.8,5.8,2621260,15490167.55,31-MAR-2015,3119,INE748C01020,
3MINDIA,EQ,8001.05,8150,7900.05,7935.3,7925.9,8034.2,178,1421292.4,31-MAR-2015,83,INE470A01017,
8KMILES,EQ,730,744,702,711.55,708.45,722,71042,51481004.25,31-MAR-2015,3552,INE650K01013,
A2ZINFRA,EQ,15.5,15.9,15.2,15.3,15.4,15.25,156555,2424829,31-MAR-2015,602,INE619I01012,
AARTIDRUGS,EQ,658,665,630,649.95,649.9,654.5,41964,27394744.15,31-MAR-2015,2049,INE767A01016,
AARTIIND,EQ,352,364,343.95,350.8,346.9,349.2,116529,41496490.55,31-MAR-2015,3368,INE769A01020,
AARVEEDEN,EQ,40.8,41.95,40.8,41.1,41.05,40.75,721,29945,31-MAR-2015,18,INE273D01019,</code></pre></figure>
<h1 id="getting-from-daily-stock-details-to-a-timeseries-not-literally">Getting from daily stock details to a timeseries (not literally)</h1>
<p>To keep things simple, I wrote a small function to load ‘all’ csv files in a given directory and merging the contents into one huge dataframe. The function is shown below:</p>
<p>I keep this and similar utility functions in ~/.Rprofile</p>
<p>While checking the TIMESTAMP column in the frame, it is clear that it was loaded as plain text and needs to be converted to a Date. You can check the class of each column using</p>
<figure class="highlight"><pre><code class="language-csv" data-lang="csv">sapply(dataframe, class)</code></pre></figure>
<h1 id="the-charting-part">The charting part</h1>
<p>Since this is a huge table with all the stock information, I have to filter out the ones corresponding to a given symbol and the plot it.
In my case, I chose to use INFY stock (Infosys Technologies) and entire 2015 data.</p>
<p>Plotting a line chart shows a surprise drop in share price around mid-year. Turns out that was during a 1:1 bonus stock issue. Atleast, it is not due to a bug in my code!</p>
<p><img src="/images/INFY_line.png" alt="INFY - Line Chart" /></p>
<p>To make things a bit more glamorous, I tried to plot the candle-chart. This turns out to be a bit more harder.
We need to know the net-trend (up/down) to decide the color. In addition, we need the decide the rectangle for the candle body and then the length of the wicks. Much of the code is to figure out those numbers.</p>
<p><img src="/images/INFY_candle.png" alt="INFY - Candle Chart" /></p>
<p>Let me know what you think. I am sure it is easier to get this view / info by visiting some popular charting sites. But then, I wouldnt be learning R :)</p>
<p><a href="https://www.diyd2.in/articles/nse-india-stock-chart-in-r/">National Stock Exchange - India chart using ggplot in R</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on April 29, 2016.</p>
https://www.diyd2.in/articles/go-game2016-03-13T00:00:00+00:002016-03-13T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<p>On Saturday, March-12th 2016, history was made when AlphaGo/DeepMind won the series against Lee Sedol 3-0 in a best-of-five series. After losing the 3 straight game, Lee Sedol apologized with characteristic humility for not doing better and meeting the expectations of his fans. After-all, Go is a serious business, especially in China, Japan and Korea where players dedicate themselves from childhood to mastering the game.</p>
<p><img src="/images/go-game-in-progress.png" alt="Go Game in Progress" /></p>
<ul id="markdown-toc">
<li><a href="#interesting-facts-about-go" id="markdown-toc-interesting-facts-about-go">Interesting facts about Go</a></li>
<li><a href="#parallels-in-my-view-with-real-life" id="markdown-toc-parallels-in-my-view-with-real-life">Parallels (in my view) with real-life</a></li>
<li><a href="#some-resources-to-get-you-started" id="markdown-toc-some-resources-to-get-you-started">Some resources to get you started</a></li>
</ul>
<p>The goal of this post is to two-fold:</p>
<ol>
<li>Intrigue You with facts about Go (and maybe you will start playing and enjoying it)</li>
<li>Draw parallels between the game and real-life - I find this fascinating.</li>
</ol>
<p>Go is a two player, turn-based game, played on a grid of 19x19 points. One of the players plays black while the other white. This is where the similarities
with Chess ends.</p>
<p>It is amazing how few (relatively speaking) know about the game of Go. Variously referred to as Baduk, Weiqi and Go, it is probably one of the oldest
strategic games. When I listed “Go” as one of my hobbies in
an introduction questionnaire in my company, the proof reader assumed I had forgotten to complete some sentence! This game deserves better.</p>
<h1 id="interesting-facts-about-go">Interesting facts about Go</h1>
<ul>
<li>Simplicity: There is only one ‘type’ of game piece and less than a dozen core rules.</li>
<li>Complexity: Ironic that I am listing this after “simplicity”. It has been repeatedly mentioned how the number of possible positions in Go are 10^171 whereas the number of atoms in the known Universe is just 10^80. Even the best/fastest computers cannot fully analyze and comprehend what the game will evolve into.</li>
<li>Popularity: Though not apparent, there are supposedly close to 60 Million players world-wide, though mostly in China, Korea and Japan. The game makes an appearance regularly in movies and other forums. For e.g. in the movie “A Beautiful Mind”</li>
</ul>
<h1 id="parallels-in-my-view-with-real-life">Parallels (in my view) with real-life</h1>
<ul>
<li>Very few rules to learn</li>
<li>Success or Failure is not instantaneous. No King to capture/kill.</li>
<li>Intuition and Gut feeling: There are many things happening at the same time. Your moves are most likely driven by intuition rather than hard-logic.</li>
<li>You win some battles and you lose some. But, you can still hope to win the War (overall game)</li>
<li>Patience and perseverance - Games can go on for hours and in earlier days even days - though today’s professional games are time limited.</li>
</ul>
<p><img src="/images/go-game-picture.jpg" alt="Japanese Painting depicting two Go players" /></p>
<h1 id="some-resources-to-get-you-started">Some resources to get you started</h1>
<p>Watch the tournaments (recordings and the rest of it): <a href="https://www.youtube.com/channel/UCP7jMXSY2xbc3KCAE0MHQ-A">DeepMind channel</a></p>
<p>Websites: <a href="http://senseis.xmp.net/">Sensei’s Library</a>, <a href="https://en.wikipedia.org/wiki/Go_(game)">Wikipedia</a></p>
<p>iOS app: <a href="https://itunes.apple.com/us/app/smartgo-player/id314506629?mt=8">SmartGo Player</a></p>
<p>Generic Game engine: <a href="https://www.gnu.org/software/gnugo/download.html">GnuGO</a> - Available on many platforms
There are many UI apps that use this engine on different platforms</p>
<p>Online gaming:
<a href="http://www.kiseido.com/index.htm">Kiseido Game Server</a>
<a href="http://pandanet-igs.com/communities/pandanet">PandaNet</a></p>
<p>I hope you Enjoy this as much as I do!</p>
<p><a href="https://www.diyd2.in/articles/go-game/">Go... Play the game of Life (and Death)</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on March 13, 2016.</p>
https://www.diyd2.in/articles/docker-jetty-mysql2016-02-14T00:00:00+00:002016-02-14T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<p>I personally find it much easier to learn new things by looking at a working system; as opposed to getting a new system up-and-running. Probably a hang-over from my childhood days dismantling anything that works…</p>
<p>In this article, I am sharing the quickest way to get a small jsp page connect to mysql instance, both running on docker containers.</p>
<ul id="markdown-toc">
<li><a href="#assumptions" id="markdown-toc-assumptions">Assumptions</a></li>
<li><a href="#get-the-code" id="markdown-toc-get-the-code">Get the code</a></li>
<li><a href="#setting-up-the-mysql-server-container-instance" id="markdown-toc-setting-up-the-mysql-server-container-instance">Setting up the MySQL server container instance</a></li>
<li><a href="#setting-up-the-jetty-side" id="markdown-toc-setting-up-the-jetty-side">Setting up the jetty side</a></li>
<li><a href="#deploy-an-app-and-test-jetty--mysql-connectivity" id="markdown-toc-deploy-an-app-and-test-jetty--mysql-connectivity">Deploy an app and test jetty / MySQL connectivity</a></li>
<li><a href="#how-does-the-jetty-server-container-access-the-mysql-server-container" id="markdown-toc-how-does-the-jetty-server-container-access-the-mysql-server-container">How does the Jetty server container access the MySQL server container?</a></li>
<li><a href="#accomplishing-the-container-wiring-through-docker-compose" id="markdown-toc-accomplishing-the-container-wiring-through-docker-compose">Accomplishing the container wiring through ‘docker-compose’</a></li>
<li><a href="#newbie-tips" id="markdown-toc-newbie-tips">Newbie tips</a></li>
</ul>
<h1 id="assumptions">Assumptions</h1>
<p>For purposes of this article, I am assuming:</p>
<ol>
<li>To avoid any confusion, I recommend that you should NOT have mysql (server/client) or jetty on the local system. And local ports 8080, 443, 3306 are available for use.</li>
<li>Java, ant build system is available</li>
<li>Docker engine is available. In case of Ubuntu, you can <a href="https://docs.docker.com/engine/installation/linux/ubuntulinux/">find more information here</a>.</li>
<li>git is installed</li>
<li>You understand that the code is meant to be simple (clear-text passwords, poor error-handling). Do not use it for anything beyond learning …</li>
</ol>
<h1 id="get-the-code">Get the code</h1>
<p>Pull the latest bare minimum code (jars, Dockerfiles, jsp) from github</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">git clone https://github.com/p2c2e/min-docker-jetty-mysql.git</code></pre></figure>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">cd </span>min-docker-jetty-mysql</code></pre></figure>
<p>Ensure there is no other local instance of mysql (port 3306 should be free)
Note that there is no data folder at this point</p>
<h1 id="setting-up-the-mysql-server-container-instance">Setting up the MySQL server container instance</h1>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker pull mysql
./bin/dmysqlserver.sh </code></pre></figure>
<p>Check if the server is up and running</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker ps <span class="nt">-a</span> | <span class="nb">grep </span>mysql-server
docker logs mysql-server</code></pre></figure>
<p>Look for “mysqld : ready for connections”</p>
<p>At this point, not only is the mysql-server up, it should also have a new database called ‘TestDB’. You can verify that by running below commands:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">./bin/dmysql.sh </code></pre></figure>
<p>In the MySQL client prompt, run:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">show databases<span class="p">;</span></code></pre></figure>
<p>The TestDB database was created for you because there is a ‘init’ sql file in the ./startup folder. The mysql-server container executes all the files in that folder as part of initialization.</p>
<h1 id="setting-up-the-jetty-side">Setting up the jetty side</h1>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker pull jetty</code></pre></figure>
<p>Now launch jetty using the shell/batch script</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">./bin/djetty.sh</code></pre></figure>
<p>Verify jetty is up by checking the following:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker ps <span class="nt">-a</span> | <span class="nb">grep </span>jetty-server</code></pre></figure>
<p>Verify if the site is up and running by visiting <a href="http://localhost:8080/">the Jetty Root</a></p>
<p>Since there is no ROOT.war distributed, there should be a context listing page with no apps deployed. Jetty is checking for webapps in the ./webapps folder</p>
<h1 id="deploy-an-app-and-test-jetty--mysql-connectivity">Deploy an app and test jetty / MySQL connectivity</h1>
<p>Now deploy a bare-minimum webpp consisting of a index.jsp connecting to a DB and dropping/creating/listing the table row count.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">ant </code></pre></figure>
<p>This should create and copy over a ‘war’ file to the ./webapps/ folder</p>
<p>Check if the war was successfully deployed using :</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker logs jetty-server</code></pre></figure>
<p>Now visit: <a href="http://localhost:8080/java-mysql-1.0/">http://localhost:8080/java-mysql-1.0/</a></p>
<p>It should display a page like below:</p>
<p><img src="/images/simple-jetty-page.png" alt="Minimal Jetty MySQL Docker Homepage" /></p>
<h1 id="how-does-the-jetty-server-container-access-the-mysql-server-container">How does the Jetty server container access the MySQL server container?</h1>
<p>Legacy Links -</p>
<ol>
<li>The primary way by which containers expose their services is through ports. The docker engine can expose pre-defined ports automaticaly or we can do that explicitly using the -p to:from options</li>
<li>For one container to talk to another, the simplest way is to a) expose the mysqld port to the outside world and b) share the mysql servers IP to the jetty-server. This is done via –link option (see the ./bin/djetty.sh)</li>
</ol>
<p>Though the wiring of the two containers were done in the shell scripts, we can ‘compose’ the two containers together so that the dependencies are clearer.</p>
<h1 id="accomplishing-the-container-wiring-through-docker-compose">Accomplishing the container wiring through ‘docker-compose’</h1>
<p>First verify that docker-compose is installed properly. Use <a href="https://docs.docker.com/compose/install/">this link</a> to get you set-up.</p>
<p>Let us kill the existing containers before we can try out the docker-compose version of the application.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker <span class="nb">kill </span>jetty-server mysql-server
docker <span class="nb">rm</span> <span class="si">$(</span>docker ps <span class="nt">-a</span> <span class="nt">-q</span> <span class="nt">-f</span> <span class="nv">status</span><span class="o">=</span>exited<span class="si">)</span></code></pre></figure>
<p>Ensure you are in the root of the clone repository (root directory should contain the docker-compose.yml file). Now, run the docker-compose version of the application using:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker-compose up </code></pre></figure>
<p>If you used the non-daemon version, you will clearly see when the system is ready to accept requests. You can shutdown the servers by Ctrl-D.</p>
<p>Alternatively (if you want to run everything as a daemon)…</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker-compose up <span class="nt">-d</span> </code></pre></figure>
<p>In this case, you would need to shutdown the servers using</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker-compose down</code></pre></figure>
<p>After about 20 seconds, you will be able to visit <a href="http://localhost:8080/java-mysql-1.0/">http://localhost:8080/java-mysql-1.0/</a>.</p>
<h1 id="newbie-tips">Newbie tips</h1>
<ol>
<li>When building images using ‘docker build’, there are many occasions where there are intermediate images that might be left behind. Use ‘docker rmi <imageid>' to remove them</imageid></li>
<li>When you build and tag a new image - if there is an image already exists with that name, it gets untagged. Importantly, the image still remains in the cache. You will need to manually remove them</li>
<li>Containers that have exited (normally or due to getting killed), continue to remain on disk. If not cleaned up, they continue to take up significant space. Remove them using ‘docker rm <containerid>'</containerid></li>
<li>alias the following command to make things easier. The command removes any containers that are lingering after they have exited:</li>
</ol>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker <span class="nb">rm</span> <span class="si">$(</span>docker ps <span class="nt">-a</span> <span class="nt">-q</span> <span class="nt">-f</span> <span class="nv">status</span><span class="o">=</span>exited<span class="si">)</span></code></pre></figure>
<ol>
<li>You can copy-over/run the d*.sh scripts in the repo and use it for generically<br />
If you run the dmysqlserver.sh from any directory, it initializes a brand new instance of mysql and stores the data in the ./data folder. It also creates a ‘startup’ folder where we can store any bootstrap / db-dumps to initialize the database
Using the djetty.sh from any directory, it creates / uses the webapps/ subfolder to deploy apps
Due to port and container name conflicts, you can only run one instance of jetty, mysql-server at any one time</li>
</ol>
<p><a href="https://www.diyd2.in/articles/docker-jetty-mysql/">Minimal Docker Jetty - Mysql application</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on February 14, 2016.</p>
https://www.diyd2.in/blog/dell-install-linux-mint2016-02-09T00:00:00+00:002016-02-09T00:00:00+00:00Sudhanhttps://www.diyd2.insudhan@diyd2.in
<p>After a <em>long</em> hiatus, I recently decided to follow developments on the technology front. Most of my recent work had not given me either the time nor the need to do this.</p>
<p>For starters, I figured I would not want to mess with our family’s shared desktop and buy myself a new laptop (What better excuse could I come up with?).</p>
<p>Choice of laptops:</p>
<h2 id="basic-specifications">Basic specifications</h2>
<p>i5+, 8G+, 500G+ - without Windows preferably (to keep the costs low). No point paying for Windows and then installing Linux anyway.</p>
<h2 id="brands-considered">Brands considered</h2>
<p>Dell, Asus, Lenovo, Acer</p>
<p>These brands were listed on ease of availability in India/locally, cost, non-windows options and finally my prioir experience with them.</p>
<p><b>Acer</b>: We could get an i7 at the cost of i5 from the rest of the brands. Cons: Lack of availability and pre-sales response - imagine what would happen after getting stuck with them!</p>
<p><b>Lenovo</b>: Pros: I still think of them as ThinkPad. Cons: Cost and the feedback on build quality + service in recent times</p>
<p><b>Asus</b>: Pros: Hardware and in some cases, the build Cons: Availability in India + servicing (my most recent laptop was an Asus bought in US and I had issues trying to upgrde the RAM)</p>
<p><b>Dell</b>: Pros: Prior exp (my desktop is 8 yrs old - it was bleeding edge when I bought it. Now, other thn the motherboard everything is changed. It still keeps chugging on…) Cons: Cost - Compared to Acers of the world. Though they technically have Ubuntu versions in the line-up, none are available easily (really sad to see this).</p>
<h2 id="decision">Decision:</h2>
<p>Went with the Dell due to sweet spot of price, availability and prior experience.
Dell 5558 - i5 - 15in - 1TB - non-touch/non-HD version w/ Windows 8.1</p>
<h2 id="prepping-the-machine">Prepping the machine:</h2>
<p>Since I had paid the Windows tax, I might as well ensure that I can use it if required. So, first step was to complete the install and activate 8.1</p>
<p>Next: Upgrade to Windows 10 while the free-upgrade offer exists
I wanted to note down the Serial details in case I needed to reinstall. Here are some tips:</p>
<p><em>The serials listed by various tools might all be slightly different</em>. Not sure what is the ‘correct’ one. I made note of everything once in 8.1 and then once again in Window 10.</p>
<p>From what I understand once Windows 10 is activted using “Digital Entitlement”, we can re-install Window 10 anytime in the future by just clicking “Skip” in the serial screen. Once the system boots up, it will connect to MS servers and reactivate based on some set of system parameters.</p>
<h2 id="choice-of-linux-flavors">Choice of Linux flavors:</h2>
<p>I have always been partial to Fedora - I have been using some flavor of RedHat on my personal machines since 1998 (when the my other main option of interest was Slackware - that will give you some perspective)</p>
<p>Finally decided to go with one of either Fedora or Linux Mint. While Fedora would have given me bleeding edge stuff and helped me with tinkering, I want to tke this opp to learn something new. With yum given way to dnf, I might as well learn apt-get/aptitude!
Linux Mint Cinnamon was the choice</p>
<h2 id="install-mint-on-5558">Install Mint on 5558</h2>
<p>After a flawless install (custom layout), I found the machine was not booting up! Something about not being able to find a bootable partition. After racking my brain about active/primry partition, layout etc. I decided to peek into the BIOS settings.</p>
<p>Dell ships with Windows w/ UEFI enabled. I suspected that some kind of security check was causing the issues.
Changed BIOS settings as followws and installed / booted without issues:</p>
<ul>
<li>Disable UEFI and enable Legacy Mode for accessing the disks</li>
<li>Ubuntu that ships on Dell supposedy has tweaks to get things working. So, it is not same as the one we would download from the ubuntu/mint sites.</li>
</ul>
<p>Back in the days - I remember grappling nVidia (free/non-free) driver issues all the time. I had to have custom patches to get things to work. Pleasantly surprised to see the screen at full resolution off the bat!</p>
<h2 id="teething-issues">Teething issues:</h2>
<p>I have been noticing that ‘suspend’ seems to work, ‘resume’ works only occasionally. On resumption, the screen displays the last view (whether it was the lock screen or desktop) and there is no cursor, any screen refreshes.</p>
<p>Based on a hunch and some googling, I have since changed the <em>Administration -> Login Window -> Options -> Default Session</em> to have “Cinnamon (Software Rendering)”. I have since not seen issues for the past couple of resumes.
IF you try the last tip, you should logout and log-back into the system after the change.</p>
<h2 id="cool-parts">Cool parts:</h2>
<p>Around 2006 is when I last had a primary Linux machine. THere was lot of discussion on <a href="https://en.wikipedia.org/wiki/Cgroups">cgroups</a> at the time when I was with qemu, kvm and virtual box. I did not quite understand what the use case was.
Glad to note that I was wrong. The whole <a href="http://www.docker.com">docker</a> ecosystem would not be possible without cgroups. Will jot down notes on how to set things in layman terms</p>
<h2 id="finding-your-windows-serial-number">Finding your Windows serial number:</h2>
<ul>
<li>If you have a Certified Windows sticker on your machine, you are good to go1</li>
<li>If not, you may want to try one of these tools: Speccy, Belarc Advisor ; Many of the tools “feel” shady.</li>
<li>There are some ‘generic’ serial numbers that are floating around. Not sure what they are meant for.</li>
<li>My serial number reported by tools before/after the Windows 10 upgrade were different.</li>
</ul>
<p><a href="https://www.diyd2.in/blog/dell-install-linux-mint/">A decent developer laptop running on Linux</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on February 09, 2016.</p>
https://www.diyd2.in/articles/magic-in-files2016-02-01T00:00:00+00:002016-02-01T00:00:00+00:00Sudharsan Rangarajanhttps://www.diyd2.insudharsan.rangarajan@gmail.com
<p>Back in olden days, we used to use a command called “file” in *nix and related systems. While the upstart OS2, DOS and Windows operating systems encouraged file extensions by design, *nix always let the user decide what to name each file. A file name ending with “.exe” need not mean it can/cannot be executed. Given the more heterogenous environments and files being copied back-n-forth with total disregard, we had to check whether the file is really what it seems to be… To take a trivial case, text files copied back and forth between DOS and *nix…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ file README
README: ASCII Text
$ file OTHERFILE.TXT
OTHERFILE.TXT: ASCII text, with CRLF line terminators
</code></pre></div></div>
<p>The file command was magical. It could not only tell you a file contains text, but whether it is shell script of a particular type etc. If it was an executable/binary, how it was linked, what target architecture it was meant for etc.</p>
<p>What made the command so smart? Turns out that among other things it uses the concept of “Magic Numbers”. This is a frequently used technique that involves adding a header section to a file or block to identify the type of the file/data. By reading the first few bytes, if not a bit more, we can guess what a file is meant to contain.</p>
<p>So, imagine my surprise when reading up on “BitCoin” and the underlying block-chains. As the name suggests, BitCoin and other crypto currencies that rely on distributed ledger mechanisms, use ‘blocks’ of data. And lo and behold, the first 4 bytes of each block is always “0xD9B4BEF9”. Does that mean anything?</p>
<p>In the early days, the magic numbers were selected by geeks to represent something meaningful to them. Here is a sampling …</p>
<table>
<tbody>
<tr>
<td>BitCoin Block</td>
<td>0xD9 B4 BE F9</td>
</tr>
<tr>
<td>Java .class files</td>
<td>0xCA FE BA BE</td>
</tr>
<tr>
<td>NxtSTEP - Motorola</td>
<td>0xFE ED FA CE</td>
</tr>
<tr>
<td>NxtSTEP - Intel</td>
<td>0xCA FE BA BE (Nxt Obj-C bears similarity to Java - and common folks)</td>
</tr>
<tr>
<td>PalmOS calendar arch</td>
<td>0xBE BA FE CA (oddly reverse ordering of Java)</td>
</tr>
<tr>
<td>Hist. Uninitialized mem</td>
<td>0xDE AD BE EF</td>
</tr>
<tr>
<td>iOS Crash files</td>
<td>0xDE AD FA 11</td>
</tr>
<tr>
<td>Nintendo ‘Normal’ boot code</td>
<td>0xD1 5E A5 E*</td>
</tr>
</tbody>
</table>
<p>Even now-a-days, I occasionlly have to check the leading part of some files when I encounter a corrupt PDF (maybe it is a zip/rar or a postscript file…).</p>
<p><a href="https://www.diyd2.in/articles/magic-in-files/">There is magic in the files...</a> was originally published by Sudhan at <a href="https://www.diyd2.in">Pithy Notes</a> on February 01, 2016.</p>