I’ve started working in a lab on a genomics project for grad school. The goal is investigating the relationships between transposons and antibiotic resistance in Drosophelia(fruit fly).
I’m not going to go into the details of the biology piece of the project in this post but want to detail the informatics pipeline I’ve been building and some additional adjacent important pieces of info I’ve learned along the way. This post may be a work in progress over the semester.
I’m really liking NixOS. I originally installed it as my daily driver on my personal machine in order to have reproducible, declarative operating system builds on top of an immutable operating system so that, in the spirit of infrastructure as code (IaC), I can look into a file (or files) and see every single configuration detail of my operating system. Operating systems accumulate cruft over time, and I’ve seen running “mystery machines” that have been up for a long time, hosting important applications, often changing ownership, with numerous people configuring them over time. No one fully understands the system, but it’s working, so just don’t touch it…!!
NixOS, with its IaC and immutable nature, solves the mystery machine problem. With all important OS files read-only to users and configuration ONLY allowed through the config files, it forces us flawed humans to be responsible… (even if we only have to do just this ONE little thing real quick). Ansible and other config management tools are great, but they are still vulnerable to drift if people still have root access to the host. Human nature being what it is, this is just laying the groundwork for a future mystery machine.
( Note: this post is purely for experimentation and learning purposes. Sites have rate limits specifically to avoid data collection, because their data is propietary and core to their business. Additionally, on smaller sites especially, excessive scraping can DDOS the site and cause expensive resource utilization. Please respect a web site’s wishes. )
Most websites have API request limits based on IP which make scraping or bulk data collection on their sites prohibitive.
Scraping and data collection requires a LOT of requests for each new page of data which would be rather quickly rate limited, slowing down your data collection to unsatisfactory speeds.
So what’s the solution? Send each request through a unique IP!
There exists companies which specialize in Bulk IP proxying, and have bulk amounts of IP’s, which, for a small fee, will let you proxy all of your requests through their network which, if all goes well, should give you unique IP per request goodness.
I found one of these IP providers called IPRoyal.
IPRoyal claims to have a network of 8 million residential IP’s!!!
I signed up for IPRoyal and forked over the 7 bucks to see what happens.
In order to find places to live or visit where I can maximize the amount of time I’m able to comfortably spend outside I wrote a script to calculate the locations in the US with temperatures between 30 degrees and 75 degrees for 6 months or more.
I stumbled on this written as a joke somewhere on the web and found it hilariously terrible.
What a way to turn a O(1) modulus operation into the funny abomination you see here.
This algorithm has a runtime of O(2^n).