If you've ever been waiting for a webpage to update then you know how time consuming it can be to constantly have to refresh the page.

I had a similar problem right before the Eclipse of 2017. I was attempting to get eclipse glasses from a local observatory but they were sold out. Fortunately, the observatory's website said a new shipment of glasses were arriving in a few days, and they would update their website when they go on sale. These eclipse glasses were going like hotcakes so I knew I had to be on the ball as soon as the website updated. Obviously I wasn't going to spend all day checking their website so I wrote this bit of Python to help.

This code detects changes in a webpage's content by getting the initial HTML of a webpage and then repeatedly getting the HTML and checking it against the initial HTML every ten seconds. If the webpage has changed it prints "Changed", if not it prints "Not Changed". You can change these print statements to email you, or run whatever other python code you like.

# Hunter Thornsberry
# http://www.adventuresintechland.com

# WebChange.py
# Alerts you when a webpage has changed it's content by comparing checksums of the html.

import hashlib
import urllib3
import random
import time

# url to be scraped
url = "http://raw.adventuresintechland.com/freedom.html"

# time between checks in seconds
sleeptime = 60

def getHash():
    # random integer to select user agent
    randomint = random.randint(0,7)

    # User_Agents
    # This helps skirt a bit around servers that detect repeaded requests from the same machine.
    # This will not prevent your IP from getting banned but will help a bit by pretending to be different browsers
    # and operating systems.
    user_agents = [
        'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11',
        'Opera/9.25 (Windows NT 5.1; U; en)',
        'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
        'Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Kubuntu)',
        'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.142 Safari/535.19',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0) Gecko/20100101 Firefox/11.0',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0.1) Gecko/20100101 Firefox/8.0.1',
        'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.151 Safari/535.19'
    ]

    http = urllib3.PoolManager()
    request = http.request(
        "GET",
        url,
        headers = {
            "User-agent": user_agents[randomint]
        }
    )

    return hashlib.sha224(request.data).hexdigest()

current_hash = getHash() # Get the current hash, which is what the website is now

while 1: # Run forever
    if getHash() == current_hash: # If nothing has changed
        print("Not Changed")
    else: # If something has changed
        print("Changed")
        break
    time.sleep(sleeptime)

YouTube is a great place to find great content, unfortunately you have to dig to find it. "Surface" YouTube is a terrible cocktail of reaction videos, clickbait, and low effort content.

Luckily I've been able to find a decent number of great channels, so here is my list:

CGP Grey

Probably one of the best YouTube creators around, he makes videos over a broad range of topics.

Baz Battles

Does a great video game like representation of historic battles with tons of background and history mixed in.

engineerguy

Bill Hamock does a great job explaining engineering and technology principals and designs.

kaptainkristian

Pop culture "love letters"

Kurzgesagt

Much like CGP Grey (they do twin videos sometimes), covers a lot of different topics

Lessons from the Screenplay

Discussions about cinema

Lindybeige

Covers topics from dance to military history, but mainly focuses on history

Military History Visualized

Great creator who covers many topics of warfare

NativLang

Videos about language, how they work, and their history

Now You See It

Another channel about cinema

Primitive Technology

A man who never speaks builds the most incredible things out of mud and muscle

Real Engineering

Covers a ton of topics about various engineering topics and issues

The Great War

This is more of a series than a collection of videos, it covers World War I, week by week

The School of Life

Some of their more recent stuff is meh, but their Philosophy, History, and Politics videos are great

Tom Scott

Great videos about things you might not know

Wendover Productions

Videos about engineering topics that changed the world

I recently saw a program that gathered the 10 most common words on a webpage and displayed them in a window along with their word count. I decided to build my own using Python and some code I had written before to scrape data from webpages.

# Gives a list of the most common words
# Hunter Thornsberry - hunter@hunterthornsberry.com
from BeautifulSoup import BeautifulSoup
import urllib2
import random
import time

#limit on the number of top words we want to know the count of
limit = 10

#random integer to select user agent
randomint = random.randint(0,7)

#random interger to select sleep time
randomtime = random.randint(1, 30)

#urls to be scraped
urls = ["http://raw.adventuresintechland.com/freedom.html"]

#user agents
user_agents = [
    'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11',
    'Opera/9.25 (Windows NT 5.1; U; en)',
    'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
    'Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Kubuntu)',
    'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.142 Safari/535.19',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0) Gecko/20100101 Firefox/11.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0.1) Gecko/20100101 Firefox/8.0.1',
    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.151 Safari/535.19'
]

words = []

index = 0
while len(urls) > index:
    opener = urllib2.build_opener()
    opener.addheaders = [('User-agent', user_agents[randomint])]
    response = opener.open(urls[index])
    the_page = response.read()
    soup = BeautifulSoup(the_page)

    #Search criteria (is an html tag). Example <p>, <body>, <h1>, etc.
    text = soup.findAll("body")

    #Runs until it has an index out of range error and breaks, this will return every response
    while True:
        try:
            i = 0
            while True:
                #print text[i].text
                words.append(text[i].text)
                i = i + 1
        except IndexError:
            print "--End--"
            break
    index = index + 1

words = words[0].split(" ")
words = [element.lower() for element in words]
sort = []
for word in set(words):
    sort.append(str(words.count(word)) + " " + word)

x = 0
for item in sorted(sort, reverse=True):
    print item
    if x == limit:
        break
    x = x + 1

This code basically comes in two parts, the first part gets the data from the webpage. I've got a whole blog post dedicated just to that.

This is the second part of the code:

words = words[0].split(" ")
words = [element.lower() for element in words]
sort = []
for word in set(words):
    sort.append(str(words.count(word)) + " " + word)

x = 0
for item in sorted(sort, reverse=True):
    print item
    if x == limit:
        break
    x = x + 1

Here I am using .split(" ") to find all of the words. Then I am making every word lower case (as to get a true count of the words, since technically "The" and "the" are two different words). Next the first for loop uses set(words) to get the unique words and appends a string representation of the number of times that word appears in the words list and the word itself.

The second for loop sorts the list and prints the results. Notice sorted() is not a defined function, it is actually built into Python, and we are also passing "reverse=True" so the word with the highest count returns first.

Output

--End--
9 programmers
9 other
9 one
9 new
9 few
9 code
8 when
8 says
8 print
8 first
8 didn't

I decided to go through and list the topics I tweet about so far this year (as of 02/12/2016). I plan on creating a twitter start chart like the one done here: https://www.hella.cheap/twitter-star-chart.html

I've redacted some of my friends names.

Kanye West,25
Technology,16
Society,9
College,8
Canada,6
Fashion,5
Norm Kelly,5
Video Games,3
Economics,2
Networking,2
Wu-Tang,2
Beach House,2
Pokemon,1
Science,1
Hometown,1
Sports,1
Breaking Bad,1
Dune,1
Parents,1
Politics,1
Playstation,1
GonzoHacker,1
Wu-Tang-Financial,1
Daft Punk,1
Rick and Morty,1
Friend1,1
Friend2,1
Travis Scott,1
Grimes,1
Florence + The Machine,1
Drumming,1
Friend3,1
New Years,1

If you browse the internet everyday like I do there are a select group of websites you visit almost everyday, so why not make your new tab page have links to these websites to help you save time? Even if you choose not to use a custom built new tab you can use any website as your new tab.

Why

The easiest way to show you why you should use a custom new tab page is to show you what my homepage and new tab looks like.

This is what I see any time I open a new tab or a new window in my browser. As you can see there are three boxes broken down by category, each with their own title and items. Each one of these items is a link to a webpage I commonly use in that category. Above those boxes I put the Arch Linux logo (as Arch Linux is my default OS).

This is just a sample of what you can do. This works by loading a webpage (either local or remote) upon the opening of a new tab or window, meaning you can wield the full force of any HTML, JavaScript, PHP, etc you can normally use in a webpage because it is a webpage. Like I said above, this also means you can use ANY webpage such as Google, or a random Wikipedia page (https://en.wikipedia.org/wiki/Special:Random).

How

1. Firefox
2. Chrome
3. Safari
4. Internet Explorer

If you have updated to Windows 10 and have an nVidia Graphics Card (in my case, two 970s) and are having issues with installing driver software follow these steps.

Before we begin, download the latest drivers from nVidia here, extract them somewhere you can locate easily, but do not install yet!

1. Navigate to Control Panel > System and Security > System > Advanced system settings (on the left hand panel)

2. Once there under the Hardware tab select Device Installation Settings and make them match the image below.
   

3. Navigate to Control Panel > Hardware and Sound > Device Manager and uninstall and drivers of devices with a "!" icon, it will likely say "Display" as the device.

4. On the Device Manager select Action > Scan for hardware changes, then you should see the graphics card (in my case, GTX 97)0 under "Display adapters". Right click the device and select "Properties", change to the "Driver" tab and click "Uninstall". Make sure to select "Delete the driver software for this device", then click okay.

5. Restart your machine.

6. Now when your computer has booted you should have generic drivers, meaning your display resolution will be quite small. Now using the files we downloaded and extracted, run the installation now.

7. If all went according to plan your drivers should have installed correctly and now you need to restart your machine, after the restart you should be all good to go!

Google Rewards is an Android app that pays you to take surveys. Most of the payouts range from $0.05 all the way up to $1.00, just for taking simple surveys. But as the old adage goes, if you're not paying for the product, you are the product.

Here's how the story goes: I take my lunch break and walk to a local restaurant, on my way I change my mind and decide that I want Jimmy John's, so I change my path and walk to Jimmy John's, grab my Turkey Tom (no tomato) and then walk back to my office. A few minutes later I get a notification of a new survey available.

Google says this data is anonymized, which very well may be but in the end Google got the data it wants, another walking route to my local Jimmy John's.

This post on the Ghost forums ask for a way to automatically, and periodically tweet links to blog posts that are X days old. So, I made one.

GhostTweeter, is written in Python and uses tweepy and feedparser to cycle through a Ghost blog's RSS feed, get post information such as Title, URL, tags, and publish date. If a minimum age is set GhostTweeter will only tweet blog post older than the minimum age. GhostTweeter also turns your blog post tags into hashtags.

Tweets are sent in the following format:

<post title>: <url> #<tag1> #<tag2>

1. Go to https://apps.twitter.com/app/new and create a new app
2. Once created, copy and paste your API key and secret (will be under "Keys and Access Tokens")
3. Create an access token, copy and paste those as well

This is the Twitter authentication part of the Python code, anywehere you see 'yourshere' you must place the correct token.

# Twitter Auth
CONSUMER_KEY = 'yourshere'
CONSUMER_SECRET = 'yourshere'
ACCESS_KEY = 'yourshere'
ACCESS_SECRET = 'yourshere'
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth)

Now it will ask you for your Ghost url, as well as how often you want to tweet (in hours). Once that it all set you are ready to go!

Note: Depending on your OS, if you want to run this in the background you can simply run python GhostTweeter.py & and the code will do the rest of the work.

GhostTweeter on GitHub

This code is very easy to change and can possibly be used with any RSS feed with some simple modifications.

Please pardon the mess, recently had to do a fresh install of Ghost.

Adventures in Techland is now on twitter as @AiTechland. Make sure to follow us to keep up-to-date with the latest post.

FitNotes is a fitness app aimed at weight lifting/body building. You can track number of reps on individual exercises, view progress graphs, record body weight, even keep notes about the day. FitNotes - Today
Custom Exercises

One of the great things about FitNotes is the ability to create custom exercises. The default set exercises has a lot of the basics but I've found that I needed to add a few exercises. When you add an exercise you can specify the name, weight increments, and even notes about that exercise. Each exercise keeps track of reps, and weight. If you set a new maximum weight/reps a trophy will appear next to that rep (see below).

FitNotes - Exercise
Rest Timer

One of the handiest features of FitNotes is the rest timer. After you add an exercise you can start a rest timer either automatically or with the click of a button, and this will give you a specific rest period. This is also great for motivation when you hear the alarm go off, you know it's time to lift.
FitNotes - Timer

Whether you're a programmer, or just the average user these Mac OS X applications will improve your experience with OS X.

Note: Applications are free unless noted otherwise
General

Browser - Google Chrome, Firefox
Text Editor - Sublime Text ($70), Atom
Music - Spotify
Notes - SimpleNote, Evernote
Media Player - VLC
Communication - Skype
File Syncing - Dropbox
App Uninstaller - AppCleaner
Screen Dimmer - Flux

If you have any apps that you would like to suggest, tell us down in the comments below!
Programmer

Terminal - iTerm 2
FTP - Transmit ($34)
Text Editor - Sublime Text ($70), Atom
Virtualization - VirtualBox
Git - GitHub for Mac
SQL - SequalPro

If you have any apps that you would like to suggest, tell us down in the comments below!

With a world of Raspberry Pi, Arduino, Beaglebones, Banana Pi, and a foray of other SoC's taking over the world it's hard to see how anyone can hope to get a large slice of the... Pi.
Enter Intel

Intel needs no introduction. The tech giant has now set its sights on the micro-computer world and has announced the Intel Compute Stick. Interestingly, there are two models of the "Compute Stick". The Windows 8.1 model priced at $149 boast 2GB of RAM, while the Linux version at $89 has 1 GB. It will be interesting to see if Intel will lock the hardware to the OS, thus barring someone from installing Linux on the 2 GB ram model.
Intel Compute Stick
Specs

Operating System: Windows 8.1
Price: $149
RAM: 2GB
Storage: 32 GB + microSD
Power: MicroUSB
Display: HDMI
Network: Wireless (WiFi)
Bluetooth: Yes
USB Ports: 1

Operating System: Linux
Price: $89
RAM: 1GB
Storage: 8 GB + microSD
Power: MicroUSB
Display: HDMI
Network: Wireless (WiFi)
Bluetooth: Yes
USB Ports: 1

(Source: TechReport)

It is possible to have one server host multiple domains, this of course increases the chance of failure (if one server goes down so do ALL of your domains). That being said this saves you tons of money, if uptime is not crucial for you, then this can save you tons of money.
This guide assumes you are on Linux, using either Apache or Nginx!
Apache

There are two files you need to edit to get Apache to server webpages on different ports. We are using "/var/www/html" on port 80 as the default settings and "/var/www/html2" on port 88 as the domain we are going to add.

1. "/etc/apache2/ports.conf"

Add "Listen 88" directly below "Listen 80" in this file.
2. "/etc/apache2/sites-enabled/000-defaut.conf"

Add the follow directly below the closing of the first Virtual Host.

<VirtualHost *:88>
        # The ServerName directive sets the request scheme, hostname and port t$
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        #ServerName www.example.com

        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html2

        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".
        #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

Nginx

We are using "/usr/share/nginx/html" on port 80 as the default and "/usr/share/nginx/html2" on port 88 as the domain we are adding.

Edit "/etc/nginx/sites-enabled/default" and add the following directly below the first server text.

server {
        listen 88 default_server;
        listen [::]:88 default_server ipv6only=on;

        root /usr/share/nginx/html2;
        index index.html index.htm;

        # Make site accessible from http://localhost/
        server_name localhost;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules
        }
}