What and why

I am a fan of Radio ROKS - their Hard'n'Heavy playlist is just perfect for me. Recently I came up with an idea of creating an app with it's playlist, but... Not so fast.

It seemed that they didn't have an API (at least, a public one), and any direct communication was no good. But I decided that I would go the hard way (which I hadn't tried before) and parse the website myself. (some real life Python practice at last)

Part 1: Getting the data with BeautifulSoup

First of all, I needed to discover the principle of how playlist is shown on their website. Through some poking, I found out that there were neither any outside requests nor JavaScript related to the playlist. It looked like the playlist was being loaded from a (slightly crummy) /playlist/<date>.html file. I found out the elements with neseccary information, and went on with actual parsing.

For this task, I went confidently with BeautifulSoup.

    soup = BeautifulSoup(page, 'html.parser')
    song_lists_html = soup.find_all(class_='song-list')

    songs = []
    for song_list_html in song_lists_html:
        time = song_list_html.find(class_='songTime').text
        song_html = song_list_html.find('div',
            class_='play-button-youtube')

        if song_html:
            singer = song_html['data-singer']
            title = song_html['data-song']
            video = song_html['data-video']
            singer_url = song_html['data-singer-url']
            song_url = song_html['data-song-url']

            song = Song(time, singer, title,
                        video, singer_url, song_url)
            songs.append(song)

When I was done, I got myself a Playlist class with all the relevant methods. Now this data must be made accessible - with an API.

Part 2: RESTful API with Flask

For API design, I wanted to go with something lightweight - I won't even render anything at all! So I dropped Django this time in favor of Flask microframework.

API design with flask is super easy. For example, this snippet gets date from the URL and an optional count from a GET parameter, build the playlist for that date with playlist#_get_song_list() and returns error-coded JSON as a response:

@app.route(API_ROOT + '/playlist/<date_str>')
def playlist(date_str):
    c = 0  # return code
    count_arg = request.args.get('count')
    count = int(count_arg) if count_arg else 0

    res = [
        song._asdict()
        for song in _get_song_list(date_str, count)
    ]
    if not res:
        c = -2

    return jsonify({'c': c, 'res': res})

Part 3: Hosting the work with PythonAnywhere

For the app to be usable, it needs to be hosted. When there is a question of hosting a Python project, I prefer going with PythonAnywhere. It has a great free plan that may suit most of the hobbyist's needs (but sadly, free plan doesn't allow for outbound requests, like I have here, so I went with their $5 plan. Worth it!)

They have a dedicated web interface for creating apps on Django, Flask and much more. So, it was just a job of cloning the GitHub repo, pointing the service at the api.py file and... Well, that's all! After that, I had my app up and running smoothly.

This concludes my story of API-like design with BeautifulSoup and Flask. Python was a great tool for the job indeed :smiley:

Resources

Further reading:

Source code is available on GitHub.