{"id":2687,"date":"2019-03-26T21:07:54","date_gmt":"2019-03-26T21:07:54","guid":{"rendered":"http:\/\/www.ralphstraumann.ch\/blog\/?p=2687"},"modified":"2019-03-27T16:58:57","modified_gmt":"2019-03-27T16:58:57","slug":"scraping-data-from-a-gis-web-forum-using-python","status":"publish","type":"post","link":"https:\/\/www.ralphstraumann.ch\/blog\/2019\/03\/scraping-data-from-a-gis-web-forum-using-python\/","title":{"rendered":"Scraping data from a GIS web forum using Python"},"content":{"rendered":"\r\n

For a private project of mine as well as in the context of a data analysis foray on behalf of <\/strong>GeoBeer<\/strong><\/a>, I wanted to use the content that has been published on the GIS-centric message board <\/strong>Geowebforum<\/strong><\/a>. In order to achieve this, I had to do some webscraping. To that end, I used Python and BeautifulSoup.<\/strong><\/p>\r\n\r\n\r\n\r\n

Two principal news sources<\/h2>\r\n\r\n\r\n\r\n

The GIS community of Switzerland has two forums for discussions and the exchange of ideas, I’d say: First, there is the #SwissGIS list on Twitter<\/a>: This is a Twitter list of “Switzerland-based people who talk about spatial data science, GIS, geoviz, carto and geo” that is curated by me.<\/p>\r\n\r\n\r\n\r\n

At the time of writing, the list features about 400 members and about 100 subscribers. I use the list as a means to follow Switzerland-related GIS news. To that end, I display the list as a column in Tweetdeck<\/a>. Over time, the SwissGIS list has been the subject of<\/a> several<\/a> analyses<\/a>. To me, this Twitter list is an invaluable resource for learning, sharing, and community building. It is also the nucleus of the GeoBeer event series<\/a> that Stephan<\/a> and I started in 2013 \u2013 after one of those analyses that are linked to above. But this is a topic for another post.<\/p>\r\n\r\n\r\n\r\n

The second forum for GIS in Switzerland, in my perception, is the Geowebforum<\/a>. It is a message board-style online forum in which, after free registration, anybody can share their news, announcements and ideas around GIS. The content is greenlit by a small group of moderators. This makes the forum a more static marketplace of ideas. As opposed to Twitter, it’s rare to see a thread growing into a discussion of 2, 3 or even more posts.<\/p>\r\n\r\n\r\n\r\n

(Of course, there are some other relevant channels besides Twitter and Geowebforum. In my case, e.g.: personal interactions, community events such as GeoBeer, and LinkedIn.)<\/p>\r\n\r\n\r\n\r\n

Getting the data<\/h2>\r\n\r\n\r\n\r\n

For private projects of mine, I wanted to use the content that has been published on Geowebforum<\/a> since its initiation in 2003. While the content is public (anybody can read the entirety of the forum contents, even without registration), it is not available in structured form (other than the RSS feed as was pointed out to me; but the feed only contains a small subset of all posts, namely the most recent ones). Thus, in order to get the content of Geowebforum into a usable form, I did some webscraping and data structuring in a database.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/a>\r\n
The “Jobs” topic on the Geowebforum<\/figcaption>\r\n<\/figure>\r\n\r\n\r\n\r\n

To that end, I used Python, the urllib2<\/a> library for making HTTP requests and the BeautifulSoup<\/a> library for analysing and extracting the page content. You can find all code associated with my project (with an open licence) (as well as the resulting database of Geowebforum content) on my GitHub: github.com\/rastrau\/geowebforum-scraper<\/a>. The work is done in scrape-geowebforum.py<\/a>. This script sets up an SQLite database and scrapes the forum content into it.<\/p>\r\n\r\n\r\n\r\n

The script first iterates through ‘topics’ \u2013 what I called the highest level in the information architecture of the Geowebforum. In the screenshot above you can see, for example, the first page of the topic ‘Jobs’, listing its threads<\/a>.<\/p>\r\n\r\n\r\n\r\n

The Python code to request that page and to find all links to threads looks as follows:<\/p>\r\n\r\n\r\n\r\n

page = urllib2.urlopen('%s&offset=%s' % (topic_url, offset))\r\nsoup = BeautifulSoup(page, 'lxml')\r\nthread_links = soup.find_all('a', {'class': ['threadtitle']})<\/pre>\r\n\r\n\r\n\r\n
With the call to soup.find_all(‘a’, {‘class’: [‘threadtitle’]})<\/strong> we can extract all hyperlink DOM elements (<a href=”….”> tags) of class “threadtitle” in the page. Later, when iterating through the collected thread links, we can access the name of the thread as well as the hyperlink to the thread as follows:<\/p>\r\n\r\n\r\n\r\n
thread_name = thread_link.string \r\nthread_url = 'https:\/\/geowebforum.ch\/%s' % thread_link['href']<\/pre>\r\n\r\n\r\n\r\n
Similarly, the code iterates through all topics, all threads within each topic and all posts within each thread. It records the URLs and the titles of these items as well as the information architecture of the website. For posts, it extracts a lot of additional information such as the post title, the post content with and without HTML tags, the post time and the anonymised post author.<\/p>\r\n
There are some peculiarities I had to take care of: e.g., upon paging too far through a thread, Geowebforum does not display an error page or return an HTTP error code, but rather shows an emptyish page \u2013 the structure of which I detect using BeautifulSoup in order to stop the paging process.<\/p>\r\n\r\n\r\n\r\n
Structuring the data<\/h2>\r\n\r\n\r\n\r\n
The data that is obtained from scraping is structured and filled into an SQLite database that was set up at the start of the processing. This is done using the sqlite3 <\/strong>library of Python. For example, given an active connection to the database and a cursor, the code for ingesting thread data into the database looks as follows:<\/p>\r\n\r\n\r\n\r\n
def ingest_thread(cursor, thread_id, thread_name, thread_url, topic_id):\r\n    cursor.execute(\"INSERT INTO threads VALUES (?, ?, ?, ?, ?)\",\r\n                   [thread_id, thread_name, thread_url, topic_id, 0])\r\n    return<\/pre>\r\n\r\n\r\n\r\n
The database is further set up with triggers, using Python. This way, upon adding a post into the table of posts, a trigger fires and updates the post count per thread in the table pertaining to forum threads.<\/p>\r\n
As a further add-on, I included a classifier of the language of posts. This is done using the langdetect<\/a><\/strong> Python library. This library lets me compute the probability that a post is written in German, French, Italian or English. Out of the four Swiss national languages the (by far) smallest, Romansh, is not covered by langdetect. But I have my doubts that there are any posts in Romansh in the Geowebforum. Besides the language-specific probabilities, I store the most likely language in the database as well. But keep in mind that probabilities can be a very sensible metric to look at too, as, in my impression, a non-insignificant amount of forum posts combine two or more languages.<\/p>\r\n\r\n\r\n\r\n
Getting the data<\/h2>\r\n\r\n\r\n\r\n
You can get all the data that I extracted from the Geoebforum between 2003 and 2018 from github.com\/rastrau\/geowebforum-scraper<\/a>. Specifically, you’ll want to download the SQLite database data.sqlite<\/a>. You can look at, and analyse, this dataset with any SQLite compatible tool, for example DBeaver<\/a> or R using RSQLite<\/a>.<\/p>\r\n\r\n\r\n\r\n
The database contains four tables:<\/p>\r\n\r\n\r\n\r\n