Looking for a house (Part 1)

looking for a house... or somehow a way to collect real estate market data. It could be interesting to know the prices trend during this pandemic period.

First of  let's build a spider to collect data from some real estate websites. In this article I just focus on one site: casa.it

As I have already done with the jobs web sites I will collect all the data in a database through a pipeline, but this is a topic for the my next article.

I use the famous scrapy framework to build the spider. I am interested to get the information marked in the red squares:

The crawler looks like that:

The start_urls contains a list comprehension where urls are generated from a python list containing only some relevant cities.

In this case it is easy to generate a url for each city:

 https://www.casa.it/srp/?tr=vendita&ft=<city_name>

for example

https://www.casa.it/srp/?tr=vendita&ft=roma

The crawler rules follow the button "seguente"(next ) at the end of the webpage:

in the second gist the xpath-queries.

All togheter:

from this point now ready to store through a dedicated pipeline(an example here)

How to save the scraped data from Scrapy to a mysql DB

After a simple example of a web spider( written  here) I will write a pipeline which will stored the data in a mysql database.

from the scrapy documentation:

After an item has been scraped by a spider, it is sent to the Item Pipeline which processes it through several components that are executed sequentially.

In order to process the item, in our case, we need a custom pipeline which stores the item in a mysql database.

I choose Sqlalchemy to store the items.



I have written here 3 methods, the first establishes a connection to the database (if you want to try it don't forget to insert your credentials), the second makes the table in the database and the third one create the model "job".

In Sqlalchemy the user-defined class rapresents the database table and the istance of those classes the table's rows. In our case the model "job" represents the table in the database

The Sqlalchemy Object relational Mapper presents a method of associating user-defined Python classes with database tables, and instances of those classes (objects) with rows in their corresponding tables. (from here).


After that we need to write our custom pipeline.

The init method provide a connection with the database.
The next method , process_item, is called for each item parsed. What it does, is simply to instantiate an object Job for each item and check if the item already exists in database or not. If it exists, a new line is added to the log file otherwise the pipeline inserts the item in database.

Now we just need to modify the file settings.py in this way:

That's all

A webs-spider for Monster

Here we are...

In this post I will write a web-spider.
If you don't know what a spider is you may check the link  here: https://en.wikipedia.org/wiki/Web_crawler

Which kind of spider?
We can scrap everything from the web: links, photo, prices... but right now I need a job. So I will write a spider to crawl a job-website.

https://www.monster.de/ this one for example.

Yes I know... there is the newsletter, but... have you ever tried to subscribe you into a newsletter?
All you get are only junk mails and  the useful ones usually  arrive too late .
And also the target of those web sites is not actually to find you a job but they only try to push the users to spend as much time as possible on their sites , just like any other web-sites...

For these reasons many web site are so hostile to spiders.

Generally scaping a web site in a non- aggressive way does not create any problem , at least in most  of the cases...



I use here python 2.7 + scrapy

What am I doing here?
I am just scraping the first page, ordered by date of monster.ch, monster.de, monster.it and I extract from them the ad's title, the link and the location.

Monster uses json to show the jobs-ads in the web site. What I am doing here is just to extract every json element in the page and take from them what I am looking for.

Where store the results of our scraping session?

You can try to have a look in the log file, or if you prefer another format the Scrapy frameworks offers many ways to store the output in a more confortable file to read:
https://docs.scrapy.org/en/latest/topics/feed-exports.html#topics-feed-exports

Next time I will show how to store the results in a mysql database.

Stay tuned!