Database update

Development discussion about the hostip.info codebase, libraries, and database

Moderator: Moderators

Database update

Postby Teo on Wed Jul 13, 2005 12:29 pm

There are two sources of IP-address data
- When people are good enough to identify their city, it is stored in the DB and associated with the /24 they are currently using. This is obviously the best info. source.
- There are (currently 10) robots on the server, tracerouting to IP addresses within /24's that we haven't yet identified. There are then several hundred regular-expression rules (from sarangworld) which match parts of the names detected to countries/cities..


The Sarangworld database was built in 1999-2000.
The technic used by him is a basic IP ping and tracert system with filters with a lot of holes.
Since these last 5 years, several hundred of thousand IP have changed.
Each month, some thousand of IP change.

The update of hostip database come only from some monthly dozen manual changes by visitor lost ?
Or several server run every days to update this system ?
Teo
Expert
 

Re: Database update

Postby bfolkens on Wed Jul 13, 2005 12:53 pm

Teo wrote:
There are two sources of IP-address data
- When people are good enough to identify their city, it is stored in the DB and associated with the /24 they are currently using. This is obviously the best info. source.
- There are (currently 10) robots on the server, tracerouting to IP addresses within /24's that we haven't yet identified. There are then several hundred regular-expression rules (from sarangworld) which match parts of the names detected to countries/cities..


The Sarangworld database was built in 1999-2000.
The technic used by him is a basic IP ping and tracert system with filters with a lot of holes.
Since these last 5 years, several hundred of thousand IP have changed.
Each month, some thousand of IP change.

The update of hostip database come only from some monthly dozen manual changes by visitor lost ?
Or several server run every days to update this system ?


There are actually quite a few updates to the system on a regular basis.
bfolkens
Site Admin
 
Posts: 154
Joined: Tue Feb 22, 2005 8:09 am

Postby Teo on Wed Jul 13, 2005 1:42 pm

But where come from datas about cities ?
Because Sarangworld work only on US city, not on other countries.

Datas must come from another database systems because you have not the technology to extract them yourself.

But where have you find the original files ?
(provider list of Geo Ip is very short)

Can you certified than your contents don't come from some hacked files (from other providers without licences paid) ?
Teo
Expert
 

Postby Teo on Wed Jul 13, 2005 3:47 pm

If city links have been done by you, how do you have retrieve them ?

The Community GEO IP Project will be interesting to use the same methods to update the datas.

(note : if you have done that yourself, you are under the software patent that we speak on other topics because there are not a lot of systems to do that)
Teo
Expert
 

Postby Teo on Wed Jul 13, 2005 4:01 pm

I have found that on the website to explain where these files come from.

The city data comes from various places around the 'net as well as from a major data donation.


major data donation = full database from a geo provider ?
:?

It's really a GPL licence if all datas come from another private provider ?
Teo
Expert
 

Postby bfolkens on Thu Jul 14, 2005 7:33 am

Teo wrote:I have found that on the website to explain where these files come from.

The city data comes from various places around the 'net as well as from a major data donation.


major data donation = full database from a geo provider ?
:?

It's really a GPL licence if all datas come from another private provider ?


Major data donation = people, especially ISPs donating their block information, locale, etc. We rely on the community for updates, and as such, the community has free access to the data.
bfolkens
Site Admin
 
Posts: 154
Joined: Tue Feb 22, 2005 8:09 am

Postby bfolkens on Thu Jul 14, 2005 7:42 am

Teo wrote:But where come from datas about cities ?
Because Sarangworld work only on US city, not on other countries.

Datas must come from another database systems because you have not the technology to extract them yourself.

But where have you find the original files ?
(provider list of Geo Ip is very short)

Can you certified than your contents don't come from some hacked files (from other providers without licences paid) ?


Teo, look at the website. If you want more information, do an SVN checkout or just browse it. Everything is there for you to see.

People volunteer their information (city, etc) because they *want* to and they understand that it will help everyone in the long run. That's how these community projects work. If everyone gives a little over time, the project really begins to mature - that's how it has in this case. Time, patience, and generosity begin to pay off.
bfolkens
Site Admin
 
Posts: 154
Joined: Tue Feb 22, 2005 8:09 am

Postby Teo on Thu Jul 14, 2005 11:51 am

It mean than some millions of user have updated the database or that 1 user as put inside an database provided by other geo provider ?

Because we speak about several hundred of thousand city position.

There are not a lot of source of datas,
when you see than 1 user put inside several hundred of thousand geo location ... maybe he has not retrieve it manualy ...

If I send you a copy of my Win xp, you will put it on download server because it's a given software then free without to check the source right ?
Teo
Expert
 

Postby Teo on Thu Jul 14, 2005 4:00 pm

If the guy who has retrieve himself all the datas, can explain how he has done that it will be nice.

Community would like his method to implement and try it too.

:? I'm curious to know how many computers and months he has used to do that and what is his original method to avoid the US patent.
Teo
Expert
 

Postby bfolkens on Thu Jul 14, 2005 6:39 pm

Teo wrote:It mean than some millions of user have updated the database or that 1 user as put inside an database provided by other geo provider ?

Because we speak about several hundred of thousand city position.

There are not a lot of source of datas,
when you see than 1 user put inside several hundred of thousand geo location ... maybe he has not retrieve it manualy ...

If I send you a copy of my Win xp, you will put it on download server because it's a given software then free without to check the source right ?


I had a hard time understanding this, but I'll do my best to reply:
1) Again, lots of visitors AND lots of ISP's updating entire blocks of information at will over a long period of time.
2) WinXP is not just common and free "information" - it is copyrighted. Furthermore, users of it are granted an End-User License Agreement (EULA) that they must comply with - so you sending that data to me would violate your EULA.
bfolkens
Site Admin
 
Posts: 154
Joined: Tue Feb 22, 2005 8:09 am

Re: Database update

Postby ifsca on Mon Jul 21, 2008 10:54 pm

I may be missing something but I find no reference to this in the discussion board.

Why not have a file available for download that contains only the changes to the database over the previous 30 days?

As I understand your current policy, users of this database must download the entire database each month to get the latest updates.

Did I miss something or is there a reason this is not being done?

Amazing project by the way. Hugely ambitious. Thank you for making this available.
ifsca
n00b
 
Posts: 1
Joined: Mon Jul 21, 2008 10:46 pm

Re: Database update

Postby Gaspar on Thu Jan 22, 2009 5:58 pm

bfolkens wrote:There are actually quite a few updates to the system on a regular basis.


When was the last update?
Gaspar
n00b
 
Posts: 10
Joined: Sat Jan 17, 2009 3:32 pm


Return to Hostip.info code/database devel

Who is online

Users browsing this forum: No registered users and 0 guests

cron