Fetching a description, title and keywords

Data Mining & Php

Description, title & keywords of a Web Page

The following script fetch the description, title and keywords from an url even if there are no meta tags. It also take care of the charset.
This kind of script can easily be implemented and adapted inside a search engine robot.

Copy and paste the following code into your editor and run it (code comments are included).
If you want to see the live result, try it here (new window).

The source code:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> </head> <body> <? /*Simple URL checking*/ if(!isset($_POST['url']) || $_POST['url'] == '' || $_POST['url'] == 'http://') { echo 'Please enter an url.'; } else { $url=strip_tags($_POST['url']); /*we first retrieve the contente of the page using the Curl library*/ $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt ($ch, CURLOPT_FILETIME, 1); curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 2); $store = curl_exec ($ch); /*Error handling*/ if (curl_errno($ch)) { print curl_error($ch); } else { $info = curl_getinfo($ch); curl_close($ch); print <<<END <h1>Description for a given website</h1> Website: $url END; $ok=1; $title=''; $description=''; $keywords=''; /*We extract the Title from the head tags:*/ preg_match("/<head.*>(.*)<\/head>/smUi",$store, $headers); if(count($headers) > 0) { /*Fetch the charset of the page*/ if(preg_match("/<meta[^>]*http-equiv[^>]*charset=(.*)(\"|')>/Ui",$headers[1], $results)) $charset= $results[1]; else $charset='None'; if(preg_match("/<title>(.*)<\/title>/Ui",$headers[1], $titles)) { if(count($titles) > 0) { /*If the charset information has been extracted, we convert it to UTF-8 - Otherwise we assume it's already UTF-8*/ if($charset == 'None') $title=trim(strip_tags($titles[1])); else $title=trim(strip_tags(iconv($charset, "UTF-8", $titles[1]))); } else { /*If there is no title given we take the url as a title*/ if(strlen($url) > 30) $title=trim(substr($url,30)).'...'; else $title= trim($url); } } else { /*If there is no title given we take the url as a title*/ if(strlen($url) > 30) $title=trim(substr($url,30)).'...'; else $title= trim($url); } } else { $ok=0; echo 'No HEAD - That might not be an HTML page!'; } /*Let's fetch the META description or give a description is there is not description available*/ preg_match("|<meta[^>]*description[^>]*content=\"([^>]+)\"[^>]*>|Ui",$headers[1], $matches); if(count($matches) > 0) { if($charset != 'None') $description= trim(strip_tags(iconv($charset, "UTF-8",$matches[1]))); else $description= trim(strip_tags($matches[1])); } else { preg_match("/<body.*>(.*)<\/body>/smUi",$store, $matches); if(count($matches) > 0) { if($charset != 'None') $description= trim(substr(trim(strip_tags(iconv($charset, "UTF-8",$matches[1]))),0,150)); else $description= trim(substr(trim(strip_tags($matches[1])),0,150)); } else { if($charset != 'None') $description= trim(substr(trim(strip_tags(iconv($charset, "UTF-8",$store))),0,150)); else $description= trim(substr(trim(strip_tags($store)),0,150)); } } /*Now the META keywords or some keywords which we extract from the description*/ preg_match("|<meta[^>]*keywords[^>]*content=\"([^>]+)\"[^>]*>|Ui",$headers[1], $matches); if(count($matches) > 0) { if($charset != 'None') $keywords= trim(strip_tags(iconv($charset, "UTF-8",$matches[1]))); else $keywords= trim(strip_tags($matches[1])); } else { /*We shall avoid the stopwords from the keywords*/ $stopwords= array(' the ',' in ',' a ',' and ',' an ',' of ',' about ',' are ',' as ',' at ',' be ',' by ',' com ',' de ',' en ',' for ',' from ',' how ',' in ',' is ',' it ',' la ',' on ',' or ',' that ',' this ',' to ',' was ',' what ',' when ',' where ',' who ',' will ',' with ',' und ',' www ',' you ',' your ',' our '); $keywords=str_replace($stopwords," ",strtolower($description)); $keywords=str_replace(" ",",",$keywords); } /*We print out the results*/ if($ok) echo '<hr>Title: '.$title.'Description:'.$description.'Keywords:'.$keywords; else { echo 'No title/description...'; } } } ?> </body> </html>

As we take care of multilingual possibilities, here is short list of different charset:
- charset=big5 - Chinese Traditional (Big5)
- charset=iso-8859-1 - Western Alphabet
- charset=iso-8859-6 - Arabic Alphabet (ISO)
- charset=utf-8 - Universal Alphabet (UTF-8)
- charset=windows-1252 - Western Alphabet (Windows)
- charset=gb2312 - Chinese Simplified (GB2312)
- charset=iso-8859-15 - Latin 9 (ISO)

A more important list is available at International/O-charset-list.html

For details about the different Php functions implemented here, there are interesting pages:
- Sitemap Generator (another Curl example)
- preg_match examples
- php.net/iconv (iconv function)

Comments

No comment yet, write the first now here or onto our webmaster forum