Sindice Blog

How to get some unexpected big data satisfaction.
First: build an infrastructure to process millions of document documents. Instead of just doing it home-brew, however, do your big data homework, no shortcuts. Second: unclog some long standing clogged pipe


The feeling is that of “it all makes sense” and it happened to us the other day when we started the dataset indexing pipeline with a queue of a dozen of large datasets (see later for the explanation). After doing that, we just sit back and watched the Sindice infrastructure, that usually takes 1-2 million documents per day, reason and index 50-100 times as much in the same timeframe, no sweats.

The outcome showing on the homepage shortly after:

But what is exactly this “Dataset Pipeline” ?  
Well its simply a by dataset approach.. to ingest datasets.
In other words, We decided to do away with basic URL to URL crawling for the LOD cloud websites and instead are taking a dataset by dataset approach, choosing just datasets we consider value to index.


This unfortunately, is a bit of a manual work. Yes, it is sad that LOD (which is an initiative that is born to make data more accessible) hasn’t really succeded at adoption of a way to collect the “deep semantic web” in the same way that a simple sitemap.org initiative has for the normal web. But that is the case.


So what we do now in terms of data acquisition is:

• For normal websites that expose structured data (RDF, RDFa, Microformats, Schema etc etc), we support Sitemaps. Just submit it and you’ll see it indexed in Sindice. For these web sites we can be in sync (24h delay or less) e,g, just expose a simple sitemap “lastmod” field and we’ll get your  latest data,
• For RDF datasets (Eg. LOD) you have to ask and send us a pointer to the dump:) . And we’ll add you to a manual watchlist and possibly to the list of those that also make it to SPARQL. Refresh can be scheduled at regular intervals

SPARQL is really hard to beat as a tool to experiment with data integration across datasets.  For this reason we get many request on what data we have in our sparql.sindice.com, frequency of  updates, etc.

We admit there was a bit of disappointment as in the past months we were not able to keep up with the promise of having the whole content also available as SPARQL and real time updated. Sorry, it worked under certain conditions but extending those has so far been elusive.

While the technology is steadily improving, we are happy to report recent improvements that should bring back SPARQL.sindice.com as a useful tool in your data explorations.

First of all  there are 2 different kinds of datasets living in Sindice

1) Websites (3 billion triples) are loaded based on the current ranking within Sindice + on a manual list (please ask if you wish a specific website to be added).  Current list can be read here by the way.

2) Datasets (1.5 billion triples) are all loaded in full, see the list below.

prefix dataset: <http://vocab.sindice.net/dataset/1.0/>

SELECT ?dataset_uri ?dataset_name ?dataset_type ?triples ?snapshot
FROM <http://sindice.com/dataspace/default/dataset/index>
WHERE {
  ?dataset_uri dataset:type ?dataset_type ;
  dataset:name ?dataset_name .
  OPTIONAL {
    ?dataset_uri dataset:void ?void_graph ;
                 dataset:snapshot ?snapshot . 
 GRAPH ?G { 
 ?dataset_uri void:triples  ?triples . 
 } 
 } 
}

As a result of this new setup and of the new procedures, we believe the reliability of the SPARQL endpoint is now much greater. We look forward to your feedback on this.

Also, the SPARQL endpoint at the moment will be updated once a month or on request. In the future we’ll switch to twice a month at least.

Realtime updated results are of course always available via the Sindice regular API (courtesy of Siren)

List of DUMP datasets

The original Sig.ma

The http://sig.ma service was created as a demonstration of live, on the fly Web of Data mashup. Provide a query and Sig.ma will demonstrate how the Web of Data is likely to contain surprising structured information about it (pages that embed RDF, RDFa, Microdata, Microformats)

By using the Sindice search engine Sig.ma allows a person to get a (live) view of what’s on the “Web of Data” about a given topic. For more information see our blog post. For academic use please cite Sig.ma as in [1].

Introducing  Sig.ma Enterprise Edition (EE)

We’re happy today to introduce Sig.ma EE , a standalone, deployable, customisable version of Sig.ma.

Sig.ma EE is deployed as a web application and will perform on the fly data integration from both local data source and remote services (including Sindice.com) ; just like Sig.ma but mixing your chosen public and private sources.

Sig.ma EE currently supports the following data providers:

• SPARQL endpoints (Virtuoso, 4store )
• YBoss + the Web (Uses YBoss to search then Any23 to get the data)
• Sindice (with optionally the ability to use Sindice Cache API for fast parallel data collection)

It is very easy to customise visually and to implement new custom data providers (e.g. for your relational or CMS data) by following documentation provided with Sig.ma EE.

Want to give it a quick try? Sig.ma EE now also powers the service on the http://sig.ma homepage so you can add a custom datasource (e.g. your publicly available SPARQL endpoint) directly from the “Options” menu you get after you search for something. It is very easy to implement new custom data source (e.g. for your relational or web CMS data) by implementing interfaces following the examples.

Sig.ma EE is open source and available for download. The standard license under which Sig.ma EE is distributed is the GNU Affero General Public License, version 3. Sig.ma EE is also available under a commercial licence. Please contact us to discuss further.

Links

• Download Page
• Source Code
• Documentation page.

If you would like to share any feedback with our team, at this point please use the sindice-dev google group.

[1] Giovanni Tummarello, Richard Cyganiak, Michele Catasta, Szymon Danielczyk, Renaud Delbru, Stefan Decker “Sig.ma: Live views on the Web of Data”, Journal of Web Semantics: Science, Services and Agents on the World Wide Web – Volume 8, Issue 4, November 2010, Pages 355-364

Today we’re happy to share two big news:

1) We’re launching today our startup company set to drive commercialization of Sindice services. See press release below. Hurray!

2) We’re making public our main Sindice SPARQL endpoint, containing the entire Sindice dataset. This currently indexes over 12 billion triples and is live updated as data comes in. This is the result of over 12 months of engineering. See technical post (below the press release)

——————————————

Sindice.com: Leading Semantic Data Technology Providers Establish Sindice Ltd.

Sindice launches database-like access to the Web of Data for developers and businesses

Galway, Ireland – 14th June 2011 – For immediate release

An ever growing number of Web sites contain rich meta-data about the page contents, e.g. product prices and features in shopping sites, opening hour information for restaurants, or artist details for concerts. This data is also set to explode as search engines recently agreed on metadata terminology with the schema.org initiative. The resulting  “Web of Data” allows for new mobile applications, browser functionality, or search engines. A critical bottleneck for business solutions, however, is a consolidated and customizable way of accessing all of this data at Web scale.

This business need is the target for Sindice.com, the main service provided by Sindice LTD. With the rich meta-data from more than 260 million pages in its index, Sindice.com currently offers the world’s largest repository of continuously updated Web of Data content, and it’s making it available to current and new generation horizontal and vertical market applications. Key service features include: a focused crawler which allows quick but sustainable data indexing and delta detection, and proprietary data indexes which are updated in realtime.

Today we also announce the latest addition to the Sindice APIs: a Web Scale SPARQL access point. A long awaited feature for data oriented Web developers, this functionality allows third parties to  develop novel applications for desktop, mobile, browser, or browser hosted extensions that leverage structured data from a vast collection of Web sites in parallel, eliminating the need to crawl, extract, transform, cleanse, and load on a per Web application basis.

Sindice LTD Solidly Backed by Leading Semantic Data Technology Providers

Sindice LTD is a newly established enterprise with its headquarters in Galway, Ireland. It is a joint venture by the founders of the Sindice.com team (DERI research institute at the National University of Ireland Galway and the FBK research institute from Trento Italy) led by Dr. Giovanni Tummarello and Dr. Renaud Delbru,  OpenLink Software Inc., the makers of the Virtuoso Database for InterWeb scale Linked Data and other leading-edge data integration technology, and Hepp Research GmbH, leading experts in RDFa technology for e-commerce applications, already supported by most major search engines. Sindice LTD will have privileged access to relevant expertise and intellectual property from all of its members.

“Sindice.com turns any existing Web site – offering semantic markup – into a table in a giant Web-scale database”, says Dr. Giovanni Tummarello, the CEO of Sindice LTD. “Based on this, Sindice.com offers advanced search functionality, data cleansing and data consolidation services and private dataspaces. Sindice.com is also focusing on quite practical products set to immediately reward those who have and are investing in semantic markup already e.g. adopting GoodRelations for ecommerce pages, Opengraph or the new Schema.org.”, he adds.

Professor Martin Hepp, the inventor and lead developer of the GoodRelations standard for e-commerce, stresses the potential for targeted shopping, location-based services, and B2B scenarios: “The potential of semantic markup for e-commerce is now firmly recognized by all three major search companies on the planet – after Yahoo (2008) and Google (2010), Bing has recently also announced plans to support the GoodRelations standard. Today’s search engines, however, harvest only the tip of the iceberg of this data solely for a better rendering of the search results. Sindice’s technology allows much more sophisticated novel commerce applications”. Prof. Hepp is backing Sindice LTD via his Hepp Research GmbH, a semantic-data consulting firm.

“Highly scalable and semantically rich linked data spaces has been a pragmatic pursuit at OpenLink Software for a long time”, says Kingsley Idehen, its Founder and CEO.  “Tuning and delivering linked data spaces as big, data-oriented cloud services is extremely challenging. With Sindice, we deliver live Data as a Service (DaaS), in a way that will provide tremendous strategic advantages to the use of Semantic technologies at InterWeb and Enterprise scales”, he explains. “Web-scale semantic dataspaces have been discussed for ages”, he continues, “but tuning the technology and allowing cloud scalability is still extremely difficult. Sindice LTD will provide effortlessly scalable live semantic dataspaces to enable a new wave of data intelligence in applications and within enterprises”

Contact Details

Sindice LTD
DERI IDA Business Park Lower Dangan, Galway Ireland

info@sindice.com

OpenLink Software Inc.
10 Burlington Mall Road, Suite 265, Burlington, MA 01803, U.S.A.

http://virtuoso.openlinksw.com/

general.information@openlinksw.com

Hepp Research GmbH
Karlstrasse 8, 88212 Ravensburg, Germany
contact@heppresearch.com

http://www.heppresearch.com/

———————

Ok this said, lets get to the technicals

Our SPARQL endpoint sparql.sindice.com is now open in public beta.

Lets take it out for a spin!

e.g. lets take this page http://www.deri.ie/about/team/ which happens to be RDFa marked up ( check it). What are the most common names of people working in DERI?

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT  distinct ?name, count(?name) as ?conte
WHERE {
GRAPH <http://www.deri.ie/about/team/> {
?person
a foaf:Person ;
foaf:givenName ?name;
foaf:phone ?phone .
}
} ORDER BY  DESC(?conte) ?name

http://sparql.sindice.com/sparql?default-graph-uri=&query=PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0ASELECT++distinct+%3Fname%2C+count%28%3Fname%29+as+%3Fconte%0D%0AWHERE+{%0D%0A++GRAPH+%3Chttp%3A%2F%2Fwww.deri.ie%2Fabout%2Fteam%2F%3E+{%0D%0A++++%3Fperson%0D%0A++++a+foaf%3APerson+%3B%0D%0A+++++foaf%3AgivenName+%3Fname%3B%0D%0A++++++++foaf%3Aphone+%3Fphone+.%0D%0A++}+%0D%0A}+ORDER+BY++DESC%28%3Fconte%29+%3Fname%0D%0A&format=text%2Fhtml&debug=on

Or why don’t we look up the phone numbers of everybody named “Brian” ?

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT  ?person ?phone
WHERE {
GRAPH <http://www.deri.ie/about/team/> {
?person a foaf:Person ;
foaf:givenName “Brian” ;
foaf:phone ?phone .
}
}

http://sparql.sindice.com/sparql?default-graph-uri=&query=PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0ASELECT++%3Fperson+%3Fphone%0D%0AWHERE+{%0D%0AGRAPH+%3Chttp%3A%2F%2Fwww.deri.ie%2Fabout%2Fteam%2F%3E+{%0D%0A%3Fperson+a+foaf%3APerson+%3B%0D%0Afoaf%3AgivenName+%22Brian%22+%3B%0D%0Afoaf%3Aphone+%3Fphone+.%0D%0A}%0D%0A}&format=text%2Fhtml&debug=on

but the cool stuff obviously comes when you start joining other datasets e.g. papers we wrote, places attended etc.

E.g. just a few months ago the the linked data workshop took place. Its homepage, http://events.linkeddata.org , is also marked with RDFa .

So we can ask Sindice, Is there anyone from the current DERI team page who presented a paper at that workshop and if so what is the title of the paper and his current deri profile picture picture?

which is naive SPARQL sounds something like this:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT  ?name, ?workshop_paper_title, ?deri_picture
WHERE {
GRAPH <http://events.linkeddata.org/ldow2011/> {
?ldowPerson a foaf:Person;
foaf:name ?name.
?paper dc:creator ?ldowPerson;
dc:title ?workshop_paper_title.

}
GRAPH <http://www.deri.ie/about/team/> {
?deriPerson a foaf:Person;
foaf:givenName ?givenName;
foaf:familyName ?familyName;
foaf:img ?deri_picture.
}

FILTER (regex(?name, ?givenName))
FILTER (regex(?name, ?familyName))
}

http://sparql.sindice.com/sparql?default-graph-uri=&query=PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0APREFIX+dc%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%3E%0D%0ASELECT++%3Fname%2C+%3Fworkshop_paper_title%2C+%3Fderi_picture%0D%0AWHERE+{%0D%0A++++GRAPH+%3Chttp%3A%2F%2Fevents.linkeddata.org%2Fldow2011%2F%3E+{%0D%0A++++++%3FldowPerson+a+foaf%3APerson%3B%0D%0A+++++++++++++++++++++foaf%3Aname+%3Fname.%0D%0A+++++++%3Fpaper+dc%3Acreator+%3FldowPerson%3B%0D%0A++++++++++++++++++dc%3Atitle+%3Fworkshop_paper_title.%0D%0A%0D%0A+++++}%0D%0A++++GRAPH+%3Chttp%3A%2F%2Fwww.deri.ie%2Fabout%2Fteam%2F%3E+{%0D%0A+++++++++%3FderiPerson+a+foaf%3APerson%3B%0D%0A++++++++++++++++foaf%3AgivenName+%3FgivenName%3B%0D%0A+++++++++++++++foaf%3AfamilyName+%3FfamilyName%3B%0D%0A+++++++++++++++foaf%3Aimg+%3Fderi_picture.%0D%0A+++++}%0D%0A+++++FILTER+%28regex%28%3Fname%2C+%3FgivenName%29%29%0D%0A+++++FILTER+%28regex%28%3Fname%2C+%3FfamilyName%29%29%0D%0A}&format=text%2Fhtml&debug=on

But the best is: All the data is LIVE, updated every few minutes, currently holding and refreshing more than 12 billion triples. These are less triple than what Siren – our frontend – currently hold, because we feed Siren also with materialized reasoning, by the way.

Imagine the implications: any website compliant with sindice effective data acquisition practices can becomes a datasource for you to use and interlink. You can therefore join linked data sources, content management systems, ecommerce websites, offering microformats etc at will at will and pretty much in real-time as new data appears or is updated on the web.

Of course this is a beta so there will certainly be issues to report, we look forward to heard feedback. Rest assured we’re constantly working to improve the service and on our way to a pretty exciting roadmap.

Gio on behalf of the team.

Having streamlined several procedures inside Sindice, rebuilding the sindice index from scratch now takes just a few hours.

Over the weekend, we built a new Sindice index based on the latest updates of Siren and improvements to the pipelines. This is now in production and sports the following enhancements:

Ranking

• no more big docs first (sorry guys, this was an issue in the last weeks)
• properties are weighted differently

Preprocessing

• Improved support of encoded URIS: Decode encoded characters in URIs prior to indexing. The two versions of the URIs, the decoded and encoded one, are indexed. As a consequence, if you look for an URI (or keyword) with any special characters (e.g., Knud_Möller), this will match also encoded URIs (e.g., http://dblp.l3s.de/d2r/resource/authors/Knud_M%C3%B6ller).
• Improved tokenisation of URI localname: Previously an URI localname was tokenised, but the non-tokenised version of the localname was not indexed. For example, given the URI http://rdf.data-vocabulary.org/#startDate, we now index as part of the local name: start, date, and startDate (the later one was not possible previously).
• Improved support of mailto URI: Previously the URI mailtotest@test.com was improperly tokenised and indexed. Now you can search either for mailto:test@test.com or test@test.com. Both will match the mailto URI.

Query Language and processing

• Improved support of special character in URI. E.g. previously, a tilde ‘~’ in a URI was invalidating the query.
• Various bug fixes as reported by users.
• Improved query processing for Ntriple Query. Huge performance benefits in certain situations.

Index Data Structure

• Improved index compression.

These are the technicals.

In practice the thing I love the most is that my favorite queries (the group by dataset ones = open web dataset finding queries) now work easily 10 times faster .

As always, follow the development of our core open source Semantic Information Retrieval Engine SIREn on github: https://github.com/rdelbru/SIREn.

This is mainly a test post to verify that the Sindice blog continues to work after migrating it to a new server. But it is also a good opportunity to briefly mention the upgrades we are making to the Sindice infrastructure. I’m happy to report that Sindice has been suffering from some growing pains over the last while – as a result of this, we are in the process of migrating to an improved infrastructure increasing both the capacity of individuals servers and introducing failover support for critical parts of the infrastructure. These measures combined, should, over the coming months significantly increase the amount of data we can both fetch from the web of data and, in turn, process, transform and publish back to the semantic web community. Watch this space for more information over the next few months on our expanding capabilities.

So far semantic web search engines and semantic aggregation services have been inserting datasets by hand or have been based on “random walk” like crawls with no data completeness or freshness guarantees.

After quite some work, we are happy to announce that Sindice is now supporting effective large scale data acquisition with *efficient syncing* capabilities based on already existing standards (a specific use of  the sitemap protocol).

For example if you publish 300000 products using RDFa or whatever you want to use (microformats,  303s etc), by making sure you comply to the proposed method, Sindice will now guarantee you

a) to crawl your dataset completely (might take some time since we do this “politely”)

b) ..but only crawl you once and then get just the updated URLs on a daily bases! (so timely data update guarantee)

So this is not “Crawling” anymore, but rather a live “DB like” connection between remote, diverse dataset all based on http. in our opinion this is a *very* important step forward for semantic web data aggregation infrastructures.

The specification we support (and how to make sure you’re being properly indexed) are published here  (pretty simple stuff actually!)

http://sindice.com/developers/publishing

and results can be seen from websites which are already implementing these (you might be already doing that indeed without knowing..)

http://sindice.com/search?q=domain:www.scribd.com+date:last_week&qt=term

Why not make sure that your site can be effectively kept in sync today?

As always  we look forward for comments, suggestions and ideas on how to serve better your data needs (e.g. yes, we’ll also support Openlink dataset sync proposal once the specs are finalized). Feel free to ask specific questions about this or any other Sindice related issue on our dev forum http://sindice.com/main/forum

Giovanni,

on behalf of the team http://sindice.com/main/about. Special credits for this to Tamas Benko and Robert Fuller.

p.s. we’re still interested in hiring selected researchers and developers

Hi. Due to an expansion of one of our datacentres (and the electrical work that this implies), Sindice and related services such as sig.ma will be down from 1730 GMT+1, 11-Jun-2010 (Friday) to 1730 GMT+1, 12-Jun-2010 (Saturday). This major upgrade will give us increased room to grow the Sindice infrastructure over time. On 27-May-2010 we hit a major milestone in Sindice of having indexed 100 million documents (over 6.5 billion triples) from the semantic web. At current rates of data acquisition, we expect to hit the 200 million document mark before Christmas – so we’ll need that extra room!

If you have any further queries on this downtime, please leave a comment or contact us via the Sindice Developers group.

Update: Thanks to the efforts of NUIG’s ISS team sindice.com is back online ahead of schedule. All services (including sig.ma should now be operational).