Over the weekend, we built a new Sindice index based on the latest updates of Siren and improvements to the pipelines. This is now in production and sports the following enhancements:

Ranking

• no more big docs first (sorry guys, this was an issue in the last weeks)
• properties are weighted differently

Preprocessing

• Improved support of encoded URIS: Decode encoded characters in URIs prior to indexing. The two versions of the URIs, the decoded and encoded one, are indexed. As a consequence, if you look for an URI (or keyword) with any special characters (e.g., Knud_Möller), this will match also encoded URIs (e.g., http://dblp.l3s.de/d2r/resource/authors/Knud_M%C3%B6ller).
• Improved tokenisation of URI localname: Previously an URI localname was tokenised, but the non-tokenised version of the localname was not indexed. For example, given the URI http://rdf.data-vocabulary.org/#startDate, we now index as part of the local name: start, date, and startDate (the later one was not possible previously).
• Improved support of mailto URI: Previously the URI mailtotest@test.com was improperly tokenised and indexed. Now you can search either for mailto:test@test.com or test@test.com. Both will match the mailto URI.

Query Language and processing

• Improved support of special character in URI. E.g. previously, a tilde ‘~’ in a URI was invalidating the query.
• Various bug fixes as reported by users.
• Improved query processing for Ntriple Query. Huge performance benefits in certain situations.

Index Data Structure

• Improved index compression.

These are the technicals.

In practice the thing I love the most is that my favorite queries (the group by dataset ones = open web dataset finding queries) now work easily 10 times faster .

As always, follow the development of our core open source Semantic Information Retrieval Engine SIREn on github: https://github.com/rdelbru/SIREn.

                          

After quite some work, we are happy to announce that Sindice is now supporting effective large scale data acquisition with *efficient syncing* capabilities based on already existing standards (a specific use of  the sitemap protocol).

For example if you publish 300000 products using RDFa or whatever you want to use (microformats,  303s etc), by making sure you comply to the proposed method, Sindice will now guarantee you

a) to crawl your dataset completely (might take some time since we do this “politely”)

b) ..but only crawl you once and then get just the updated URLs on a daily bases! (so timely data update guarantee)

So this is not “Crawling” anymore, but rather a live “DB like” connection between remote, diverse dataset all based on http. in our opinion this is a *very* important step forward for semantic web data aggregation infrastructures.

The specification we support (and how to make sure you’re being properly indexed) are published here  (pretty simple stuff actually!)

http://sindice.com/developers/publishing

and results can be seen from websites which are already implementing these (you might be already doing that indeed without knowing..)

http://sindice.com/search?q=domain:www.scribd.com+date:last_week&qt=term

Why not make sure that your site can be effectively kept in sync today?

As always  we look forward for comments, suggestions and ideas on how to serve better your data needs (e.g. yes, we’ll also support Openlink dataset sync proposal once the specs are finalized). Feel free to ask specific questions about this or any other Sindice related issue on our dev forum http://sindice.com/main/forum

Giovanni,

on behalf of the team http://sindice.com/main/about. Special credits for this to Tamas Benko and Robert Fuller.

p.s. we’re still interested in hiring selected researchers and developers

                          

If you have any further queries on this downtime, please leave a comment or contact us via the Sindice Developers group.

Update: Thanks to the efforts of NUIG’s ISS team sindice.com is back online ahead of schedule. All services (including sig.ma should now be operational).

                          

While technically possible to host a second Sindice site in a second datacentre (we do have access to such a datacentre) – it’s not a scenario we have examined in detail to date due to the overall high availability of our current datacentre. In addition, Sindice currently runs on 14 servers – meaning we’d have to have 14 servers sitting around in standby mode. While we’d like to have 100% availability through fail-over, we prefer to offer the higher performance on a day to day basis using those 14 servers in other ways (including a new Hadoop cluster which we’re in the process of bringing online – more about that in a future posting).

In summary, the current schedule for our datacentre is to be down from 0000 GMT, 05-Mar-2010 (Friday) to 2000 GMT, 08-Mar-2010 (Monday). We’re hoping the people working on the datacentre upgrades can get things done sooner but the schedule is pretty aggressive already. Obviously if we get things back up sooner, we’ll let you know.

If you have any further queries on this downtime, please leave a comment or contact us via the Sindice Developers group.

                          

While the demo is probably.. agreably cool there is more to talk about.

While Sig.ma is by no mean the first data aggregator for the Semantic Web, its contribution is to show that the sum is really bigger than the single parts and exciting possibilities lie in a holistic approach for automatic semistructured data discovery and consolidation.

In Sig.ma, elements such as large scale semantic web indexing, logic reasoning, data aggregation heuristics, pragmatic ontology alignments and, last but not least, user interaction and refinement, all play together to provide entity descriptions which become live, embeddable data mash ups.

An interesting example:

when we first saw the B&W pictures (e.g. see the demo ) pop up automatically the first time we ran Sigma we were really excited: that DERI data had been there forever yet never meaningfully used or integrated.. let alone automatically! That DERI RDF file does no reuse the right URI for people , doesn’t use Inverse Functional Properties such as “emails”, and  uses only one of many ways to say “author”.

But here it was! That file was there, discovered automatically and  contributing marvelously to the mashup providing information about papers,  (including technical reports that would not be listed otherwise) an extra picture, the phone number, a confirmation of the personal homepage, research projects and more.

Note: this doesn’t mean that the DERI file is bad at all actually. It’s simply not unrealistically great, in other words it was created with a realistic effort, the same that we can expect from any data publisher.

There was no way to get that very useful data with classic  Semantic Web inference and rule consolidation alone. All it took was instead the mix of semantic web practices and tricks with pragmatic and elements of soft computing (quite basic indeed).

In our opinion it all makes sense and inspires the following thoughts:

1. A little semantic might in fact go a long way:  no way there could be something comparable to Sig.ma had we not had a large core of semantically structured data (the Web of Data itself). Publish way more please!  Be this in whatever format can be consolidated to RDF.
2. … it goes in fact even more a long way when the user is involved, and can with pragmatic actions (e.g. “reject” or “approve”) to  steer and validate the results.
3. For data publishers: just like on the HTML web you can simply care only about your site.  If you don’t reuse other people URIs or you don’t put “sameAs” links or you don’t really use the ontology everyone else is using then..  it can work all the same most likely and for most applications!
4. .. but overdescribe
   Be verbose with your semantic descriptions,  more than what you would be for a human. A well described entity will be the best possible “entity” identifier that one could think. It will automatically generate invisible but robust links to others entity descriptions. So dont just write name = fooguy, make sure you expose all you have (and are willing to share) and let aggregation engines use this data to at least do the best consolidation possible.  Good descriptions will also make you show more often in semantic aggregations, foster new applications and make people more likely to integrate with you.
5. For data consumers: We are working for you really and willing to do the hard work.
   This is again very similar to the HTML world. How difficult is to make sense of all the broken HTML out there? Very! How many people have to do it really? just a few, the browser makers.  Others can reuse their efforts and concentrate on other aspects. Sig.ma and Sindice are engines that do the hard part for you as a Web of Data developer.  We provide open services and open source components (heck, at the end of the week we’re even releasing our index open source , next the reasoning engine).  If there is interest and market others will come and there will be more choice

So let me conclude with a good-fortune  Sig.ma of Stefan Decker (50 sources, sigma “Stefan Decker” + add info “Stefan Decker DERI”, with a  couple of manual sources added or deleted)

And the rest follows from the small FAQ in the Sig.ma about page. Cheers!

Why is this potentially revolutionary?

As appropriate data sources become available (pages annotated with RDFa or Microformats), Sig.ma is in a different league in terms of information richness and precision compared to methods solely based on web text analysis.

Sig.ma can be used by humans and software agents alike to obtain structured data about any entity.

Is Sigma noise free?

Not yet. Sig..ma still employs heuristics for many aspects and has to deal with heterogeneous data in the current Web of Data – a very early stage environment! What we can say however is:

1. Sig.ma is interactive and can learn from its usage: when a user deletes a piece of information or a source, Sigma writes it down and that piece of information is less likely to show back at a later time.
2. We have deliberately chosen very simple strategies at this point to test the general idea more than advanced strategies: the potential for improvement is tremendous.
3. The Web of Data itself is very new: until very recently there was basically no way to see this data in action and markup has been done on a best effort-hacker enthusiastic-leap of faith way. Now that Google and Yahoo are starting to recognize the value of page markup, it is realistic to expect improvements in data coverage and quality.

Why does my phone number/picture/favourite movie not appear?

Pages exposing RDF, RDFa or Microformats will appear. If you or your company want information to be found on the web of data, it is very simple to mark up your HTML using RDFa, then submit it to Sindice. You will find it returned by Sig.ma within 10-15 minutes.

How is Sig.ma built? Can I build applications like Sig.ma?

Sig.ma is enabled by Sindice, an index of the web of data. Thanks to Sindice, Sig.ma can accurately locate sources of web data using not only text but also precise attribute value searches and more. Sindice is alive and growing, constantly finding new information, receiving “pings” and immediately adding new documents etc. Where to start? Please write on our forum.

Acknowledgements

Sig.ma and Sindice are built at DERI mainly within the OKKaM Project (ICT-215032) but also with the support of the Science Foundation Ireland under Grant No. SFI/02/CE1/I131, of the ROMULUS project(ICT-217031) and the iMP project.

R&D by  Michele Catasta, Richard Cyganiak, Szymon Danielczyk and Giovanni Tummarello.

                          

Sindice was awarded 1st prize at the European Semantic Web Techology Conference ,

Business Idea Contest ( http://www.estc2008.com/index.php/contest ) unfortunately it is not reported on the homepage just yet, but you can find a nice account here.
Stefan Decker presented on behalf of the team and apparently the pitch was very well received in fact.  He receved the prize itself during the ESTC gala dinner.

In a somehow related news, a  few weeks earlier, “Sindice.com: A
Documented-oriented Lookup Index for Open Linked Data” , by E. Oren, R. Delbru, M. Catasta, R. Cyganiak, H. Stenzhorn and G. Tummarello recevied the Saltlux prize for the best 2007 DERI paper.

As a team, we wish to thank all for such recognitions.

As it should be clear by now, the Semantic Web is all but an easy thing to get started.

We at Sindice have taken our own route and decided to invest a lot of time in getting a quality infrastructure up for everybody’s use. In an academic setting, this is a somehow bold choice. As we know Academia is mostly based on publications and therefore actual infrastructures (That cost a LOT of time, sweats and have much more uncertainty associated than, say, a conference submission) are generally speaking an investment that few do.

It is therefore particularly nice, and useful in fact, to receive these recognitions. They do and will help us continue our commitment toward a more and more useful Web of Data.

                          

Sindice now indexes over 20+ millions Semantic Web documents (21,5 m as i type) and will index your submitted RDFs in usually less than 30 minutes. This great result is entirely due to the dedication of the Sindice development team.

Some of the geeky bits . We have now the version 2 of the indexing pipeline up and running (Renaud, Eyal, Michele).

The Sindice indexing pipeline does a job which is all but trivial. And does it at an amazing speed.

Basically each document is integrated by recursively resolving the URIs of the properties and classes in use, thus calculating a “Web closure” of the explicitly or implicitly imported ontologies. Once this is performed, reasoning happens using RDFS and some OWL ( e.g. FunctionalProperty, TransitiveProperty, sameAs, inverseOf, InverseFunctionalProperty, SymmetricProperty). Sindice has done this for each of the 22 million source independently,in less than 3 weeks (plus the actual indexing and all sort of other processes) on a relatively small cluster (4-6 xeon cores). Not bad?

Thanks to this processing, we can be as precise and complete as possible in solving tasks such computing the IFP index, composing human legible descriptions of documents and powering at best the forthcoming entity based APIs.

Notably, all large datasets (e.g. the huge UniProt) are now proudly processed using our brand new Hadoop based Semantic Sitemap processor, specific courtesy of Holger Stenzhorn who has joined the team last month.

Sindice is Hiring!

In the context of the EU project OKKAM, to start Jan 2008, we are now looking for candidates who’re interested in developing highly scalable and innovative Semantic Web infrastructures and applications. Positions include Interns, Masters, Ph.D, and Postdocs and Scientific Developers.

While we of course highly value academic brilliance, we’re expecially looking for candidates who, like us, believe that it is through clever but hard core software engineering and development that we can make the difference on the Semantic Web.

Successful candidates will be rewarded with top salaries and working conditions.

                          

Sindice is focused on scalability: most of the Semantic Web, to the best of our knowledge, is already indexed (currently 11M documents). Sindice includes big datasets like DBLP in RDF, DBPedia, Geonames, Uniprot, plenty of FOAF and more.

Lots of data to index?

Sindice manages big datasets the smart way. You can easily create a Sitemap using the Semantic Sitemap Extension [1] which Sindice then accepts to process huge databases quickly and without overloading your server.

Data changes frequently?

Sindice is live! Ping us and see your Semantic Web source indexed within 15 minutes.

Ontology Import and Reasoning at Web Scale!

Sindice does Web scale, online reasoning: for each source, it will recursively resolve, compose and reason on top of all the related ontologies. Because of this we can accurately index things like document labels, Inverse Functional Properties and others independently for each RDF file published on the Semantic Web.

Applications:

* As an application developer, you can use our scalable API to serendipitously find data on the Semantic Web: find sources and automatically ping us with your data to make sure that it is found by others!

* As a Semantic Web user, it is already pretty useful to find appropriate URIs around. Please read well: no excuses now for minting yet more URIs for SW people:)

Credits:The results are achieved thanks to the ideas and dedication of the Sindice Beta 0 team Eyal Oren, Renaud Delbru, Michele Catasta and Giovanni Tummarello.

Sindice is made possible thanks to the investment from the DERI Galway, supported by the Science Foundation Ireland (www.sfi.ie).

Sindice is available at http://www.sindice.com

[1] http://sw.deri.org/2007/07/sitemapextension/