2006-07-18 Invited Keynote Tuesday morning
\nArgmax Search in Natural Language Processing
\nDaniel Marcu<\/p>\n
<\/p>\n
The goal is to convince us that search is hard, and by not worrying
\nabout it (just throwing in an argmax without thinking about it much)
\nwe have a problem. <\/p>\n
For example with ISI’s syntax based MT system, the big problem they
\nhave is search errors because the search space is very complex. When
\nthey moved to training on long sentences, they started to have many
\nsearch errors. A very interesting high-level talk about the problems
\nstill in search. Many good examples, and encouraging us as a field to
\nlook at search more compared to models where we do spend a lot of time
\nalready. <\/p>\n
—<\/p>\n
Summarization I session<\/p>\n
Bayesian Query-Focused Summarization
\nHal Daume III, Daniel Marcu (ISI, University of Southern California)<\/p>\n
Extractive summarization in a mostly-unsupervised environment.<\/p>\n
Assume a large document collection, a query, and relevance judgments
\n(a few representative examples) to the query over the documents.
\nVery interesting talk, compared to a variety of extraction methods.
\nBasically it works like a relevance feedback method for IR over
\nsentences. <\/p>\n
—<\/p>\n
Extractive Summarization using Inter- and Intra- Event Relevance
\nWenjie Li, Mingli Wu, Qin Lu, Wei Xu, Chunfa Yuan<\/p>\n
Their events are verbs and action nouns appearing between two named
\nentities. They also look at associated named entities (person,
\norganization, location, date) as well. They build a graph of named
\nentities and event terms. They use PageRank to run over the graph and
\ncreate summaries. <\/p>\n
Intra-event relevance is defined as relevance that is direct between
\ntwo nodes on named entities. Inter-event relevance is links on the
\ngraph that are not direct? I’m not sure, this isn’t very clear in the
\npresentation. <\/p>\n
They use WordNet for some similarity for “semantic relevance” and also
\na topic-specific relevance from the document set. Basically it looks
\nlike just a count of how many times the events occur together (the
\nnumber of named entities that they share.) They also cluster named
\nentities based on descriptive text to see how they are related. <\/p>\n
Basically I don’t think they are doing anything new here; I’ve seen
\nall the things they are doing before. They put a lot of different
\napproaches together perhaps, but I don’t see any new big ideas here. <\/p>\n
They use some DUC data for evaluation with ROUGE scores of different
\nfeatures of their system, but I don’t think they differences they
\npresent are statistically significant from what I remember of DUC
\nscores. <\/p>\n
There was a long pause before any questions, but then one was asked
\nabout how PageRank was used. I don’t think that is an interesting
\nquestion, because Drago (at least, there are others) have been using
\nPageRank for sentence ranking for a few years. Of course, some sort
\ndetails can be given on their usage and adoption of it. <\/p>\n
—<\/p>\n
Models for Sentence Compression: A Comparison across Domains, Training
\nRequirements and Evaluation Measures
\nJames Clarke, Mirella Lapata<\/p>\n
(I met James Clarke last night and had a few drinks with him. A
\nreally nice guy!) <\/p>\n
This paper concentrates on word deletion. They had 3 annotators
\nremove tokens from a broadcast news transcript. Analysis of their
\ncorpus and an existing one (Ziff-Davis by Knight and Marcu I think,
\ncreated from looking at abstracts and looking for full sentences in
\nthe news article.) <\/p>\n
They train a decision tree model to learn operations over the parse
\ntree. Compared to a word-based model. <\/p>\n
60 unpaid annotators rated compressed sentences on a single 1-5
\nscale. <\/p>\n
They looked at two automatic evaluation metrics to see if they are
\ncorrelated with the human scores. The more complicated metric (they
\nsay F-measure, but probably should specify as F-measure over what)
\ncorrelated better than simple string accuracy. The decision list
\nmethod didn’t perform as well as the word-based model. <\/p>\n
I really liked this presentation as well, and will make an effort to
\nsee his poster. <\/p>\n
Kathy commented that he missed out on some previous work (Hongyan Jing
\nnamely and Bonnie Dorr on headline generation.) This paper is more
\nstatistically oriented, but didn’t seem to include the document
\ncontext to determine what is important. The poster describing work
\nusing Integer programming to pull in more linguistic rules.
\nAnother comment on document-level context usage.
\nA question about trying to retain information that is important to
\nsomeone (so, query-focused compression.) A question from Inderjeet
\nMani (I think) on the quality of the internet-based free annotators. <\/p>\n
—<\/p>\n
A Bottom-up Approach to Sentence Ordering for Multi-document
\nSummarization
\nDamushka Bollegala, Naoki Okazaki, Mitsuru Ishizuka
\n(The University of Tokyo)<\/p>\n
They propose an automatic evaluation metric for sentence ordering
\n(average continuity.) They do a hierarchical ordering based on
\npairwise comparison of all sentences (or blocks) to others. They have
\na variety of heuristics for sentence ordering that go into the
\nevaluation functions, and learn the combination of all the features
\nfrom a set of manual sentence orderings. Chronological, topical
\nrelevance using cosine similarity, precedence criteria (compare
\nsentences in their document that come before the sentence they have
\nextracted, and look at info in extracted set), succession criteria is
\nthe same. They create training instances with the 4 features over
\nmanually ordered summary. <\/p>\n
Average continuity computed n-grams of orderings and looks at how many
\nof those are in the target ordering. <\/p>\n
I didn’t really think that this is groundbreaking, but just has put
\ntogether a few features for sentence ordering. It is a nice piece of
\nengineering to pull together the different sentence ordering
\napproaches. <\/p>\n
—<\/p>\n
Went to lunch with Kathy McKeown and Min-yen Kan.<\/p>\n
—<\/p>\n
Skipped the first session so I could check email and brush my teeth.<\/p>\n
—<\/p>\n
Direct Word Sense Matching for Lexical Substitution
\nIdo Dagan, Oren Glickman, Alfio Gliozzo, Efrat Marmorshtein, Carlo
\nStrapparava<\/p>\n
The main point of this talk was that WSD does not have to be defined
\nas picking out the wordnet synset number for each word we are
\ninterested in. It can be reduced to a binary categorization task:
\ndoes the word sense of the target word match the word sense of the
\nsource word? Moving to that formalization will make things clearer
\nfor actual use in applications (a less complicated task than what
\npeople have been solving unnecessarily) and open up the field to new
\nmethods from classification. <\/p>\n
—<\/p>\n
Segment-based Hidden Markov Models for Information Extraction
\nZhenmei Gu, Nick Cerone<\/p>\n
I’m having a hard time understanding what the main thrust of this talk
\nis 11 slides in. <\/p>\n
—<\/p>\n
Information Retrieval I Session<\/p>\n
An Iterative Implicit Feedback Approach to Personalized Search
\nYuanhau Lv, Le Sun, Junlin Zhang, Jina-Yun Nie, Wan Chen, Wei Zhang<\/p>\n
Using click-through information to gather terms for query expansion
\nthat is used to re-rank scores for the user. They use a HITS-like
\nalgorithm analogously with “search results” and “query terms” for
\n“authorities” and “hubs”. The interesting thing is that the
\nre-ranking and query expansion is performed at the same time (upon
\nconvergence, take top terms as QE, top results to re-rank.)<\/p>\n
Out-performs Google in both English and Chinese on precision at 5, 10,
\n20, and 30 documents. <\/p>\n
A question on what documents are used to determine the terms used to
\npick for QE. The accusation is that by using terms from unclicked
\ndocuments to determine which terms to select, they are not using a
\ngood discrimination set. <\/p>\n
Atsushi Fujii from NII asked whether they have tried pseudo-relevance
\nfeedback, which should have similar performance. There is some
\nquestion about whether they have used a fair baseline. <\/p>\n
—<\/p>\n
The Effect of Translation Quality in MT-Based Cross-Language
\nInformation Retrieval
\nJiang Zhu, Haifeng Wang (Toshiba R&D China)<\/p>\n
The idea is to use translated queries from an MT system that has been
\nartificially degraded by reducing knowledge (rule \/ dictionary) base
\nsize, then try to correlate translation quality with result quality.
\nThey are correlated, so better MT means better IR results. Degrading
\nthe rule base leads to syntax errors and word sense errors. Degrading
\nthe dictionary causes only word sense errors, to which the IR systems
\nare more sensitive. <\/p>\n
Comments: much CLIR does not even use MT since it isn’t really
\nrequired, and often systems throw in all senses of words. The
\nquestion is did they try to determine which categories of words seemed
\nto have the most impact (Named Entities, nouns, verbs, etc.) They
\ndidn’t look at that, they focused on translation quality and search
\neffectiveness. Similar question on syntax rules, but they only
\nrandomly selected rules for dropping. <\/p>\n
—<\/p>\n
A Comparison of Document, Sentence, and Term Event Spaces
\nCatherine Blake<\/p>\n
The conclusion is that the document IDF and and inverse sentence
\nfrequency (and inverse term frequency) are correlated. IDF values
\nappear to be very stable. Language used in abstracts are different
\nfrom language for full text (for this data anyway.) <\/p>\n
A kind of interesting investigation of IDF applied to different ways
\nof using it. There were some questions from (my notes abruptly end here.)<\/p>\n","protected":false},"excerpt":{"rendered":"
2006-07-18 Invited Keynote Tuesday morning Argmax Search in Natural Language Processing Daniel Marcu The goal is to convince us that search is hard, and by not worrying about it (just throwing in an argmax without thinking about it much) we have a problem. For example with ISI’s syntax based MT system, the big problem they […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[4,10],"tags":[],"_links":{"self":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/posts\/123"}],"collection":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/comments?post=123"}],"version-history":[{"count":0,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/posts\/123\/revisions"}],"wp:attachment":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/media?parent=123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/categories?post=123"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/tags?post=123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}