Notes from Friday’s sessions at the last day of COLING\/ACL. <\/p>\n
<\/p>\n
Friday 2007-07-21 Notes from COLING\/ACL
\nLanguage, gender and sexuality: Do bodies always matter?
\nSally McConnel-Ginet Cornell University (Invited talk)<\/p>\n
Co-author of book “Language and Gender” – sounds interesting.<\/p>\n
Gave an example of John Summers (sp?) the Harvard guy and his talk
\nabout women and science. There is a difference between the overt and
\ncovert messages, and we have to be aware of both of those, as well as
\nthe ways that our messages can be interpreted. The majority of the
\ntalk was about gender as a performance vs. gender as what we do, and
\nvarious ways that linguistics can be a resource for that performance.
\nInteresting talk about identity and how we realize our identities, and
\nthe frameworks around us for socializing them. <\/p>\n
—
\nQuestion and Answering Session<\/p>\n
Answer Extraction, Semantic Clustering, and Extraction Summarization
\nfor Clinical Question Answering
\nDina Demner-Fushman, Jimmy Lin (University of Maryland)<\/p>\n
They present a system that answers the question “What is the best
\ntreatment type for X?” and provide bullets that list treatment, along
\nwith context to show support, and links to the source articles where
\nthe answers came from. Extract answers, do semantic clustering to put
\nanswers into a hierarchy, then extract answers for each semantic
\ncategory. <\/p>\n
They have a template to search medline for drug therapy type
\nquestions. They then tag the returned data with MetaMap (a NE \/
\nsemantic tagger for medical domain) to find interventions (drug
\ntreatments) and cluster the interventions and extract a sentence to
\nshow support. Their extractive summarizer uses supervised machine
\nlearning to detect outcome statements using a bunch of classifiers
\n(rule based, naive Bayes, n-gram, position, semantics, length) that
\nare combined linearly to detect best “outcome statement”. <\/p>\n
Seems like a reasonable approach, I would be interested in seeing how
\nthis work compares to Min-yen Kan’s centrifuser system, or the work
\nthat Noemie has done. I’m not familiar enough with work in this area
\nto be able to make a good evaluation of this work. It doesn’t look
\nlike anything amazing to me though. I think evaluation in this
\nscenario is tough, because what are they going to compare to? They
\ncompare to a graded scale from 1-10 of how effective each treatment
\nwas (but I’m not sure if this comes from medline or elsewhere for
\nsimilar questions.) They are using ROUGE to evaluate their summaries,
\nbut what are they using as the reference summary?<\/p>\n
So far their results have been color coded instead of hard numbers
\nwith significance testing. They do say that the answers that come
\nfrom their clustering system outperform just the results from the IR
\nsystem at a statistically significant level, but I don’t know what
\ntheir model summaries are.<\/p>\n
Kathy asks about how they deal with contradictions across articles.
\nThey do not deal with contradictions. Question from someone else
\nabout what version of ROUGE and what the reference summaries were (me
\ntoo!) Used ROUGE 1.5 with ROUGE-1 precision, summaries are the one
\nprovided as the abstract of the papers used to generate their
\nextractive summary (this is a very strange and poor evaluation
\napproach I think.) <\/p>\n
Another question about comparatives and negation (“this is not the
\nbest treatment”) they use SEMREP which deals with negations. It isn’t
\nquite the same as Kathy’s question, but is somehow related. In the
\ndata there does not seem to be many comparative summaries, so it
\nhasn’t been much of a problem. <\/p>\n
Question to explain more about the features used in the classifier for
\nsentence extraction. It looks to me like they have basically some
\npretty normal features. The rule-based classifier was created by a
\nregistered nurse who looked at about 1000 good articles and tried to
\nidentify what the bottom line sentence would be, and indicators to
\nfind it. What is the naive Bayes classifier trained on? N-gram &
\nsemantic also… They have a paper on that (JAMIA 2005) it looks
\nlike, so we can go there for more information. Another question on
\nthe extractor: have you compared the output of the meta-classifier
\nagainst expert opinions on extracts. Yes, it is fairly accurate (from
\n85 – 92%) but didn’t say about how they evaluated that. There is
\nanother work that looks at the relations from the abstract, and they
\ndidn’t want to duplicate that work. <\/p>\n
—<\/p>\n
Exploring Correlation of Dependency Relation Paths for Answer
\nExtraction
\nDan Shen, Dietrich Klakow (Saarland University presented by Dan Shen –
\na female)<\/p>\n
They are looking at syntactic matching between question parse tree and
\ncandidate answer sub-trees. They want to avoid strict parse tree
\nmatching because the answer can be realized in different forms. They
\nuse Minipar dependency relation parser. They introduce a path
\ncorrelation measure to check the correlation between they question
\nparse and candidate answer parse. They use a Dynamic Time warping
\nalgorithm to align the two paths. They need to know what the
\ncorrelation is between types of relations, so they compute a
\ncorrelation measure over the TREC 02 and 03 data using a mutual
\ninformation measure. <\/p>\n
They also have a phrase similarity metric that works over noun
\nphrases. It takes some morphological variation (via stemming) into
\naccount and some format variations. Semantic similarity is done as
\nper Moldovan and Novischi 2002. <\/p>\n
TREC 1999-2003 for training data, test data is TREC 2004. Built the
\ntraining set by grabbing relevant documents for each question and
\nmatching one keyword to find the answer. They used this for training?
\nIsn’t there some better gold standard? I am not clear on their
\ntraining data and what they are trying to train at all. They use
\nkernel trees to do the syntactic matching, but say that they don’t
\nlearn the similarity features? They present a bunch of different
\nmethods, but spent too much time early on explaining their system, and
\nI’m not clear what they are comparing against. <\/p>\n
This talk is going a bit long, but there were some mike problems
\nearlier on (picking up interference from another mike somewhere?) that
\ndelayed it. I don’t think there will be time for questions though.
\nShe might not finish. For a 20 minute talk, she has 32 slides…
\nThey found that overall their correlation based metric outperformed
\ncompeting methods, and performed well on “difficult question” where NE
\nrecognizers might not help. <\/p>\n
There is time for at least one question. The correlation was used in
\nboth answer extraction and answer ranking. How good of a job does
\nthis correlation do in capturing additional candidates and wrong
\ncandidates? (Where are false positives and negatives?) Using exact
\nmatching degraded performance by 6%, and lose 8% if they remove the
\nMaximum Entropy model. <\/p>\n
Seems like an interesting paper, but I think the presentation should
\nhave been more polished. There were real timing issues, and I’m not
\nsure if the real impact was presented. I think the main contribution
\nhere is the correlation measure and not using strict matching only.
\nAgain, I don’t have a large background in this area though. I’m
\ninterested in the similarity measures though, since I think variation
\nmakes taking different forms into account important. <\/p>\n
—<\/p>\n
Question Answering with Lexical Chains Propagating Verb Arguments
\nAdrian Novischi, Dan Moldovan (Language Computer Corp.)<\/p>\n
Dan Moldovan (session chair) is giving the next paper, so he was
\nintroduced by someone else. <\/p>\n
The main point is that you should propagate syntactic structures along
\nyour lexical chains. They needed to extend WordNet by matching to
\nVerbNet patterns. Not all VerbNet classes have syntactic patterns, so
\nthey had to learn them. Didn’t present the results of the learning,
\nbut probably there is another paper about that somewhere previously. <\/p>\n
Presented a method to propagate the structure along through verbs.
\nSee Moldovan and Novischi 2002. Different relations modify the
\nsyntactic structure around in some way, and they have a table showing
\nthe different classes and what changes. They have some weights for
\nhow to order the relations, which were set by trial and error. I’m
\nnot really clear on what the ordering is used for. Examples for
\ndifferent relation types. There is also an example for how this can
\nbe used to map an answer to a question. <\/p>\n
They have some numbers on how many arguments can be propagated across
\n106 concept chains. By adding this syntactic structure propagation
\nthey improved the performance of their Q&A system. <\/p>\n
eXtended WordNet is available from University of Texas at Dallas. It
\nhas word senses disambiguated in the glosses and they have been
\ntransformed into logic forms. <\/p>\n
In general, I am not sure what this talk is really about. It is about
\nlexical word chains (using WordNet concepts instead of words) and how
\nyou can transform them by following the relations in WordNet\/VerbNet (along
\nwith some learned relations) to help with Q&A by transforming
\nanswer parses to match question parses. You can do some sophisticated
\nreasoning with this approach. It seemed like it was really a bit all
\nover the place though. I would have preferred that the presentation
\nfocused on the algorithm and how it can be used with Q&A. I
\ndidn’t see enough on the learning for 20,000 syntactic patterns added
\nto VerbNet. <\/p>\n
Questions: How do you handle modifications that have things like
\n“considered hitting the ball”. That would be up to the semantic
\nparser to figure out. In cases of entailment the same thematic roles
\nare retained – what about in cases like “buy and sell”, do the
\nthematic roles stay the same? Possibly they flip in that example.<\/p>\n
—<\/p>\n
Methods of Using Textual Entailment in Open-Domain Question Answering
\nSanda Harabagiu, Andrew Hickl (LCC)<\/p>\n
Focused on using textual entailment in a Q&A system. They have a
\nsophisticated system for parsing questions and paragraph answers.
\nThey extract features, then a classifier does a yes\/no classification
\non whether the question entails the answer given with some belief.
\nSanda has really been going through the slides quickly. They do
\ndetect negative polarity. Their system is built upon a large QUAB
\n(Question Answer Bank – a big database of questions and answers.)
\nGenerally they are looking for an entailment relationship between a
\nnew question, and a question they have already seen in their
\ndatabase. They use a Maximum Entropy classifier to predict if two
\nquestions can be related by entailment. They use a variety of
\nfeatures for the classifier. <\/p>\n
Trained classifier over 10,000 alignment chunk pairs as positive or
\nnegative. MaxEnt did better than a hill climber on the same training
\nset. They extracted 200,000 sentence pairs as positive examples from the ACQUAINT corpus
\nand used them as positive examples. Used headline to entail a
\nsentence that is the first pair, filtered on if there were at least
\none entity in common between the two. Negative examples took
\nsentences with even though, in contrast, but etc. between two
\nsentences. The two were used as negative examples for not entailing
\nbetween them. Performance improved significantly over their 10,000
\ndevelopment corpus. <\/p>\n
Also uses paraphrase alignment in their system. Use WWW data to get
\nclusters of potential paraphrases following Barzilay and Lee 2003. <\/p>\n
She’s really been whipping through the slides, there is a lot of
\ncontent here, but it is getting hard to follow. She explained the
\nfeatures used in the entailment classifier. Their experiments use the
\ntextual entailment in three ways: to filter the output answers, to
\nrank candidate passages, to select automatically generated question –
\nanswer pairs from the QUAB database. Evaluated it over 600 randomly
\nselected questions from TREC. Also using PASCAL Recognizing Textual
\nEntailment 2006 data for evaluation? Their textual entailment system
\nis about 75% accurate based on that PASCAL data. They can improve
\nQ&A accuracy from .30 to .56 using entailment for known answer
\ntypes. Also improves with unknown answer types but not as much. <\/p>\n
Lots of interesting things in here, but there was too much in the
\npresentation I think. I’ll have to read the paper. In general
\nthough, it looks like textual entailment was a big help… The talk
\nwent long, maybe 10 minutes? Since it is just before lunch though, it
\nisn’t a problem. <\/p>\n
Questions: It wasn’t clear how the actual entailment was done? The
\nquestion was the hypothesis. <\/p>\n
—
\nMachine Translation Session (Dekai Wu chairing)<\/p>\n
Scalable Inference and Training of Context-Rich Syntactic Translation
\nModels
\nMichel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve
\nDeNeefe, Wei Wang, Ignacio Thayer
\n(Columbia University, USC\/ISI, Language Weaver, Google)<\/p>\n
I thought Michel’s talk was pretty good. I’m sure he’ll get some
\ntough questions though. I liked the rule analysis, and that he didn’t
\nspend too much time talking about BLEU scores. <\/p>\n
Dan (Melamed) asked a question about non-contiguous phrases, and what
\nwas meant by that. Asked a question about the power of the formalism,
\nif it is the same as ITG or not? Question by Dekai Wu about that time
\ncomplexity. <\/p>\n
—<\/p>\n
Modelling lexical redundancy for machine translation
\nDavid Talbot, Miles Osborne (School of informatics) (which one?
\nwhere? University of Edinborough) <\/p>\n
There is redundancy in the lexicon used for translation that is
\nunimportant for translation purposes. The model becomes more complex
\n(e.g., seaweed -> nori, konbu, wakame, etc. in Japanese) which makes the
\nlearning problem more difficult. Remove the distinction in the
\ntraining corpus to improve distributions for learning, reducing data
\nsparsity, etc.<\/p>\n
I had troubling following the math used to cluster terms together that
\nshould be collapsed. The model prior is a markov random field. This
\nis used somehow to indicate how likely words should be assigned to the
\nsame cluster (has features over the words, for example two words that
\ndiffer only in that they start with a c or g.) They use an EM
\nalgorithm to tune the parameters using bilingual occurrence
\ninformation to guide the E step. That looks pretty interesting to
\nme. David gave an overview of other related work, and there is some
\nother related work in morphology that might do more of that. <\/p>\n
Their experiments use Czech, Welsh, and French. Phrase-based decoder
\nwith Giza for alignment. Showed some improvement in BLEU scores (2-4
\npoints) and examples where their system helped. Also looked at vocab
\nsizes, which are maybe .25 reduced. <\/p>\n
Questions: BLEU should be thrown away. Would like to see 10
\nsentences before and after. Did you look at distributional features
\nin the source language? No, not yet, but he has considered it.
\nThought of using monolingual features in the MRF prior, but didn’t
\nimplement it. Initially talked about how more refined word senses
\ncould be tackled here, but examples were all morphological. The
\nfeatures in the MRF don’t capture thesauri information, but had an
\nexample of secret, secretive and private translated into welsh where
\nsome collapsing was done. <\/p>\n
I thought this was an interesting presentation and will have to read
\nthe paper. <\/p>\n
—<\/p>\n
Moved over to the generation session.<\/p>\n
Learning to Predict Case Markers in Japanese
\nHisami Suzuki, Kristina Toutanova (Microsoft Research)<\/p>\n
In one of the examples she gives as incorrect output from her MT
\nsystem, I would probably have made the same mistake… First
\npresented results on predicting case markers in a monolingual task.
\nThe system makes errors with ha \/ ga, similar to how humans make the
\nmistake… <\/p>\n
Second part did prediction in a bilingual reference settings. Given a
\nsource English sentence, and a Japanese translation of the sentence
\nmissing case markers. They have a dependency parse of the English
\nfrom the NLPWIN (Qurick et. al 2005 from MS research) parser then
\nalign with Giza++, and derive syntactic information on the Japanese
\nside from the English parse projected via alignment links, and POS
\ntags on Japanese. <\/p>\n
They have a variety of features for this task. A 2-word left and
\nright context window, POS tags and dependency tree info, words from
\nthe English that are aligned, source syntactic features. Improves
\nover the monolingual Japanese model only, in each case using syntactic
\nfeatures helped. <\/p>\n
Questions: Japanese case markers often do not have English
\nequivalents, so what do they align to in English using Giza? Usually
\nalign to null, or ha sometimes goes to copula (is). <\/p>\n
—<\/p>\n
Improving QA Accuracy by Question Inversion
\nJohn Prager, Pablo Duboue, Jennifer Chu-Carrol (IBM T.J. Watson)<\/p>\n
What other questions should you ask a QA system when you don’t know if
\nan answer is correct? By asking other questions they can find
\ninformation that can be used to invalidate bad answers (e.g., if you
\nknow the birth and death year of a person, all things relating to
\ntheir achievements must be bounded by those years.) <\/p>\n
They take a question Q1, generate more questions (inverted questions)
\nQ2 to find answers A2 that can be used to invalidate answers A1 in
\nresponse to Q1. Looks like I got this description a bit wrong, but it
\nshould be more clear as the presentation goes on. <\/p>\n
Interesting, they use the generated questions to get answers that they
\nuse to validate the answers they already had. <\/p>\n
—<\/p>\n
Learning to Say It Well: Reranking Realizations by Predicted Synthesis
\nQuality
\nCrystal Nakatsu, Michael White (Department of Linguistics, Ohio State
\nUniversity)<\/p>\n
This is synthesis in the text to speech meaning. The job is to choose
\ntext realizations for the synthesizer that are predicted to sound
\ngood. They rated a variety of different sentence realizations, and
\nused an SVN to rank the plans. Their system improved output quality
\nto statistically significant levels. Raised average ratings from “ok”
\nto “good”. <\/p>\n
Questions: There was a question about where effort should go to
\nimprove synthesis? The answer is that you should work on both
\nsynthesis and generation. Did the reviews hit a fatigue effect where
\nthey got used to the synthesizer? Spread effort out over days, and
\nstarted from different ends of the lists. How was inter-annotator
\nagreement, and did the annotators really judge only on synthesis and
\nnot sentence structure? Agreement was like .66. More interested in
\nextremes of what was really good and really bad, so not too worried
\nabout that. <\/p>\n","protected":false},"excerpt":{"rendered":"
Notes from Friday’s sessions at the last day of COLING\/ACL. Friday 2007-07-21 Notes from COLING\/ACL Language, gender and sexuality: Do bodies always matter? Sally McConnel-Ginet Cornell University (Invited talk) Co-author of book “Language and Gender” – sounds interesting. Gave an example of John Summers (sp?) the Harvard guy and his talk about women and science. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[4,10],"tags":[],"_links":{"self":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/posts\/125"}],"collection":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/comments?post=125"}],"version-history":[{"count":0,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/posts\/125\/revisions"}],"wp:attachment":[{"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/media?parent=125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/categories?post=125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fugutabetai.com\/blog\/wp-json\/wp\/v2\/tags?post=125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}